Community Engagement for Education Impact Evaluation 2012-2013
Pakistan, 2012 - 2013
Identification
PAK_2012_CEEIE-BL_v01_M
Community Engagement for Education Impact Evaluation 2012-2013
Baseline Survey
| Name | Country code |
|---|---|
| Pakistan | PAK |
The main objective of the interventions supported by this impact evaluation is to strengthen linkages between communities and school to improve education outcomes. Rigorous evidence generated from the research will provide valuable information to Pakistani policy makers, donors and development practitioners on the ways in which school based management reforms can be strengthened in low-governance environments like Sindh, Pakistan. The findings of this research are valuable for the ongoing dialogue with the GoSindh on school based management, one of the critical reform area supported under the Second Sindh Education Sector Program (SEP-II).
The impact evaluation is a component of the World Bank's ongoing technical and advisory support to the Government of Sindh for improving the quality and performance of government primary schools as part of its medium-term, multi-pronged Sindh Education Sector Reform Program (SERP-II). An important subprogram under SERP and SERP-II has been the revitalization of school management committees (SMCs) in government schools, with the provision of annual school improvements grants and basic guidelines on SMCs rights, roles and responsibilities across Sindh province. An area of concern in these early efforts has been poor or dissipating community interest and engagement. The interventions piloted in select districts of rural Sindh were designed by the World Bank in partnership with the Reform Support Unit, which is the implementation arm of the Education and Literacy Department of GoSindh. The aim of these interventions was to explore concrete ways to elicit meaningful and sustained local community engagement in improving education outcomes.
Both the baseline survey and the interventions were implemented in three pilot districts in 2012 and 2013. The core intervention being evaluated is community engagement to revitalize SMCs under two distinct mechanisms:
- a community-level meeting to engage the community in a dialogue for school improvement via SMCs;
- a virtual network of community members to engage in a similar dialogue supported through text messages on mobile phones.
The first intervention arm makes use of an existing social platform, enabling community members to participate in traditional meetings to acquire information and engage the community in dialogue and discussion on school-related issues. The second arm has created an innovative virtual platform through which registered community members receive school-related information, anonymously send text messages about these issues and receive a summary of key observations or issues twice every month.
The baseline survey, documented here, was implemented in January 2012 - January 2013. There is no midline survey for this study. The endline survey will start in January 2015.
Sample survey data [ssd]
The unit of randomization for the intervention is a village.
Administered questionnaires have the following units of analysis: individuals (teachers, students, parents), households, schools, and communities.
Version
v01
2013-06-30
The data uploaded is cleaned so any missing or unknown data is replaced with a missing value. Annex 1 and 2 in the document "Impact Evaluation Background Notes" represent a complete list of the datasets with the unique identifier for each instrument and details of the survey instruments.The baseline household data is at individual level where each individual is marked with a unique ID at household level. Similarly for school level instruments, the respondents (teacher, head teacher and students) are marked at an individual level with a unique ID at school level.
The school observation questionnaire is administered at school level with a unique identifier for each school. Any duplicates at data entry stage have been removed.
For the student achievement data, questions that were not attempted by a child were entered with the code "NA". For scoring the test, all questions not attempted were replaced by a missing value.
Scope
The scope of the study includes:
- School-level surveys:
- enrollment,
- attendance,
- teacher on-task,
- facilities,
- school committees,
- funding and expenditure
- Household-level surveys:
- demographic and socioeconomic characteristics,
- parent choices about child's school,
- School Management Committees (SMC),
- parent engagement with SMC,
- adult perceptions of returns to schooling and quality of learning
- household assets,
- health,
- use of media sources
- Adult Literacy Test
- Child Surveys:
- child's basic information,
- activities and work,
- perceptions about learning and school
- Learning Assessment Test (LAT)
Coverage
Mirpur Khas, Mitiari and Sanghar districts in Sindh province.
All primary schools and rural households in Mirpur Khas, Mitiari and Sanghar districts in Sindh province.
Producers and sponsors
| Name | Affiliation |
|---|---|
| Salman Asim | World Bank |
| Name |
|---|
| Education Sector Development Program Trust Fund |
| Bank Supervision Budget for Sindh Education Sector Project |
| Strategic Impact Evaluation Fund |
Sampling
The districts chosen for the study were based on district ranks in terms of school density in the district and school participation rates from the Pakistan Social and Living Standards Measurement Survey (PSLM) and Administrative School Census (ASC) data respectively. One district each was chosen from the low, middle and top category to make an overall representative sample of rural Sindh. By this method, the final districts selected were Mirpur Khas, Mitiari and Sanghar. Using the ASC data in terms of number of schools, Mitiari was ranked the third smallest district, Mirpur Khas was ranked at number twelve (middle rank) and Sanghar at number eighteen (top rank). Using the PSLM for education indicators (proportion of adults who ever attended school and school participation rate of primary-age children), Sanghar ranked at the top followed by Mitiari (median) and lastly by Mirpur Khas.
The Administrative School Census (ASC) data is collected by the Government of Sindh every year to provide an updated list of primary schools in all districts of Sindh. The census data for 2010-2011 was used to randomly draw 300 villages within our sample districts. However, because of poor quality of administrative census data, researchers conducted a census listing of all households and also mapped all primary schools in these 300 villages to set the population frame for the study.
- School Sampling Strategy
The school sampling strategy was primarily to target all primary schools in the main settlement that were either open on the day of visit or closed for a period of less than one year. In addition, 15% of the remaining schools in these villages were also surveyed to capture spillover effects. For villages with no school in the main settlement, all schools located out of the main settlement were surveyed1. For villages that did not meet these criteria, all schools were sampled even if the school was closed for more than one year. 4 villages had to be dropped because no school was found in village-level mapping of primary schools.
- Household Sampling Strategy
The household sampling strategy for each village was to randomly select 20 households from the main settlement and 8 households from the peripheral settlements conditional on the household having at least one child of school going age (5-16 years). From this list, the first 16 households were to be surveyed and in case the head of the household was not available, the household was substituted from the list of four buffer households. For the peripheral settlement, any 4 out of the 8 households were surveyed2. In addition, household questionnaires were also administered to all SMC members from the target schools, approximately 4 households in a village.
Overall, on the school level 514 school, 454 head teacher, 409 teacher and 4,573 student questionnaires were administered. On the household level, 6,505 head of the household, 6,503 spouse, 5,281 child and 901 school management committee questionnaires were administered.
Survey instrument
School Surveys
Detailed data on school-level variables such as enrollment, attendance, teacher on-task, facilities, school committees, funding and expenditure were collected through a set of four questionnaires: School Observation, Teacher Roster, Head Teacher and Teacher Questionnaire. In addition, a list of School Management Committees (SMC) members was enumerated at the school-level for household surveys.
School Observation Questionnaire
School questionnaire consisted of five sections and was based on the observation of the enumerator about school building, facilities, hygiene conditions, on-going classroom practices and teacher activities. The questionnaire also required the enumerator to record school GPS coordinates and school visit details.
Head Teacher Questionnaire
Head Teacher questionnaire compromised of two parts: information based on the head teacher’s knowledge and information based on official school records. The first part gathered data on the respondent’s personal and professional background as well as his knowledge of students, school facilities and SMC. The second part collected official school details on school improvement plan, enrollment, attendance, fee, SMC funds and expenditures.
School Teacher Questionnaire
Teacher questionnaire consisted of nine sections and was administered to all teachers present in the school . It gathered the personal and professional information of the teacher as well as his perceptions on SMC functionality, student learning and returns to education.
Teacher Roster Questionnaire
Teacher Roster collected information on teachers that are currently teaching in the school and those that left or transferred over the last two years. The survey recorded teacher information on attendance, contact number, gender, contract type, pay scale and class taught. For teachers that have left, it also covered information on reasons for leaving school. The information for the roster is to be provided by the head teacher or the senior most teacher in the school.
Household Surveys
The baseline survey also covered households to gather information on demographic and socioeconomic characteristics, parent choices about child’s school, parent engagement with school’s SMC, adult perceptions of returns to schooling and quality of learning through four set of questionnaires: Household Roster, Household Head Questionnaire, Spouse of Head Questionnaire and SMC Member Questionnaire.
Household Roster Questionnaire
The household roster questionnaire collected information about gender, age, marital status, education and job status of all members of the household. This roster information was filled by the head of the household but in case of his absence, the survey was filled by other members that were required to explain their relationship to the head.
Head of the Household Questionnaire
The head of the household questionnaire consisted of fifteen sections and collected detailed information on family members, education, consumption pattern, business details, household expenditures and incomes. It also recorded information on about the respondent’s aspirations, awareness about the SMC, trust in the education system and perceptions about returns to education and quality of learning in the respective school.
Questionnaire for Female
The Female questionnaire consisted of fourteen sections and was to be filled by the spouse of the head of the household. It collected information on SMC awareness, perceptions about returns and quality of education and aspirations. In addition, it also covered topic on children education, health, use of media sources and household assets.
Adult Literacy Test
A specialized adult literacy test designed for this study was administered to the male and female heads of the household with the purpose to identify average literacy levels in the village. The test had three components that evaluated respondents' proficiency in thinking skills, numeracy and prose literacy at three difficulty levels: basic, intermediate and advanced. For each question, the enumerator read out the instructions, showed flash cards and recorded the response of the respondent. In order to move onto a higher difficulty level, the respondent had to answer at least 2 questions correctly on the given level of difficulty.
The purpose of the adult literacy test was to group villages as literate and illiterate. This will help understand where the results of the intervention are likely to be strong.
Questionnaire for Child and Learning Assessment Test
A child questionnaire was administered at both household and school level. It recorded the child’s basic information, activities and perceptions about learning and school. For children in the household, there was an additional section to record the activities and work undertaken by a child throughout the day.
The Learning Assessment Test (LAT) was also conducted to benchmark the performance of the child with regional and international standards. The test was made up of multiple choice and short-answer questions encompassing concepts from Grade 1 through Grade 5 and was based on three sections: English, Mathematics and Sindhi.
At the household level, the test was administered to children between ages 7 to 13 years regardless of their enrollment status. At the school level, the test was administered to students of grade 3 and 4 but because of multi-grade classes within the school, some students from other grades were also tested.
School Management Committees (SMCs) Questionnaire
A list of SMC members was enumerated at the school-level for household surveys and an SMC instrument was to be filled by a household member who is an Executive Member of the SMC. This questionnaire gathered personal information about the respondent and recorded detailed information on the functioning and awareness of the SMC.
Data collection
| Start | End | Cycle |
|---|---|---|
| 2012-01-01 | 2013-01-30 | Baseline |
| Name |
|---|
| Weitek Group |
The initial timeline for the survey had to be delayed because of the floods in Sindh causing the school level surveys were conducted in two rounds: before and after the summer vacations. The household level surveys including the head of the household, spouse and child questionnaire were successfully completed in this year but for the school level survey, the school, teacher and student questionnaires continued to be administered until January 2013. Some households that were flagged for inconsistencies were also re-surveyed during this time. On the intervention side, the community engagement material and community dialogue portal was designed in 2013 and the intervention was fully implemented by 2013.
Data processing
The data uploaded is cleaned so any missing or unknown data is replaced with a missing value. The census household listing data is at household level, the community data is at settlement level and school mapping data at school level. The baseline household data is at individual level where each individual is marked with a unique ID at household level. Similarly for school level instruments, the respondents (teacher, head teacher and students) are marked at an individual level with a unique ID at school level. The school observation questionnaire is administered at school level with a unique identity for each school. Any duplicates at data entry stage have been removed.
For the LAT test, questions that were not attempted by the child were entered with the code 'NA'. For scoring the test, all questions not attempted were replace by a missing value.
Depositor information
| Name | Affiliation |
|---|---|
| Education - GP | World Bank |
Data Access
Use of the dataset must be acknowledged using a citation which would include:
- the Identification of the Primary Investigator
- the title of the survey (including country, acronym and year of implementation)
- the survey reference number
- the source and date of download
Salman Asim, World Bank. Pakistan Community Engagement for Education Impact Evaluation 2012-2013 (CEEIE-BL), Baseline Survey. Ref. PAK_2012_CEEIE-BL_v01_M. Dataset downloaded from [URL] on [date].
The study will be available in the external Microdata Library (microdata.worldbank.org) in July 2016.
Disclaimer and copyrights
The user of the data acknowledges that the original collector of the data, the authorized distributor of the data, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses.
Contacts
| Name | Affiliation | |
|---|---|---|
| Salman Asim | World Bank | sasim@worldbank.org |
Metadata production
DDI_PAK_2012_CEEIE-BL_v01_M_WB
| Name | Affiliation | Role |
|---|---|---|
| Development Data Group | World Bank | Study documentation |
| Education - GP | World Bank | Study documentation |
2014-12-08
Metadata version
v01 (December 2014)