Formal Technical Education 2009-2013, Independent Impact Evaluation

With a budget of nearly $20 million, the Formal Technical Education Sub-Activity was designed to strengthen technical and vocational educational institutions in the Northern Zone of El Salvador. By improving schools and offering scholarships, the sub-activity financed efforts to increase youths' access to high-quality technical education in the region, thus increasing their achievement levels, secondary (and post-secondary) school graduation rates, and prospects for gainful employment. By 2012, the Formal Technical Education Sub-Activity was scheduled to invest $3.8 million in scholarships for students enrolled in secondary and post-secondary technical schools in the Northern Zone. According to preliminary budgets, the sub-activity would also provide $9 million to improve 20 technical secondary schools in the Northern Zone with infrastructure investments and additional teacher training programs. In addition, the sub-activity was scheduled to invest $7 million to strengthen ITCHA, an existing post-secondary institute in the Northern Zone.

In conducting the evaluation of the Formal Technical Education Sub-Activity-which includes secondary and post-secondary school improvements and scholarships-Mathematica will address the following research questions regarding Sub-Activity investments from 2009 to 2012:

1.Program design/implementation. How were the secondary school strengthening and scholarship programs, and the ITCHA strengthening program designed and implemented? Did implementation meet original targets and expectations? Why or why not?

2.Description of participants. What are the characteristics (age, gender, initial household income, etc.) of scholarship recipients? What are the basic characteristics of secondary school and ITCHA students?

3.Impact/Results. What is the impact of FOMILENIO's strengthening secondary school program on students' education and labor market outcomes, including secondary school enrollment, grade completion, graduation, and further education, employment, and income? What is the impact of the offer of scholarships in some programs within strengthened schools on student educational and labor outcomes? Did ITCHA graduates obtain jobs and experience increased income following graduation? Did ITCHA students who graduated from secondary school MEGATEC programs have better academic and labor market outcomes than students who did not attend secondary school MEGATEC programs?

4.Impacts/Results by key target subgroups. Were impacts/results different for girls versus boys? What types of participants experienced positive impacts?

5.Explanation for impact findings and results. What was the ex-post statistical power, and can this explain the lack of impacts (in cases where no impacts are found)? What aspects of implementation could explain the impacts/results? If impacts/results were different for girls versus boys, why?

6.Sustainability. Are secondary school improvements and scholarships being maintained? Are ITCHA improvements being maintained? Are they likely to be maintained in the medium to long term?

To answer all research questions regarding the design, implementation, and sustainability of the strengthening efforts and scholarships (Topics 1, 2, 5, and 6), Mathematica will use a mixed-methods evaluation design that uses qualitative and quantitative methods (see Table III.1). With this approach, researchers will use qualitative methods-namely, qualitative interview data and programmatic reports-to help understand processes and activities, provide information on setting or context, and communicate the perspectives and experiences of key participants through direct quotes. In addition, Mathematica will use quantitative information on program outputs and costs, participant characteristics, and budget outlays to summarize the intervention, describe its participants, and analyze the sustainability of its original investments.

To answer research questions regarding impacts and results (3, 4, and 5), Mathematica will use a variety of designs. To determine the impact of secondary school scholarships, researchers designed and implemented a random assignment design, by which some eligible applicants were randomly selected to receive scholarships. To determine the impact of secondary school strengthening investments, Mathematica designed and implemented a matched comparison group approach using propensity score methods, by which students at the 20 strengthened schools are compared to students at 20 similar non-strengthened schools. Finally, to measure key results of the ITCHA intervention-including graduation and employment rates-researchers used a mixed-methods approach that featured a follow-up survey of ITCHA students.

All impact and results analyses rely on in-person surveys, including panel surveys of scholarship applicants, cross-sectional baseline and follow-up surveys of secondary school students, and a follow-up survey of ITCHA students.