Evaluating a test-based subsidy program for low-cost private schools: Regression-discontinuity evidence from Pakistan Felipe Barrera-Osorio Dhushyanth Raju (The World Bank)

Overview of the study: Strategy and Results Strategy of estimation – Since entry into the program is a function of a minimum student pass rates in the test (average at the school level), we used a Regression Discontinuity approach in two entry school waves (Phases 3 and 4) – We used both administrative and own collected data via phone survey to schools near the discontinuity point Results – For Phase-4 applicants (sharp design), we find large positive impacts on the number of students, teachers, classrooms and blackboards. Students to teacher ratio remains constant. Impact emerging within a short period – Using Phase-3 (fuzzy design), we did not find statistically significant impacts. 2

Program setting In general, Pakistan has poor performance in educational indicators – Participation rate in formal school (grade 1+) of children ages 6-15 was 65.7% – While 91.8% of individuals ages completed primary school (grade 5), only 40.7% completed secondary school (grade 10) – Mean scores in national assessments less than 50% Public sector the dominant provider of education with weak accountability and weak incentive systems. Dramatic growth since mid 90’s of the private school sector (Andrabi et al 2007; Andrabi et al 2006; Das, 2006) – 25% of school-goers (6-17 year olds) in private school – Specially for the poor and in rural areas – Fees in private school are generally low: median annual fees per student in 2000 were 960 (rural area) and 751 (urban) rupees The government formed an partnership with the private sector in order to increase enrollment and quality of education, specially for disadvantaged households 3

Foundation-Assisted Schools (FAS) program structure and coverage Administration: Designed and administered by the Punjab Education Foundation, a semi-autonomous organization Objectives: Increase school participation and achievement levels among children from disadvantaged households Timeline: Program established in 2005 and expanded in phases (4 phases completed). Coverage: – Districts: 18 out of the 35 districts in Punjab; 87% of schools in 7 districts; – Schools: 1,082 low-cost private schools; – Students: 474,000 4

FAS program: eligibility conditions 1.Initial benefit eligibility conditions a.Minimum enrollment size of 100 students. b.Pass a qualitative inspection conducted by PEF. c.At least 67% of students tested have to score 33% or higher on the test. 2.Continued benefit eligibility conditions a.Maintain minimum enrollment size of 100 students. b.Eliminate all tuition and fees for all students: signboard outside school gate which announces tuition-free schooling and provides PEF contact information. c.Report enrollment figures to PEF every month. d.At least 67% of students tested have to score 40% or higher on the test.  First violation: Penalties (e.g., frozen enrollment).  Second violation: Permanent program disqualification. 5

FAS program: Benefits Subsidy: Rs. 300 (US$4.3) per student per month. Subsidy level set at upper-end of price range for low-cost sector. Use of subsidy largely unfettered. Teacher bonus: Rs. 10,000 (US$143) per teacher per year for 5 teachers in schools in which at least 90% of students in tested classes obtain a score of 40% or higher in the Quality Assurance Test. 370% of mean monthly teacher salary at baseline. School bonus: Rs. 50,000 (US$714) to the school in each district with the highest pass rate in the QAT per year. 76% of mean monthly subsidy payment to schools given mean enrollment size at baseline. 6

FAS program: Tests The entry test: Short-listing Quality Assurance Test (SLQAT): – To be part of the program, at least 67% of students tested have to score 33% or higher in the SLQAT – Pared-down version of QAT – Developed and administered by PEF. – Offered to two to three grades in a unannounced visit to the school Continuation test: Quality Assurance Test (QAT): – For continuation in the program, at least 67% of students in tested grades have to score 40% or higher in the QAT – Based on samples of syllabi and textbooks used in low-cost private schools. – Selected subjects: English, Urdu, Mathematics, and Science. – Offered twice a year in two or three grades 7

FAS Program: Application process and Number of schools 8

Identification Entry process: – Schools apply when a call for applications is issued. – Schools with qualifying applications are subject to a physical inspection – In phase-3 and phase-4, schools that pass the physical inspection are offered the SLQAT Assignment to FAS program (treatment) based ultimately on school’s pass rate on SLQAT – To be part of the program, at least 67% of students tested have to score 33% or higher in the SLQAT – Virtually all schools that attained at least the cutoff, accepted treatment (take-up rate of 94% in Phase 3 and 98%, in Phase 4) – No treatment dropouts to date 9

Local smoothness in baseline mean outcomes Comparison of means for those that marginally fail the SLQAT and for those that marginally pass: no statistically- significant differences, except for mean SLQAT score (scores positive correlated with pass rates). LLR estimates: Some statistically-significant breaks but nothing systematic to suggest a breakdown of the conditional smoothness condition for RD identification. 10

Data used for the evaluation Data before the program – Source: Administrative data. – Data on school characteristics and non-learning outcomes obtained from applications maintained electronically by PEF. – Data on SLQAT pass rates and average school test performance obtained from individual student test score data maintained electronically by PEF – Number of schools: 747 (Phase 3) and 830 (Phase 4) Data after the program – Source: Phone interviews (schools phone numbers in applications) – Data collected on school characteristics – Number of schools (just passers and just failers): Phase 3: 192. Phase 4: 236 – Data collected 14 and 10 months after first subsidy payment to phase- 3 and phase-4 entrants (partially spans two academic years) 11

Main findings Large positive effects on enrollment, teachers, classrooms, and blackboards; within short treatment period of 10 months Conservative estimates: +85 children (37% relative to baseline mean); +3.4 teachers (37%); +4 classrooms (47%); and +2.8 blackboards (27%) The annual cost per student of increasing enrollment by 1% is 97 rupees (US$1.4) – Among the lowest cost-effectiveness ratios estimated

Ongoing studies More evaluation research is planned with several data sources – Annual public school censuses – The Punjab Examination Commission test score data – Additional FAS program administrative data from PEF – Primary survey data from samples of low-cost private schools Some example of key questions – What are the impacts of the FAS program on student learning – Does the FAS program have any impacts on the socioeconomic composition of their students (particularly in terms of gender and economic background) – Does the FAS program induce the displacement of students from neighboring public schools and to what extent