Improving School Quality in East Africa: Randomized Evaluation of Policies to Create Local Accountability under Free Primary Education in Kenya and Uganda Germano Mwabu University of Nairobi, Kenya Frederick Mugisha Economic Policy Research Centre, Uganda RESEARCH PROPOSAL Presented to PEP-AusAid Policy Impact Evaluation Research Initiative

Overview  Randomized controlled trial of institutional interventions in the Kenyan and Ugandan primary education sector  Project will take place in close collaboration with the Ministries of Education (MoE) in both countries  Both countries characterized by large increases in enrollment following introduction of free primary education and (perceived) decline in educational ‘quality’  Research will focus on two key interventions Hiring local contract teachers to address shortage of government teachers in Kenya, and funding locally determined projects in Uganda. Community-Based Monitoring System (CBMS)  Interventions correspond to the two challenges in FPE – input shortages and lack of parental ownership.

Research Question  Expansion of the primary education sector under FPE has led to influx of new students creating shortage of teachers undermined the ability of schools to raise funding locally due to the abolition of fees weakened community based monitoring of schools. (FACTS)  Main question: does local accountability require local fundraising or is strengthening CBMS sufficient? (question: what is local accountability)

Hypotheses in Detail Hiring additional contract teachers in Kenya will have a significant effect on student learning achievements. (Either by lowering teacher-pupil ratios or changing the nature of the contract.) Providing school lunches in Uganda will have a significant effect on student attendance and learning. Giving parents a financial stake in the school (soliciting co- funding to hire contract teachers) will increase local accountability and community monitoring relative to pure top-down funding. Informing school management committees of parents’ rights and teachers’ responsibilities and providing them with tools to monitor school performance will improve school management and student performance. Community-based monitoring systems are particularly effective when parents have a financial stake in the school – i.e., there are complementarities between the interventions.

Description of Interventions  School Management Committee Score Cards completed by SMC on a termly basis, gathering information on… teacher performance: teacher attendance, preparation of schemes and lesson plans, class room activities measured via direct class room observation and spot checks financial administration: correspondence between budgeted and actual expenditures, appropriate input purchases, perceived wastage or inefficiency in budget use School facilities and maintenance: hygiene and sanitation, classroom conditions and repairs  School meals (Uganda only) matching grants for locally funded school lunches Create cooperative decision-making across geographically proximate schools

Description of Interventions  Contract Teachers (Kenya only) Provide co-funding to SMCs to hire contract teachers Addresses two issues:  Acute teacher shortage due to constraints placed on local fundraising by Free Primary Education  providing for contract teachers will test the importance of local control over teaching staff in creating accountability and quality service provision  Hypothesis: local contract teachers -- hired on a fixed term, renewable contract in which the SMC has control over hiring and firing -- will be more responsive to local demands

Research design: pop & outcomes  Population to be studied: 196 Kenyan schools with a total of 3299 students, nationally representative, based on Southern and Eastern African Consortium for Monitoring Education Quality Survey (SACMEQ) Ugandan sample is representative of rural primary school students in four districts selected to represent poor- performing districts of the 4 regions  Outcome of interest: Student achievement 9 months – 1 year after start of the intervention: measured by national exams as well as custom-made literacy and numeracy exams (in conjunction with Kenya Institute of Education in Kenya project, Uganda National Examinations Board) Changes in enrolment and transition to secondary school 9 months after start of the intervention.

Research design: randomization  Control schools chosen randomly and simultaneously with the treatment. In Kenya, use SACMEQ sample stratification so treatment and control schools have equal representation from urban and rural areas. Successful randomization will be tested using pre-intervention information. In Uganda, sample from primary data collected at district level. Schools representative of rural sub-counties, with ‘blocking’ of the sample based on pre-intervention test results to ensure representation of low-performing schools.  Use cross-cutting design: treatment group for each intervention assigned orthogonally  Difference-in-differences model exploiting the randomized design of the intervention, comparing mean changes in academic performance between treatment and control schools.

Research design: analytical framework  Cross-cutting interventions to test hypotheses about interactions For instance, is a CBMS program enough to increase accountability, or is it primarily effective when parents have more control over teachers (through contract &/or fundraising)  If Y is individual exam performance, i indexes individuals and j indexes schools, TEACH and CBMS are treatment dummies then:  The B x parameter will test the interaction between local ownership and community monitoring.  We plan to go further and test two levels of funding as follows:

Similarly for Uganda: No meals intervention Meals intervention No co- funding (full) Co-funding (partial) No SMC intervention17(no Meals No SMC), Pure Control 88 SMC score- cards Bottom-up accountability 178 X X Top-down accountability 178 X X

Minimum detectable effects for alternative outcome measures and designs, Uganda (1)(2)(3) Design parameters j=100, n=30, P=.5 j=400, n=30, P=.125 j=100, n=50, P=.5 Controlsdistrict, gender District, gender district, gender OutcomeEnglish0.69 (0.36)0.52 (0.27)0.68 (0.35) Math0.53 (0.34)0.40 (0.25)0.52 (0.33) Avg score2.46 (0.37)1.84 (0.28)2.43 (0.37) Pass indicator 0.12 (0.30)0.09 (0.22)0.12 (0.29) Note: Minimum detectable effect sizes in parentheses.

Minimum Detectable Effect Sizes for alternative outcome measures and test parameters, Kenya SAQMEQ Parameter estimated (assumptions) Estimated intra-class correlation MDES (P=.5, j=196, n=28) MDES (P=.25, j=196, n=28) OutcomeReading Math Average

Data Collection/Sources Kenya: Admin data:use of Education Management information System, Kenya Certificate of Primary Education exam data, Southern and Eastern African Consortium to Monitor Education Quality (SACMEQ) II and III; the Kenya Integrated Household Budget Survey (KIHBS). Follow-up to SACMEQ II to be conducted with MoE Uganda: Admin data: use of Primary Leaving Examination results and administrative data on school funding and district-level inputs. Survey of school management and pupils to be collected by EPRC researchers in four districts. Tests prepared by UNEB (National Assessment for Planning in Education) to be administered to a sample of students

Example of MoE Database: nationwide coverage over time

Consultations What have we done to include stakeholders in the planning process?  Kenya Meetings held with Teacher Service Commission, Kenya Institute of Education and MoE (Quality Assurance and Basic Ed Directorates) Focus groups discussions with teachers and head teachers plus church sponsor Training workshop with MOE staff to bring them into the project as active participants  Uganda Preliminary meetings with District Education officials in each of the 4 districts, plus MOE staff in Kampala. School visits to meet with teachers & head teachers in each district 3-day intervention design and training workshop in Kampala with MoE staff, UBOS, DEOs., and civil society organizations.

Dissemination Plan  Standard research model: get data, do analysis, try to convince policymakers you’re right  Our approach in this project: Train government officials in methods of analysis Rely as much as possible on administrative data sources Involve gov officials in the implementation and analysis  Final results are fully owned by policymakers as their own collaborative product

Project staff Lead Institutions a.University of Nairobi, School of Economics, Kenya Germano Mwabu, Faculty member Ms Mumia Phillis Machio, Graduate student Ms Racheal Nakhumicha Musitia, Graduate student Ms Alice Muthoni Nga’ng’a, Graduate student b.Economic Policy Research Centre (EPRC), Uganda Fredrick Mugisha, Senior Research Fellow Lawrence Bategeka, Research Fellow Ms Madina Guloba, Assistant Research Fellow c. Ministry of Education, Kenya Ministry officials d.Ministry of Education and Sports Ministry officials

Project staff… Collaborating Institutions University of Connecticut, Economics Department: Samson Mwangi Kimenyi University of Oxford, Centre for the Study of African Economies:Tessa Bold, Justin Sandefur, Roxana Gutierez, Andrew Zeitlin, Stefan Dercon --- All the above researchers are involved in a larger research project funded by DFID to improve educational outcomes in Africa and Asia

Budget  Two thirds of the requested resources will fund project interventions directly

Thank you