1. Operationalizing IE: Case Study example: Textbooks and Teacher Training in Sierra Leone APEIE Workshop, May 10-14 2010

2. Operational Key Steps 2. IE Design 3. Baseline Survey 4. Intervention Starts 5. Survey 6. Analysis, Reports, Policy Advice 1. Policy

3. IE Design – methodological issues Available data and required data Identify control group and treatment group – Randomization – Transparency Sampling and sample sizes – Power calculations – Budget and access constraints Identify outcome indicators

4. Data Instruments Utilize survey experts Who do you survey? – Households, community, facilities, government… Pilot instruments – Clear questions and coding – Includes all information needed for analysis and covariates

5. Intervention Roll Out Possible contamination of evaluation results! – Intervention benefits rolled out to control and treatment schools  evaluation compromised – Intervention or treatment schools only receive additional benefit  evaluation compromised Monitor roll out – Train implementers – they need to know why! – Hire supervisors – Continue supervision throughout roll out

6. Background In June 2007, the World Bank and a team from the Sierra Leone Ministry of Education and Sports (MOES) met to rigorously evaluate the impact of two pilot studies on student performances. Two questions – What is the effect of textbooks on student performance? – What is the effect of teacher training on student performances?

7. Random Selection – 5 districts in Sierra Leone Kailahun, Kambia, Pujehun, Western Rural, Western Urban – Neediest schools in district (using EMIS data) Student to teacher ratio, presence of latrine, school construction 360 schools total Random Assignment – Comparison schools: receive regular allotment of books – Books schools: receive additional books – Books and training schools: receive additional books + teacher training How do we design the study to get rigorous results?

8. Timeline Sept: WB and MOE agree on pilot Oct: Met with local councilors to select districts Nov: Baseline data collection and teacher training manual developed Nov: Mid-term data collection June: Baseline data collection Dec: Roll out of textbooks and teacher training June: Roll out of textbooks 20072008 2009 Dec: Mid-term data collection NO teacher training up to this point REALITY TARGET

9. What happened? Met with several challenges on the ground that caused significant delays in the project: – Textbook distribution – Teacher Training – Data Collection

10. Challenge 1: Textbook Distribution Target date: December 2007 Actual date: June 2008 Number of books: – MOES accidentally distributed original books, so had to use Sababu books – had fewer books than project initially called for Communication – information not filtering down – Sababu not properly informed – Service Providers unwilling/unable to change distribution – Headteachers in comparison schools unaware of project

11. Challenge 1: Textbook Distribution continued Textbook distribution: June 2008 Data collection: December 2008 Consequences: – Not enough books distributed in some districts to make a statistical comparison – Distribution happened very late and at a bad time (right before summer vacation) Allowing for only a small window for students to utilize the extra textbooks – Some Comparison schools received books

12. Challenge 2: Teacher Training Target date: December 2007 Actual date: has not yet occurred Time and budget constraints The ‘+’ part of the ‘intervention +’ so took a back seat to the textbook distribution

13. Challenge 3: Mid-term Data Collection Target date: November 2008 Actual date: December 2009 Delayed funding disbursement Non-existent schools (poor quality of initial data) Headteachers unaware of project – not willing for enumerators to collect data

14. Solutions: Back up list of schools on hand for enumerators Better communication with all levels of organization involved Constant project sensitization Greater monitoring of distribution Inclusion of Process Surveys Allow for sufficient time for procurement and disbursement

15. Next Steps Analyze mid-term data – Was there a significant effect due to textbooks? – Has education in Sierra Leone improved in the past 2 years? – Where in the textbook distribution are there major informational stop gaps? Move forward on teacher training Write up findings and circulate report

16. Conclusion Impact evaluations a useful tool for policy considerations But need to understand every process involved in the impact evaluation – Textbook distribution, not just textbooks Study can be ineffective if you don’t have full buy-in and understanding at all levels of government, all service providers, and headteachers: – SENSITIZATION! – COMMUNICATION! – MONITORING!

17. THANK YOU