1. Sierra Leone Education Impact Evaluation Implementation Challenges & Solutions Philip F. Kargbo Philip F. Kargbo Field Coordinator

2. Overview of the Project June 2007:Team met at Abuja workshop and developed a study focusing on three separate interventions June 2007:Team met at Abuja workshop and developed a study focusing on three separate interventions Textbooks Textbooks Textbooks and Teacher Training Textbooks and Teacher Training Expert Teacher VCD+TV+Solar panel Expert Teacher VCD+TV+Solar panel

3. Challenge 1: Losing an intervention 1. September 2007: team decided to focus on the first two interventions: textbooks and textbooks + teacher training 2. The third intervention was suspended i) No electricity through out the country ii) Solar panels were expensive to be bought a)Expected aid program to provide them was delayed 3. The remaining interventions still answer key questions in the Education Sector Plan

4. Challenge 2: Convince local government officers to participate in the evaluation 1. October 2007:Meeting of representatives from every local council in the country 2. Initially: few were convinced to participate in the study 3. All became willing i) Extensive explanation ii) Hearing out all their concerns 4. Transparent lottery in same meeting: 4 councils were selected to participate in the pilot study, one from each of Sierra Leone’s four regions

5. Challenge 3: Delays in data collection 1. Initial date for collecting baseline data: November 2007 a) Final collection: May 2008 2. Key reasons a) Ongoing negotiations with WAEC on test design Finally: had to contract international consultant Finally: had to contract international consultant b) Funding disbursement (need to allow sufficient time for procurement and disbursement!) One more delay… in an enumerator reaching home! Two enumerators crossing the river. One fell out but was rescued. The perils of data collection! Two enumerators crossing the river. One fell out but was rescued. The perils of data collection!

6. Challenge 4: Rolling out textbooks to schools Challenge 4: Rolling out textbooks to schools 1. Roll-out timeline a) Original roll-out of textbooks planned for December 2008 b) BUT Ministry official accidentally released books to all schools (treatment & control) in November c) SO we identified other books (from a World Bank program) that could be used d) Plan: distribute in June 2008 e) As of June 2008: 3 of 4 had begun distribution f) As of present: Still not all disributed 1. A few comparison schools received 2. Some treatment schools still not received (including one district) g) Actual distribution: June 2008 (end of school year): Textbooks distribution delayed 2. Reasons for challenges a) Insufficient follow-up with Ministry book officer b) Service providers (NGOs that distribute the books) had limited understanding of the project at start 1. Poor communication between service providers and government

7. Challenge 5: Teacher Training 1. November 2007: Initial manual developed along with budget, but pilot and training not done Delay due to i) Ministry officials having other work engagements ii) Teacher Trainers who develop manual find it difficult to convene a meeting to revise manual iii) Funds were not available to revise and to train 2. Updated timeline developed in October 2008: piloting in December, training in January 3. Last week: Revised manual delivered! 4. Funding still delayed: Parliament delay in endorsing the ESP

8. Solutions 1. Local Councils i) Need repeated sensitization workshops to thoroughly explain the project (because councils are new and they have a lot in their hands) 2. Delays in baseline data collection i) Allow sufficient time for contracting and disbursement 3.Textbook distribution i) Field Coordinator monitor textbooks distribution (even travelling to the provinces) ii) Convene one, two day(s) workshop to sensitize service providers on guidelines iii) Provide letters from Ministry introducing team to Service Providers and seeking their cooperation.

9. 4. Teacher Training i) Make funds available on time for the teacher training ii) Set realistic timelines and keep to them 5. General i) Set regular, repeating time for team meetings (e.g., audio once a month and video once a month) ii) Set regular time for contact between field coordinator and technical advisors THANK YOU ALL FOR LISTENING