1. IMPROVING EDUCATION MANAGEMENT IN MADAGASCAR (AGEMAD) ACCRA, GHANA Mai 11, 2010 Results of an Impact Evaluation Préparée par Jee Peng et Cornelia Paul RANDRIANIRINA Présentée par

2. Primary Education in Madagascar: Much Progress, but still Many Challenges Signs of progress: Primary completion rate doubled from 35% in1999 to 71% in 2008 Evidence of weak sector performance: ½ of each cohort of 1 st graders does not finish the primary cycle; Repetition rate still high at 18% in 2005 (30% in 2000) Low quality: in 2004-5 PASEC, average test score of 50% in Maths and Malagasy and 32% in French; deteriorated since 1997-98 Multiple systemic causes : Inconsistencies in teacher allocation across schools; Ineffective management of pedagogical processes at school and classroom levels

3. Les défis de lAGEMAD Améliorer lallocation des ressources entre les écoles. Assurer que les ressources allouées sont transformées en résultats au niveau des élèves. Identifier des interventions qui permettent de renforcer la gestion du système. Tester les interventions et évaluer leur impact afin déclairer les choix sur des généralisations possibles

4. Une démarche en 4 étapes : 1. Identifier acteurs du système éducatif = ceux qui ont des responsabilités à assurer 2. Déterminer quelles tâches ils ont à accomplir 3. Elaborer des outils de travail pour quils accomplissent ces tâches: outils (procédures, tableaux de bord, statistiques) rationalisés, adaptés, conçus et testés en collaboration avec les utilisateurs 4. Clarifier les conditions incontournables pour la bonne exécution du système de gestion.

5. Tighter Management to Improve Accountabilty Conceptual Intervention Framework & IE Design: Workflow tools to clarify tasks and internal accountabilities; Facilitation of meetings between school and community; Better information flows within school and between school and community; Structured training for teachers and school heads Leading to: improvement in actors behavior through bottom up and top down accountability better managed school increased school quality higher student learning

6. Key Questions for Policymakers What is the impact of tighter management of processes on school functioning and student performance? At what administrative level are management interventions the most effective (school, district or inspection level)?

7. Impact Evaluation Design (1) Method: Randomized experimental design over 2 school years Interventions: Specify actors responsibilities & their mutual accountability processes through: Management Tools and Guides for key tasks (e.g. pedagogical, administrative) Training Focus attention on results by clarifying goals through: Report cards: School, district and inspection report cards School meetings: Facilitated school meetings & development of school improvement plans based on school report cards

8. School & District Report Cards for Better Information Flow Report cards for school directors, sub-district and district levels officers: Complement the tools and processes Draw attention to schooling outcomes Include comparative data, allowing a school to compare its outcomes with those of other schools Serve as basis for dialogue and accountability

9. Impact Evaluation Design (2) 9 303 Schools AGEMAD TREATMENT 3 303 Schools ZAP AGEMAD TREATMENT 2 15 CISCO AGEMAD 89 ZAP AGEMAD 80 ZAP CONTROL 303 Schools CONTROL 15 CISCO CONTROL 84 ZAP NON-AGEMAD 303 Schools CISCO AGEMAD TREATMENT 1

10. Collecting Data Actors Behavior (direct effects): Questionnaire from impromptu school visits in 1,200 schools, with information for 4,000 teachers Questionnaires for District and Community admin. level Collection and analysis of tools used in 40 schools (850 tools) Schooling outcomes (indirect effects): Test scores from standardized tests in 3 subjects National year-end school census data: flow rates, repetition, CEPE pass rate Timeline: 2 school years, 2005-2007 Baseline survey/test and post-intervention survey/test

11. What tasks are deemed essential? Teacher: Takes daily roll call Prepares daily lesson plan Prepared bi-monthly lesson plans Monitors student learning Has tested pupils during the past two months Helps lagging students Discusses student learning issues with the director School director: Keeps a register of enrollments Signs off on daily roll call Analyzes student absences on a monthly or bi-monthly basis Reviews pupils test results Takes stock of teacher absences Informs sub-district or district officer about teacher absences Follows up with teachers on lesson planning

12. Results: Effects on Actors Behavior AGEMAD schools Control schools Teacher absence (%)8.79.2 Teacher completes all key tasks (%)63.0*42.4 All teachers in school perform their key tasks (%) 42.9**22.1 Well managed schools (%)36.5**15.2 *significant at 5% level; **significant at 1% level

13. 13 Results: Effects on Schooling Outcomes AGEMAD schools Control schools Student attendance (%)90.7*86.6 Repetition (%)17.5*22.6 Drop out (%)5.56.1 Success rate at CEPE exam73.061.9 Student test results (post-test) Math51.249.4 Malagasy50.948.5 French30.029.4 All subjects43.541.9 *significant at 5% level

14. Policy Implications Prioritize school-level actors Cascade training model alone, as currently defined, doesnt work Though results are encouraging, better management essentially entails changing peoples behaviors, which takes time and effort Mainstream IE results into MoE activities Need a champion from the start Need early involvement of a national team, with good technical support Necessary to sustain change in actors attitudes & behaviors Use existing structures and mechanisms for scale up: Tools, guides and training modules integrated into teacher training Tool distribution, training and facilitated school meetings funded through the local catalytic funds based on regional, district and school performance plans and needs Develop leaders to drive change in management practices Discussion underway on collaboration in leadership training between Madagascar MoE and partner organization in another country

15. Stay tuned…Publications forthcoming Africa Human Development Working Paper Series «Améliorer la gestion de l'enseignement primaire à Madagascar - Résultats d'une expérimentation randomisée » Journal Article undergoing peer review «Managing for results in primary education in Madagascar: Evaluating the impact of selected workflow interventions »

16. It takes a village… Government commitment: Stable counterpart team (15 staff from MoE with coordinator) Partner commitment: Financial and technical assistance from AFD (via two staff) WB team lead by Jee-Peng Tan and Cornelia Jesse, consisting of Gérard Lassibille and Trang van Nguyen (with in-country field coordinators) Local NGO Aide et Action to assist with training Financing: WB, AFD, MoE, EFA-FTI (EPDF), Irish Aid, Ministry of Foreign Affairs of Norway Timeline: 2004 – 2007 Total number of people involved: 50

17. The Perils of Data Collection…