The role of civil society to strengthen accountability and to build evaluation systems The experience of Fundación IDEA November 2013

Content About Fundación IDEA The Indigenous Education Gap Index PEC FIDE (School based management program): evaluation method and results Civil society’s role in M&E M&E and policy-making 2

Fundación IDEA is a non-partisan think tank that provides rigorous analysis and sound solutions. 3 Our vision: informed policy decisions Rigorous analysis and sound solutions 8 years of experience Projects in Mexico, Guatemala, Colombia, USA, Italy, Peru Impartial and non-partisan Work on a variety of topics: education, economic development…

“Evaluación Transversal: políticas y programas para el desarrollo de la juventud” “Estrategia nacional para la vivienda sustentable” “¿Qué hacer con la diabetes?” “¿Quiénes son los emprendedores innovadores mexicanos?” “Propuesta de programa de inclusión laboral para jóvenes” “Estudio sobre calidad docente en zonas rurales e indígenas en México” “Índice compuesto de eficacia de los sistemas escolares” “Índice de equidad educativa indígena” “Evaluación de impacto PEC-FIDE (Fortalecimiento en inversión directa a escuelas)” 4 Fundación IDEA works on a variety of topics. Fundación IDEA has developed expertise on education.

There is a gap of 34%. Implications of the study: Showing the gap to push for targeted policy. 6 Fundación IDEA found that indigenous people in Mexico have achieved 66% of the educational achievement compared to non-indigenous.

Impact evaluation of the federal School-Based Management Program PEC FIDE (PROGRAMA ESCUELAS DE CALIDAD-FORTALECIMIENTO EN INVERSIÓN DIRECTA A ESCUELAS) Pilot variation of program School zone approach Focus on very low-income neighborhoods Focus on at risk students Focus on training (shift from insfrastructure) More money per school than average PEC Dropout rates Graduation rate Learning outcomes Relevant SBM indicators Short-runLong-run

8 Fundación IDEA was the external evaluator. PEC FIDE Impartial, rigorous approach, publish results; Usefulness of recommendations; Implications of evaluation: design programs based on evidence, keep programs that work, improve programs that do not work. Evaluation context: 50% of schools were beneficiaries (selection bias). Info available: annual school census; ENLACE test; population census. Methodology (mixed methods): quantitative: diff-in-diff w/ propensity score matching. qualitative: interviews, participant observation and document analysis in 10 schools.

Tasa de aprobación Tasa de aprobación* Efectos después de tres ciclos es escolares Efectos de un ciclo escolar Resultados de Skoufias y Shapiro (2006a) Resultados de Murnane, Willett y Cárdenas (2006) 1.6 pp Tasa de aprobación Efectos de un ciclo escolar Resultados de Abreu y Santibañez (2010) 0.24 pp0.11 pp PEC FIDEPEC vs The evaluation showed positive impact for PEC FIDE ( ) in graduation rates. Other results: Approx 20 more points in ENLACE test (reading comprehension); approx 1/3 of standard deviation. Approx -4 percentage points of students at insufficient level in ENLACE test.

Inicio de ciclo Fin de ciclo puntos de ENLACE/año 12.8 puntos de ENLACE/año The evaluation showed impact on reading comprehension at the elementary school level (ENLACE test).

Fundación IDEA presented results to administrators of the program. Qualitative analysis showed increased motivation. Investment focused on infraestructure, hard shift to training. Limited participation: how to promote more and better participation? The importance of targeting neighborhoods? Importance on school zone program design? Targeting students at risk?

Content About Fundación IDEA The Indigenous Education Gap Index PEC FIDE (School based management program): evaluation method and results Civil society’s role in M&E M&E and policy-making 12

13 Rigorous analysis Credibility of conclusions Usefulness of recommendations Public policy based on evidence Programs may be implemented because of: -Evidence -Experience -Ideology -Politics -Budget -Technical capacity Civil society needs to promote an evidence based policy culture.

Evaluation should be used as a tool for decision making and efficient spending. We are experiencing a paradigm shift in M&E. Evaluation is a tool for accountability and transparency - programs use public resources and so society needs to know if objectives are accomplished and if programs are effective Evaluation as a learning process - to improve design and operation of programs - to generate knowledge about complex problems and about interventions that work - to know if a program works or not - “bang for the buck”

15 M&E is used for decison making t No M&E Some M&E activities Good consumer and capacity to perform M&E activities M&E, if appropriately used, can improve how programs are managed and the outcomes they can have. The use of M&E by organizations

16 Impact evalImplementation evalSound indicatorsNeeds assessment There are some misconceptions on M&E practice. Types of evaluation: Needs assessment Program theory (theory of change) Design evaluation Implementation or process evaluation Results (change) Impact (causality) Efficiency (CBA, cost effectiveness)

Impact evaluation can change policy if implemented based on a good understanding of M&E practice. Key lessons for capacity building: Valid Counterfactual - understand when to evaluate impact and when not - many organizations decide to evaluate before-after or compare beneficiaries with non-beneficiaries: not always good use of resources Theory of Change - important for understaning what the organization is doing and how they are measuring it - sometimes there is no clarity/consensus of the purpose of a program among stakeholders ObjectivesInputsProducts Intermediate results Impact

18 Thank you