1. DIRECCIÓN DE PLANEACIÓN Y NORMATIVIDAD DE LA POLÍTICA DE EVALUACIÓN The Importance of the Institutionalization of M&E: The Mexican Case Gonzalo Hernández Licona 2013

2. Institutional milestones Mexico 1997-2009  2000 Congress’ Decree: annual external evaluations to all federal programs  2001 Evaluation Units within ministries  2005-6 Social Development Law  CONEVAL. National Council for the Evaluation of Social Policy  Multidimensional Poverty Methodology  2006 Budget Law: Performance Evaluation System  2007 Evaluation Guidelines  CONEVAL, Ministry of Finance, National Audit Office  2009 Government Accountability Law

3. Challenges for constructing an Evaluation system:  Institutional: It’s almost impossible to have a public Evaluation system without a proper institutional arrangement: evaluation mandate, evaluation unit, feedback procedures for policy improvements, norms about transparency.  Technical: Suitable methodologies, trained evaluators, administrative records, information. Constructing an M&E system is a political and institutional task with technical elements….not the other way round….

4. CONEVAL  Mandate:  Measurement of Poverty at the National, State and Municipality level  Evaluation of social programs and policies  Governance  CONEVAL is part of the Executive, but  The Board has 8 seats. The majority of Board members (6) are academic researchers elected by all the States, representatives from Municipalities, Congress and the Executive (44 votes)  Other two seats: M. of Social Development and Finance.  Technical and managerial independence

5. THE M&E SYSTEM

6. Identify the main social problems to prioritize strategies and resources. Public Policy Actions  Which programs can be related to the social priorities?  Improvement of Programs and Policies Measuring Poverty (various dimensions) Evaluation of Programs and Policies M&E: Budget decisions based on Results.

7. Measuring poverty: Social Development Law Dimensions for poverty measurement Current income per capita Educational gap Access to health services Access to social security Quality of living spaces Housing access to basic services Access to food

8. Evaluation Guidelines for all institutions, together with the Ministry of Finance: The Demand for evidence was clearer now Ministries’ Strategic objectives Logical Framework: All Programs (SYS) Results Recommendations’ follow-up (SYS) Consistency & Results Evaluations (SYS) Process Evaluations Policy Evaluations Impact Evaluations Annual Evaluation Plan PlanningEvaluation National Development Plan Annual Performance Reports (SYS)

9. Uses of the Information from the Performance Evaluation System Recommendations´ follow-up 4 Release to press and civil society Release to press and civil society 3 Information for the budget allocation process 2 Congress: Budget Comission M Finance + Explanatory meetings + Meetings with Congressmen Website Press Civil society Work documents Handing in to relevant actors 1 Congress M Finance Presidency States

10. Products and facts  There are poverty figures at a national, state and municipality level  Almost 600 programs have Log Frameworks. All social programs plus others…  We hired ECLAC to help us with the capacity building for the Log models.  We have offered an evaluation course every year for policy makers and every 2 for researchers.  25% of all indicators are oriented to measure Results  140 programs are evaluated every year (now every two). There are around 273 social programs and actions. This is 90% of the total budget.  258 one-page summaries every year.  Between 2-4 impact evaluations are done every year.  CONEVAL budget is around $23 millions on even years (household survey); $16 millions on odd years. But there is more budget for evaluation within ministries.  We can find on the internet:  Poverty estimates  All the evaluations  The program’s point of view about its evaluation  Each program’s Work Plan

11. INFORMATION: EXAMPLES

12. Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2008 y 2010 Population whose income is below the wellbeing threshold Access to food Access to basic services in the dwelling Quality and spaces of the dwelling Access to social security Access to health services Educational gap Millions of persons Extreme Poverty 2008 44.5 % 48.8 millions 2010 46.2 % 52.0 millions 2008 10.6 % 11.7 millions 2010 10.4% 11.7 millions 6 4 2 0 -2 -4 -6 -8 -10 -9.0 -2.9 -2.5 -2.3 -0.8 4.1 3.5 3.2 0.0 Social Deprivations 4.8 Population whose income is below the minimum wellbeing treshold Evolution of multidimesional poverty, Mexico 2008-2010

13. Percentage of population living in poverty by municipality. Mexico, 2010 Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2010 and the Census 2010

14. Performance Assessment of Social Programs The Traffic Lights System Each program’s evaluations results are rated according to the following categories: Results Design Results Oriented Planning Coverage and targeting Operation Perception of targeted population Follow-up of recommendations Logical framework Budget compliance Distributional impact Each program’s evaluations results are rated according to the following categories: Results Design Results Oriented Planning Coverage and targeting Operation Perception of targeted population Follow-up of recommendations Logical framework Budget compliance Distributional impact

15. PROCAMPO’s Performance Assessment

16. Performance Assessment of Social Programs

17. USING THE RESULTS FROM M&E IN SOCIAL POLICY

18. Changes in social policy due to the M&E System  The government launched in January 2013 an aggressive strategy to reduce hunger and extreme (multidimensional) poverty.  In September 2013 the President sent a bill to Congress to reshape the way social security has been working for almost 50 years.  The fiscal reform of 2013 changed the way resources will be allocated between States and Municipalities.  Since 2009, the Ministry of Finance uses information from the evaluation process to elaborate the annual budget.  There is a constant public and in-house debate and analysis (Presidency, ministries, programs, Congress, media) about the public evaluations of social programs and policies.  CONEVAL published that poverty and food deprivation increased between 2008 and 2012, and the Government acknowledged it.  Multidimensional poverty estimations showed that the larger deprivation is the access to social security (61%)  The reform has the goal to reduce extreme poverty according to the new multidimensional poverty methodology.  There are now annual information for almost 270 social programs. For some of them there are impact, process, design evaluations.  All evaluations are discussed with policy makers and all of them are public.

19. Changes in social policy due to the M&E System  The budget for the Cement Floor program was increased nearly 400% between 2007 and 2012.  The program Employment for the Youth was cancelled in 2009.  The food supplement of the Progresa- Oportunidades program was modified in order to have a better iron formulae in 2003.  The rural program Procampo is now less regressive, due to a design change in 2009. program were changes due to the evaluation.  Due to the design evaluation of the National Crusade Against Hunger, the strategy improved definitions, diagnosis and coordination tools  The impact evaluation demonstrated reductions of gastrointestinal diseases for children do to the program  The design evaluation of the program showed various design and operational flaws at a considerable cost.  The impact evaluaton showed the iron had not been absorbed by children.  The distributional analysis of various programs illustrated that the program was very regressive.  In July 2013 CONEVAL launched and evaluation to the CNCH, which was discussed with the Ministry of Social Development and also was public. The goal: On going improvement of the strategy.

20. /1 Information from the Diagnostics of the Indicators Matrixes for Results 2008 and 2010. /2 Information from the Follow-up of Aspects Susceptibles of Improvement System. The estimation considers the aspects susceptibles of improvement attenden by a 100% to march 2012. /3 The estimations includes two programas that finished their obligations but are not in operation since 2011. Various social programs and their budget have changed due to the “Evaluation- Improvement Mechanism”: Type of Improvement 2010 /1 2011-2012 /2 Programs Relative Participation Programs /3 Relative Participation Improvement in programs´ activities or processes 1916.8%3547% Improvement in the services offered by programs 2219.5%1014% Reorientation of programs7162.8%2635% Programs were merged- - 34% The program was cancelled10.9%00% TOTAL113100.0%74100%

21. CHALLENGES

22.  Better coordination between evaluation institutions and budget allocation decisions based on results: Planning vs Budget; CONEVAL, Ministry of Finance and Ministry of Audit  More capacities on evaluation and measurement tecniques  Improve indicators of various programs  Indicators of quality of educational and health services  List of beneficiaries  Evaluation of legislation and norms’ changes  Evaluation of Sectors/Ministries. Strategic objectives.  Rigorous evaluations and transparency in states and municipalities.  Take more into account the information from evaluations in budgetary, operative and strategic decisions.  A closer approach with the Executive Challenges for Mexico

23. SOME LESSONS

24. Lessons  Countries should find their own institutional arrangement: Chile, Colombia, South Africa, USA, Canada, Mexico, China, etc.  The balance of power between Congress and the Executive is important  Public pressure helps  Credibility is at the center of the institutional arrangement  Champions are always a key factor  Internacional help is important…but try to build your own path  Impact evaluations are not always the first step, more basic evidence is sometimes more important  South-South exchanges of knowledge and innovation  Keep the fine balance between Transparency and Policy Improvement.  On of the main users: Ministry of Finance

25. ANNEX

26. Why all these changes in a country where citizens usually mistrust government information and where political parties fight for resources?  Since 1997 there is a balance of power between Congress and the President.  Congress demanded the creation of an independent institution, CONEVAL, for the measurement of poverty and for evaluation.  Information on poverty and evaluation is public and transparent.  The Finance Ministry has been an important ally in the process.  CONEVAL and the Ministry of Finance produced Guidelines for the Evaluation of Programs.  The center of the Guidelines: Results  The Guidelines also created an “Evaluation-Improvement Mechanism”  Most ministries have cooperated in the design of the M&E system.  Fine balance between: Accountability and Policy Improvement.  The (autonomous) Statistical Office has invested a lot on data collection.  Good academic skills and suitable evaluation methodologies have been increasingly disposable for the past 15 years.

27. Evidence must be part of a planning system  Identify and measure social challenges  Analysis  What works?  Program design  Budget  Implementation  Monitoring & Evaluation Why do we need evidence?  Improve social policy  Make better decisions (management, design, budget…)  Accountability

28. Multidimensional Poverty Measurement (National and State Estimations) o Economic wellbeing o Education lag o Access to health services o Access to social security o Quality and spaces of the dwelling o Access to basic services in the dwelling o Access to food o Social cohesion Social Gap Index Poverty Labor Trend Index (ITLP) Poverty Maps Income Poverty Measurement Annual Evaluation Plan Log Framework: Programs Consistency & Results Evaluations Impact Evaluations Complementary Evaluations Performance Evaluations Programs’ Performance Summary Social Development Programs’ Inventory Thematic Evaluations Policy Evaluations Recommendations’ Follow-up Document for Budgetary Considerations Social Programs Evaluation Poverty Measurement and Analysis Main Products