1. DIRECCIÓN DE PLANEACIÓN Y NORMATIVIDAD DE LA POLÍTICA DE EVALUACIÓN An Independent Evaluation Unit in Mexico: CONEVAL Gonzalo Hernández Licona 2014

2. Evaluation. Main Objectives  Improve social policies (programs, actions, strategies), through the generation of evidence.  An external eye can detect problems that managers cannot see  Accountability.  It is common for governments (and individuals) to say that everything is OK. Big political pressure  These 2 objectives require a delicate balance……

3. Where do we come from in Mexico?  Mexico was a rather authoritarian state, despite elections...  The Mexican State was unreliable  Elections  Economic and social figures  The political process changed in 1997, when Congress was no longer dominated by the President’s political party.  It started then a real balance of power between Congress and the Executive. This is the beginning of Democracy in Mexico..

4. Institutional milestones Mexico 1997-2014  1997 Progresa Evaluation  2000 Congress’ Decree: annual external evaluations to all federal programs  2001 Evaluation Units within ministries  2001 National Audit Office  2004 Law of Transparency and Public Access to Information  2005 Social Development Law  CONEVAL. National Council for the Evaluation of Social Policy  2006 Budget Law: Performance Evaluation System

5. CONEVAL  Mandate:  Measurement of Poverty at the National, State and Municipality level (multidimensional methodology)  Evaluation of social programs and policies  Governance  CONEVAL is part of the Executive, but  The Board has 8 seats. The majority of Board members (6) are academic council members elected by: all the States, Municipalities, Congress and the Executive (44 votes)  Technical and managerial independence

6. CONEVAL. Academic Council Members  The importance of the academic council members:  They are not appointed by the Executive  They are a majority on the Board  We have been able to say what we think, through evidence, about the social policy, including that poverty increased 2008-2012: Credibility  The Executive Secretary (CONEVAL’s director) is appointed by the President, but the academic council members are in fact a shield against any temptation from the government/political parties/state governors

7. USING THE RESULTS

8. Changes in social policy due to reliable information (evidence)  The government launched in 2013 an strategy to reduce hunger and extreme poverty.  The budget for the Cement Floor program was increased nearly 400% between 2007 and 2012.  The program Employment for the Youth was cancelled in 2009.  The food supplement of the Progresa- Oportunidades program was modified in order to have a better iron formulae in 2003.  The rural program Procampo is now less regressive, due to a design change in 2009.  CONEVAL published that poverty and food deprivation increased between 2008 and 2012.  The impact evaluation demonstrated reductions of gastrointestinal diseases for children do to the program  The design evaluation of the program showed various design and operational flaws at a considerable cost.  The impact evaluaton showed the iron had not been absorbed by children.  The distributional analysis illustrated that the program was very regressive.

9. SOME LESSONS

10. Lessons  The balance of power between Congress and the Executive was important in Mexico  Credibility and Trust are at the center of the institutional arrangement  In CONEVAL, the academic council members are the main source of independence.  Publish in advance the agenda for releasing evaluations and poverty estimates to the public  Transparent and replicable poverty figures  Keep the fine balance between Accountability and Policy Improvement.  Preferably release evaluations together with the improvements already made by programs  Each country will trace its own path

11. ANNEX

12. CHALLENGES

13.  Better coordination between evaluation institutions: CONEVAL, Ministry of Finance  More capacity building  Improve results’ indicators in many programs  Evaluation of Sectors/Ministries.  Evaluations and transparency in states and municipalities.  Use more evidence in budgetary, operative and strategic decisions. Challenges for Mexico

14.  Since 2014 CONEVAL is an autonomus institution.  The institution electing the council members changed the balance of power between political parties.  The previous party in power (now in the opposition) was not pleased with the poverty figures  The new government found in 2013 an uknown body evaluating it’s policies CONEVAL, a constitutional autonomus institution

15. /1 Information from the Diagnostics of the Indicators Matrixes for Results 2008 and 2010. /2 Information from the Follow-up of Aspects Susceptibles of Improvement System. The estimation considers the aspects susceptibles of improvement attenden by a 100% to march 2012. /3 The estimations includes two programas that finished their obligations but are not in operation since 2011. Various social programs and their budget have changed due to the “Evaluation- Improvement Mechanism”: Type of Improvement 2010 /1 2011-2012 /2 Programs Relative Participation Programs /3 Relative Participation Improvement in programs´ activities or processes 1916.8%3547% Improvement in the services offered by programs 2219.5%1014% Reorientation of programs7162.8%2635% Programs were merged- - 34% The program was cancelled10.9%00% TOTAL113100.0%74100%

16. Products and facts  There are poverty figures at a national, state and municipality level  Almost 550 programs have Log Frameworks. All social programs plus others…  We hired ECLAC to help us with the capacity building for the Log models.  We have offered an evaluation course every year for policy makers and every 2 for researchers.  20% of all indicators are oriented to measure Results  150 programs are evaluated every year (now every two). There are around 179 social programs. This is 90% of the total budget.  270 one-page summaries every year.  Between 3-4 impact evaluations are done every year.  CONEVAL budget is around $23 millions on even years (household survey); $16 millions on odd years. But there is more budget for evaluation within ministries.  We can find on the internet:  Poverty estimates  All the evaluations  The program’s point of view about its evaluation  Each program’s Work Plan

17. Important elements  Information on poverty and evaluations is open and transparent.  The Finance Ministry has been an important partner in the process.  CONEVAL and the Ministry of Finance produced Guidelines for the Evaluation of Programs.  The (autonomous) Statistical Office has invested a lot on data collection.  Capacity building

18. Monitoring and Evaluation System Ministries’ Strategic objectives Logical Framework: Programs Results Recommendations’ follow-up Consistency & Results Evaluations Impact Evaluations Policy Evaluations Process evaluations Annual Evaluation Plan Planning Evaluation National Development Plan Annual Performance Report Multidimensional Poverty Estimations