WHAT CAN WE LEARN FROM SUCCESSFUL COUNTRY SYSTEMS? PHILIPP KRAUSE PRMPR 1

Overview 2  Why bother?  A selection of country cases  How do differences matter?  What do “successful” systems have in common?

Why bother? 3  Two schools of thought  The “best practice” school  The “It’s all about the context” school  The truth lies somewhere in between – there are important characteristics shared by successful cases, but differences between countries matter tremendously  Most important reason: Countries do M&E for different reasons and different users – what you need depends on what you want to do with it  Our definition of success:  High degree of utilization  M&E information meets quality standards and is reliable  System is sustainable over time

Overview 4  Why bother?  A selection of country cases  How do differences matter?  What do “successful” systems have in common?

Australia: Main Features 5  Mandate to evaluate each program every 3-5 years  Portfolio Evaluation Plans to be prepared annually for the following three years  Department of Finance: Steering & quality control  Sector Departments: planning, implementation  Evaluation results primarily used for budgetary decisions: allocations of funds for new policies and reallocation of savings (i.e. the discretionary part of annual budget process)  System lasted from – sustainability?

Australia: How does it work? 6 Sector Departments (and outrider agencies) Treasury Department of Prime Minister and Cabinet Department of Finance Parliament Citizens Inputs to PEPs Inputs to Portfolio Evaluation Plans (PEPs) Formal notification of PEPs Involvement in evaluations Publication of department’s evaluation reports Reporting of key Evaluation findings in department’s budget paper (prospective) and in their annual reports (retrospective) Source: Mackay 2011

United Kingdom: Main Features 7  Comprehensive system of performance monitoring and targets: Spending Reviews (multiannual budgets) and Public Service Agreements (“resources for delivery”) in return for (relative) managerial flexibility  Oversight and leadership from central executive (Prime Minister, PM’s Delivery Unit, Treasury)  No systematic evaluations in the executive government  Problem: “Gaming in Targetworld”  Value-for-Money Audits: 60 per year from National Audit Office

Delivery Unit* UK: How does it work? 8 HM Treasury Parliament audits Prime Minister’s Office is accountable to reports to National Audit Office Ministers Spending Unit monitors priority areas reporting/monitoring is accountable to Negotiations over PSAs * Moved to HM Treasury in 2007, abolished 2010

Mexico: Main Features 9  M&E of social policies delegated to a specialized technical agency – CONEVAL  CONEVAL in charge of evaluation portfolio, development of methods, dissemination  Key oversight decisions by inter-ministerial committees – system involves many stakeholders (Congress, Finance, Presidency, Public Admin Ministry, Sector Ministries, CONEVAL)  Implementation of evaluation and day-to-day operation of monitoring done by ministries

Mexico: How does it work? 10 Inter-ministerial Committee Sector Ministry CONEVAL Sector Ministry Sector Ministry CONEVAL Define the evaluation plan Decide what and how to evaluate Coordinate and steer evaluation system Maintain quality control Define monitoring parameters Select and hire evaluators Supervise evaluation implementation Monitor indicators and targets Programs are evaluated by ministries Collection of evaluation results Dissemination of evaluation results Utilization of evaluation results Congress, Presidency, Sector Ministries, Finance

Chile: Main Features 11  System is highly centralized – closely linked to and based on a highly centralized, top-down budget process  Budget office manages most details, and is main user of information – utilization fostered by close link to budget process  Management control by hierarchical oversight  Little buy-in from other (potential) stakeholders  Some impact on allocations and program management Chile: Utilization of Government Evaluations—2000–05 Effect on program Minor adjustment of program, for example, improved processes or information systems Major adjustment of management processes, for example, changed targeting criteria, or new information systems Substantial redesign of program or organizational structure Institutional relocation of program Program termination Programs affected 24%38%25%5%8% Percentage of all evaluated programs. Source: Guzman 2007

Chile: How does it work? 12 Budget Office Budget Office Budget Office Programs Budget Office Define the evaluation plan Decide what and how to evaluate Coordinate and steer evaluation system Maintain quality control Define monitoring parameters Programs are evaluated directly by Budget Office Select and hire evaluators Supervise evaluation implementation Monitor indicators and targets Collection of evaluation results Dissemination of evaluation results Utilization of evaluation results Budget Office Congress

Centralization vs. Delegation 13 Inter-ministerial Committee Mexico Sector Ministry CONEVAL Sector Ministry Sector Ministry Chile Budget Office Budget Office Budget Office Programs Budget Office CONEVAL Congress, Presidency, Sector Ministries, Finance Define the evaluation plan Decide what and how to evaluate Coordinate and steer evaluation system Maintain quality control Define monitoring parameters Programs are evaluated directly by Budget Office Select and hire evaluators Supervise evaluation implementation Monitor indicators and targets Programs are evaluated by ministries Collection of evaluation results Dissemination of evaluation results Utilization of evaluation results Budget Office Congress

Overview 14  Why bother?  A selection of country cases  How do differences matter?  What do “successful” systems have in common?

Institutional Differences Matter: Index of Legislative Budgetary Powers Chile Mexico Source: Wehner 2007

Difference in Purpose 16  Budgetary: To inform budgetary decisions, best allocation of resources between sectors and programs, also to enforce operational savings in annual budget  Accountability: External accountability towards legislature, stakeholders and public. To make systematic information on performance available and strengthen the public evidence base of policy decisions  Control: To develop better central government information on implementation and service delivery of public programs as tools to hold managers to account

Matrix of Country Comparison 17 CountryPurposeLeadershipOperationUsers Chile Budget/ControlDirectCentralizedSingle Mexico Budget/AccountabilityDelegatedDecentralizedMultiple Australia BudgetDirectDecentralizedSingle Canada BudgetDirectDecentralizedSingle UK/NAO* AccountabilityDirectCentralizedMultiple * Supreme Auditor

Features that Matter 18  Centralization requires the right institutional structure – a centralized M&E system design in a fragmented public sector will fail  In a system with multiple stakeholders delegation to an impartial agency might be a viable option – but beware of over-engineering and objective overload  Who is to gain and who has to worry about buying into a M&E system: The senior civil service, the legislature, ministries, service delivery units, the finance ministry, the head of government (PM or President)?  Staying in control of overall steering and quality of outputs does not equal having to internalize all aspects of M&E implementation – strategic delegation might be smart for buy-in and workload  Is it possible to imagine a long-term sustainable, well utilized M&E system that does not have a stable link to budgetary decisions?

Overview 19  Why bother?  A selection of country cases  How do differences matter?  What do “successful” systems have in common?

Lessons Start M&E Systems Successfully* (1) 20  Somewhere in government is substantive demand for M&E information. This is necessary to start and sustain an M&E system  Actors need to have incentives to engage with M&E. They are key for M&E to be conducted and for the information produced to be utilized  Simple is better – successful M&E systems tend to deliver just what users want, not more. They also serve only those objectives that result in utilization  Success is more likely with a powerful champion(s) to lead the push for institutionalization of M&E – rather than a legislative or technical exercise *very liberally adapted from Mackay 2010

Lessons Start M&E Systems Successfully (2) 21  It is important to have the stewardship of a central, capable ministry that can design, develop, and manage the system  Some reforms may start with a bang, but it requires patience, determination, and a long-term effort to build an effective M&E system  For donors: It helps the process to start with a diagnosis of what M&E functions already exist in the country (and why other M&E functions do not exist – they usually don’t for a reason)

22 THANKS! For further information, and to access the sources cited here, please visit: