Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Presentation to a LAC PREM BBL 11 January 2007 Keith Mackay Independent Evaluation Group

Similar presentations


Presentation on theme: "1 Presentation to a LAC PREM BBL 11 January 2007 Keith Mackay Independent Evaluation Group"— Presentation transcript:

1 1 Presentation to a LAC PREM BBL 11 January 2007 Keith Mackay Independent Evaluation Group

2 2 Structure of Presentation 1.What does success look like? 2.Chile 3.Colombia 4.Australia 5.Definition of good practice M&E system 6.Lessons 7.Useful resource materials

3 3 1. What Does Success Look Like? -- Why Countries Want an M&E System To support budget decision-making = performance-based budgeting To support budget decision-making = performance-based budgeting To support national and sectoral planning To support national and sectoral planning To design policies and programs To design policies and programs To assist sector ministries / agencies in their management To assist sector ministries / agencies in their management To strengthen accountability relationships To strengthen accountability relationships

4 4 2. Chiles M&E System -- Architecture 2. Chiles M&E System -- Architecture Designed, managed and used by Hacienda Designed, managed and used by Hacienda Developed incrementally, over past decade Developed incrementally, over past decade Performance indicators ( 1,600) for all government programs (1994) Performance indicators ( 1,600) for all government programs (1994) Government program evaluations ( 185) – these are desk reviews (1996) Government program evaluations ( 185) – these are desk reviews (1996) Rigorous impact evaluations ( 18) (2001) Rigorous impact evaluations ( 18) (2001) Comprehensive Spending Reviews ( 8) – desk reviews of all programs in a functional area (2002) Comprehensive Spending Reviews ( 8) – desk reviews of all programs in a functional area (2002)

5 5 Chiles M&E System -- Strengths (i) Graduated approach to M&E Graduated approach to M&E Evaluations conducted externally, in fully transparent process, and are highly credible Evaluations conducted externally, in fully transparent process, and are highly credible All M&E findings reported publicly and sent to Congress All M&E findings reported publicly and sent to Congress M&E system closely linked to the budget information needs of Hacienda M&E system closely linked to the budget information needs of Hacienda Performance information used to set targets for ministries – these are largely met Performance information used to set targets for ministries – these are largely met Hacienda closely monitors extent of utilization of evaluation findings Hacienda closely monitors extent of utilization of evaluation findings

6 6 Chiles M&E System -- Strengths (ii) High utilization of M&E findings by Hacienda in the budget process and to impose management improvements on ministries / agencies (see Table) High utilization of M&E findings by Hacienda in the budget process and to impose management improvements on ministries / agencies (see Table) Utilization of government evaluations to 2005 Minor adjustment of program, e.g. improved info system Major change in mgment processes e.g. new targeting criteria, new MIS Substantial redesign of program or of organiz- ational structure Institutional relocation of program Program termination TOTAL 24%38%24%5%9%100%

7 7 Chiles M&E System -- Weaknesses Unevenness in quality of evaluations – due to cost and time constraints Unevenness in quality of evaluations – due to cost and time constraints Chile probably not spending enough on evaluations Chile probably not spending enough on evaluations Low utilization – low ownership – of Haciendas evaluations by sector ministries Low utilization – low ownership – of Haciendas evaluations by sector ministries

8 8 3. Colombias M&E System (SINERGIA) -- Architecture SINERGIA is managed by the Department of National Planning (DNP), with strong support from the Presidencia SINERGIA is managed by the Department of National Planning (DNP), with strong support from the Presidencia On-line sub-system – SIGOB – for monitoring and reporting government progress vis-à-vis Presidential Goals On-line sub-system – SIGOB – for monitoring and reporting government progress vis-à-vis Presidential Goals Ambitious agenda of impact and other evaluations ( 15 underway) Ambitious agenda of impact and other evaluations ( 15 underway) DNP provides technical assistance to a few ministries/agencies to develop M&E, and to municipalities to pilot SIGOB and performance-based budgeting DNP provides technical assistance to a few ministries/agencies to develop M&E, and to municipalities to pilot SIGOB and performance-based budgeting

9 9 Colombias M&E System -- Strengths Very high utilization of SIGOB by President for oversight of ministers and ministries – via performance targets – and for accountability, i.e. social control Very high utilization of SIGOB by President for oversight of ministers and ministries – via performance targets – and for accountability, i.e. social control Rigorous impact evaluations conducted externally, and have high credibility Rigorous impact evaluations conducted externally, and have high credibility Collaborative approach between DNP and sector ministries/ agencies, and with municipalities Collaborative approach between DNP and sector ministries/ agencies, and with municipalities

10 10 Colombias M&E System -- Weaknesses Insufficient reliance on M&E information to support national planning and budget decision-making – this may now be about to change Insufficient reliance on M&E information to support national planning and budget decision-making – this may now be about to change Weak coordination of M&E roles / functions within DNP and with central and sector ministries Weak coordination of M&E roles / functions within DNP and with central and sector ministries SIGOB data quality perceived as low SIGOB data quality perceived as low Too high reliance on donor funding for SINERGIA –low level of government funding support Too high reliance on donor funding for SINERGIA – low level of government funding support

11 11 4. Australias M&E System ( ) -- Architecture System was managed by Finance Department – emphasis on evaluation System was managed by Finance Department – emphasis on evaluation Evaluations mandatory – every 3-5 years for every program Evaluations mandatory – every 3-5 years for every program Sector departments required to prepare rolling, 3-year plans for major evaluations Sector departments required to prepare rolling, 3-year plans for major evaluations Broad range of evaluation types Broad range of evaluation types By mid-1990s, 160 underway at any point in time By mid-1990s, 160 underway at any point in time Much less emphasis on performance info Much less emphasis on performance info

12 12 Australias M&E System ( ) -- Strengths Evaluation findings heavily used in budget analysis, policy advice, and by the Cabinet in its budget decision-making Evaluation findings heavily used in budget analysis, policy advice, and by the Cabinet in its budget decision-making High utilization of evaluation findings by sector departments and agencies High utilization of evaluation findings by sector departments and agencies Evaluation was a collaborative endeavor between Finance Department, other central departments, and sector departments Evaluation was a collaborative endeavor between Finance Department, other central departments, and sector departments

13 13 Australias M&E System ( ) -- Weaknesses Uneven quality of evaluations Uneven quality of evaluations Insufficient attention to regular performance information Insufficient attention to regular performance information An administrative burden on departments was claimed An administrative burden on departments was claimed

14 14 5. How to Define a Government M&E System as Being Good Practice Can be dangerous concept – each countrys starting point and desired end-point are unique Can be dangerous concept – each countrys starting point and desired end-point are unique What a successful M&E system is NOT: What a successful M&E system is NOT: complex set of laws, decrees, regulations complex set of laws, decrees, regulations # performance indicators collected # performance indicators collected # rigorous impact evaluations conducted # rigorous impact evaluations conducted # rapid evaluations conducted # rapid evaluations conducted = these simply reflect the architecture of the system or are measures of M&E effort = these simply reflect the architecture of the system or are measures of M&E effort

15 15 How to Define a Government M&E System as Being Good Practice Chile, Colombia and Australia have good- practice M&E systems because Chile, Colombia and Australia have good- practice M&E systems because 1.the quality of their M&E work is generally reliable, and 2.in particular, the monitoring information and evaluation findings which they produce are used intensively High utilization reflects strong demand for M&E, and can be viewed as a predictor of an M&E systems sustainability High utilization reflects strong demand for M&E, and can be viewed as a predictor of an M&E systems sustainability

16 16 6. Lessons (i) Lessons from the 3 countries are consistent with other countries experience Lessons from the 3 countries are consistent with other countries experience 1. Key role of powerful champion of M&E 2. Opportunistic development of M&E systems, via continuous review and modification; non-linear development of the systems 3. Centrally-driven, by capable ministry 4. Incentives are key – cultural change to strengthen demand, achieve high utilization 5. Avoid competing systems – Planning, Finance, Presidencia

17 17 Lessons (ii) 6. Build reliable ministry data systems 7. Role of structural arrangements to ensure M&E objectivity and quality 8. Long-haul effort, requiring patience 9. Limitations of relying on laws, decrees, regulations 10. An M&E system can be built and operated are relatively low cost Chiles M&E system costs $0.75m p.a. Chiles M&E system costs $0.75m p.a. Colombias system costs about $2m p.a. Colombias system costs about $2m p.a.

18 18 7. Useful Resource Materials (i) Ernesto May et al. (eds.), Towards the Institutionalization of Monitoring and Evaluation Systems in Latin America and the Caribbean, World Bank/IADB, ntMDK: ~pagePK:146736~piPK:146830~theSitePK:258554,00.html (Disponible en Español) ntMDK: ~pagePK:146736~piPK:146830~theSitePK:258554,00.html Fernando Rojas et al, Chile: Análisis del Programa de Evaluación del Gasto Publico, World Bank, wAttachments?ReadForm&ID=85256D CC CB26B& wAttachments?ReadForm&ID=85256D CC CB26B& World Bank, A Diagnosis of Colombias National M&E System, SINERGIA, World Bank, 2007 (forthcoming). Keith Mackay, Two Generations of Performance Evaluation and Management System in Australia, World Bank, h/ A1A709BD85256E AA8/$file/ecd_wp_11.pdf h/ A1A709BD85256E AA8/$file/ecd_wp_11.pdf

19 19 Useful Resource Materials (ii) World Bank website on Building Government M&E Systems: Chile: Colombia: Keith Mackay, Institutionalization of Monitoring and Evaluation Systems to Improve Public Sector Management, World Bank, (Disponible en Español) Teresa Curristine, Performance Information in the Budget Process: Results of the OECD 2005 Questionnaire, OECD Journal on Budgeting 5(2), 2005, pp


Download ppt "1 Presentation to a LAC PREM BBL 11 January 2007 Keith Mackay Independent Evaluation Group"

Similar presentations


Ads by Google