Module 5.2 Measuring the performance of PFM systems INTRODUCTION TO PUBLIC FINANCE MANAGEMENT (PFM) Module 5.2 Measuring the performance of PFM systems

Module map 1.2 Budget Cycle 2.1 Macroeconomics of the Budget Planning and budgeting 2.4 Budget Execution 1.1 Introduction 2.1 Macroeconomics of the Budget 4.1 Revenue Administration 3.1 Payroll, Procurement & IT 1.4 Budget Classification 4.3 Accounting & Reporting 4.2 Treasury Management 5.2 Assessing & Recapitulation 3.2 Internal Control & Audit 5.1 External Scrutiny & Oversight 2.2 MTEF and performance budgeting 1.3 The Budget and budget preparation 1.2 Budget Cycle

Module outline PFM Assessment tools PEFA Operating Design Principles Comprehensiveness of PEFAs PEFA scoring NB. there is a dedicated course on PEFA assessments

Why PFM assessment tools Concerns about achievement of development objectives Fiduciary and accountability concerns Need to link governance with development Move towards aid modalities using national PFM procedures Need to measure progress Inform decision-making on amount, timing and aid modality Alignment of processes Build capacity and ownership of reform agenda When deciding whether or not to use country public financial management (PFM) and procurement systems for delivering their programmes, most donors are required to carry out their own fiduciary assessments.  However, there is an increasing trend towards joint diagnostic exercises, using standard methodologies and internationally agreed criteria.  A common diagnostic process helps partner countries and donors to agree on reform and capacity building priorities.  It also puts the onus on donors to make use of country systems if objective assessments show reasonable capacity and a track record on reform. From old slides: Importance of development as well as fiduciary objectives - complementary not opposites Best fiduciary assurance comes from a government committed to a sound PFM system Risk is to country funds as well as donor/IFI funds These tools are not audits, or seeking to trace individual transactions or a “pass/fail” certification of adequacy of the PFM system for budget support lending Tools are knowledge tools about the environment into which donor/IFI funds are going Going into lending operations with “eyes open” Influencing decisions on amount, timing and form (budget support, sectoral support, investment projects) of assistance Budget support may be appropriate in a weak PFM environment where there is clear government commitment to PFM improvement – and evidence of improvements Improved PFM may therefore be an outcome of or a precondition for financial assistance Public Expenditure Analysis and Management Course Zimbabwe 2014

PFM Assessment PFM Reform Strategy PFM Reform action plan Capacity Development

Toolkit Do you know a tool to assess a PFM system

PEFA PER CFAA PETS MAPS DeMPA Performance Audit Compliance Audit 1 policy design & review 2 strategic Planning 3 budget Preparation 4 budget Execution 5 accounting & Reporting 6 external audit & scrutiny PEFA CFAA PETS MAPS DeMPA Compliance Audit

Guidance on use PER FRA - CFAA MAPS PETS PEFA - ROSC DEMPA Are public expenditures efficient/effective? What are fiduciary risks? How does procurement work? Are there leakages/corruption? Are the country systems in line with good practices? How is Public Debt managed? PER FRA - CFAA MAPS PETS PEFA - ROSC DEMPA

Core features of PFM assessment tools   PER CFAA MAPS/ CPAR Fiscal ROSC PETS DeMPA FRA PEFA Identification of PFM strengths & weaknesses X Focused on part of the Budget Cycle Integrated Focus In-depth analysis of capacity factors Recommendations for reform Assess fiduciary risk to public/external funds Track progress over time Linked to reform cycle (see Brussels slide PEFA training) The PEFA framework identifies strength and weaknesses and therefore provides useful inputs for the Dutch PFM Appraisal Framework (see next slides). But its Performance Report (PR) does not give recommendations for PFM Reform. Based on the PEFA results an assessment can be made of the risks for all funds, including external funds. But the PR does not focus explicitly on external funds. Read more at: http://web.worldbank.org/WBSITE/EXTERNAL/PEFA/0,,contentMDK:22687080~menuPK:7355896~pagePK:7313176~piPK:7327442~theSitePK:7327438,00.html Public Expenditure Analysis and Management Course Zimbabwe 2014

OECD Methodology for Assessing Procurement Systems (MAPS) How does the procurement work? Background Limited coverage of procurement issues in PEFA Need for a detailed procurement tool and consistent approach Introduced in 2006 following the PEFA approach 12 indicators, 54 sub-indicators Base Line Indicators (BLIs) Compliance/Performance Indicators (CPIs) Allows to track progress over time Under the auspices of the joint World Bank / OECD Development Assistance Committee (DAC) Procurement Round Table initiative, developing countries and bilateral and multilateral donors worked together from 2003 – 2004 to develop a set of tools and standards that provide guidance for improvements in procurement systems and the results they produce. The Methodology for Assessment of National Procurement Systems has emerged from the work since 2005 of the OECD-DAC Joint Venture on Procurement.  Its assessment methodology provides a standard set of indicators for measuring procurement legislation, institutional capacity, performance and accountability.  It has been piloted in 22 countries. Version 4 of the Benchmarking and Assessment methodology has been approved for testing and ongoing application of lessons learned through field experience. The methodology for assessment of national procurement systems is intended to provide a common tool which developing countries and donors can use to assess the quality and effectiveness of national procurement systems. The understanding among the participants in this process is that the assessment will provide a basis upon which a country can formulate a capacity development plan to improve its procurement system. Similarly, donors can use the common assessment to develop strategies for assisting the capacity develop plan and to mitigate risks in the individual operations that they decide to fund. The long term goal is that countries will improve their national procurement systems to meet internationally recognised standards enabling greater effectiveness in the use of funds to meet country obligations. Public Expenditure Analysis and Management Course Zimbabwe 2014

Debt Management Performance Assessment (DeMPA) How is public debt managed? Background Limited coverage of debt management in PEFA Framework No insight into the underlying causes of weak performance Methodology designed by WB 2006-2009 Guideline revised in 2009 DeMPA modelled on PEFA 6 core functions 15 indicators, 35 dimensions Allows to track progress over time Debt Management Performance Assessment (DeMPA) Tool, a methodology for assessing performance through a comprehensive set of performance indicators spanning the full range of government debt management (DeM) functions. The indicator set is intended to be an internationally recognized standard in the government DeM field and may be applied in all developing countries. It is based on the principles set out in the International Monetary Fund (IMF) and World Bank’s Guidelines for Public Debt Management, initially published in 2001 and updated in 2003.1 It is modeled after the Public Expenditure and Financial Accountability (PEFA) framework for performance measurement of public financial management. The DeMPA evaluates strengths and weaknesses in public debt management, through a comprehensive set of 15 performance indicators covering six core areas of public debt management: (1) governance and strategy development; (2) coordination with macroeconomic policies; (3) borrowing and related financing activities; (4) cash flow forecasting and cash balance management; (5) operational risk management; and (6) debt records and reporting. Its scope is central government public debt management and closely related functions such as issuance of loan guarantees, on-lending and cash flow forecasting and cash balance management. A DeMPA can help guide the design of actionable reform programs, facilitate monitoring of performance over time, and enhance donor harmonization based on a common understanding of priorities. The DeMPA is modeled after the Public Expenditure and Financial Accountability (PEFA) indicators. It can be considered a more detailed and comprehensive assessment of government DeM than is currently reflected in the PEFA indicators.4 The two frameworks are complementary: the DeMPA can be used to undertake a detailed assessment of the underlying factors leading to poor PEFA ratings in the area of DeM; alternatively, if the DeMPA exercise precedes a PEFA assessment, the latter can use the DeMPA results to inform its assessment of the relevant indicators. Read more on: http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTDEBTDEPT/0,,contentMDK:22764818~menuPK:4876300~pagePK:64166689~piPK:64166646~theSitePK:469043~isCURL:Y,00.html Public Expenditure Analysis and Management Course Zimbabwe 2014

Concluding remarks Choice of tool depends on specific needs Coordinate assessment as much as possible to avoid assessment fatigue Assessment = platform for dialogue versus pass/fail exam Country leadership / ownership essential Debt Management Performance Assessment (DeMPA) Tool, a methodology for assessing performance through a comprehensive set of performance indicators spanning the full range of government debt management (DeM) functions. The indicator set is intended to be an internationally recognized standard in the government DeM field and may be applied in all developing countries. It is based on the principles set out in the International Monetary Fund (IMF) and World Bank’s Guidelines for Public Debt Management, initially published in 2001 and updated in 2003.1 It is modeled after the Public Expenditure and Financial Accountability (PEFA) framework for performance measurement of public financial management. The DeMPA evaluates strengths and weaknesses in public debt management, through a comprehensive set of 15 performance indicators covering six core areas of public debt management: (1) governance and strategy development; (2) coordination with macroeconomic policies; (3) borrowing and related financing activities; (4) cash flow forecasting and cash balance management; (5) operational risk management; and (6) debt records and reporting. Its scope is central government public debt management and closely related functions such as issuance of loan guarantees, on-lending and cash flow forecasting and cash balance management. A DeMPA can help guide the design of actionable reform programs, facilitate monitoring of performance over time, and enhance donor harmonization based on a common understanding of priorities. The DeMPA is modeled after the Public Expenditure and Financial Accountability (PEFA) indicators. It can be considered a more detailed and comprehensive assessment of government DeM than is currently reflected in the PEFA indicators.4 The two frameworks are complementary: the DeMPA can be used to undertake a detailed assessment of the underlying factors leading to poor PEFA ratings in the area of DeM; alternatively, if the DeMPA exercise precedes a PEFA assessment, the latter can use the DeMPA results to inform its assessment of the relevant indicators. Read more on: http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTDEBTDEPT/0,,contentMDK:22764818~menuPK:4876300~pagePK:64166689~piPK:64166646~theSitePK:469043~isCURL:Y,00.html Public Expenditure Analysis and Management Course Zimbabwe 2014

Module outline PFM Assessment tools PEFA Operating Design Principles Comprehensiveness of PEFAs PEFA scoring NB. there is a dedicated course on PEFA assessments

The PEFA PMF: Operating Design Principles Focus on high level performance indicators applied to the central government level Widely applicable: relevant to countries at all levels of development Assessment must be evidence based Capable of calibration to capture progress over time Use data that can be collected cost effectively Comprehensive: cover all aspects of the PFM cycle Define the achievement of sound PFM to be based upon the three PFM objectives

The PEFA PMF: Operating Design Principles Specify six critical dimensions of a sound PFM system as: Credibility of the budget Comprehensiveness and transparency of the budget Policy based budgeting Predictability and control in budget execution Accounting, recording and reporting External scrutiny and audit Incorporate a measure of the impact of donor practices on PFM performance

The PEFA PMF: Operating Design Principles An open and orderly PFM system that supports the 3 PFM objectives: Aggregate fiscal discipline Strategic allocation of resources Efficient service delivery Six core dimensions: Credibility of the budget Comprehensiveness and transparency Budget Cycle Policy-based budgeting Predictability and control in budget execution Accounting, recording and reporting External scrutiny and audit In the context of this Framework, measurement of the operational performance of the key elements of the PFM systems (by means of 28 indicators) should lead to assessments of the extent the existing PFM systems, processes and institutions: Support the 3 objectives Meet the 6 core dimensions PEFA is not only a tool to measure quality of PFM systems throughout the budget cycle, BUT also to look at budget principles at least to some extent (like transparency, credibility, etc). Key elements of operational PFM performance: 31 performance indicators: 28 for governments and 3 for donors

PEFA Model

Scope of the PFM PMF The Central Government Budget

PEFA Framework, transparency and corruption A comprehensive and proper PFM system makes it harder to divert or misuse funds PEFA framework: Assesses whether real basics of PFM system are operating well (‘getting the basics right’) Provides technical summary of PFM system Less focusing on political and cultural aspects relevant for corruption Core focus on ‘What’ question, not on ‘Why’ of ‘How’ ‘what’: identification of weaknesses in PFM system NOT: ‘why’: insight in causes of budgetary dysfunctions ‘how’: how can these causes be tackled?  No in-depth analysis of underlying causes  No reform recommendations made in PEFA Performance Report Transparent and good fiscal information, available to the public, makes misuse easier to detect, as does effective internal and external control. Nevertheless, the PEFA framework is inadequate for capturing corruption. The indicators are insufficient to judge whether corruption is likely to be high of low in the country assessed. In particular, indicators of central government accountability focus largely on the characteristics of control mechanisms rather than the ability of other agents to sanction government officials where misuse of funds occurs. PEFA contributes to more transparency through its way of questioning. This transparency should help to reduce future corruption. PEFA is also not meant to be fiduciary risk assessment tool. Furthermore, PEFA focuses primarily on the Central government, while most corruption takes place at lower level governments. Focus is explicitly on the system, hence underlying causes may not be covered such as capacity of financial managers and staff. PEFA framework does not provide in-depth analysis of underlying causes of poor performance (i.e. it does not answer the why and how question) and therefore the PEFA Performance report does not make recommendations (insufficient info to build PFM reform plan) PEFA focuses on real basics of PFM systems are operating well, not on more fancy PFM stuff like activity based costing, performance budgeting and full fledged MTEFs. For this reason, no prioritisation of indicators has been made. Sources: Understanding the Politics of the budget: what drives change in the budget process? DFID practice paper, January 2007. Should corrupt countries receive budget support, CMI Brief, November 2005, Volume 4, No.4.

Module outline PFM Assessment tools PEFA Operating Design Principles Comprehensiveness of PEFAs PEFA scoring

Comprehensiveness of the PEFA PMF PFM

The PEFA Iceberg – how comprehensive? PEFA Assessment Fixed asset register Supply chain management Financial administrative network Capacity Procurement Political context Market Quality of expenditure management Engaged civil society

Module outline PFM Assessment tools PEFA Operating Design Principles Comprehensiveness of PEFAs PEFA scoring

PEFA Scoring Calibration and Scoring Two scoring methods: Specific calibration of scores using a four point ordinal scale (A, B, C and D) Intermediate scores (B+, C+, D+) possible only for multi-dimensional indicators, where dimensions score differently Arrow ▲ can indicate an improvement not reflected in change of indicator score Two scoring methods: Method M1 ‘weakest link among dimensions’ Method M2 ‘average of dimensions’

Interdependencies PFM linkages are analogous to foundation footing linkages – the strength of a foundation is determined by it weakest footing That is the reason why the Tower of Pisa leans. The performance of a PFM system is determined by the strength of its weakest PFM activity As a general rule PFM scores, like the strengths of foundation footings, cannot be averaged

Measuring PFM system performance: a comparison of tools CFAA: Country Financial Accountability Assessment (WB) CPAR: Country Procurement Assessment (WB) ROSC: IMF’s Report on Fiscal Standards and Codes (ROSC) FRA: Fiduciary risk assessment (DfID) NPA: National Program for Action

Consequences of applying High Level Indicators for managing support Movements in scorings may vary only slowly over time The proper rating being based upon the lowest sub- indicator scoring (M1 method) will contributes further to slow changes in indicator scorings A number of factors may contribute to a sub- indicator so that even though there may be significant improvements these may not be reflected in a changes score. (see for example PI- 17 (i))

Budget Support and PEFA Agreed reviewable milestones Requirements for demonstrated progress to facilitate BS disbursements Measurements of progress over time using PEFA PFM Progress over time

Key messages The PEFA Framework measures the performance of PFM systems in achieving their objectives It generally focuses on the “basics” of PFM systems It is used to monitor progress in strengthening systems It does not deal with policy issues There is a dedicated PEFA course