1. Module 5.2 Measuring the performance of PFM systems
INTRODUCTION TO PUBLIC FINANCE MANAGEMENT (PFM)
Module 5.2 Measuring the performance of PFM systems

2. Module map 1.2 Budget Cycle 2.1 Macroeconomics of the Budget
Planning and budgeting
2.4 Budget Execution
1.1 Introduction
2.1 Macroeconomics of the Budget
4.1 Revenue
Administration
3.1 Payroll, Procurement & IT
1.4 Budget Classification
4.3 Accounting & Reporting
4.2 Treasury Management
5.2 Assessing & Recapitulation
3.2 Internal Control & Audit
5.1 External Scrutiny & Oversight
2.2 MTEF and performance budgeting
1.3 The Budget and budget preparation
1.2 Budget Cycle

3. Module outline PFM Assessment tools PEFA Operating Design Principles
Comprehensiveness of PEFAs
PEFA scoring
NB. there is a dedicated course on PEFA assessments

4. Why PFM assessment tools
Concerns about achievement of development objectives
Fiduciary and accountability concerns
Need to link governance with development
Move towards aid modalities using national PFM procedures
Need to measure progress
Inform decision-making on amount, timing and aid modality
Alignment of processes
Build capacity and ownership of reform agenda
When deciding whether or not to use country public financial management (PFM) and procurement systems for delivering their programmes, most donors are required to carry out their own fiduciary assessments.  However, there is an increasing trend towards joint diagnostic exercises, using standard methodologies and internationally agreed criteria.  A common diagnostic process helps partner countries and donors to agree on reform and capacity building priorities.  It also puts the onus on donors to make use of country systems if objective assessments show reasonable capacity and a track record on reform.
From old slides:
Importance of development as well as fiduciary objectives - complementary not opposites
Best fiduciary assurance comes from a government committed to a sound PFM system
Risk is to country funds as well as donor/IFI funds
These tools are not audits, or seeking to trace individual transactions or
a “pass/fail” certification of adequacy of the PFM system for budget support lending
Tools are knowledge tools about the environment into which donor/IFI funds are going
Going into lending operations with “eyes open”
Influencing decisions on amount, timing and form (budget support, sectoral support, investment projects) of assistance
Budget support may be appropriate in a weak PFM environment where there is clear government commitment to PFM improvement – and evidence of improvements
Improved PFM may therefore be an outcome of or a precondition for financial assistance
Public Expenditure Analysis and Management Course Zimbabwe 2014

5. PFM Assessment
PFM Reform Strategy
PFM Reform action plan
Capacity Development

6. Toolkit
Do you know a tool to assess a PFM system

7. PEFA PER CFAA PETS MAPS DeMPA Performance Audit Compliance Audit
1 policy design & review
2 strategic Planning
3 budget Preparation
4 budget Execution
5 accounting & Reporting
6 external audit & scrutiny
PEFA
CFAA
PETS
MAPS
DeMPA
Compliance Audit

8. Guidance on use PER FRA - CFAA MAPS PETS PEFA - ROSC DEMPA
Are public expenditures efficient/effective?
What are fiduciary risks?
How does procurement work?
Are there leakages/corruption?
Are the country systems in line with good practices?
How is Public Debt managed?
PER
FRA - CFAA
MAPS
PETS
PEFA - ROSC
DEMPA

9. Core features of PFM assessment tools
PER
CFAA
MAPS/ CPAR
Fiscal
ROSC
PETS
DeMPA
FRA
PEFA
Identification of PFM strengths & weaknesses X
Focused on part of the Budget Cycle
Integrated Focus
In-depth analysis of capacity factors
Recommendations for reform
Assess fiduciary risk to public/external funds
Track progress over time
Linked to reform cycle (see Brussels slide PEFA training)
The PEFA framework identifies strength and weaknesses and therefore provides useful inputs for the Dutch PFM Appraisal Framework (see next slides). But its Performance Report (PR) does not give recommendations for PFM Reform. Based on the PEFA results an assessment can be made of the risks for all funds, including external funds. But the PR does not focus explicitly on external funds.
Read more at:
Public Expenditure Analysis and Management Course Zimbabwe 2014

10. OECD Methodology for Assessing Procurement Systems (MAPS)
How does the procurement work?
Background
Limited coverage of procurement issues in PEFA
Need for a detailed procurement tool and consistent approach
Introduced in 2006 following the PEFA approach
12 indicators, 54 sub-indicators
Base Line Indicators (BLIs)
Compliance/Performance Indicators (CPIs)
Allows to track progress over time
Under the auspices of the joint World Bank / OECD Development Assistance Committee (DAC) Procurement Round Table initiative, developing countries and bilateral and multilateral donors worked together from 2003 – 2004 to develop a set of tools and standards that provide guidance for improvements in procurement systems and the results they produce.
The Methodology for Assessment of National Procurement Systems has emerged from the work since 2005 of the OECD-DAC Joint Venture on Procurement.  Its assessment methodology provides a standard set of indicators for measuring procurement legislation, institutional capacity, performance and accountability.  It has been piloted in 22 countries.
Version 4 of the Benchmarking and Assessment methodology has been approved for testing and ongoing application of lessons learned through field experience.
The methodology for assessment of national procurement systems is intended to provide a common tool which developing countries and donors can use to assess the quality and effectiveness of national procurement systems. The understanding among the participants in this process is that the assessment will provide a basis upon which a country can formulate a capacity development plan to improve its procurement system. Similarly, donors can use the common assessment to develop strategies for assisting the capacity develop plan and to mitigate risks in the individual operations that they decide to fund. The long term goal is that countries will improve their national procurement systems to meet internationally recognised standards enabling greater effectiveness in the use of funds to meet country obligations.
Public Expenditure Analysis and Management Course Zimbabwe 2014

11. Debt Management Performance Assessment (DeMPA)
How is public debt managed?
Background
Limited coverage of debt management in PEFA Framework
No insight into the underlying causes of weak performance
Methodology designed by WB
Guideline revised in 2009
DeMPA modelled on PEFA
6 core functions
15 indicators, 35 dimensions
Allows to track progress over time
Debt Management Performance Assessment (DeMPA) Tool, a methodology for assessing performance through a comprehensive set of performance indicators spanning the full range of government debt management (DeM) functions. The indicator set is intended to be an internationally recognized standard in the government DeM field and may be applied in all developing countries.
It is based on the principles set out in the International Monetary Fund (IMF) and World Bank’s Guidelines for Public Debt Management, initially published in 2001 and updated in It is modeled after the Public Expenditure and Financial Accountability (PEFA) framework for performance measurement of public financial management.
The DeMPA evaluates strengths and weaknesses in public debt management, through a comprehensive set of 15 performance indicators covering six core areas of public debt management: (1) governance and strategy development; (2) coordination with macroeconomic policies; (3) borrowing and related financing activities; (4) cash flow forecasting and cash balance management; (5) operational risk management; and (6) debt records and reporting. Its scope is central government public debt management and closely related functions such as issuance of loan guarantees, on-lending and cash flow forecasting and cash balance management. A DeMPA can help guide the design of actionable reform programs, facilitate monitoring of performance over time, and enhance donor harmonization based on a common understanding of priorities.
The DeMPA is modeled after the Public Expenditure and Financial Accountability (PEFA) indicators. It can be considered a more detailed and comprehensive assessment of government DeM than is currently reflected in the PEFA indicators.4 The two frameworks are complementary: the DeMPA can be used to undertake a detailed assessment of the underlying factors leading to poor PEFA ratings in the area of DeM; alternatively, if the DeMPA exercise precedes a PEFA assessment, the latter can use the DeMPA results to inform its assessment of the relevant indicators.
Read more on:
Public Expenditure Analysis and Management Course Zimbabwe 2014

12. Concluding remarks Choice of tool depends on specific needs
Coordinate assessment as much as possible to avoid assessment fatigue
Assessment = platform for dialogue versus pass/fail exam
Country leadership / ownership essential
Debt Management Performance Assessment (DeMPA) Tool, a methodology for assessing performance through a comprehensive set of performance indicators spanning the full range of government debt management (DeM) functions. The indicator set is intended to be an internationally recognized standard in the government DeM field and may be applied in all developing countries.
It is based on the principles set out in the International Monetary Fund (IMF) and World Bank’s Guidelines for Public Debt Management, initially published in 2001 and updated in It is modeled after the Public Expenditure and Financial Accountability (PEFA) framework for performance measurement of public financial management.
The DeMPA evaluates strengths and weaknesses in public debt management, through a comprehensive set of 15 performance indicators covering six core areas of public debt management: (1) governance and strategy development; (2) coordination with macroeconomic policies; (3) borrowing and related financing activities; (4) cash flow forecasting and cash balance management; (5) operational risk management; and (6) debt records and reporting. Its scope is central government public debt management and closely related functions such as issuance of loan guarantees, on-lending and cash flow forecasting and cash balance management. A DeMPA can help guide the design of actionable reform programs, facilitate monitoring of performance over time, and enhance donor harmonization based on a common understanding of priorities.
The DeMPA is modeled after the Public Expenditure and Financial Accountability (PEFA) indicators. It can be considered a more detailed and comprehensive assessment of government DeM than is currently reflected in the PEFA indicators.4 The two frameworks are complementary: the DeMPA can be used to undertake a detailed assessment of the underlying factors leading to poor PEFA ratings in the area of DeM; alternatively, if the DeMPA exercise precedes a PEFA assessment, the latter can use the DeMPA results to inform its assessment of the relevant indicators.
Read more on:
Public Expenditure Analysis and Management Course Zimbabwe 2014

13. Module outline PFM Assessment tools PEFA Operating Design Principles
Comprehensiveness of PEFAs
PEFA scoring
NB. there is a dedicated course on PEFA assessments

14. The PEFA PMF: Operating Design Principles
Focus on high level performance indicators applied to the central government level
Widely applicable: relevant to countries at all levels of development
Assessment must be evidence based
Capable of calibration to capture progress over time
Use data that can be collected cost effectively
Comprehensive: cover all aspects of the PFM cycle
Define the achievement of sound PFM to be based upon the three PFM objectives

15. The PEFA PMF: Operating Design Principles
Specify six critical dimensions of a sound PFM system as:
Credibility of the budget
Comprehensiveness and transparency of the budget
Policy based budgeting
Predictability and control in budget execution
Accounting, recording and reporting
External scrutiny and audit
Incorporate a measure of the impact of donor practices on PFM performance

16. The PEFA PMF: Operating Design Principles
An open and orderly PFM system that supports the 3 PFM objectives: Aggregate fiscal discipline
Strategic allocation of resources
Efficient service delivery
Six core dimensions:
Credibility of the budget
Comprehensiveness and transparency
Budget Cycle
Policy-based budgeting
Predictability and control in budget execution
Accounting, recording and reporting
External scrutiny and audit
In the context of this Framework, measurement of the operational performance of the key elements of the PFM systems (by means of 28 indicators) should lead to assessments of the extent the existing PFM systems, processes and institutions:
Support the 3 objectives
Meet the 6 core dimensions
PEFA is not only a tool to measure quality of PFM systems throughout the budget cycle, BUT also to look at budget principles at least to some extent (like transparency, credibility, etc).
Key elements of operational PFM performance:
31 performance indicators: 28 for governments and 3 for donors

17. PEFA Model

18. Scope of the PFM PMF
The Central Government Budget

19. PEFA Framework, transparency and corruption
A comprehensive and proper PFM system makes it harder to divert or misuse funds
PEFA framework:
Assesses whether real basics of PFM system are operating well (‘getting the basics right’)
Provides technical summary of PFM system
Less focusing on political and cultural aspects relevant for corruption
Core focus on ‘What’ question, not on ‘Why’ of ‘How’
‘what’: identification of weaknesses in PFM system
NOT: ‘why’: insight in causes of budgetary dysfunctions
‘how’: how can these causes be tackled?
 No in-depth analysis of underlying causes
 No reform recommendations made in PEFA Performance Report
Transparent and good fiscal information, available to the public, makes misuse easier to detect, as does effective internal and external control. Nevertheless, the PEFA framework is inadequate for capturing corruption. The indicators are insufficient to judge whether corruption is likely to be high of low in the country assessed. In particular, indicators of central government accountability focus largely on the characteristics of control mechanisms rather than the ability of other agents to sanction government officials where misuse of funds occurs.
PEFA contributes to more transparency through its way of questioning. This transparency should help to reduce future corruption. PEFA is also not meant to be fiduciary risk assessment tool. Furthermore, PEFA focuses primarily on the Central government, while most corruption takes place at lower level governments.
Focus is explicitly on the system, hence underlying causes may not be covered such as capacity of financial managers and staff. PEFA framework does not provide in-depth analysis of underlying causes of poor performance (i.e. it does not answer the why and how question) and therefore the PEFA Performance report does not make recommendations (insufficient info to build PFM reform plan)
PEFA focuses on real basics of PFM systems are operating well, not on more fancy PFM stuff like activity based costing, performance budgeting and full fledged MTEFs. For this reason, no prioritisation of indicators has been made.
Sources:
Understanding the Politics of the budget: what drives change in the budget process? DFID practice paper, January
Should corrupt countries receive budget support, CMI Brief, November 2005, Volume 4, No.4.

20. Module outline PFM Assessment tools PEFA Operating Design Principles
Comprehensiveness of PEFAs
PEFA scoring

21. Comprehensiveness of the PEFA PMF
PFM

22. The PEFA Iceberg – how comprehensive?
PEFA Assessment
Fixed asset register
Supply chain management
Financial administrative network
Capacity
Procurement
Political context
Market
Quality of expenditure management
Engaged civil society

23. Module outline PFM Assessment tools PEFA Operating Design Principles
Comprehensiveness of PEFAs
PEFA scoring

24. PEFA Scoring Calibration and Scoring Two scoring methods:
Specific calibration of scores using a four point ordinal scale (A, B, C and D)
Intermediate scores (B+, C+, D+) possible only for multi-dimensional indicators, where dimensions score differently
Arrow ▲ can indicate an improvement not reflected in change of indicator score
Two scoring methods:
Method M1 ‘weakest link among dimensions’
Method M2 ‘average of dimensions’

25. Interdependencies
PFM linkages are analogous to foundation footing linkages – the strength of a foundation is determined by it weakest footing
That is the reason why the Tower of Pisa leans. The performance of a PFM system is determined by the strength of its weakest PFM activity
As a general rule PFM scores, like the strengths of foundation footings, cannot be averaged

26. Measuring PFM system performance: a comparison of tools
CFAA: Country Financial Accountability Assessment (WB)
CPAR: Country Procurement Assessment (WB)
ROSC: IMF’s Report on Fiscal Standards and Codes (ROSC)
FRA: Fiduciary risk assessment (DfID)
NPA: National Program for Action

27. Consequences of applying High Level Indicators for managing support
Movements in scorings may vary only slowly over time
The proper rating being based upon the lowest sub- indicator scoring (M1 method) will contributes further to slow changes in indicator scorings
A number of factors may contribute to a sub- indicator so that even though there may be significant improvements these may not be reflected in a changes score. (see for example PI- 17 (i))

28. Budget Support and PEFA
Agreed reviewable milestones
Requirements for demonstrated progress to facilitate BS disbursements
Measurements of progress over time using PEFA
PFM Progress over time

29. Key messages
The PEFA Framework measures the performance of PFM systems in achieving their objectives
It generally focuses on the “basics” of PFM systems
It is used to monitor progress in strengthening systems
It does not deal with policy issues
There is a dedicated PEFA course