1. SAI Performance Measurement Framework
Presented by: Yngvild H. Arnesen, INTOSAI Development Initiative (IDI)
Date: CReCER, Managua, October, 2012
INTOSAI Working Group on the Value and Benefits of SAIs

2. What is the SAI Performance Measurement Framework (PMF)?
A Performance Measurement Framework tailored to Supreme Audit Institutions (SAIs)
Intended to give a high level assessment of SAI performance against established INTOSAI good practice:
The International Standards of Supreme Audit Institutions (ISSAI) framework
The Framework on the Value and Benefits of SAIs
Other INTOSAI guidance material
Combination of objective measurement and qualitative assessment
Analyzes 7 domains of SAI performance in the country context and aims at assessing the value and benefits of the SAI to society (impact)

3. 3 Approaches to Conducting a SAI PMF Assessment:
Using the SAI PMF will be voluntary for the SAIs – the tool will be a public good which SAIs can make use of if it helps them achieve their objectives
The SAI will decide whether or not to publish results
3 Approaches to Conducting a SAI PMF Assessment:
There are three main approaches to conducting an assessment using the SAI PMF. The PMF will be equally well suited for all three:
1. SAI Self Assessment
Trained Team from within the SAI
2. INTOSAI Peer Review
Peer SAI
INTOSAI Regional Body
3. External Assessment
Professional PFM Body
Donor Agency
Consultant
External auditor .
SAI Involvement

4. Responsibilities and Reporting Lines
INTOSAI Congress (2013)
Approval
Mandate from INCOSAI XX, 2010
INTOSAI Working Group on the Value and Benefits of SAIs
Oversight and Direction
INTOSAI Capacity Building Committee
INTOSAI-Donor Steering Committee
Other Relevant INTOSAI Committees
In Consultation with:
WGVBS SAI PMF Task Team
Development of SAI Performance Measurement Framework
Coordinator: INTOSAI-Donor IDI
SAI PMF Reference Group
Technical Advice, Review and Quality Assurance
INTOSAI Capacity Building Committee Members
INTOSAI-Donor Steering Committee Members
Other Groups:
PEFA Secretariat, International Budget Partnership
INTOSAI Capacity Building Committee Members
INTOSAI-Donor Steering Committee Members

5. SAI PMF Timetable    WGVBS: Set Direction
2011, Q3
2011, Q4
2012, Q1
2012, Q2
2012, Q3
2012, Q4
2013, Q1
2013, Q2
2013, Q3
2013, Q4
Mapping Existing SAI Measurement Tools
Concept Note for SAI PMF
Develop SAI PMF Version 2.0
Phase 1 Pilots in 3 SAIs & Comments from Stakeholders
Develop SAI PMF Exposure Draft
Phase 2 Piloting and Pre-Consultation (ongoing into 2014 & 2015) 
WGVBS: Approve way forward  
WGVBS: Approve Exposure Draft
Exposure Draft at INCOSAI

6. Why should SAIs use the PMF (1)
As a step towards ISSAI implementation
Increased understanding in the INTOSAI community of what good practice for SAIs entails
Individual SAIs can benchmark their performance against good practice – and get an indication of where there is room for improvement
They can use the assessment as a basis for more detailed ISSAI compliance assessments
To demonstrate progress and value and benefits to society
The framework will be well suited for measuring progress over time and demonstrating this to external stakeholders
Show to stakeholders how the SAI contributes to strengthening public financial management

7. Why should SAIs use the PMF (2)
Internal performance measurement / annual reporting
Help SAIs better manage their own performance
ISSAI requirement to assess and report on own performance – lead by example and become a model organization
Inform and secure support for capacity development initiatives
Identify strengths and weaknesses, increase understanding of reasons for good and weak performance
Help build a culture for performance improvement
SAIs can use assessment results to get support from donors and other external stakeholders

8. SAI PMF: Evidence Based Assessment of Performance
Qualitative assessment (Performance Report)
Measurable indicators
Holistic assessment of the SAI and its environment – including factors which are not covered by the indicators
Objective measurement to inform qualitative assessment and track progress over time

9. SAI Performance Measurement Framework – Structure
Under the control of the SAI
Under the control of the SAI
SAI Performance
SAI Environment
SAI Capability
Impact
Foundational factors set by the legislative and/or executive, but which can be influenced by the SAI in the long term
SAI STRATEGY
Measured using other assessments. Impact can not be directly attributed to SAI results, but the SAI’s likely contribution should be analyzed in the SAI performance report.
SAI Performance
3 indicators
C. Strategy for Organizational Development
1 indicator
SAI CORE BUSINESS
D. Audit Standards and Methodology
3-5 indicators
B. Independence and Legal Framework
3 indicators
Contribution to Public Financial Management, Governance and Tackling Corruption
E. Management & Support Structures
2 indicators
F. Human Resources
3 indicators
G. Communication and Stakeholder Management
3 indicators
SAI SUPPORT SERVICES
Public Financial Management Environment
Influences the effectiveness of the SAI but not under direct control of the SAI.
Measured through other assessments but referred to explicitly in the narrative SAI performance report.

10. SAI Performance Report - end product from an assessment (30-40 pages?)
Provides a coherent and holistic assessment of SAI performance
Informed by objective and evidence based performance indicators
Analyzed within context
Considered as a whole – not the sum of its indicator scores
Enables SAI heads and senior management to
Identify barriers to improved performance
Track performance changes over time
Promote and communicate the value and benefits of the SAI
Explores how SAI external performance is constrained by
Country context and PFM environment
SAI environment
SAI internal capability

11. Potential risks related to having a SAI PMF
Revelations of performance weaknesses
It will be up to the SAI to decide if results are to be shared
Openness and commitment to reform likely to encourage external partners to support capacity development
May help draw attention to constraints outside the SAI’s control
Possible negative effect of scoring: Use of performance league tables
Intention is not to aggregate scores to the level of the entire SAI – as all indicators may not be of equal importance
Scores partly linked to achievements of SAI’s own mandate
Narrative report provides context for scoring
Scoring is already applied in INTOSAI (AFROSAI-E)
Scoring very useful for measuring progress over time within an SAI
Profusion of Assessment Tools
Risk that the SAI PMF becomes an additional tool
Donors and SAIs continue to develop & use other tools

12. 6. Further Information www.idi.no
Mapping of Tools for Assessing Performance of Supreme Audit Institutions
IDI, May 2012
Supreme Audit Institutions Performance Measurement Framework
Concept Note
IDI, May 2012