SAI Performance Measurement Framework Event Location, Date Presented By:

What is SAI PMF ? SAI PMF is a tool for measuring, monitoring, managing and reporting the performance of a Supreme Audit Institution.

Purpose of a SAI PMF Assessment Why might SAIs want to measure their performance Identify strengths and weaknesses Introduce / strengthen internal performance management and accountability Enable external (annual) reporting to stakeholders Demonstrate progress in capability and performance Demonstrate the value and benefits of the SAI to citizens Support ISSAI implementation / improve audit standards & quality Obtain & maintain external support for capacity development MULTIPLE PURPOSES FOR SAI PMF ASSESSMENTS Note here that the SAI PMF has been designed as a GENERAL PURPOSE FRAMEWORK. It can be used for any of the above purposes. But the intended purpose of an assessment in a specific country has a profound implication on how the assessment should be best carried out to meet that purpose.

SAI PMF: Evidence Based Assessment of Performance Holistic assessment of the SAI and its environment – including factors which are not covered by the indicators Qualitative assessment (Performance Report) Measurable indicators Objective measurement to inform qualitative assessment and track progress over time PEFA style drill down into SAI Performance

Why Develop a Global Framework for Measuring SAI Performance?

ISSAI Framework adopted in 2010 NEW Level 3: Updated audit principles, consistent with level 4 1 2 3 4 Founding Principles Prerequisites for the functioning of SAIs Level 4: First agreed, international public sector external audit guidelines Fundamental auditing Principles For the first time, the ISSAIs provided a set of agreed international standards which most countries aspire towards, and against which performance could be measured. Prior to the adoption of the ISSAI framework, it was not possible to get agreement on what constitutes good performance, making development and agreement of a measurement tool impossible. Auditing guidelines

Why SAI PMF? ISSAI 20 calls for SAIs to assess and report publically on their performance No internationally agreed common framework for measuring SAI’s performance The INTOSAI Congress (2010), decided to develop a framework for measuring SAI performance (PMF) Key Points: SAIs globally are facing the challenge of how to assess their own performance, for both internal and external purposes. Includes the challenge of measuring performance changes over time, as well as how to best learn from the experiences of peers.

Progress made in the development of SAI PMF? Version 2.0 published 2012 and piloted in Djibouti, Sierra Leone & Norway Global consultation on version 2.0 in 2012 Updated SAI PMF Pilot Version published July 2013 11 SAI PMF training courses delivered in Europe, Africa, North & South America, Asia, Middle East – in English, French, Spanish and Arabic Coming soon – Portuguese and Russian, the Pacific

Phase 2 Pilots - Volunteers Completed: Planned: Slovakia Argentina Armenia Guatemala Bangladesh Brazil Burkina Faso Costa Rica In Progress: Dominican Republic Barbados Haiti Bhutan Ecuador Latvia India Maldives Mexico Mozambique Expressed Interest: Oman SAIs across the Pacific Nepal Palestine

Adoption by INTOSAI Congress in 2016 What is next? 20 more pilot tests More SAI PMF training courses Further improve SAI PMF Adoption by INTOSAI Congress in 2016

Part 1 - SAI PMF Indicator Framework

SAI Performance Measurement Framework Structure 7 Domains SAI Capability C. Strategy for Organizational Development SAI Reporting Value and Benefits of Supreme Audit Institutions B. Independence and Legal Framework D. Audit Standards and Methodology E. Management & Support Structures F. Human Resources and Leadership G. Communication and Stakeholder Management Note for facilitator – everything within the dotted black lines is under the control of the SAI (providing it is adequately resourced). If it is not within the control of the SAI due to specific independence issues, then we do not score it here and the limitations on SAI independence is reflected in the scores under domain B. Thus the part within the dotted lines follows the principle of only measuring things which the SAI is responsible for. Everything within the red line is measured. Things outside the red line form part of the narrative assessment, because they are important, but are not directly measured – being difficult to measure against a standardised framework. Country Context, Governance and Public Financial Management

Domain Indicator 1 Dimension 1 Criteria 1 Criteria 2 Criteria 3 Dimension 2 Dimension 3 Dimension 4 Indicator 2 Criteria are taken directly from the ISSAIs or other international good practices

Part 2 - SAI PMF Assessment Report

From Evidence to Executive Summary II Quality Assurance IV Management Use of Assessment Results Judgment I Executive Summary Analysis III. Observations on SAI Performance & Impact Within (III) there are four parts, as follows: a) Is about understanding linkages between domains & what constrains SAI performance b) Goes a step further than (a), to identify how the SAI’s performance makes a difference to the lives of citizens – specifically looking at linkages between SAI PMF indicators and the V&B framework, as well as identifying specific examples from the SAIs own reports (annual report, performance audit reports etc.) c) Examines how factors largely inside or outside the SAI’s control support or undermine the SAIs ability to deliver impact, including constraints in its mandate and independence, weaknesses in the PFM and governance systems, country limitations, and the availability and effectiveness of external support d) Looks at capacity development prospects, based on ch.4 and challenges identified in ch.3 Evidence Ch.1 Introduction Ch.2 Country & SAI Background Ch.3 Assessment (Indicator Scores & Evidence) Ch.4 SAI Capacity Development Main Report

Part 3 – Initiating a SAI PMF Pilot Assessment

Six Key Decisions to be Made by the SAI Undertake SAI PMF? Assessment owner? Purpose(s) of the Assessment? Assessment Approach? 5. Timing? 6. Use of Results? 1 2 3 4 5 6 There are more decisions during the course of the assessment, but these are the six key ones to be taken at the outset, and made specifically by the SAI.

Learning More SAI PMF Training Course Part A – for assessors, 2.5 days Part B – deeper understanding and future trainers, 2 days www.idi.no Request a course: INTOSAI.Donor.Secretariat@idi.no The training course covers many issues not dealt with here, e.g. Determining the scope of assessments, including dealing with regional offices and sub-national audit entities Steps in the assessment process Planning assessments and designing terms of reference Quality assurance arrangements Writing the SAI performance report