1. Performance Auditing Alta Fölscher CABRI/WBI PFM Training Seminar 18-20 June 2007

2. What is a Performance Audit Differs from regularity audit –Different approach, more ad hoc, less rigid, different skills –But same insofar an independent, objective assessment Audit of 3 Es –Economy –Efficiency –Effectiveness All three or any combination of three Of wide range of issues Aim to lead to improvements Two dominant approaches –Performance oriented –Problem oriented

3. An example of a performance audit Does the Drug Abuse Resistance Education (DARE) Program Work? –DARE is a drug abuse prevention program that aims to reduce drug use among school-age children. The performance audit of the program measured the extent to which it was achieving its goals. The audit used an experimental design, comparing juvenile arrest rates for youth in the program (the intervention group) with those who did not (the control group). The demographic profiles (ethnicity, income levels, age) were identical. The audit found that students who participated in the program were arrested more frequently than the control group, for both drug-related and non-drug-related offences. The design of this audit was heavily dependent on the existence of sufficient reliable data for determining student involvement in the program and identifying their arrest information in the local juvenile correctional system. These conditions are often difficult to meet, unless such comparisons have been planned from the start. Source: Location fictionalized from the 1994 audit of the Austin DARE Program, Office of the City Auditor, Austin, Texas as reported in Waring C and Morgan S, 2007. Public Sector Performance Auditing in Shah, A (ed): Performance Accountability and Combating Corruption, WBI, Washington DC

4. Why is it important? Based on need for public accountability –Insight into activities of public agency on behalf of stakeholders Reliable independent information on performance strengthens legitimacy and trust Basis for future decisions Incentives for change Better government spending, better services, better public accountability and better public management Are things done in the right way; are the right things being done?

5. The 3 Es For same level of quality Economy –Is about keeping the cost low –Minimising the cost of resources used or required Are means chosen most economical use of resources? Have human, financial and material resources been used economically? Is management of programme in line with sound admin and management? –Spending less Efficiency –Relationship between outputs and the goods, services and resources used to produce them –Can I produce more from given resources, can I use less resources for a level of outputs? –In practice often difficult to distinguish from economy –Spending well

6. 3 Es Effectiveness –Relationship between intended and achieved results of public spending Are aims being met by the means employed, outputs produced and impacts? Are the impacts really result of policy rather than other circumstances? –Spending wisely

7. 3Es Assessment always relative to audit criteria (standards against which actual performance measured) Important choice in audit design Standards used to determine whether programme meets or exceeds expectations Audit scope and design interprets economy, efficiency and effectiveness concretely Criteria should be relevant, reasonable, attainable eg –Historical; Similar programmes or other locations; Legislation; Accepted standard; Professional opinion Assessment of effectiveness particularly tough –Starts with programme design and clear, concrete goals –Ideally measurement before and after + control group –In practice difficult –Less ambitious Assess plausibility of assumptions in programme design and achievement of goals as set down (goal achievement analysis) Have objectives been reached, target groups reached, what is level of performance?

8. Mandate of Performance Auditor Important underpinning for work Mandate and goals should be defined in legal framework Specific regulations Auditor must be free to select audit areas within mandate and refuse commissions Professional quality of work important: sufficient resources to acquire skills Differs across countries –Coverage of mandate –Scope of audits –Approach

9. Performance Auditor’s domain Input Output chain Commitment Input Action/production OutputOutcome Purpose defined Resources assigned Activity undertaken Goods and services provided Objectives met Performance dimension Economy Physical and financial Amount Timing Quantity Quality Efficiency Productivity Unit cost Operating ratios Output effectiveness Level Quantity Timeliness Quality Price Cost Customer satisfaction Outcome effectiveness Mission and goal achievement Financial viability Cost benefit Cost effectiveness Programme dimension Auditor should be careful not to stray into political terrain But, audit findings can include reflections on policies Cross-cutting performance aspects Compliance with laws and regulations; validity, reliability and availability of information; maintaining underlying values; continuous improvement Source: Waring C and Morgan S, 2007. Public Sector Performance Auditing in Shah, A (ed): Performance Accountability and Combating Corruption, WBI, Washington DC

10. What to audit? Determine selection criteria country by country Added value important Important problem / problem area Risk –Substantial budgets, sudden increase in budgets –Traditional risk areas –New, urgent activities –Complex management arrangements –Performance information not updated

11. Standards for Performance Auditing Well thought out plan very important Pre-study Audit proposal –Identify issue and audit objectives (what will be audited and why) –Develop scope and design of audit What issues, what are hypotheses, what type of study? Is issue auditable and worth auditing? Understanding audited programme Defining audit criteria Methodological planning –Quality assurance, timetable, audit team and resources Field standards –Communicative and analytical process –Sufficiency, validity, reliability, relevance, reasonableness of evidence –Scepticism –Record-keeping and oversight Accessible, comprehensive reporting

12. Performance auditing chain Audit criteria – what should be Audit evidence (condition) – what is Audit finding – what is compared with what should be Determine the causes and effects of the finding Develop audit conclusions and recommendations Estimate likely impact of recommendations Audit cause – the programme (intervention)

13. Examples of audit findings TypeElementsExample DescriptiveCondition only Annual cost to incarcerate a prisoner was $67,800 in 2005. NormativeCondition and criteria Annual cost to incarcerate a prisoner was $67,800 in 2005, compared with $52,000 at comparable prison Traditional/ causal Criteria, condition, cause, and effect Annual cost to incarcerate a prisoner was $67,800 in 2005. Budget authorized $58,000, resulting in a deficit of $17.8 million. The additional costs were caused primarily by increase in labour and benefit costs following the May 2005 union contract. ImpactCondition with cause compared with condition without cause Recidivism rates among alcohol-dependent inmates who participated in the alcohol treatment before release were significantly lower than rates among alcohol-dependent inmates who did not receive the treatment. Source: Waring C and Morgan S, 2007. Public Sector Performance Auditing in Shah, A (ed): Performance Accountability and Combating Corruption, WBI, Washington DC

14. Performance auditing in Sub- Saharan African Performance auditing can help build foundations for performance system But –Keep it simple at start –Basics should be in place (rule of law, clearly defined institutions with understood roles and responsibilities, budget systems and accounting systems) –Financial audit function in place first –Organisational independence, legal mandate, sufficient funding, strong leadership, ability to recruit and retain skilled staff, stakeholder support and professional audit standards –Credibility –Support at highest levels of government