Self Review Training 24 June 2010 Performance Improvement Framework

Central Agencies DPMC SSC Treasury 2 PIF challenge: Showing value to a sceptical public (or reviewer)

Central Agencies DPMC SSC Treasury Value for Money is hard to assess directly … but poor results at ANY corner of the ‘triangle’ shows under-performance

Central Agencies DPMC SSC Treasury #1. PRICE, QUANTITY & STANDARDS (TECHNICAL EFFICIENCY) 1.How did prices of big outputs change? 2.Did output volume or quality change proportionately? 3.Did quality change? 4.Did o’heads grow? 5.Can we show citizens actually benefit from this? ‘Did new spending actually increase or improve output ?’

Central Agencies DPMC SSC Treasury #2. DID OUTPUT GET TO THE PROBLEM AREA, & EFFECT CHANGE  Can we link extra $$$ & output to priority groups, areas, needs & opportunities?  Who has / is the problem?  Who got interventions or $$$?  Who missed out?  How well do allocative rules really work?  Can we deliver more in areas that will benefit more, & limit waste? Allocative Efficiency ‘ Can we get better results without spending more ?’

Central Agencies DPMC SSC Treasury #3. INCREASED PRICE (OR OUTPUT) IMPROVES CORE OUTCOMES 1.Did end outcomes improve much? 2.Did gains mirror / equal spending? 3.Did gains precede or lag changes? 4.Intermediate outcomes improved? ‘Are results good, bad or simply indifferent ?’ Look for diminishing returns. Scale & timing ($ vs. outcome)?

ERO Approach - Context  ERO evaluates the performance of schools and early childhood services  ERO’s self-review was undertaken as a pilot review for the PIF (July 2009)  The context for ERO’s self-review was “the alignment of services and supporting infrastructure to better support Government priorities for education”  This organisation-wide context meant that ERO tested all aspects of the framework Brent McPherson- June 2010

ERO Approach - Team The team:  was appointed by the Chief Executive  had knowledge about the business and governance  had access to information  had evaluation skills Level Position (tier 2) National Manager Corporate Services (tier 2) National Manager Review Services (Central) (tier 3) Strategy & Performance Manager (tier 3)Manager Information Services (tier 3)Chief Finance Officer (tier 3) Manager Standards & Contracts Brent McPherson- June 2010

ERO Approach - Method  Critical areas identified in the framework were allocated between review members - in pairs (allocated areas were independent to current portfolios)  Each pair recorded their assessment, action points & evidence in a template (to justify their judgement)  The team synthesised results & discussed judgements (moderation)  The report (and recommendations) was discussed with the Senior Management team Brent McPherson- June 2010

Benefits to ERO The self-review:  reinforced a commitment to all staff to promote a culture of continuous improvement  reinforced ERO’s own credibility (ERO promotes evaluation capacity externally)  identified a set of recommendations for action (the recommendations have been accepted by SMT and integrated into the work programme)  encouraged the Chief Executive to think about ERO’s future approach to internal evaluation Brent McPherson- June 2010

Why Do A Self-Review?  respond to Govt priorities (ie how to “maximise the value received from its education services”)  promote continuous improvement through a formal and structured approach  promote accountability – and a balance of internal and external review (ERO ‘s approach to schools)  prepare for a formal review  review the organisation (do it in chunks?)  basis for engaging with the Minister (PIAs, SOI) Brent McPherson- June 2010

Issues and Challenges To improve the value of the self-review, next time ERO would….. plan ahead  schedule at a time appropriate to ERO (and set time aside to conduct the self-review)  engage managers and staff in the planning (and engage the senior management team throughout)  identify priorities – to reduce time spent on areas where there are high levels of comfort  incorporate a greater field component - to test the cascade of initiatives to the field (ERO is a field agency)  develop an associated communications plan Brent McPherson- June 2010

Issues and Challenges  using a moderation process to synthesise the results ERO removed bias by:  ensuring that the views expressed were backed up by supporting evidence  ensuring that review team members did not review their own portfolio Brent McPherson- June 2010

Issues and Challenges  ensure the team has access to evaluation skills Other recommendations/challenges:  plan for conflicting opinion  treat it as a formal review (ready the documents)  what is the current culture of the agency? Brent McPherson- June 2010