The Challenge of Designing ‘Good Enough’ Performance Measures & Results Framework Targets & Performance Management World Bank’s Public Sector Performance Global Expert Team 9 May Gwyn Bevan

Models of delivery of public services Knights / Knaves Econs / Humans Prospect theory Trust & Altruism (T&A) KnightsHumansReward failure? Hierarchy & Targets (H&T) Knights & Knaves EconsSack staff? Public Ranking (PR) KnightsHumansReputation? Choice & Competition (C&C) Knights & Knaves EconsWinners & losers?

UK’s natural experiment: before & after devolution (1999) NHSsSchools BeforeAfterBeforeAfter England T&A H&T PR C&C PR WalesT&AC&C PR C&C

H&T+PR: ‘naming & shaming’ spotlight + prospect theory  Staff in zero-rated hospital ‘devastating …hit right down to workforce -- bad reports usually hit senior management upwards nurses demand changing rooms.. because being accosted in streets’ Source: Mannion et al (2005) ‘dirty dozen’ zero-rated CEOs (English NHS)

Hospital waiting times: T&A  H&T+PR Numbers waiting elective admissions (England) (‘000s) H&T+PR

Hospital waiting times: T&A No/‘000 waiting > 3 months GP  1st outpatient appointment

Hospital waiting times. Natural experiment: T&A vs H&T+PR No/‘000 waiting > 3 months GP  1st outpatient appointment H&T+PR

Ambulances:T&A Source: Bevan & Hamblin (2009) % responses life-threatening emergencies < 8 minutes Target E & W 

Ambulances. Natural experiment: T&A vs H&T+PR Source: Bevan & Hamblin (2009) % responses life-threatening emergencies < 8 minutes H&T+PR Target E  Target W 

% pupils achieving > 5 good grades at 16 Note: data are missing for 1998, 2001 & 2005 & these have been estimated as the mean values from adjacent years Schools: C&C+ PR

Schools. Natural experiment: C&C+PR vs C&C Note: data are missing for 1998, 2001 & 2005 & these have been estimated as the mean values from adjacent years Wales stops PR % pupils achieving > 5 good grades at 16

H&T & PR: design Residual domain R Gages on P No measures on P Tin openers on P Selected domain P

H&T & PR : problems of gaming Selected domain T: incentives apply to measures M[T] Excluded domain N : unimportant or cannot be measured Holmstrom & Milgram (1991)

H&T & PR: types of gaming  N ? Worsen performance?  M[T] ? Hit the target & miss the point Misreporting & cheating Collusion Target M[T] Domain N Domain T

For discussion  Theory of contingency mix of models  T&A performance measurement no gaming  ineffective  PR + H&T powerful  gaming Design to counter gaming  C&C choice attractive in principle Characteristics influence effectiveness

For discussion C&C: Requirements UniversitiesSchoolsHospitals Money follows choice Supply-side flexibility  Information  Transaction costs  Management freedom 

COFEPRIS: Commission for Protection against Sanitary Risks 9,000 registrations (value $1bn) required 150 registered in 2010

Elements for transformation of failing organisations English NHSCOFEPRIS Spotlight effect Naming & shaming in media Inform media of failure ‘Broken windows’ Waiting timesRegistrations Prospect theory Sack ‘failing’ CEOsSack underperforming middle managers RewardsName elite organisations Promote young able managers

COFEPRIS: Sanitary registrations Source: Mikel Arriola Peñalosa (2012) Federal Commissioner for the Protection against Sanitary Risks