Governance by Targets & Terror: Synecdoche, Gaming & Audit Gwyn Bevan (LSE) & Christopher Hood (All Souls College, Oxford) Westminster Economic Forum 20 April 2005

Governance by Targets & Terror New approach to health care governance in 2000s Decisive breakthrough in governance – or repeat of history of Soviet Union? Conclusion

Governance by Targets & Terror New approach to health care governance in 2000s targets & indicators linked to negative feedback Voltaire: ‘ici on tue de temps en temps un amiral pour encourager les autres’ Decisive breakthrough in governance – or repeat of history of Soviet Union? Conclusion

Three phases of governance of public health care system UK Concordat clinical autonomy & resource constraints no real command or control from the centre 1980s: various attempts to empower managers 1990s: attempt to control public health-care professionals through quasi-markets separating providers & purchasers England only from 2001- new ‘concordat’ of higher spending accompanied by P.I.s & targets monitored from centre by multiple & overlapping units (& ‘terror’?)

NHS spend as % of GDP % GDP Sources: Office of Health Economics, HM Treasury, & official projection of 9.4 per cent by 2008

Institutional arrangements Prime Minister’s Delivery Unit 22 targets Treasury 130 PSA targets– c. 10 for health Audit Commission & NAO Audit finance & vfm Dept of Health 50 targets by trust type Healthcare Commission Quality regulator develops & publishes ratings Health care ‘trusts’ (c.700) money reporting dialogue

Some underlying assumptions Synecdoche: a part can meaningfully stand for the whole What is measured is a good indicator of performance (e.g. ‘threshold effects’ at the top of the quality range either will not occur or do not matter) Gaming effects (the ‘knights-to-knaves’ problem) are either small or unimportant

Defining priorities Residual domain β Domain α: government’s priorities

Measuring priorities Domain αn: no measures Residual domain β Domain αg: good measures M[αg]: no false positives or negatives Domain αi: imperfect measures M[αi]: large numbers false positives & negatives

The synecdoche assumption: the part can represent the whole Residual domain β+: omitted because unimportant or cannot be measured Domain α-: government’s priorities for which good & imperfect measures exist

‘Threshold Effects’ either don’t matter or won’t Happen Success Frequency Failure? Waiting time In months Target

Knights either will not turn into Knaves or Knavery can be Controlled ‘Saints’ may not share mainstream goals public service ethos so high that they voluntarily disclose shortcomings to central authorities ‘Honest triers’ broadly share mainstream goals do not voluntarily draw attention to their failures but do not attempt to spin or fiddle data in their favour ‘Reactive gamers’ broadly share mainstream goals, but aim to spin or fiddle data if they have a motive or opportunity to do so ‘Rational maniacs’ do not share mainstream goals aim to manipulate data to conceal their operations

Target & PI systems against 4 types of actors ‘Saints’ ‘Honest triers’ ‘Reactive gamers’ ‘Rational maniacs’ Expected effect of targets NO CHANGE

What is measured is a good indicator of performance? Works for saints Problems for honest triers Vulnerable to Reactive gamers Fails for rational maniacs Gresham’s law: saints & honest triers  Reactive gamers?

What is measured is a good indicator of performance? Agent satisfied signals M[α-] All is well? Domain α-  but domain β+ ? Domain α- ? Failure on M[α-] concealed by problems of definition & measurement? gaming? Domain α- Domain β+ Signals M[α-]:

Governance by Targets & Terror New approach to health care governance in 2000s Decisive breakthrough in governance – or repeat of history of Soviet Union? Some evidence from the English NHS Conclusion

Development of star ratings in England 2001 2002 from 2003 Acute (156) Specialist (20)  Ambulances (31) Mental Health (88) PCTs (304)

Evidence of impact of key targets Ambulances 75% category A calls < 8 minutes Hospitals total time in A&E < 4 hours waiting times for elective inpatient admission Reported successes Problems of measurement & gaming

Reported success: 75% category A calls<8 minutes Before After

Problems of measurement & gaming: third ‘corrected’ response times ‘Corrections’’ only 2% to 6% 75% < 8 minutes 75% < 8 minutes Source: http://www.chi.nhs.uk/eng/cgr/ambulance/index.shtml

Impact: A&E total time < 4 hours 2002-03 2003-04 2004-05 & a 20% increase in numbers in A&E Source: http://www.nao.org.uk Improving Emergency Care in England

Problems of measurement & gaming in A&E Government figures: mid-2004, target met by 96% of patients Healthcare Commission survey (55,000 patients): 77% of patients stayed < four hours in A&E Problems of gaming Queues of ambulances outside Moving staff & cancelling operations over period of measurement

Impact of key targets on hospital waiting times for elective inpatient admission Target waiting time (months) for elective inpatient admission

Impact: hospital waiting times elective admission Numbers waiting elective admissions (‘000s) Star ratings published Source: http://www.dh.gov.uk Chief Executive’s Report to the NHS – Statistical Supplement (2004)

Impact of key target: hospital waiting times elective admission Numbers waiting elective admissions (‘000s) Star ratings published Source: http://www.dh.gov.uk Chief Executive’s Report to the NHS – Statistical Supplement (2004)

Impact of key target: hospital waiting times elective admission % patients waiting for hospital admission > 12 months Source: http://www.statistics.gov.uk National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38

Impact of key target: hospital waiting times elective admission % patients waiting for hospital admission > 12 months Source: http://www.statistics.gov.uk National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38

Impact of key target: hospital waiting times elective admission % patients waiting for hospital admission > 12 months Source: http://www.statistics.gov.uk National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38

Impact of key target: hospital waiting times elective admission % patients waiting for hospital admission > 12 months Source: http://www.statistics.gov.uk National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38

Natural experiment in UK countries England (from 2001): annual performance (star) rating Zero to  ‘naming & shaming’ Wales, Scotland, Northern Ireland no ranking no incentives Spend per capita on health care (UK = 100) Source: www.hm-treasury.gov.uk/media//B4887/pesa04_chapter08_190404.pdf

Waiting times: problems of measurement & gaming Audit Commission: reporting errors at least one PI in 19 trusts Problems of gaming NAO: 9 NHS trusts inappropriately adjusted their waiting lists Audit Commission: 3 cases of deliberate misreporting of waiting list information Sources: www.nao.gov.uk/publications/nao_reports/01-02/0102452.pdf www.audit-commission.gov.uk/health/index.asp?catId=english^HEALTH.

‘Synecdoche’ problems over quality of care 2 failures that resulted in major public inquiries Bristol case of paediatric cardiac surgery Shipman All difficult to detect from outside All could plausibly have sent ‘satisfactory’ M[α-] signals under star ratings regime And M[α-] ‘failures’ might nevertheless provide excellent quality of care

Governance by Targets & Terror New approach to health care governance in 2000s Decisive breakthrough in governance – or repeat of history of Soviet Union? Conclusion How far did English NHS satisfy assumptions? Repeating Soviet history: 1939? 1969? 1989? Policy implications?

How far did English NHS satisfy assumptions? Synecdoche: a part can meaningfully stand for the whole? What is measured is a good indicator of performance? Gaming effects (the ‘knights-to-knaves’ problem) are either small or unimportant?

Repeating Soviet history: 1939? 1969? 1989? 1939: waiting times? 1969: ? 1989: balanced scorecard?

Policy implications: targets retained, limiting gaming? Limits on transparency Designing & resourcing systems of validation Sources of information other than data reported by those being monitored