Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Governance of Health Care by Targets and Indicators: Can it be both Proportional and Transparent? Gwyn Bevan (LSE) and Christopher Hood (All Souls.

Similar presentations


Presentation on theme: "The Governance of Health Care by Targets and Indicators: Can it be both Proportional and Transparent? Gwyn Bevan (LSE) and Christopher Hood (All Souls."— Presentation transcript:

1 The Governance of Health Care by Targets and Indicators: Can it be both Proportional and Transparent? Gwyn Bevan (LSE) and Christopher Hood (All Souls College, Oxford) © ESRC Public Services Programme and Christopher Hood/Gwyn Bevan

2 1.A new approach to health care governance in the 2000s: targets and indicators linked to negative feedback (Voltaire: ‘ici on tue de temps en temps un amiral pour encourager les autres’) 2.Was it a decisive breakthrough in governance – or a partial repeat of the some of the history of the Soviet Union? Some evidence from the English NHS 3.Implications: how far can you combine transparency and proportionality in a world of reactive gaming?

3 Three phases of governance of the public health care system in England (  UK) 1.40 years of rule by professionals with weak managers (often loosely termed ‘command and control,’ but with no real command or control from the centre, albeit with various attempts to empower managers in the 1980s) 2.Attempt to control public health-care professionals through quasi-markets (separating providers and purchasers of health care) in the early 1990s 3.The era from 2001- a ‘concordat’ of higher spending accompanied by P.I.s and targets monitored from the centre by multiple and overlapping units (and a form of ‘terror’?)

4 Some Assumptions of Target and PI Governance Systems: Measurement is Unproblematic and Gaming is Unimportant 1.A part can meaningfully stand for the whole 2.‘Threshold effects’ at the top of the quality range either will not occur or do not matter 3.Gaming effects of targetry (the ‘knights-to-knaves’ problem) are either small or unimportant

5 Domain β for which no measures exist Domain α for which measures exist M[α] The Underlying Assumptions of The Target/PI Regime 1: Synecdoche: a Part can Meaningfully Stand for the Whole

6 Domain β for which no measures exist Domain α g for which good measures exist M[α g ]: no false positives or negatives Domain α i for which imperfect measures exist M[α i ]: large numbers of false positives and false negatives Three Domains of Performance

7 % poor averageexcellent Step 1: Performance Distribution across Delivery Units before Targets Underlying Assumptions of The Target Regime 2: ‘Threshold Effects’ either Don’t Matter or Won’t Happen

8 % poor averageexcellent Step 2: Target imposed

9 % poor averageexcellent Step 3: Excellence is squeezed

10 Underlying Assumptions of the Target Regime 3: Knights either will not turn into Knaves or Knavery can be Controlled ‘Saints’: who may not share mainstream goals, but whose public service ethos is so high that they voluntarily disclose shortcomings to central authorities ‘Honest triers’: who broadly share mainstream goals and do not voluntarily draw attention to their failures, but do not attempt to spin or fiddle data in their favour ‘Reactive gamers’: who broadly share mainstream goals, but aim to spin or fiddle data if they have a motive or opportunity to do so ‘Rational maniacs’: who do not share mainstream goals and aim to manipulate data to conceal their operations e.g. in the following extension of the knights/knaves distinction

11 Target and PI systems considered against 4 types of actors ‘Saints’: who may not share mainstream goals, but whose public service ethos is so high that they voluntarily disclose shortcomings to central authorities ‘Honest triers’: who broadly share mainstream goals and do not voluntarily draw attention to their failures, but do not attempt to spin or fiddle data in their favour ‘Reactive gamers’: who broadly share mainstream goals, but aim to spin or fiddle data if they have a motive or opportunity to do so ‘Rational maniacs’: who do not share mainstream goals and aim to manipulate data to conceal their operations Type Expected effect of targets NO CHANGE

12 The No-Gaming Assumption Revisited Types of conduct: Saints, Honest triers, Reactive gamers, Rational maniacs Problem of relying on signals: agent satisfied signals (M[α]): 1.All is well? 2.domain α  but domain β  3.Signals M[α]  but actions on domain α  4.failure on M[α] concealed by gaming Works for saints, problems for honest triers, vulnerable to Reactive gamers, fails for rational maniacs Gresham’s law: saints & honest triers  Reactive gamers?

13 1.A new approach to health care governance in the 2000s: targets and indicators linked to negative feedback (Voltaire: ‘ici on tue de temps en temps un amiral pour encourager les autres’) 2.Was it a decisive breakthrough in governance – or a partial repeat of the some of the history of the Soviet Union? Some evidence from the English NHS 3.Implications: how far can you combine transparency and proportionality in a world of reactive gaming?

14 England (since 2001): annual performance (star) rating –Zero to  –‘naming & shaming’ (hanging the admirals) Wales, Scotland, Northern Ireland –no ranking –no incentives The Case of Health Care Governance in England by Targets and PIs

15 Prime Minister’s Delivery Unit (since 2001, monitoring of c. 22 targets for PM by ‘war room’ approach) Treasury (since 1998, setting and monitoring over 130 PSA targets attached as conditions of funding to spending depts – c. 10 for health) Dept of Health (funds trusts, appointments, monitoring, guidance, setting targets additional to PSAs – c. 50 for each type of trust) Quality regulators (HCC, Monitor etc) (standards, inspections, monitoring, and since 2001 ratings of trusts in England) Health care ‘trusts’ (c.600) (for hospitals, ambulances, primary care: 1.3m staff, 100,000 doctors, 300,000 nurses, €85bn per year): DH money goes to c.300 PCTs which contract for services with other trusts Auditors (Audit Commission, NAO, plus nominated auditors): (financial and vfm audit) money reporting dialogue Institutions: Basics of the Targets and PI System

16 Elements of governance ‘Targets’ (include PIs & standards) –Agreed by government (priorities) & ‘independent’ regulator (credible measures?) Transparent regulation –Agents know targets being used –Results are published (to put pressure on agents) Proportionate ‘performance management’ –Government rewards ‘successes’ (‘light touch’ / ‘earned autonomy’ and penalises ‘failures’ (more intense monitoring / sack chief executives)

17 Some of the Key Targets Ambulances (2002) –75% category A calls < 8 minutes Hospitals –trolley waits in A&E: <12 hours admission –total time in A&E < 4 hours –waiting times for elective inpatient admission Hospitals: waiting times for elective inpatient admission Target waiting time (months)

18 20012002 from 2003 Acute (156) Specialist (20)  Ambulances (31)  Mental Health (88)  PCTs (304)  The Rise of ‘Star Ratings’

19 Effects of the Targets -Apparently convincing evidence that the introduction of targets affected reported performance in health care, independently of public spending level But how much of that performance change was ‘gamed’? -Evidence of gaming is serendipitous, because of a mix of ‘over-auditing’ and ‘under-auditing’ (just coincidence or ‘Nelson’s eye’?) -But undoubtedly there was a non-trivial amount of gaming

20 Exhibit A: target hospital waiting times before and after star ratings Numbers waiting elective admissions (‘000s) Star ratings published Source: Chief Executive’s Report to the NHS – Statistical Supplement (2004)

21 Exhibit B: Hospital waiting times in England over time compared to waits in other UK countries without star rating regimes Source: National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38 % patients waiting for hospital admission > 12 months

22 Spend per capita on health care (UK = 100) Source: www.hm-treasury.gov.uk/media//B4887/pesa04_chapter08_190404.pdf

23 Public Expenditure on health care (% GDP) Sources: 1990 to 2002 OECD & & official projection for UK of 9.4 per cent by 2008

24 NHS spend as % of GDP % GDP Sources: Office of Health Economics, HM Treasury, & official projection of 9.4 per cent by 2008

25 Exhibit C: Per cent of A & E cases seen within 4 hours 2002-03 2003-042004-05 & a 20% increase in numbers in A&E

26 EXAMPLE Ambulances (2001): target of 75% category A calls met in less than 8 minutes (category A calls are those in which there is an immediate threat to life, and are claimed to save 1,800 lives a year) But evidence of Soviet-type Gaming Too…

27 Exhibit C: Ambulance Category A Calls within 8 Minutes ‘Corrections’’ only 2% to 6% BeforeAfter

28 Response to target (75% of category A calls in less than 8 minutes) by 31 ambulance trusts in England in 2001 half failed to reach target reports of bullying gaming –Relocation of ambulance depots from rural to urban areas –third ‘corrected’ response times

29 Ambulance response times: ‘corrected’ in response to targets 75% < 8 minutes http://www.chi.nhs.uk/eng/cgr/ambulance/index.shtml

30 Other (variably documented) examples of gaming in the paper -Gaming over A & E wait times (wheels off trolleys, tents in car parks, patients kept in ambulances parked outside hospital, other activities cancelled during anticipated audits) -Gaming over elective surgery wait times (e.g. surgery offered during known vacation times) -Gaming over outpatient wait time targets for new patients (e.g. by cancellation and delay of follow-up appointments, which were not targeted - clinical incident forms showed that 25 patients lost their vision over two years as a result) -Gaming by Primary care trusts over 24 hour access target (removal of forward appointment systems, especially in London)

31 Definitional ambiguity and inconsistency –75% life-threatening emergencies < 8 minutes. massive variation between trusts in the proportion of calls that were classed as emergencies Data accuracy issues –Audit Commission ‘spot checks’ at 41 trusts reporting errors in at least one PI in 19 trusts (Audit Commission 2003). Performance distribution issues –Discrepancy in hospital waiting time targets –Relocation of ambulance depots from rural to urban areas How Robust were the Measures?

32 How Much Evidence of Gaming? (that is, failure on M[α] concealed by gaming) National Audit Office (2001): 9 NHS trusts ‘inappropriately’ adjusted their waiting lists –3 for 3 years or more, affecting nearly 6,000 patient records. –5 whose adjustments only came to light following patient, health authority or MP complaints, or adverse publicity –4 cases identified by the trusts concerned Audit Commission ‘spot checks’ 41 trusts 2002 –3 further cases of deliberate misreporting of waiting list information Such evidence suggests non-trivial but limited (7 per cent or less) detected gaming: but how hard are inspectors looking? (capacity for quality of data investigation arguably reduced with shift of responsibility from AC to Healthcare Commission)

33 …and non-trivial ‘synecdoche’ problems in health care quality measures 3 examples of failures, 2 of which produced major political fallout: –St George’s –BRI –Shipman All difficult to detect from outside –All could plausibly have sent ‘satisfactory’ M[α] signals under star ratings regime –And M[α] ‘failures’ might nevertheless provide excellent quality of care

34 Some Policy Conclusions Gaming is little discussed or acknowledged by architects and operators of PI and target systems, but (a)If gaming is significant, transparency and proportionality don’t always mix (b)If gaming is significant, there is a huge hole in the audit system between financial and value for money audit (that is, no data quality auditing – cf. Robert Behn’s advocacy of an Office of Performance Assessment). Is this hole accidental or deliberate? (c)If we are repeating the history of the USSR, is it 1945 or 1991?

35 -the principle of transparency in the sense of perfect certainty about the link between P.Is and rewards/sanctions appears to be incompatible with the principle of proportionality -The conflict might be reconciled by either (a) adopting a restrictive definition of transparency (e.g. retrospective rather than real time, process rather than event (Heald)), or (b) arguing that proportionality is not violated by random measures where risk is not assessable -But in either case random monitoring and possibly some element of uncertainty about weightings for KPIs or targets seems to be necessary for the operation of such a system of governance Transparency and Proportionality in Governance by Targets


Download ppt "The Governance of Health Care by Targets and Indicators: Can it be both Proportional and Transparent? Gwyn Bevan (LSE) and Christopher Hood (All Souls."

Similar presentations


Ads by Google