1. Bayesian insights into health-care performance monitoring in the UK David Spiegelhalter MRC Biostatistics Unit, Cambridge Thanks to Nicky Best, Clare Marshall, Ken Rice, Paul Aylin, Gordon Murray, Stephen Evans, Vern Farewell, Robin Kinsman, Olivia Grigg, Tom Treasure.

2. Some background 1997: General Medical Council finds 2 Bristol surgeons guilty of ‘Gross Professional Misconduct’ after high mortality rates 2000: Dr Harold Shipman convicted of murdering 15 of his patients 2001: Bristol Inquiry recommends establishing ‘Office of Information on Health-Care Performance’ as part of CHI 2002: ‘Office’ set up, responsible for performance assessment, ‘star ratings’ etc.

3. Some statistical issues not to be discussed Selection and use of performance indicators Data quality Risk-adjustment Aggregating multiple indicators Evaluation of programme Performance management vs quality improvement etc

4. To be discussed (briefly) ‘League tables’ and ranking Role of shrinkage estimation? ‘Extreme’ or ‘divergent’ cases? ‘Null distributions’ Sequential analysis (False discovery rates) (risk-adjusted dynamic models)

8. Criticisms of ‘league tables’  Spurious ranking – ‘someone’s got to be bottom’ · Encourages comparison when perhaps not justified · 95% intervals arbitrary · No consideration of multiple comparisons · Risk-adjustment is always inadequate · Single-year cross-section – what about change?

10. Funnel plot No ranking Data displayed No preset threshold for ‘out-of-control’ Visual relationship with volume Emphasises increased variability of smaller centres Links to Shewhart control chart / SPC methods / six-sigma etc

14. Can make inferences on ranks using MCMC

15. ‘Dr Foster’ - commercial publisher

16. Can fit hierarchical / ‘random-effects’ models

17. Care with random effects / shrinkage models Outlying centres, if included, can be very influential in estimating between-centre variability But smaller outlier centres can be shrunk and so remain undetected Should check plausibility of random- effects distribution before shrinkage

18. Need to be clear about hypotheses A ‘shrunk’ interval tests whether Centre X > average, assuming X is ‘similar’ (exchangeable) with rest But we are interested in whether X is ‘similar’ ‘Divergent’ rather than simply ‘extreme’

19. ‘Null distributions’ Y i ~ p( y i |  i ) H 0 :  i ~ N( ,  2 ) H 1 :  i ~ ‘divergent’ Bayesian perspective: partial exchangeability with unknown group membership SPC perspective: over-dispersed ‘in-control’ distribution Problem: are ,  estimated from data, or specified as ‘acceptable’ target variability?

20. Bristol model r i ~ Binomial(  i, n i ) logit(  i ) =  i H 0 :  i ~ N( ,  2 ) Uniform priors on ,  Cross-validation: 1.each centre i left out in turn 2.  i rep simulated 3.r i rep |  i rep simulated and compared with r i obs

21. (a) Leave-one-out Cross-validation

22. But there may be many ‘divergent cases’: e.g. re-admission rates in acute hospitals

23. (b) Model H 1 Want ‘robust’ estimates of ,  ; uninfluenced by potential divergent centres Many options: trimming, censoring, mixtures, e.g.  i ~ p 0 N( ,  2 ) + (1 – p 0 ) N( ,  2 ) Estimate parameters (Bayes, ML) Get P-values P i = P(Y>y i obs | H 0 ) Allow for uncertainty about ,  Can put P-values into FDR analysis

24. Two-sided P-values for re-admission data: null Normal random-effects distribution

25. One-sided P-values allowing for over-dispersed null distribution;  = 0.26 (0.08 to 0.56)

26. Informative prior distributions Often limited evidence on ‘null’ sd  Over-dispersion of ‘in-control’ group Represents unexplained heterogeneity, due to inadequate risk-adjustment etc Often on logit or log scale Interpretation important

27. Interpretation of  : sd of logit(  i )  ‘Range’ of true ORs: ie 97.5% / 2.5% exp(3.92  ) Median ratio of two random ORs exp(1.09  ) 01.00 0.22.191.24 0.57.101.72 150.402.97 225408.84 Suggests  = 0.2 is reasonable in many contexts,  = 0.5 is quite high

28. Distributions for  : sd of logit(  i )

29. Informative prior distributions For Bristol, mixture model with uniform prior for  gives  = 0.26 (0.08 to 0.56); p-Bristol = 0.006 Half-normal prior with upper 90% point at  = 0.5 gives  = 0.22 (0.09 to 0.44); p-Bristol = 0.002 Informative prior should be an acceptable input in this context

31. Shipman Inquiry July 2002: 215 definite victims, 45 probable

32. (NB: Shipman Inquiry total of definite or probable victims: 189 female > 65, 55 male over 65)

33. Developed in 1940’svs Most powerful sequential test: H 0 vs H 1 LLR = log [ p(data| H 0 ) / p(data| H 1 ) ] Contribution to log(likelihood ratio) is O log r - ( r -1) E To detect doubling of risk on death: LLR = O log 2 - E Horizontal thresholds set by error rates Sequential probability ratio test (SPRT)

35. Risk-adjusted CUSUM Same steps as SPRT Never drops below 0 Will always eventually ‘signal’ so need to check performance over limited periods

36. Aylin, Best, Bottle, Marshall (2003) Retrospective analysis of 1000 GPs, including Shipman Risk-adjusted CUSUM (same steps as SPRT, constrained to be > 0 ) Transformation to approximate Normality Adjustment for over-dispersion (although  estimated including Shipman etc) Evaluation of system over limited time period, using simulated sensitivity and FDR Retrospective identification of ‘divergent’ GPs

37. Aylin et al (2003), Fig1 CUSUM charts for the 12 GPs signalling at any time between 1993 and 1999. Charts designed to detect increase of four standard deviations in standardised excess mortality (K=4) using an alarm threshold of h=3 (h=5 is also shown). Harold Shipman’s CUSUM chart is shown in bold.

38. Conclusions Wide range of issues Caution in using hierarchical models There is enthusiasm for robust methods with good graphical output Partial exchangeability can translate to ‘over-dispersed in-control distribution’ Tie-in with SPC techniques attractive Bayesian insights useful ‘Bayesian’ / ‘non-Bayesian’ division not so useful

39. References P Aylin, NB Best, A Bottle, EC Marshall EC (2003) Following Shipman: a pilot system for monitoring mortality rates in primary care. Lancet O Grigg, VT Farewell and DJ Spiegelhalter (2003) A comparison of approaches to sequential monitoring of risk-adjusted health outcome Statistical Methods in Medical Research 12, 147--170 E C Marshall and D J Spiegelhalter (2003) Approximate cross-validatory predictive checks in disease-mapping model. Statistics in Medicine 22, 1649--1660 D J Spiegelhalter, P Aylin, S J W Evans, G D Murray, and N G Best. Commissioned analysis of surgical performance using routine data: lessons from the Bristol Inquiry (with discussion). Journal of the Royal Statistical Society, Series A, 165:191–232, 2002. D. J. Spiegelhalter (2002) Funnel plots for institutional comparisons. Quality Safety in Health Care, 11, 390—391. DJ Spiegelhalter (2002) An investigation into the relationship between mortality and volume of cases: an example in paediatric cardiac surgery between 1991 to 1995. British Medical Journal 324, 261--263 DJ Spiegelhalter, R Kinsman, O Grigg and T Treasure. (2003) Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman, and adult cardiac surgery. International Journal for Quality in Health Care 15:7–13, DJ Spiegelhalter, K Abrams, and JP Myles. Bayesian Approaches to Clinical Trials and Health Care Evaluation. Wiley, Chichester, 2003.