© Imperial College LondonPage 1 A method for estimating the cost of reducing the false alarm rate in multi- institution performance monitoring using CUSUM charts Alex Bottle Imperial College London Dr Foster Unit

© Imperial College LondonPage 2 Overview Background: cumulative sum charts and nationwide NHS mortality monitoring tool Extent of multiple testing Factors affecting false alarm rate Simulation for false alarm and successful detection rates Estimation of ‘cost’: worked example for AMI Summary

© Imperial College LondonPage 3 CUSUM chart essentials Plots one patient at a time Chart statistic (log-likelihood ratio) goes up if patient dies and down if patient survives Chart rises faster if low-risk patient dies If crosses preset threshold, chart ‘signals’ Threshold choice involves consideration of type I and type II error rates

© Imperial College LondonPage 4 CUSUM charts

© Imperial College LondonPage 5 Mortality monitoring tool In use in ~100 acute hospitals in England Compares each hospital’s case-mix adjusted mortality rate with national average Tests for an odds ratio of at least 2 Displayed using cumulative sum charts Data are updated monthly

© Imperial College LondonPage 6 Mortality monitoring tool: opening screen

© Imperial College LondonPage 7 Extent of multiple testing Over time: threshold handles this element But… At each hospital trust each month: 78 diagnosis groups >100 procedure groups National monitoring incurs further ‘cost’: ~150 acute hospital trusts Consultant-level monitoring?

© Imperial College LondonPage 8 Factors affecting the false alarm rate Threshold: the higher this is set, the lower the false alarm rate Length of monitoring: number of patients varies by hospital and diagnosis Expected mortality rate: e.g. 5% rates will have high FAR than 1% rates Size of increase (OR) to be detected (not considered here)

© Imperial College LondonPage 9 Research question A higher chart threshold -> lower FAR but slower detection of high mortality rates Compared with the conventional 5% false alarm rate, what is the ‘cost’ of having a lower false alarm rate (1% or 0.1%) to deal with all the multiple testing?

© Imperial College LondonPage 10 Simulation: FAR and SDR For FAR, generate 5,000 artificial hospitals with mortality rate p Do this for various p, p=0.1% to 30% Calculate FAR after t patients, t in steps of 5 from 5 to 20,000 Do this for different thresholds h, h=0.5 to 15 For SDR, generate hospitals with rate 2p/(1-p)

© Imperial College LondonPage 11 Using the simulation to estimate ‘cost’ For each dx, work out the threshold h needed for FAR of 5% at average hosp Find number of monitored patients t needed for SDR of 80% using threshold h Knowing the dx’s expected death rate and OR to be detected, convert t into a number of deaths Repeat for FAR of 1% and 0.1% Find the difference in number of deaths between the pairs of FAR values

© Imperial College LondonPage 12 ‘Cost’ calculation for AMI in England (1) National death rate=11.8%. Average number of AMIs per hospital=467 For FAR=5%, h=5.2 -> t=185 for SDR=80% At rate p, this means 21.8 deaths At rate 2p/(1-p), this means 39.0 deaths ‘Excess’ deaths: 39.0 – 21.8 = 17.2

© Imperial College LondonPage 13 ‘Cost’ calculation for AMI in England (2) For FAR=0.1%, h=8.6 -> t=305 for SDR=80% At rate p, this means 36.0 deaths At rate 2p/(1-p), this means 64.4 deaths ‘Excess’ deaths: 64.4 – 36.0 = 28.4 ‘Cost’ of lowering FAR to 0.1% = 28.4 – 17.2 = 11.2 extra deaths at average hosp

© Imperial College LondonPage 14 Findings for AAA repair and CABG AAA repair: less common but high risk ‘Cost’=6.3 for FAR=0.1%, 2.4 for FAR=1% CABG: common but low risk ‘Cost’= 6.8 for FAR=0.1%, 2.8 for FAR=1% These are all figures for an average hospital

© Imperial College LondonPage 15 Summary Multiple testing can be addressed by lowering the false alarm rate: raise the threshold for CUSUM charts Other approaches include minimising ‘loss function’ or maximising ‘desirability function’ The proposed measure of ‘cost’ depends on mortality rate and hospital volume The ‘cost’ can be derived from simulation and is intuitive to less-technical users