1. Outcomes surveillance using routinely collected health data Paul Aylin Professor of Epidemiology and Public Health Dr Foster Unit at Imperial College London p.aylin@imperial.ac.uk 16 th July 2015

2. Heart operations at the Bristol Royal Infirmary “Inadequate care for one third of children” Harold Shipman Murdered more than 200 patients Key events

4. Final report of the Inquiry “Bristol was awash with data. There was enough information from the late 1980s onwards to cause questions about mortality rates to be raised both in Bristol and elsewhere had the mindset to do so existed” The report of the public inquiry into children's heart surgery at the Bristol Royal Infirmary 1984-1995: learning from Bristol http://webarchive.nationalarchives.gov.uk/+/www.dh.gov.uk/en/Publicationsandstatistics/Pu blications/PublicationsPolicyAndGuidance/DH_4005620

8. CUSUM charts for the GPs that signalled as having unusually high mortality rates

9. Hospital Episode Statistics - HES Electronic record of every inpatient or day case episode of patient care in every NHS (public) hospital in England 19 million records a year 300 fields of information including Patient details such as age, sex, geographical area of residence Diagnosis using ICD10 Procedures using OPCS4 Admission method Discharge method More recently outpatient, A&E attendance, and diagnostic and imaging dataset.

10. Why use Hospital Episode Statistics Comprehensive – collected by all NHS trusts across country on all patients Coding of data separate from clinician Access Current (we receive monthly updates)

11. Challenges - Case mix adjustment Limited within HES? Age Sex Emergency/Elective

12. How successful is the casemix adjustment?

13. ROC curve areas comparing ‘simple’, ‘intermediate’ and ‘complex’ models derived from HES with models derived from clinical databases Aylin P; Bottle A; Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334: 1044

14. Current casemix adjustment model for each diagnosis and procedure group Adjusts for age sex elective status socio-economic deprivation (Carstairs) Diagnosis subgroups (3 digit ICD10) or procedure subgroups co-morbidity – Charlson index number of prior emergency admissions palliative care year month of admission Source of admission

15. Comparison of HES vs clinical databases Vascular surgery HES = 32,242 National Vascular Database = 8,462 Aylin P; Lees T; Baker S; Prytherch D; Ashley S. (2007) Descriptive study comparing routine hospital administrative data with the Vascular Society of Great Britain and Ireland's National Vascular Database. Eur J Vasc Endovasc Surg 2007;33:461-465 Bowel resection for colorectal cancer HES 2001/2 = 16,346 ACPGBI 2001/2 = 7,635 ACPGBI database, 39% of patients had missing data for the risk factors Garout M, Tilney H, Aylin, P. Comparison of administrative data with the Association of Coloproctology of Great Britain and Ireland (ACPGBI) colorectal cancer database. International Journal of Colorectal Disease 2008;23(2):155-63

16. Other Challenges Trends Transfers Transfers linked. All spells (admissions) linked into superspells For diagnosis, outcome based on discharge method at end of superspell Diagnosis on admission No diagnosis on admission exists within HES/SUS We use primary diagnosis given on completion of first episode, unless a “vague symptoms and signs” diagnosis, in which case we examine subsequent episode Co-morbidity Palliative care If treatment specialty in any episode in the admission coded to palliative care or includes ICD10 code Z515, accounted for in risk model

17. Using data to identify outliers and produce alerts “Even if all surgeons are equally good, about half will have below average results, one will have the worst results, and the worst results will be a long way below average” Poloniecki J. BMJ 1998;316:1734-1736

18. Adjusted (EuroSCORE) mortality rates for primary isolated CABGs by centre (3 years data up to March 2005) using SCTS data with 95% and 99.8% control limits based on mean national mortality rates

19. Risk-adjusted Log-likelihood CUSUM charts STEP 1: estimate pre-op/admission risk for each patient, given their age, sex etc. This may be national average or other benchmark STEP 2: Order patients chronologically by date of operation STEP 3: Choose chart threshold(s) of acceptable “sensitivity” and “specificity” (via simulation) STEP 4: Plot function of patient’s actual outcome v pre-op risk for every patient, and see if – and why – threshold(s) is crossed

20. More details Based on log-likelihood CUSUM to detect a predetermined increase in risk of interest Taken from Steiner et al (2000); pre-op risks derived from logistic regression of national data The CUSUM statistic is the log-likelihood test statistic for binomial data based on the predicted risk of outcome and the actual outcome Model uses administrative data and adjusts for age, sex, emergency status, socio-economic deprivation etc. Bottle A, Aylin P. Intelligent Information: a national system for monitoring clinical performance. Health Services Research (in press).

24. Re-identification service Allows re-identification of cases underlying Dr Foster analytical tools No identifiable data ever passed to Dr Foster Intelligence Only NHS trusts ever see identifiable data, and only on their own patients Facilitates case note reviews and audit of care

26. Future Increasing barriers to using data for research and quality improvement Power of linkage Move to Electronic Patient Records New technologies (e.g. “wearables”) in gathering outcome data. Global

27. International data

28. Acknowledgements Alex Bottle Steve Middleton Brian Jarman