WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW 16.3.2011 Matti Reinikainen North Karelia Central Hospital, Joensuu

THE FINNISH INTENSIVE CARE CONSORTIUM A co-operation body coordinating a quality assurance project Started in 1994 Has strongly expanded since 1998 All university hospitals (5) have been involved since 2002 Now the Consortium comprises all central hospital districts (20) on the mainland 1994 2007

APACHE II data is also collected So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS II Based on “The Severity Study” 13 152 patients (720 from 7 Finnish hospitals) Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270: 2957-63. APACHE II data is also collected

APACHE II vs. SAPS II same basic principle, values of physiologic parameters from the first 24 hrs in the ICU APACHE II (Acute Physiology And Chronic Health Evaluation II): the diagnostic category weight is added to the logit SAPS II (Simplified Acute Physiology Score II): the diagnosis is not needed; instead the type of admission (scheduled surgical, unscheduled surgical, medical) affects the score

ARE THE OLD MODELS GOOD ENOUGH? APACHE II - from 1985 - not always easy to choose the right diagnostic category SAPS II - from 1993 - advantage: no diagnosis needed - disadvantage: does not take into account the diagnosis

DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? A patient example: HR 110/min SAPs 84 mmHg Tc 38 ºC consciousness, renal function, blood cell counts, electrolytes quite OK HCO3- 18 mmol/l age 65 years no difficult chronic diseases a medical admission respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa) PROBABILITY OF IN-HOSPITAL DEATH ?

DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? A patient example: HR 110/min SAPs 84 mmHg Tc 38 ºC consciousness, renal function, blood cell counts, electrolytes quite OK HCO3- 18 mmol/l age 65 years no difficult chronic diseases a medical admission respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33,3 kPa) SAPS II score 32 points → probability 0.128

SAPS II –score 32 → probability 0.128 the database of the Finnish Consortium, 1998-2007, readmissions excluded: 2319 patients, with a SAPS II score of 32 points hospital mortality 8.4% 84/128 = 0,66

SAPS II –score 32 → probability 0,128 the database of the Finnish Consortium, 1998-2007, readmissions excluded: 2319 patients, with a SAPS II score of 32 points hospital mortality 8.4% diabetic ketoacidosis (n = 26): mort 0% drug intoxication (n = 108): mort 0.9% congestive heart failure (n = 49): mort 22.4% 84/128 = 0,66 11/49 = 0,224

CAN SAPS II STILL BE USED? It overestimates the risk of death – leads to ”grade inflation” If most intensive care units are graduating with honors, is it genuine quality or grade inflation? Popovich MJ, Crit Care Med 2002 Recalibrations are needed

SMR 1998 – 2007, FINNISH INTENSIVE CARE CONSORTIUM SMR based on new calibration SMR based on original SAPS II model Päivitetty 09.04.2008 11

CAN SAPS II STILL BE USED? It can be used for monitoring changes in a unit’s own results Can be used for benchmarking purposes if the units to be compared have similar case-mix Should not be used to compare results of units with major differences in case-mix

SAPS 3 WAS CONSIDERED IN FINLAND TOO - IS IT A GOOD ALTERNATIVE? Values of physiological parameters ± 1 h of ICU admission Reason for ICU admission documented more precisely than in SAPS II Takes into account pre-ICU care Prognostic performance? Quality of data collected??

The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344 The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344. (Part 1) Moreno et al. ICM 2005: 31:1345-1355. (Part 2) At first 22,791 admissions Exclusions: readmissions (1455), < 16 yrs (628), those without ICU admission or discharge data (1074) and those that lacked an entry in the field ”ICU outcome” (57) - SAPS 3 basic cohort: 19,577 patients

The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344 The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344. (Part 1) Moreno et al. ICM 2005: 31:1345-1355. (Part 2) SAPS 3 basic cohort: 19,577 patients More exclusions: patients with a missing entry in the field of ”vital status at hospital discharge” (2540) and those still in hospital (253) SAPS 3 Hospital outcome cohort: 16,784 patients Quality of data? – at first, 5.5% of patients excluded because of missing data; then 13% of the remaining population excluded because of missing data on vital status

The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344 The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344. (Part 1) Moreno et al. ICM 2005: 31:1345-1355. (Part 2) How about data completeness? ”Data completeness was found to be satisfactory with 1 [0-3] SAPS II parameter missing per patient” How many SAPS 3 parameters were missing? ??? Were the physiological values obtained within ± 1 h?

SAPS 3 – even if data quality in the study was less than perfect, does it work?

Ledoux D et al. SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 2008; 34: 1873-7. single-centre (Belgium), 802 patients “the SAPS 3 … model customised for Central and Western Europe … was not significantly better than the SAPS II.” Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009; 35: 1916-24. “…the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings” Intensive Care Med. 2008 Oct;34(10):1873-7. Epub 2008 Jul 1. SAPS 3 admission score: an external validation in a general intensive care population. Ledoux D, Canivet JL, Preiser JC, Lefrancq J, Damas P. Soins Intensifs Généraux, Centre Hospitalier Universitaire de Liège, Domaine Universitaire de Sart Tilman Bat B35, 4000 Liège, Belgium. dledoux@chu.ulg.ac.be Abstract OBJECTIVES: To validate the SAPS 3 admission score in an independent general intensive care case mix and to compare its performances with the APACHE II and the SAPS II scores. DESIGN: Cohort observational study. SETTING: A 26-bed general ICU from a Tertiary University Hospital. PATIENTS AND PARTICIPANTS: Eight hundred and fifty-one consecutive patients admitted to the ICU over an 8-month period. Of these patients, 49 were readmissions, leaving 802 patients for further analysis. INTERVENTION: None. MEASUREMENTS AND RESULTS: APACHE II, SAPS II and SAPS 3 variables were prospectively collected; scores and their derived probability of death were calculated according to their original manuscript description. The discriminative power was assessed using the area under the ROC curve (AUROC) and calibration was verified with the Hosmer-Lemeshow goodness-of-fit test. The AUROC of the APACHE II model (AUROC = 0.823) was significantly lower than those of the SAPS II (AUROC = 0.850) and SAPS 3 models (AUROC = 0.854) (P = 0.038). The calibration of the APACHE II model (P = 0.037) and of the SAPS 3 global model (P = 0.035) appeared unsatisfactory. On the contrary, both SAPS II model and SAPS 3 model customised for Central and Western Europe had a good calibration. However, in our study case mix, SAPS II model tended to overestimate the probability of death. CONCLUSION: In this study, the SAPS 3 admission score and its prediction model customised for Central and Western Europe was more discriminative and better calibrated than APACHE II, but it was not significantly better than the SAPS II. Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009 Nov;35(11):1916-24. Epub 2009 Aug 14. Servizio Anestesia e Rianimazione, Ospedale Civile San Martino, Belluno, Italy. danest@libero.it OBJECTIVE: To evaluate the SAPS 3 score predictive ability of hospital mortality in a large external validation cohort. DESIGN: Prospective observational study. SETTING AND PATIENTS: A total of 28,357 patients from 147 Italian ICUs joining the Project Margherita national database of the Gruppo italiano per la Valutazione degli interventi in Terapia Intensiva (GiViTI). INTERVENTIONS: None. MEASUREMENT: Evaluation of discrimination through ROC analysis and of overall goodness-of-fit through the Cox calibration test. MAIN RESULTS: Although discrimination was good, calibration turned out to be poor. The general and the South-Europe Mediterranean countries equations overestimated hospital mortality overall (SMR values 0.73 with 95% CI 0.72-0.75 for both equations) and homogeneously across risk classes. Overprediction was confirmed among important subgroups, with SMR values ranging between 0.47 and 0.82. CONCLUSIONS: The result strictly supported by our data is that the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings.

Sakr Y et al. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Br J Anaesth 2008; 101: 798-803. single-centre (Germany), 1851 patients “… the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.” Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group. Austrian validation and customization of the SAPS 3 Admission Score. Intensive Care Med 2009; 35: 616-22. 22 ICUs in Austria, 2060 patients “The SAPS 3 … general equation can be seen as a framework … For benchmarking purposes, region-specific or country-specific equations seem to be necessary...” Br J Anaesth. 2008 Dec;101(6):798-803. Epub 2008 Oct 9. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Sakr Y, Krauss C, Amaral AC, Réa-Neto A, Specht M, Reinhart K, Marx G. Department of Anaesthesiology and Intensive Care, Friedrich-Schiller-University Hospital, Erlanger Allee 103, 07743 Jena, Germany. Abstract BACKGROUND: The Simplified Acute Physiology Score (SAPS) 3 has recently been developed, but not yet validated in surgical intensive care unit (ICU) patients. We compared the performance of SAPS 3 with SAPS II and the Acute Physiology and Chronic Health Evaluation (APACHE) II score in surgical ICU patients. METHODS: Prospectively collected data from all patients admitted to a German university hospital postoperative ICU between August 2004 and December 2005 were analysed. The probability of ICU mortality was calculated for SAPS II, APACHE II, adjusted APACHE II (adj-APACHE II), SAPS 3, and SAPS 3 customized for Europe [C-SAPS3 (Eu)] using standard formulas. To improve calibration of the prognostic models, a first-level customization was performed, using logistic regression on the original scores, and the corresponding probability of ICU death was calculated for the customized scores (C-SAPS II, C-SAPS 3, and C-APACHE II). RESULTS: The study included 1851 patients. Hospital mortality was 9%. Hosmer and Lemeshow statistics showed poor calibration for SAPS II, APACHE II, adj-APACHE II, SAPS 3, and C-SAPS 3 (Eu), but good calibration for C-SAPS II, C-APACHE II, and C-SAPS 3. Discrimination was generally good for all models [area under the receiver operating characteristic curve ranged from 0.78 (C-APACHE II) to 0.89 (C-SAPS 3)]. The C-SAPS 3 score appeared to have the best calibration curve on visual inspection. CONCLUSIONS: In this group of surgical ICU patients, the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models. Intensive Care Med. 2009 Apr;35(4):616-22. Epub 2008 Oct 10. Austrian validation and customization of the SAPS 3 Admission Score. Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group. Collaborators (22) Sagmüller G, Schwameis F, Pichler B, Ernst F, Bauer T, Sterrer F, Trimmel H, Klimscha W, Linemayr D, Schuh J, Sprinzl G, Dörre K, Trimmel H, Frank G, Malle H, Schindler I, Fitzal S, Schuster R, Locker G, Schneller H, Artmann H, Schuberth O. Department of Medical Statistics, Medical University of Vienna, Vienna, Austria. OBJECTIVE: To test the prognostic performance of the SAPS 3 Admission Score in a regional cohort and to empirically test the need and feasibility of regional customization. DESIGN: Prospective multicenter cohort study. PATIENTS AND SETTING: Data on a total of 2,060 patients consecutively admitted to 22 intensive care units in Austria from October 2, 2006 to February 28, 2007. MEASUREMENTS AND RESULTS: The database includes basic variables, SAPS 3, length-of-stay and outcome data. The original SAPS 3 Admission Score overestimated hospital mortality in Austrian intensive care patients through all strata of the severity-of-illness. This was true for both available equations, the General and the Central and Western Europe equation. For this reason a customized country-specific model was developed, using cross-validation techniques. This model showed excellent calibration and discrimination in the whole cohort (Hosmer-Lemeshow goodness-of-fit: H = 4.50, P = 0.922; C = 5.61, P = 0.847, aROC, 0.82) as well as in the various tested subgroups. CONCLUSIONS: The SAPS 3 Admission Score's general equation can be seen as a framework for addressing the problem of outcome prediction in the general population of adult ICU patients. For benchmarking purposes, region-specific or country-specific equations seem to be necessary in order to compare ICUs on a similar level.

2 ICUs in Norway, 1862 patients “The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II.” A comparison of SAPS II and SAPS 3 in a Norwegian intensive care unit population. Acta Anaesthesiol Scand. 2009 May;53(5):595-600. Strand K, Søreide E, Aardal S, Flaatten H. Department of Anesthesia and Intensive Care, Stavanger University Hospital, Stavanger, Norway. stkr@sus.no Comment in: Acta Anaesthesiol Scand. 2009 Oct;53(9):1230-1. Abstract BACKGROUND: Simplified Acute Physiology Score (SAPS II) is the most widely used general severity scoring system in European intensive care medicine. Because its performance has been questioned in several external validation studies, SAPS 3 was recently released. To our knowledge, there are no published validation studies of SAPS II or SAPS 3 in the Scandinavian countries. We aimed to evaluate and compare the performance of SAPS II and SAPS 3 in a Norwegian intensive care unit (ICU) population. METHOD: Prospectively collected data from adult patients admitted to two general ICUs at two different hospitals in Norway were used. Probability of mortality was calculated using the SAPS 3 global equation (SAPS 3 G), the SAPS 3 Northern European equation (SAPS 3 NE), and the original SAPS II equation. Performance was assessed by the standardized mortality ratio (SMR), area under receiving operating characteristic, and the Hosmer and Lemeshow goodness-of-fit C test. RESULTS: One thousand eight hundred and sixty-two patients were included after excluding readmissions, and patients who were admitted after coronary surgery or burns. The SMRs were SAPS 3 G 0.71 (0.65, 0.78), SAPS 3 NE 0.74 (0.68, 0.81), and SAPS II 0.82 (0.75, 0.91). Discrimination was good in all systems. Only the SAPS 3 equations displayed satisfactory calibration, as measured by the Hosmer-Lemeshow test. CONCLUSION: The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II. Both systems considerably overestimated mortality and exhibited good discrimination, but only the SAPS 3 equations showed satisfactory calibration. Customization of these equations based on a larger cohort is recommended.

SAPS II showed better discrimination SAPS 3 equations showed better calibration “…in our experience the scoring process is more time-comsuming and complex than that for SAPS II.”

SAPS 3, CONCLUSION: Does it work? – Yes! However, prognostic performance is NOT better than that of SAPS II the scoring process is more time-comsuming and complex than that for SAPS II (experience from Norway) on the other hand: according to many studies, the calibration of SAPS II is poor and customisation is needed

QUESTION DISCUSSED IN FINLAND: Should we implement a new risk-adjustment model (SAPS 3) that is not better than the old ones is more time-consuming would require customisation Or should we go on with one of the old models (that also require customisation)?

FINNISH (at least temporary) SOLUTION: OWN CUSTOMISED PREDICTION MODEL One objective: no need to exclude patient groups for benchmarking neuro- and cardiac surgical patients are not excluded We did not want to increase the burden of data collection – no new parameters added SAPS II –based data collection preserved possible to compare the results with those of previous years possible to describe the population using a well-known scoring system

OWN CUSTOMISED MODEL Based on patients treated in 2007-2008 - M Reinikainen, P Mussalo, V Kiviniemi, V Pettilä, E Ruokonen Based on patients treated in 2007-2008 Readmissions excluded Age ≥ 18 yrs Those discharged to another ICU excluded n = 25 801

OWN CUSTOMISED MODEL Outcome variable (to be predicted) ”DEATH IN HOSPITAL” Explaining covariates: Emergency admission or planned beforehand Surgical postoperative or medical SAPS II score without admission type points ln ((SAPS II score without admission type points) + 1) Diagnostic groups having an independent impact on the probability of death First a binary variable (0,1) was made of every APACHE III –dg group; everyone of these was tested separately 31 dg groups with an independent effect were included in the model

LOGISTIC REGRESSION ANALYSIS logit = β0 + β1X1 + β2X2 + … + βiXi - the regression analysis produces the constant β0 and the coefficients βi the logit can be calculated when the parameter values Xi are known the logit (log odds) can also be expressed as and thus

CALCULATING THE RISK R

PROB = EXP(LOGIT) / (1 + EXP(LOGIT)) LOGIT = -7,796 + 0,049 x (SCORE_SAPS_WITHOUT_ADM_TYPE_POINTS) + 1,013 x (ln(SAPS_WITHOUT_ADM_TYPE_POINTS + 1)) + 0,767 (if emergency admission) - 0,219 (if post-operative admission) + 1,229 (if DG_NONOP_CARDIOGENIC_SHOCK) + 0,364 (if DG_NONOP_CARDIAC_ARREST) – 0,796 (if DG_NONOP_RHYTHM_DISTURBANCE) + 0,348 (if DG_NONOP_ACUTE_MYOCARDIAL INFARCTION) + 0,422 (if DG_NONOP_BACTERIAL_OR_VIRAL_PNEUMONIA) – 1,619 (if DG_NONOP_MECHANICAL_AIRWAY_OBSTRUCTION) + 0,306 (if DG_NONOP_OTHER_RESP_DISEASES) + 0,795 (if DG_NONOP_HEPATIC_FAILURE) + 0,703 (if DG_NONOP_GI_PERFORATION_OR_OBSTRUCTION) + 0,643 (if DG_NONOP_GI BLEEDING_DUE_TO_VARICES) + 0,431 (if DG_NONOP_OTHER_GI_DISEASES) + 0,790 (if DG_NONOP_INTRACEREBRAL_HAEMORRHAGE) + 0,654 (if DG_NONOP_SUBARACHNOID_HAEMORRHAGE) + 0,400 (if DG_NONOP_STROKE) – 1,427 (if DG_NONOP_NEUROLOGIC_INFECTION) - 1,266 (if DG_NONOP_SEIZURE) – 0,486 (if DG_NONOP_OTHER_NEUROLOGIC_DISEASES) - 0,679 (if DG_NONOP_MULTIPLE TRAUMA_WITHOUT_HEAD_TRAUMA) – 0,658 (if DG_NONOP_METABOLIC_COMA) – 2,126 (IF DG_NONOP_DIABETIC_KETOACIDOSIS) – 2,245 (if DG_NONOP_DRUG_OVERDOSE) – 1,150 (if DG_NONOP_OTHER_METABOLIC_DISEASES) – 0,752 (if DG_NONOP_OTHER MEDICAL_DISEASES) + 0,340 (if DG_POSTOP_DISSECTING_OR_RUPTURED_AORTA) – 0,701 (if DG_POSTOP_CABG) + 0,701 (if DG_POSTOP_PERIPH_ARTERY_BYPASS_GRAFT) + 0,470 (if DG_POSTOP_GI_PERFORATION_OR_RUPTURE) + 0,411 (if DG_POSTOP GI_OBSTRUCTION) – 0,522 (if DG_POSTOP_SUBDURAL_OR_EPIDURAL_HAEMATOMA) – 0,885 (if DG_POSTOP_CRANIOTOMY_FOR_NEOPLASM) – 1,620 (if DG_POSTOP_OTHER_RENAL_DISEASES) PROB = EXP(LOGIT) / (1 + EXP(LOGIT))

PATIENT EXAMPLE A patient example: HR 110/min SAPs 84 mmHg Tc 38 ºC consciousness, renal function, blood cell counts, electrolytes quite OK HCO3- 18 mmol/l age 65 years no difficult chronic diseases a medical admission respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa) SAPS II score 32 points → probability 0.128

PATIENT EXAMPLE SAPS II score 32 → probability 0.128 New customised model: If none of the diagnoses included in the model: probability 0.082 dg bacterial pneumonia: probability 0.12 dg drug intoxication: probability 0.0094

AUROC APACHE II: 0.84 SAPS II: 0.84 new customised model: 0.87 H-L test for new model: p = 0.127

CONCLUSIONS SAPS 3 works, but its prognostic performance is not better than that of SAPS II If you want to use SAPS 3, you should probably customise it If you want to use SAPS II, you should probably customise it Idea for future research: to create a Nordic risk adjustment model, predicting 6-month or 1-year mortality