Presentation on theme: "WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW"— Presentation transcript:
1 WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW Matti ReinikainenNorth Karelia Central Hospital, Joensuu
2 THE FINNISH INTENSIVE CARE CONSORTIUM A co-operation body coordinating a quality assurance projectStarted in 1994Has strongly expanded since 1998All university hospitals (5) have been involved since 2002Now the Consortium comprises all central hospital districts (20) on the mainland19942007
3 APACHE II data is also collected So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS IIBased on “The Severity Study”patients (720 from 7 Finnish hospitals)Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270:APACHE II data is also collected
4 APACHE II vs. SAPS IIsame basic principle, values of physiologic parameters from the first 24 hrs in the ICUAPACHE II (Acute Physiology And Chronic Health Evaluation II): the diagnostic category weight is added to the logitSAPS II (Simplified Acute Physiology Score II): the diagnosis is not needed; instead the type of admission (scheduled surgical, unscheduled surgical, medical) affects the score
5 ARE THE OLD MODELS GOOD ENOUGH? APACHE II- from 1985- not always easy to choose the right diagnostic categorySAPS II- from 1993- advantage: no diagnosis needed- disadvantage: does not take into account the diagnosis
6 DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? A patient example:HR 110/minSAPs 84 mmHgTc 38 ºCconsciousness, renal function, blood cell counts, electrolytes quite OKHCO3- 18 mmol/lage 65 yearsno difficult chronic diseasesa medical admissionrespiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa)PROBABILITY OF IN-HOSPITAL DEATH ?
7 DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? A patient example:HR 110/minSAPs 84 mmHgTc 38 ºCconsciousness, renal function, blood cell counts, electrolytes quite OKHCO3- 18 mmol/lage 65 yearsno difficult chronic diseasesa medical admissionrespiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33,3 kPa)SAPS II score 32 points → probability 0.128
8 SAPS II –score 32 → probability 0.128 the database of the Finnish Consortium, , readmissions excluded: 2319 patients, with a SAPS II score of 32 pointshospital mortality 8.4%84/128 = 0,66
9 SAPS II –score 32 → probability 0,128 the database of the Finnish Consortium, , readmissions excluded: 2319 patients, with a SAPS II score of 32 pointshospital mortality 8.4%diabetic ketoacidosis (n = 26): mort 0%drug intoxication (n = 108): mort 0.9%congestive heart failure (n = 49): mort 22.4%84/128 = 0,6611/49 = 0,224
10 CAN SAPS II STILL BE USED? It overestimates the risk of death – leads to ”grade inflation”If most intensive care units are graduating with honors, is it genuine quality or grade inflation? Popovich MJ, Crit Care Med 2002Recalibrations are needed
11 SMR 1998 – 2007, FINNISH INTENSIVE CARE CONSORTIUM SMR based on new calibrationSMR based on original SAPS II modelPäivitetty11
12 CAN SAPS II STILL BE USED? It can be used for monitoring changes in a unit’s own resultsCan be used for benchmarking purposes if the units to be compared have similar case-mixShould not be used to compare results of units with major differences in case-mix
13 SAPS 3 WAS CONSIDERED IN FINLAND TOO - IS IT A GOOD ALTERNATIVE? Values of physiological parameters ± 1 h of ICU admissionReason for ICU admission documented more precisely than in SAPS IITakes into account pre-ICU carePrognostic performance?Quality of data collected??
14 The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344 The SAPS 3 Study Metnitz et al ICM 2005: 31: (Part 1) Moreno et al. ICM 2005: 31: (Part 2)At first 22,791 admissionsExclusions: readmissions (1455), < 16 yrs (628), those without ICU admission or discharge data (1074) and those that lacked an entry in the field ”ICU outcome” (57)- SAPS 3 basic cohort: 19,577 patients
15 The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344 The SAPS 3 Study Metnitz et al ICM 2005: 31: (Part 1) Moreno et al. ICM 2005: 31: (Part 2)SAPS 3 basic cohort: 19,577 patientsMore exclusions: patients with a missing entry in the field of ”vital status at hospital discharge” (2540) and those still in hospital (253)SAPS 3 Hospital outcome cohort: 16,784 patientsQuality of data? – at first, 5.5% of patients excluded because of missing data; then 13% of the remaining population excluded because of missing data on vital status
16 The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344 The SAPS 3 Study Metnitz et al ICM 2005: 31: (Part 1) Moreno et al. ICM 2005: 31: (Part 2)How about data completeness?”Data completeness was found to be satisfactory with 1 [0-3] SAPS II parameter missing per patient”How many SAPS 3 parameters were missing????Were the physiological values obtained within ± 1 h?
17 SAPS 3 – even if data quality in the study was less than perfect, does it work?
18 Ledoux D et al. SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 2008; 34:single-centre (Belgium), 802 patients“the SAPS 3 … model customised for Central and Western Europe … was not significantly better than the SAPS II.”Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009; 35:“…the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings”Intensive Care Med Oct;34(10): Epub 2008 Jul 1.SAPS 3 admission score: an external validation in a general intensive care population.Ledoux D, Canivet JL, Preiser JC, Lefrancq J, Damas P.Soins Intensifs Généraux, Centre Hospitalier Universitaire de Liège, Domaine Universitaire de Sart Tilman Bat B35, 4000 Liège, Belgium.AbstractOBJECTIVES: To validate the SAPS 3 admission score in an independent general intensive care case mix and to compare its performances with the APACHE II and the SAPS II scores.DESIGN: Cohort observational study.SETTING: A 26-bed general ICU from a Tertiary University Hospital.PATIENTS AND PARTICIPANTS: Eight hundred and fifty-one consecutive patients admitted to the ICU over an 8-month period. Of these patients, 49 were readmissions, leaving 802 patients for further analysis.INTERVENTION: None.MEASUREMENTS AND RESULTS: APACHE II, SAPS II and SAPS 3 variables were prospectively collected; scores and their derived probability of death were calculated according to their original manuscript description. The discriminative power was assessed using the area under the ROC curve (AUROC) and calibration was verified with the Hosmer-Lemeshow goodness-of-fit test. The AUROC of the APACHE II model (AUROC = 0.823) was significantly lower than those of the SAPS II (AUROC = 0.850) and SAPS 3 models (AUROC = 0.854) (P = 0.038). The calibration of the APACHE II model (P = 0.037) and of the SAPS 3 global model (P = 0.035) appeared unsatisfactory. On the contrary, both SAPS II model and SAPS 3 model customised for Central and Western Europe had a good calibration. However, in our study case mix, SAPS II model tended to overestimate the probability of death.CONCLUSION: In this study, the SAPS 3 admission score and its prediction model customised for Central and Western Europe was more discriminative and better calibrated than APACHE II, but it was not significantly better than the SAPS II.Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009 Nov;35(11): Epub 2009 Aug 14.Servizio Anestesia e Rianimazione, Ospedale Civile San Martino, Belluno, Italy.OBJECTIVE: To evaluate the SAPS 3 score predictive ability of hospital mortality in a large external validation cohort.DESIGN: Prospective observational study.SETTING AND PATIENTS: A total of 28,357 patients from 147 Italian ICUs joining the Project Margherita national database of the Gruppo italiano per la Valutazione degli interventi in Terapia Intensiva (GiViTI).INTERVENTIONS: None.MEASUREMENT: Evaluation of discrimination through ROC analysis and of overall goodness-of-fit through the Cox calibration test.MAIN RESULTS: Although discrimination was good, calibration turned out to be poor. The general and the South-Europe Mediterranean countries equations overestimated hospital mortality overall (SMR values 0.73 with 95% CI for both equations) and homogeneously across risk classes. Overprediction was confirmed among important subgroups, with SMR values ranging between 0.47 and 0.82.CONCLUSIONS: The result strictly supported by our data is that the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings.
19 Sakr Y et al. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Br J Anaesth 2008; 101:single-centre (Germany), 1851 patients“… the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.”Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group. Austrian validation and customization of the SAPS 3 Admission Score. Intensive Care Med 2009; 35:22 ICUs in Austria, 2060 patients“The SAPS 3 … general equation can be seen as a framework … For benchmarking purposes, region-specific or country-specific equations seem to be necessary...”Br J Anaesth Dec;101(6): Epub 2008 Oct 9.Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit.Sakr Y, Krauss C, Amaral AC, Réa-Neto A, Specht M, Reinhart K, Marx G.Department of Anaesthesiology and Intensive Care, Friedrich-Schiller-University Hospital, Erlanger Allee 103, Jena, Germany.AbstractBACKGROUND: The Simplified Acute Physiology Score (SAPS) 3 has recently been developed, but not yet validated in surgical intensive care unit (ICU) patients. We compared the performance of SAPS 3 with SAPS II and the Acute Physiology and Chronic Health Evaluation (APACHE) II score in surgical ICU patients.METHODS: Prospectively collected data from all patients admitted to a German university hospital postoperative ICU between August 2004 and December 2005 were analysed. The probability of ICU mortality was calculated for SAPS II, APACHE II, adjusted APACHE II (adj-APACHE II), SAPS 3, and SAPS 3 customized for Europe [C-SAPS3 (Eu)] using standard formulas. To improve calibration of the prognostic models, a first-level customization was performed, using logistic regression on the original scores, and the corresponding probability of ICU death was calculated for the customized scores (C-SAPS II, C-SAPS 3, and C-APACHE II).RESULTS: The study included 1851 patients. Hospital mortality was 9%. Hosmer and Lemeshow statistics showed poor calibration for SAPS II, APACHE II, adj-APACHE II, SAPS 3, and C-SAPS 3 (Eu), but good calibration for C-SAPS II, C-APACHE II, and C-SAPS 3. Discrimination was generally good for all models [area under the receiver operating characteristic curve ranged from 0.78 (C-APACHE II) to 0.89 (C-SAPS 3)]. The C-SAPS 3 score appeared to have the best calibration curve on visual inspection.CONCLUSIONS: In this group of surgical ICU patients, the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.Intensive Care Med Apr;35(4): Epub 2008 Oct 10.Austrian validation and customization of the SAPS 3 Admission Score.Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group.Collaborators (22) Sagmüller G, Schwameis F, Pichler B, Ernst F, Bauer T, Sterrer F, Trimmel H, Klimscha W, Linemayr D, Schuh J, Sprinzl G, Dörre K, Trimmel H, Frank G, Malle H, Schindler I, Fitzal S, Schuster R, Locker G, Schneller H, Artmann H, Schuberth O.Department of Medical Statistics, Medical University of Vienna, Vienna, Austria.OBJECTIVE: To test the prognostic performance of the SAPS 3 Admission Score in a regional cohort and to empirically test the need and feasibility of regional customization.DESIGN: Prospective multicenter cohort study.PATIENTS AND SETTING: Data on a total of 2,060 patients consecutively admitted to 22 intensive care units in Austria from October 2, 2006 to February 28, 2007.MEASUREMENTS AND RESULTS: The database includes basic variables, SAPS 3, length-of-stay and outcome data. The original SAPS 3 Admission Score overestimated hospital mortality in Austrian intensive care patients through all strata of the severity-of-illness. This was true for both available equations, the General and the Central and Western Europe equation. For this reason a customized country-specific model was developed, using cross-validation techniques. This model showed excellent calibration and discrimination in the whole cohort (Hosmer-Lemeshow goodness-of-fit: H = 4.50, P = 0.922; C = 5.61, P = 0.847, aROC, 0.82) as well as in the various tested subgroups.CONCLUSIONS: The SAPS 3 Admission Score's general equation can be seen as a framework for addressing the problem of outcome prediction in the general population of adult ICU patients. For benchmarking purposes, region-specific or country-specific equations seem to be necessary in order to compare ICUs on a similar level.
20 2 ICUs in Norway, 1862 patients “The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II.”A comparison of SAPS II and SAPS 3 in a Norwegian intensive care unit population.Acta Anaesthesiol Scand May;53(5):Strand K, Søreide E, Aardal S, Flaatten H.Department of Anesthesia and Intensive Care, Stavanger University Hospital, Stavanger, Norway.Comment in:Acta Anaesthesiol Scand Oct;53(9):AbstractBACKGROUND: Simplified Acute Physiology Score (SAPS II) is the most widely used general severity scoring system in European intensive care medicine. Because its performance has been questioned in several external validation studies, SAPS 3 was recently released. To our knowledge, there are no published validation studies of SAPS II or SAPS 3 in the Scandinavian countries. We aimed to evaluate and compare the performance of SAPS II and SAPS 3 in a Norwegian intensive care unit (ICU) population.METHOD: Prospectively collected data from adult patients admitted to two general ICUs at two different hospitals in Norway were used. Probability of mortality was calculated using the SAPS 3 global equation (SAPS 3 G), the SAPS 3 Northern European equation (SAPS 3 NE), and the original SAPS II equation. Performance was assessed by the standardized mortality ratio (SMR), area under receiving operating characteristic, and the Hosmer and Lemeshow goodness-of-fit C test.RESULTS: One thousand eight hundred and sixty-two patients were included after excluding readmissions, and patients who were admitted after coronary surgery or burns. The SMRs were SAPS 3 G 0.71 (0.65, 0.78), SAPS 3 NE 0.74 (0.68, 0.81), and SAPS II 0.82 (0.75, 0.91). Discrimination was good in all systems. Only the SAPS 3 equations displayed satisfactory calibration, as measured by the Hosmer-Lemeshow test.CONCLUSION: The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II. Both systems considerably overestimated mortality and exhibited good discrimination, but only the SAPS 3 equations showed satisfactory calibration. Customization of these equations based on a larger cohort is recommended.
21 SAPS II showed better discrimination SAPS 3 equations showed better calibration“…in our experience the scoring process is more time-comsuming and complex than that for SAPS II.”
22 SAPS 3, CONCLUSION: Does it work? – Yes! However, prognostic performance is NOT better than that of SAPS IIthe scoring process is more time-comsuming and complex than that for SAPS II (experience from Norway)on the other hand: according to many studies, the calibration of SAPS II is poor and customisation is needed
23 QUESTION DISCUSSED IN FINLAND: Should we implement a new risk-adjustment model (SAPS 3) thatis not better than the old onesis more time-consumingwould require customisationOr should we go on with one of the old models (that also require customisation)?
24 FINNISH (at least temporary) SOLUTION: OWN CUSTOMISED PREDICTION MODEL One objective: no need to exclude patient groups for benchmarkingneuro- and cardiac surgical patients are not excludedWe did not want to increase the burden of data collection – no new parameters addedSAPS II –based data collection preservedpossible to compare the results with those of previous yearspossible to describe the population using a well-known scoring system
25 OWN CUSTOMISED MODEL Based on patients treated in 2007-2008 - M Reinikainen, P Mussalo, V Kiviniemi, V Pettilä, E RuokonenBased on patients treated inReadmissions excludedAge ≥ 18 yrsThose discharged to another ICU excludedn =
26 OWN CUSTOMISED MODELOutcome variable (to be predicted) ”DEATH IN HOSPITAL”Explaining covariates:Emergency admission or planned beforehandSurgical postoperative or medicalSAPS II score without admission type pointsln ((SAPS II score without admission type points) + 1)Diagnostic groups having an independent impact on the probability of deathFirst a binary variable (0,1) was made of every APACHE III –dg group; everyone of these was tested separately31 dg groups with an independent effect were included in the model
27 LOGISTIC REGRESSION ANALYSIS logit = β0 + β1X1 + β2X2 + … + βiXi- the regression analysis produces the constant β0 and the coefficients βithe logit can be calculated when the parameter values Xi are knownthe logit (log odds) can also be expressed asand thus
30 PATIENT EXAMPLE A patient example: HR 110/minSAPs 84 mmHgTc 38 ºCconsciousness, renal function, blood cell counts, electrolytes quite OKHCO3- 18 mmol/lage 65 yearsno difficult chronic diseasesa medical admissionrespiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa)SAPS II score 32 points → probability 0.128
31 PATIENT EXAMPLE SAPS II score 32 → probability 0.128 New customised model:If none of the diagnoses included in the model: probability 0.082dg bacterial pneumonia: probability 0.12dg drug intoxication: probability
32 AUROCAPACHE II: 0.84SAPS II: 0.84new customised model: 0.87H-L test for new model: p = 0.127
34 CONCLUSIONSSAPS 3 works, but its prognostic performance is not better than that of SAPS IIIf you want to use SAPS 3, you should probably customise itIf you want to use SAPS II, you should probably customise itIdea for future research: to create a Nordic risk adjustment model, predicting 6-month or 1-year mortality