Presentation on theme: "Www.pkssk.fi Pohjois-Karjalan sairaanhoito- ja sosiaalipalvelujen kuntayhtymä WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW 16.3.2011."— Presentation transcript:
Pohjois-Karjalan sairaanhoito- ja sosiaalipalvelujen kuntayhtymä WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW Matti Reinikainen North Karelia Central Hospital, Joensuu
THE FINNISH INTENSIVE CARE CONSORTIUM
So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS II –Based on “The Severity Study” – patients (720 from 7 Finnish hospitals) –Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270: APACHE II data is also collected
APACHE II vs. SAPS II same basic principle, values of physiologic parameters from the first 24 hrs in the ICU APACHE II (Acute Physiology And Chronic Health Evaluation II): the diagnostic category weight is added to the logit SAPS II (Simplified Acute Physiology Score II): the diagnosis is not needed; instead the type of admission (scheduled surgical, unscheduled surgical, medical) affects the score
APACHE II - from not always easy to choose the right diagnostic category SAPS II - from advantage: no diagnosis needed - disadvantage: does not take into account the diagnosis ARE THE OLD MODELS GOOD ENOUGH?
DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? A patient example: - age 65 years - no difficult chronic diseases - a medical admission - respiratory insufficiency, need for mechanical venti- lation, PaO 2 /FIO mmHg (33.3 kPa) - HR 110/min - SAPs 84 mmHg -Tc 38 ºC - consciousness, renal function, blood cell counts, electrolytes quite OK - HCO mmol/l PROBABILITY OF IN-HOSPITAL DEATH ?
DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? A patient example: - age 65 years - no difficult chronic diseases - a medical admission - respiratory insufficiency, need for mechanical venti- lation, PaO 2 /FIO mmHg (33,3 kPa) - HR 110/min - SAPs 84 mmHg -Tc 38 ºC - consciousness, renal function, blood cell counts, electrolytes quite OK - HCO mmol/l SAPS II score 32 points → probability 0.128
SAPS II –score 32 → probability the database of the Finnish Consortium, , readmissions excluded: 2319 patients, with a SAPS II score of 32 points - hospital mortality 8.4%
SAPS II –score 32 → probability 0,128 - the database of the Finnish Consortium, , readmissions excluded: 2319 patients, with a SAPS II score of 32 points - hospital mortality 8.4% - diabetic ketoacidosis (n = 26): mort 0% - drug intoxication (n = 108): mort 0.9% - congestive heart failure (n = 49): mort 22.4%
CAN SAPS II STILL BE USED? It overestimates the risk of death – leads to ”grade inflation” If most intensive care units are graduating with honors, is it genuine quality or grade inflation? Popovich MJ, Crit Care Med 2002 Recalibrations are needed
Tehohoidon laatupäivät Helsingissä SMR 1998 – 2007, FINNISH INTENSIVE CARE CONSORTIUM Päivitetty SMR based on original SAPS II model SMR based on new calibration
CAN SAPS II STILL BE USED? It can be used for monitoring changes in a unit’s own results Can be used for benchmarking purposes if the units to be compared have similar case-mix Should not be used to compare results of units with major differences in case-mix
SAPS 3 WAS CONSIDERED IN FINLAND TOO - IS IT A GOOD ALTERNATIVE? Values of physiological parameters ± 1 h of ICU admission Reason for ICU admission documented more precisely than in SAPS II Takes into account pre-ICU care Prognostic performance? Quality of data collected??
The SAPS 3 Study Metnitz et al ICM 2005: 31: (Part 1) Moreno et al. ICM 2005: 31: (Part 2) At first 22,791 admissions Exclusions: readmissions (1455), < 16 yrs (628), those without ICU admission or discharge data (1074) and those that lacked an entry in the field ”ICU outcome” (57) - SAPS 3 basic cohort: 19,577 patients
SAPS 3 basic cohort: 19,577 patients More exclusions: patients with a missing entry in the field of ”vital status at hospital discharge” (2540) and those still in hospital (253) –SAPS 3 Hospital outcome cohort: 16,784 patients Quality of data? – at first, 5.5% of patients excluded because of missing data; then 13% of the remaining population excluded because of missing data on vital status The SAPS 3 Study Metnitz et al ICM 2005: 31: (Part 1) Moreno et al. ICM 2005: 31: (Part 2)
How about data completeness? –”Data completeness was found to be satisfactory with 1 [0-3] SAPS II parameter missing per patient” How many SAPS 3 parameters were missing? –??? –Were the physiological values obtained within ± 1 h?
SAPS 3 – even if data quality in the study was less than perfect, does it work?
Ledoux D et al. SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 2008; 34: –single-centre (Belgium), 802 patients –“the SAPS 3 … model customised for Central and Western Europe … was not significantly better than the SAPS II.” Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009; 35: –“…the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings”
Sakr Y et al. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Br J Anaesth 2008; 101: –single-centre (Germany), 1851 patients –“… the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.” Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group. Austrian validation and customization of the SAPS 3 Admission Score. Intensive Care Med 2009; 35: –22 ICUs in Austria, 2060 patients –“The SAPS 3 … general equation can be seen as a framework … For benchmarking purposes, region-specific or country-specific equations seem to be necessary...”
2 ICUs in Norway, 1862 patients “The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II.”
SAPS II showed better discrimination SAPS 3 equations showed better calibration “…in our experience the scoring process is more time- comsuming and complex than that for SAPS II.”
SAPS 3, CONCLUSION: Does it work? – Yes! However, prognostic performance is NOT better than that of SAPS II the scoring process is more time-comsuming and complex than that for SAPS II (experience from Norway) on the other hand: according to many studies, the calibration of SAPS II is poor and customisation is needed
QUESTION DISCUSSED IN FINLAND: Should we implement a new risk-adjustment model (SAPS 3) that –is not better than the old ones –is more time-consuming –would require customisation Or should we go on with one of the old models (that also require customisation)?
FINNISH (at least temporary) SOLUTION: OWN CUSTOMISED PREDICTION MODEL One objective: no need to exclude patient groups for benchmarking –neuro- and cardiac surgical patients are not excluded We did not want to increase the burden of data collection – no new parameters added SAPS II –based data collection preserved –possible to compare the results with those of previous years –possible to describe the population using a well-known scoring system
Based on patients treated in Readmissions excluded Age ≥ 18 yrs Those discharged to another ICU excluded n = OWN CUSTOMISED MODEL - M Reinikainen, P Mussalo, V Kiviniemi, V Pettilä, E Ruokonen
OWN CUSTOMISED MODEL Outcome variable (to be predicted) ”DEATH IN HOSPITAL” Explaining covariates: –Emergency admission or planned beforehand –Surgical postoperative or medical –SAPS II score without admission type points –ln ((SAPS II score without admission type points) + 1) –Diagnostic groups having an independent impact on the probability of death First a binary variable (0,1) was made of every APACHE III –dg group; everyone of these was tested separately 31 dg groups with an independent effect were included in the model
LOGISTIC REGRESSION ANALYSIS logit = β 0 + β 1 X 1 + β 2 X 2 + … + β i X i - the regression analysis produces the constant β 0 and the coefficients β i -the logit can be calculated when the parameter values X i are known - the logit (log odds) can also be expressed as and thus
PATIENT EXAMPLE A patient example: - age 65 years - no difficult chronic diseases - a medical admission - respiratory insufficiency, need for mechanical venti- lation, PaO 2 /FIO mmHg (33.3 kPa) - HR 110/min - SAPs 84 mmHg -Tc 38 ºC - consciousness, renal function, blood cell counts, electrolytes quite OK - HCO mmol/l SAPS II score 32 points → probability 0.128
SAPS II score 32 → probability New customised model: –If none of the diagnoses included in the model: probability –dg bacterial pneumonia: probability 0.12 –dg drug intoxication: probability PATIENT EXAMPLE
AUROC - APACHE II: SAPS II: new customised model: H-L test for new model: p = 0.127
CONCLUSIONS SAPS 3 works, but its prognostic performance is not better than that of SAPS II If you want to use SAPS 3, you should probably customise it If you want to use SAPS II, you should probably customise it Idea for future research: to create a Nordic risk adjustment model, predicting 6-month or 1-year mortality