1. WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW
Matti Reinikainen
North Karelia Central Hospital, Joensuu

2. THE FINNISH INTENSIVE CARE CONSORTIUM
A co-operation body coordinating a quality assurance project
Started in 1994
Has strongly expanded since 1998
All university hospitals (5) have been involved since 2002
Now the Consortium comprises all central hospital districts (20) on the mainland
1994
2007

3. APACHE II data is also collected
So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS II
Based on “The Severity Study”
patients (720 from 7 Finnish hospitals)
Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270:
APACHE II data is also collected

4. APACHE II vs. SAPS II
same basic principle, values of physiologic parameters from the first 24 hrs in the ICU
APACHE II (Acute Physiology And Chronic Health Evaluation II): the diagnostic category weight is added to the logit
SAPS II (Simplified Acute Physiology Score II): the diagnosis is not needed; instead the type of admission (scheduled surgical, unscheduled surgical, medical) affects the score

5. ARE THE OLD MODELS GOOD ENOUGH?
APACHE II
- from 1985
- not always easy to choose the right diagnostic category
SAPS II
- from 1993
- advantage: no diagnosis needed
- disadvantage: does not take into account the diagnosis

6. DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY?
A patient example:
HR 110/min
SAPs 84 mmHg
Tc 38 ºC
consciousness, renal function, blood cell counts, electrolytes quite OK
HCO3- 18 mmol/l
age 65 years
no difficult chronic diseases
a medical admission
respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa)
PROBABILITY OF IN-HOSPITAL DEATH ?

7. DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY?
A patient example:
HR 110/min
SAPs 84 mmHg
Tc 38 ºC
consciousness, renal function, blood cell counts, electrolytes quite OK
HCO3- 18 mmol/l
age 65 years
no difficult chronic diseases
a medical admission
respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33,3 kPa)
SAPS II score 32 points → probability 0.128

8. SAPS II –score 32 → probability 0.128
the database of the Finnish Consortium, , readmissions excluded: 2319 patients, with a SAPS II score of 32 points
hospital mortality 8.4%
84/128 = 0,66

9. SAPS II –score 32 → probability 0,128
the database of the Finnish Consortium, , readmissions excluded: 2319 patients, with a SAPS II score of 32 points
hospital mortality 8.4%
diabetic ketoacidosis (n = 26): mort 0%
drug intoxication (n = 108): mort 0.9%
congestive heart failure (n = 49): mort 22.4%
84/128 = 0,66
11/49 = 0,224

10. CAN SAPS II STILL BE USED?
It overestimates the risk of death – leads to ”grade inflation”
If most intensive care units are graduating with honors, is it genuine quality or grade inflation? Popovich MJ, Crit Care Med 2002
Recalibrations are needed

11. SMR 1998 – 2007, FINNISH INTENSIVE CARE CONSORTIUM
SMR based on new calibration
SMR based on original SAPS II model
Päivitetty 11

12. CAN SAPS II STILL BE USED?
It can be used for monitoring changes in a unit’s own results
Can be used for benchmarking purposes if the units to be compared have similar case-mix
Should not be used to compare results of units with major differences in case-mix

13. SAPS 3 WAS CONSIDERED IN FINLAND TOO - IS IT A GOOD ALTERNATIVE?
Values of physiological parameters ± 1 h of ICU admission
Reason for ICU admission documented more precisely than in SAPS II
Takes into account pre-ICU care
Prognostic performance?
Quality of data collected??

14. The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344
The SAPS 3 Study Metnitz et al ICM 2005: 31: (Part 1) Moreno et al. ICM 2005: 31: (Part 2)
At first 22,791 admissions
Exclusions: readmissions (1455), < 16 yrs (628), those without ICU admission or discharge data (1074) and those that lacked an entry in the field ”ICU outcome” (57)
- SAPS 3 basic cohort: 19,577 patients

15. The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344
The SAPS 3 Study Metnitz et al ICM 2005: 31: (Part 1) Moreno et al. ICM 2005: 31: (Part 2)
SAPS 3 basic cohort: 19,577 patients
More exclusions: patients with a missing entry in the field of ”vital status at hospital discharge” (2540) and those still in hospital (253)
SAPS 3 Hospital outcome cohort: 16,784 patients
Quality of data? – at first, 5.5% of patients excluded because of missing data; then 13% of the remaining population excluded because of missing data on vital status

16. The SAPS 3 Study Metnitz et al ICM 2005: 31:1336-1344
The SAPS 3 Study Metnitz et al ICM 2005: 31: (Part 1) Moreno et al. ICM 2005: 31: (Part 2)
How about data completeness?
”Data completeness was found to be satisfactory with 1 [0-3] SAPS II parameter missing per patient”
How many SAPS 3 parameters were missing?
???
Were the physiological values obtained within ± 1 h?

17. SAPS 3 – even if data quality in the study was less than perfect, does it work?

18. Ledoux D et al. SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 2008; 34:
single-centre (Belgium), 802 patients
“the SAPS 3 … model customised for Central and Western Europe … was not significantly better than the SAPS II.”
Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009; 35:
“…the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings”
Intensive Care Med Oct;34(10): Epub 2008 Jul 1.
SAPS 3 admission score: an external validation in a general intensive care population.
Ledoux D, Canivet JL, Preiser JC, Lefrancq J, Damas P.
Soins Intensifs Généraux, Centre Hospitalier Universitaire de Liège, Domaine Universitaire de Sart Tilman Bat B35, 4000 Liège, Belgium.
Abstract
OBJECTIVES: To validate the SAPS 3 admission score in an independent general intensive care case mix and to compare its performances with the APACHE II and the SAPS II scores.
DESIGN: Cohort observational study.
SETTING: A 26-bed general ICU from a Tertiary University Hospital.
PATIENTS AND PARTICIPANTS: Eight hundred and fifty-one consecutive patients admitted to the ICU over an 8-month period. Of these patients, 49 were readmissions, leaving 802 patients for further analysis.
INTERVENTION: None.
MEASUREMENTS AND RESULTS: APACHE II, SAPS II and SAPS 3 variables were prospectively collected; scores and their derived probability of death were calculated according to their original manuscript description. The discriminative power was assessed using the area under the ROC curve (AUROC) and calibration was verified with the Hosmer-Lemeshow goodness-of-fit test. The AUROC of the APACHE II model (AUROC = 0.823) was significantly lower than those of the SAPS II (AUROC = 0.850) and SAPS 3 models (AUROC = 0.854) (P = 0.038). The calibration of the APACHE II model (P = 0.037) and of the SAPS 3 global model (P = 0.035) appeared unsatisfactory. On the contrary, both SAPS II model and SAPS 3 model customised for Central and Western Europe had a good calibration. However, in our study case mix, SAPS II model tended to overestimate the probability of death.
CONCLUSION: In this study, the SAPS 3 admission score and its prediction model customised for Central and Western Europe was more discriminative and better calibrated than APACHE II, but it was not significantly better than the SAPS II.
Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009 Nov;35(11): Epub 2009 Aug 14.
Servizio Anestesia e Rianimazione, Ospedale Civile San Martino, Belluno, Italy.
OBJECTIVE: To evaluate the SAPS 3 score predictive ability of hospital mortality in a large external validation cohort.
DESIGN: Prospective observational study.
SETTING AND PATIENTS: A total of 28,357 patients from 147 Italian ICUs joining the Project Margherita national database of the Gruppo italiano per la Valutazione degli interventi in Terapia Intensiva (GiViTI).
INTERVENTIONS: None.
MEASUREMENT: Evaluation of discrimination through ROC analysis and of overall goodness-of-fit through the Cox calibration test.
MAIN RESULTS: Although discrimination was good, calibration turned out to be poor. The general and the South-Europe Mediterranean countries equations overestimated hospital mortality overall (SMR values 0.73 with 95% CI for both equations) and homogeneously across risk classes. Overprediction was confirmed among important subgroups, with SMR values ranging between 0.47 and 0.82.
CONCLUSIONS: The result strictly supported by our data is that the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings.

19. Sakr Y et al. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Br J Anaesth 2008; 101:
single-centre (Germany), 1851 patients
“… the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.”
Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group. Austrian validation and customization of the SAPS 3 Admission Score. Intensive Care Med 2009; 35:
22 ICUs in Austria, 2060 patients
“The SAPS 3 … general equation can be seen as a framework … For benchmarking purposes, region-specific or country-specific equations seem to be necessary...”
Br J Anaesth Dec;101(6): Epub 2008 Oct 9.
Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit.
Sakr Y, Krauss C, Amaral AC, Réa-Neto A, Specht M, Reinhart K, Marx G.
Department of Anaesthesiology and Intensive Care, Friedrich-Schiller-University Hospital, Erlanger Allee 103, Jena, Germany.
Abstract
BACKGROUND: The Simplified Acute Physiology Score (SAPS) 3 has recently been developed, but not yet validated in surgical intensive care unit (ICU) patients. We compared the performance of SAPS 3 with SAPS II and the Acute Physiology and Chronic Health Evaluation (APACHE) II score in surgical ICU patients.
METHODS: Prospectively collected data from all patients admitted to a German university hospital postoperative ICU between August 2004 and December 2005 were analysed. The probability of ICU mortality was calculated for SAPS II, APACHE II, adjusted APACHE II (adj-APACHE II), SAPS 3, and SAPS 3 customized for Europe [C-SAPS3 (Eu)] using standard formulas. To improve calibration of the prognostic models, a first-level customization was performed, using logistic regression on the original scores, and the corresponding probability of ICU death was calculated for the customized scores (C-SAPS II, C-SAPS 3, and C-APACHE II).
RESULTS: The study included 1851 patients. Hospital mortality was 9%. Hosmer and Lemeshow statistics showed poor calibration for SAPS II, APACHE II, adj-APACHE II, SAPS 3, and C-SAPS 3 (Eu), but good calibration for C-SAPS II, C-APACHE II, and C-SAPS 3. Discrimination was generally good for all models [area under the receiver operating characteristic curve ranged from 0.78 (C-APACHE II) to 0.89 (C-SAPS 3)]. The C-SAPS 3 score appeared to have the best calibration curve on visual inspection.
CONCLUSIONS: In this group of surgical ICU patients, the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.
Intensive Care Med Apr;35(4): Epub 2008 Oct 10.
Austrian validation and customization of the SAPS 3 Admission Score.
Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group.
Collaborators (22) Sagmüller G, Schwameis F, Pichler B, Ernst F, Bauer T, Sterrer F, Trimmel H, Klimscha W, Linemayr D, Schuh J, Sprinzl G, Dörre K, Trimmel H, Frank G, Malle H, Schindler I, Fitzal S, Schuster R, Locker G, Schneller H, Artmann H, Schuberth O.
Department of Medical Statistics, Medical University of Vienna, Vienna, Austria.
OBJECTIVE: To test the prognostic performance of the SAPS 3 Admission Score in a regional cohort and to empirically test the need and feasibility of regional customization.
DESIGN: Prospective multicenter cohort study.
PATIENTS AND SETTING: Data on a total of 2,060 patients consecutively admitted to 22 intensive care units in Austria from October 2, 2006 to February 28, 2007.
MEASUREMENTS AND RESULTS: The database includes basic variables, SAPS 3, length-of-stay and outcome data. The original SAPS 3 Admission Score overestimated hospital mortality in Austrian intensive care patients through all strata of the severity-of-illness. This was true for both available equations, the General and the Central and Western Europe equation. For this reason a customized country-specific model was developed, using cross-validation techniques. This model showed excellent calibration and discrimination in the whole cohort (Hosmer-Lemeshow goodness-of-fit: H = 4.50, P = 0.922; C = 5.61, P = 0.847, aROC, 0.82) as well as in the various tested subgroups.
CONCLUSIONS: The SAPS 3 Admission Score's general equation can be seen as a framework for addressing the problem of outcome prediction in the general population of adult ICU patients. For benchmarking purposes, region-specific or country-specific equations seem to be necessary in order to compare ICUs on a similar level.

20. 2 ICUs in Norway, 1862 patients
“The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II.”
A comparison of SAPS II and SAPS 3 in a Norwegian intensive care unit population.
Acta Anaesthesiol Scand May;53(5):
Strand K, Søreide E, Aardal S, Flaatten H.
Department of Anesthesia and Intensive Care, Stavanger University Hospital, Stavanger, Norway.
Comment in:
Acta Anaesthesiol Scand Oct;53(9):
Abstract
BACKGROUND: Simplified Acute Physiology Score (SAPS II) is the most widely used general severity scoring system in European intensive care medicine. Because its performance has been questioned in several external validation studies, SAPS 3 was recently released. To our knowledge, there are no published validation studies of SAPS II or SAPS 3 in the Scandinavian countries. We aimed to evaluate and compare the performance of SAPS II and SAPS 3 in a Norwegian intensive care unit (ICU) population.
METHOD: Prospectively collected data from adult patients admitted to two general ICUs at two different hospitals in Norway were used. Probability of mortality was calculated using the SAPS 3 global equation (SAPS 3 G), the SAPS 3 Northern European equation (SAPS 3 NE), and the original SAPS II equation. Performance was assessed by the standardized mortality ratio (SMR), area under receiving operating characteristic, and the Hosmer and Lemeshow goodness-of-fit C test.
RESULTS: One thousand eight hundred and sixty-two patients were included after excluding readmissions, and patients who were admitted after coronary surgery or burns. The SMRs were SAPS 3 G 0.71 (0.65, 0.78), SAPS 3 NE 0.74 (0.68, 0.81), and SAPS II 0.82 (0.75, 0.91). Discrimination was good in all systems. Only the SAPS 3 equations displayed satisfactory calibration, as measured by the Hosmer-Lemeshow test.
CONCLUSION: The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II. Both systems considerably overestimated mortality and exhibited good discrimination, but only the SAPS 3 equations showed satisfactory calibration. Customization of these equations based on a larger cohort is recommended.

21. SAPS II showed better discrimination
SAPS 3 equations showed better calibration
“…in our experience the scoring process is more time-comsuming and complex than that for SAPS II.”

22. SAPS 3, CONCLUSION: Does it work? – Yes!
However, prognostic performance is NOT better than that of SAPS II
the scoring process is more time-comsuming and complex than that for SAPS II (experience from Norway)
on the other hand: according to many studies, the calibration of SAPS II is poor and customisation is needed

23. QUESTION DISCUSSED IN FINLAND:
Should we implement a new risk-adjustment model (SAPS 3) that
is not better than the old ones
is more time-consuming
would require customisation
Or should we go on with one of the old models (that also require customisation)?

24. FINNISH (at least temporary) SOLUTION: OWN CUSTOMISED PREDICTION MODEL
One objective: no need to exclude patient groups for benchmarking
neuro- and cardiac surgical patients are not excluded
We did not want to increase the burden of data collection – no new parameters added
SAPS II –based data collection preserved
possible to compare the results with those of previous years
possible to describe the population using a well-known scoring system

25. OWN CUSTOMISED MODEL Based on patients treated in 2007-2008
- M Reinikainen, P Mussalo, V Kiviniemi, V Pettilä, E Ruokonen
Based on patients treated in
Readmissions excluded
Age ≥ 18 yrs
Those discharged to another ICU excluded
n =

26. OWN CUSTOMISED MODEL
Outcome variable (to be predicted) ”DEATH IN HOSPITAL”
Explaining covariates:
Emergency admission or planned beforehand
Surgical postoperative or medical
SAPS II score without admission type points
ln ((SAPS II score without admission type points) + 1)
Diagnostic groups having an independent impact on the probability of death
First a binary variable (0,1) was made of every APACHE III –dg group; everyone of these was tested separately
31 dg groups with an independent effect were included in the model

27. LOGISTIC REGRESSION ANALYSIS
logit = β0 + β1X1 + β2X2 + … + βiXi
- the regression analysis produces the constant β0 and the coefficients βi
the logit can be calculated when the parameter values Xi are known
the logit (log odds) can also be expressed as
and thus

28. CALCULATING THE RISK R

29. PROB = EXP(LOGIT) / (1 + EXP(LOGIT))
LOGIT = -7, ,049 x (SCORE_SAPS_WITHOUT_ADM_TYPE_POINTS)
+ 1,013 x (ln(SAPS_WITHOUT_ADM_TYPE_POINTS + 1))
+ 0,767 (if emergency admission) - 0,219 (if post-operative admission)
+ 1,229 (if DG_NONOP_CARDIOGENIC_SHOCK) + 0,364 (if DG_NONOP_CARDIAC_ARREST)
– 0,796 (if DG_NONOP_RHYTHM_DISTURBANCE) + 0,348 (if DG_NONOP_ACUTE_MYOCARDIAL INFARCTION) + 0,422 (if DG_NONOP_BACTERIAL_OR_VIRAL_PNEUMONIA)
– 1,619 (if DG_NONOP_MECHANICAL_AIRWAY_OBSTRUCTION)
+ 0,306 (if DG_NONOP_OTHER_RESP_DISEASES) + 0,795 (if DG_NONOP_HEPATIC_FAILURE)
+ 0,703 (if DG_NONOP_GI_PERFORATION_OR_OBSTRUCTION) + 0,643 (if DG_NONOP_GI BLEEDING_DUE_TO_VARICES) + 0,431 (if DG_NONOP_OTHER_GI_DISEASES)
+ 0,790 (if DG_NONOP_INTRACEREBRAL_HAEMORRHAGE)
+ 0,654 (if DG_NONOP_SUBARACHNOID_HAEMORRHAGE) + 0,400 (if DG_NONOP_STROKE)
– 1,427 (if DG_NONOP_NEUROLOGIC_INFECTION) - 1,266 (if DG_NONOP_SEIZURE)
– 0,486 (if DG_NONOP_OTHER_NEUROLOGIC_DISEASES) - 0,679 (if DG_NONOP_MULTIPLE TRAUMA_WITHOUT_HEAD_TRAUMA) – 0,658 (if DG_NONOP_METABOLIC_COMA)
– 2,126 (IF DG_NONOP_DIABETIC_KETOACIDOSIS) – 2,245 (if DG_NONOP_DRUG_OVERDOSE)
– 1,150 (if DG_NONOP_OTHER_METABOLIC_DISEASES) – 0,752 (if DG_NONOP_OTHER MEDICAL_DISEASES) + 0,340 (if DG_POSTOP_DISSECTING_OR_RUPTURED_AORTA)
– 0,701 (if DG_POSTOP_CABG) + 0,701 (if DG_POSTOP_PERIPH_ARTERY_BYPASS_GRAFT)
+ 0,470 (if DG_POSTOP_GI_PERFORATION_OR_RUPTURE) + 0,411 (if DG_POSTOP GI_OBSTRUCTION) – 0,522 (if DG_POSTOP_SUBDURAL_OR_EPIDURAL_HAEMATOMA)
– 0,885 (if DG_POSTOP_CRANIOTOMY_FOR_NEOPLASM)
– 1,620 (if DG_POSTOP_OTHER_RENAL_DISEASES)
PROB = EXP(LOGIT) / (1 + EXP(LOGIT))

30. PATIENT EXAMPLE A patient example:
HR 110/min
SAPs 84 mmHg
Tc 38 ºC
consciousness, renal function, blood cell counts, electrolytes quite OK
HCO3- 18 mmol/l
age 65 years
no difficult chronic diseases
a medical admission
respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa)
SAPS II score 32 points → probability 0.128

31. PATIENT EXAMPLE SAPS II score 32 → probability 0.128
New customised model:
If none of the diagnoses included in the model: probability 0.082
dg bacterial pneumonia: probability 0.12
dg drug intoxication: probability

32. AUROC
APACHE II: 0.84
SAPS II: 0.84
new customised model: 0.87
H-L test for new model: p = 0.127

34. CONCLUSIONS
SAPS 3 works, but its prognostic performance is not better than that of SAPS II
If you want to use SAPS 3, you should probably customise it
If you want to use SAPS II, you should probably customise it
Idea for future research: to create a Nordic risk adjustment model, predicting 6-month or 1-year mortality