1. 1 Dichotomization of ICU length of stay based on model calibration Marion Verduijn, Niels Peek, Frans Voorbraak, Evert de Jonge, Bas de Mol

2. 2 ICU length of stay (LOS) Important outcome after cardiac surgery Predictive models for identification of high risk patients case load planning and resource allocation

3. 3 Main objective Development of a predictive model to estimate the risk of long ICU LOS using the method of class probability trees

4. 4 Data 2063 patients (Academic Medical Center, Amsterdam, 1997-2001) - preoperative (e.g., age, gender) - operative (e.g., surgery type, duration) - first 24h ICU stay (e.g., blood pressure, temperature) (122 patients died: 5.2%)

5. 5 Problem of outcome definition How should we define the outcome ‘long ICU LOS’? Literature: outcome dichotomized based on threshold values of 2-10 days without motivation or based on simple statistics

6. 6 Objective of this study Selection of the threshold value to dichotomize ICU LOS in a structured fashion based on data analysis

7. 7 Approach 1.Development of tree models for outcomes defined with different threshold values 2.Calculation of the model performance 3.Selection of the best model

8. 8 First results thresholdproportion events† Brier score‡ class probability tree 2 days0.5480.475 3 days0.3900.418 4 days0.2920.348 5 days0.2450.293 6 days0.2070.280 7 days0.1820.274 8 days0.1670.242 9 days0.1560.230 10 days0.1410.232 12 days0.1290.216 † patients with ICU LOS higher than the threshold value of death ‡ determined using 10-fold cross-validation

9. 9 Distances between probabilities I PxPx MxMx d RelEr d AbsEr d SqEr d KL 0.10.150.6660.050.00250.0122 0.010.0150.6660.0050.0000250.00109 0.0010.00150.6660.00050.000000250.000108

10. 10 Distances between probabilities I PxPx MxMx d RelEr d AbsEr d SqEr d KL 0.10.150.6660.050.00250.0122 0.010.0150.6660.0050.0000250.00109 0.0010.00150.6660.00050.000000250.000108

11. 11 Distances between probabilities I PxPx MxMx d RelEr d AbsEr d SqEr d KL 0.10.150.6660.050.00250.0122 0.010.0150.6660.0050.0000250.00109 0.0010.00150.6660.00050.000000250.000108

12. 12 ALOR distance Distance between two probabilities for a given x Absolute Log-Odds Ratio

13. 13 Property of ALOR: approximate proportional equivalence PxPx MxMx d RelEr d AbsEr d SqEr d KL d ALOR 0.10.150.6660.050.00250.01220.4626 0.010.0150.6660.0050.0000250.001090.4105 0.0010.00150.6660.00050.000000250.0001080.4060 Distances between probabilities II

14. 14 MALOR statistic Distance measure for all elements in F Mean value of the Absolute Log-Odds Ratio (MALOR)  quantifies model calibration

15. 15 Procedure of threshold selection 1) define a set of possible threshold values T 2) for all threshold values t in T do a) define the dichotomized outcome Y t using threshold t b) build a predictive model M t for outcome Y t c) compute D MALOR (M t, P t ) 3) select threshold value with minimal MALOR statistic

16. 16 Additional results thresholdproportion events† Brier score‡MALOR class probability tree tree ensemble 2 days0.5480.4750.4150.492 3 days0.3900.4180.3660.490 4 days0.2920.3480.3120.516 5 days0.2450.2930.2800.468 6 days0.2070.2800.2510.534 7 days0.1820.2740.2320.575 8 days0.1670.2420.2120.618 9 days0.1560.2300.2060.616 10 days0.1410.2320.1930.765 12 days0.1290.2160.1810.709 † patients with ICU LOS higher than the threshold value of death ‡ determined using 10-fold cross-validation

17. 17 Tree model for ‘ICU LOS>5 days or death’

18. 18 Discussion and conclusions Class probability trees to identify high risk groups Performance measure should be insensitive to class unbalance when comparing models for different prediction problems

19. 19 Marion Verduijn m.verduijn@amc.uva.nl