1. Division of Population Health Sciences Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn New Method for Pooling Validation Studies of Clinical Prediction Rules in Systematic Reviews Borislav D Dimitrov, Nicola Motterlini, Tom Fahey 1

2. Division of Population Health Sciences Contents 1.Introduction - predicted/observed values in validation studies 2.Example of systematic review of validation studies for CBR- 65 ( predicted values by a new method) 3.The new method and related statistical issues 4.Comparison of predicted values (Framingham) 5.Conclusions (new method) 2

3. Division of Population Health Sciences 1.Introduction - predicted/observed values in validation studies 2.Example of systematic review of validation studies for CBR- 65 ( predicted values by a new method) 3.The new method and related statistical issues 4.Comparison of predicted values (Framingham) 5.Conclusions (new method) 3

4. Division of Population Health Sciences 4

5. 5

6. 6

7. 1.Introduction - predicted/observed values in validation studies 2.Example of systematic review of validation studies for CBR- 65 ( predicted values by a new method) 3.The new method and related statistical issues 4.Comparison of predicted values (Framingham) 5. Conclusions (new method) 7

8. Division of Population Health Sciences 8

9. Confusion Respiratory rate ≥ 30/min Blood pressure (SBP≤ 90 or DBP≤60) Age ≥ 65 0 1 or 2 3 or 4 Low Risk mortality 1.2% Intermediate Risk mortality 8.13% High Risk mortality 31% CRB-65: a clinical prediction rule Likely suitable for home treatment Consider hospital referral Urgent hospital admission 9

10. Division of Population Health Sciences Studies included in meta-analysis (n=14) studysettingparticipants Barlow et al 2007inpatients419 Bauer et al 2006outpatients + inpatients1959 Bont et al 2008outpatients314 Buising et al 2007emergency department740 Capelastegui et al 2006outpatients + inpatients1776 Chalmers et al 2008inpatients1007 Ewig et al 2009inpatients388406 Kruger et al 2008outpatients + inpatients1404 Man et al 2007inpatients1016 Menendez et al 2009inpatients447 Myint et al 2006inpatients192 Schaaf et al 2007inpatients105 Schuetz et al 2008emergency department373 Zuberi et al 2008inpatients137 TOTAL398295 10

11. Division of Population Health Sciences Statistical Methods Derivation study used as predictive model! Results presented as ratio measurement: predicted deaths by CRB-65 rule observed deaths in validation study 11

12. Division of Population Health Sciences Confusion Respiratory rate ≥ 30/min Blood pressure (SBP≤ 90 or DBP≤60) Age ≥ 65 01 or 2 3 or 4 Low Risk mortality 1.2% Intermediate Risk mortality 8.13% High Risk mortality 31% CRB-65 12

13. Division of Population Health Sciences 13

14. Division of Population Health Sciences 1.Introduction - predicted/observed values in validation studies 2.Example of systematic review of validation studies for CBR- 65 ( predicted values by a new method) 3.The new method and related statistical issues 4.Comparison of predicted values (Framingham) 5.Conclusions (new method) 14

15. Division of Population Health Sciences Calculation predicted values NEW METHOD 15 SCOREDERIVATIONVALIDATION STUDY PatientsDeathRate (R) Patients (N) Observed deaths (O) Predicted deaths (P) 016721.2%1283 R*NR*N 1.54 1-2455378.13%7835863.66 3-4963031.3%1052632.87 Total8798.07 Risk ratio (RR)

16. Division of Population Health Sciences SCOREDERIVATIONVALIDATION STUDY PatientsDeathRate (R) Patients (N) Observed deaths (O) Expected deaths (E) 016721.2%1283 R*NR*N 1.54 1-2455378.13%7835863.66 3-4963031.3%1052632.87 Total8798.07 Standardized mortality ratio (SMR) 16 Ex.1: Indirect standardization (1)

17. Division of Population Health Sciences Ex.1: Indirect standardization (2) (confidence intervals) SMR 17 Observed number of deaths in the validation study Expected number of deaths if the score group specific rates were the same as those of the derivation study = = 95 % CI = SMR / Error factor TO SMR x Error factor Error Factor = Essential medical statistics (2 nd edition; 2003). Betty R. Kirkwood, Jonathan Sterne.

18. Division of Population Health Sciences 18 | SMR [95% Conf. Interval] % Weight +--------------------------------------------- | 1.770 1.420 2.206 8.31 | 0.722 0.557 0.937 7.96 | 0.712 0.565 0.897 8.22 | 1.255 1.049 1.502 8.64 | 0.932 0.764 1.137 8.49 | 1.448 1.437 1.459 9.37 | 0.751 0.584 0.966 8.04 | 0.888 0.720 1.095 8.40 | 0.853 0.616 1.182 7.33 | 0.960 0.659 1.399 6.83 | 1.124 0.622 2.030 4.89 | 1.713 1.261 2.326 7.52 | 1.657 1.044 2.631 6.01 +--------------------------------------------- | 1.074 0.889 1.297 100.00 +--------------------------------------------- Heterogeneity chi-squared = 151.18 (d.f. = 12) p < 0.001 I-squared = 92.1% Estimate of between-study variance Tau-squared = 0.0994 Test of SMR=1 : z = 0.74 p = 0.461 For software reasons analyses were performed in log scale, but results were back-transformed and presented in original scale Study | RR [95% Conf. Interval] % Weight ----------------+--------------------------------------------- Barlow | 1.756 1.249 2.467 8.16 Bauer | 0.722 0.519 1.004 8.27 Buising | 0.713 0.536 0.948 8.75 Capelastegui | 1.255 0.970 1.624 9.03 Chalmers | 0.933 0.717 1.213 8.98 Ewig | 1.448 1.430 1.466 10.54 Kruger | 0.753 0.546 1.038 8.37 Man | 0.888 0.674 1.169 8.85 Menendez | 0.857 0.560 1.312 7.24 Myint | 0.964 0.591 1.573 6.57 Schaaf | 1.100 0.488 2.479 3.96 Schuetz | 1.708 1.054 2.769 6.64 Zuberi | 1.636 0.803 3.334 4.63 ----------------+--------------------------------------------- Pooled | 1.059 0.863 1.299 100.00 ----------------+--------------------------------------------- Heterogeneity chi-squared = 90.77 (d.f. = 12) p < 0.001 I-squared = 86.8% Estimate of between-study variance Tau-squared = 0.1033 Test of RR=1 : z = 0.55 p = 0.586 Meta-analysis: RR vs. SMR (1)

19. Division of Population Health Sciences Meta-analysis: RR vs. SMR (2) Overall (I-squared = 86.8%, p <0.001) Zuberi 2008 Schuetz 2008 Myint 2006 Menendez 2009 Kruger 2008 Barlow 2007 Ewig 2009 Schaaf 2007 Man 2007 Chalmers 2007 Capelastegui 2006 Bauer 2006 Study Buising 2007 1.06 (0.86, 1.30) 1.64 (0.80, 3.33) 1.71 (1.05, 2.77) 0.96 (0.59, 1.57) 0.86 (0.56, 1.31) 0.75 (0.55, 1.04) 1.76 (1.25, 2.47) 1.45 (1.43, 1.47) 1.10 (0.49, 2.48) 0.89 (0.67, 1.17) 0.93 (0.72, 1.21) 1.26 (0.97, 1.62) 0.72 (0.52, 1.00) 0.71 (0.54, 0.95) 1.06 (0.86, 1.30) 1.64 (0.80, 3.33) 1.71 (1.05, 2.77) 0.96 (0.59, 1.57) 0.86 (0.56, 1.31) 0.75 (0.55, 1.04) 1.76 (1.25, 2.47) 1.45 (1.43, 1.47) 1.10 (0.49, 2.48) 0.89 (0.67, 1.17) 0.93 (0.72, 1.21) 1.26 (0.97, 1.62) 0.72 (0.52, 1.00) RR (95% CI) 0.71 (0.54, 0.95) 1.313.33 Overall (I-squared = 92.1%, p < 0.001) Zuberi 2008 Schuetz 2008 Myint 2006 Menendez 2009 Kruger 2008 Barlow 2007 Ewig 2009 Schaaf 2007 Man 2007 Chalmers 2007 Capelastegui 2006 Bauer 2006 Study Buising 2007 1.07 (0.89, 1.30) 1.66 (1.04, 2.63) 1.71 (1.26, 2.33) 0.96 (0.66, 1.40) 0.85 (0.62, 1.18) 0.75 (0.58, 0.97) 1.77 (1.42, 2.21) 1.45 (1.44, 1.46) 1.12 (0.62, 2.03) 0.89 (0.72, 1.10) 0.93 (0.76, 1.14) 1.25 (1.05, 1.50) 0.72 (0.56, 0.94) SMR (95% CI) 0.71 (0.57, 0.90) 1.07 (0.89, 1.30) 1.66 (1.04, 2.63) 1.71 (1.26, 2.33) 0.96 (0.66, 1.40) 0.85 (0.62, 1.18) 0.75 (0.58, 0.97) 1.77 (1.42, 2.21) 1.45 (1.44, 1.46) 1.12 (0.62, 2.03) 0.89 (0.72, 1.10) 0.93 (0.76, 1.14) 1.25 (1.05, 1.50) 0.72 (0.56, 0.94) 0.71 (0.57, 0.90) 1.3812.63 Overprediction Underprediction

20. Division of Population Health Sciences 20 Updating method with no adjustment Ex.2: Updating method with no adjustment (1)

21. Division of Population Health Sciences 21 Logistic regression model Where is the intercept and the regression coefficients from the derivation study and the predictor values in the validation study In our case, the score is the only predictor Ex.2: Updating method with no adjustment (2)

22. Division of Population Health Sciences 1.Introduction - predicted/observed values in validation studies 2.Example of systematic review of validation studies for CBR- 65 ( predicted values by a new method) 3.The new method and related statistical issues 4.Comparison of predicted values (Framingham) 5.Conclusions (new method) 22

23. Division of Population Health Sciences Framingham Risk Score (1) (derivation studies) 23

24. Division of Population Health Sciences All studies (2) 24 Paired t-test -> p=0.063 Wilcoxon signed-rank test -> p=0.032

25. Division of Population Health Sciences Studies with men only (3) (followed for 10 years) 25 Paired t-test -> p=0.986 Wilcoxon signed-rank test -> p=0.726

26. Division of Population Health Sciences 26 Overall (I-squared = 97.9%, p < 0.001) Germany angiography USA Johns Hopkins 1 USA Normative Aging Study UK Caerphilly & Speedwell 1 USA Johns Hopkins 2 UK BRHS Germany Augsberg men UK Caerphilly & Speedwell 2 Study 1.41 (0.84, 2.34) 0.60 (0.24, 1.50) 0.41 (0.26, 0.65) 0.97 (0.81, 1.16) 18.15 (12.36, 26.65) 0.56 (0.41, 0.77) 1.13 (1.02, 1.24) 2.35 (1.95, 2.83) 1.89 (1.63, 2.19) 1.41 (0.84, 2.34) RR (95% CI) 0.60 (0.24, 1.50) 0.41 (0.26, 0.65) 0.97 (0.81, 1.16) 18.15 (12.36, 26.65) 0.56 (0.41, 0.77) 1.13 (1.02, 1.24) 2.35 (1.95, 2.83) 1.89 (1.63, 2.19) 1.0375126.7 Overall (I-squared = 96.7%, p < 0.001) UK Caerphilly & Speedwell 2 UK Caerphilly & Speedwell 1 USA Johns Hopkins 1 Germany angiography USA Johns Hopkins 2 UK BRHS Germany Augsberg men Study USA Normative Aging Study 1.37 (0.91, 2.06) 1.37 (1.17, 1.60) 12.70 (8.61, 18.74) 0.38 (0.23, 0.60) 0.60 (0.24, 1.50) 0.67 (0.50, 0.91) 1.57 (1.43, 1.72) 2.00 (1.65, 2.42) 1.08 (0.91, 1.28) 1.37 (0.91, 2.06) RR (95% CI) 1.37 (1.17, 1.60) 12.70 (8.61, 18.74) 0.38 (0.23, 0.60) 0.60 (0.24, 1.50) 0.67 (0.50, 0.91) 1.57 (1.43, 1.72) 2.00 (1.65, 2.42) 1.08 (0.91, 1.28) 1.0534118.7 Underprediction Overprediction Original predicted Observed New predicted Observed Studies with men only (4) (followed for 10 years)

27. Division of Population Health Sciences 1.Introduction - predicted/observed values in validation studies 2.Example of systematic review of validation studies for CBR- 65 ( predicted values by a new method) 3.The new method and related statistical issues 4.Comparison of predicted values 5.Conclusions (new method) 27

28. Division of Population Health Sciences Conclusions: Limitations (1) The score is the only predictor The method, when building predicted values, does not take into account –Possible different incidence in the validation study –Different strength of the score as a predictor in the validation study –Possible existence of unknown significant predictors that were not taken in consideration in the derivation and construction of the original score! 28

29. Division of Population Health Sciences Conclusions: Advantages (2) There is always frequency data in the derivations study Predicted values are calculated in a simple way Comparisons by over-prediction/under-prediction are clinically intuitive and straightforward to use by a clinician The new technique is based on sound statistical logic (same or similar to the indirect standardization approach and the updating calibration method without adjustment) –0 29

30. Division of Population Health Sciences Thanks for your attention! 30