1. Illustration of the evaluation of risk prediction models in randomized trials Examples from women’s health studies Parvin Tajik, MD PhD candidate Department of Clinical Epidemiology & Biostatistics Department of Obstetrics & Gynecology Academic Medical Center, University of Amsterdam, the Netherlands FHCRC 2014 Risk Prediction Symposium June 11, 2014

2. Clinical Problem I Pre-eclampsia

3. fullPIERS model Lancet, 2011

4. Development Method Patients: 2000 women admitted in hospital for pre-eclapmsia (260 event) Outcome: Maternal mortality or other serious complications of pre-eclampsia Logistic regression model with stepwise backward elimination

5. Final model Logit P(D) = 2.68 – (0.054 × gestational age at eligibility) + (1.23 × chest pain or dyspnoea) – (0.027 × creatinine) + (0.21 × platelets) + (0.00004 × platelets 2 ) + (0.01 × AST) – (0.000003 × AST 2 ) + (0.00025 × creatinine × platelet) – (0.00007 × platelets × AST) – (0.0026 × platelets × SpO2)

6. Performance of full-PIERS model Reported good risk discrimination and calibration

7. Online calculator

8. HYPITAT trial (2005-2008) P P Women at 36-41 wks of pregnancy with mild pre-eclampsia (n=750) I I Early Induction of labor (LI) C C Expectant monitoring (EM) O O Composite measure of adverse maternal outcomes

9. HYPTAT Results (relative risk 0.71, 95% CI 0.59–0.86, p<0·0001) Management Adverse maternal outcomes Total Labor induction117 (31 % ) 377 Expectant monitoring166 (44 % ) 379

10. Modeling Logit P(D=1|T,Y) = β 0 + β 1 T + β 2 Y + β 3 TY D = 1 Adverse maternal outcome Y = fullPIERS score T = Treatment 1 Labor induction 0 Expectant monitoring

11. FullPIERS for guiding labor induction P for interaction: 0.93 fullPIERS score

12. Clinical Problem II Preterm birth

13. Cervical pessary Medical device inserted to vagina to provide structural support to cervix

14. ProTWIN trial (2009-2012) P Women with multiple pregnancy (twin or triplet) between 12 & 20 weeks pregnancy I Cervical Pessary (n = 403) C Control (n = 410) O Primary: Composite Adverse perinatal outcome

15. ProTWIN Results (relative risk 0.98, 95% CI 0.69–1.39) Management Composite adverse perinatal outcome Total Pessary53 (13 % ) 401 No pessary55 (14 % ) 407

16. Pre-specified subgroup analysis Cervical length ( = 38 mm)

17. Pre-specified subgroup analysis Trial Conclusion: Clinicians should consider a cervical pessary in women with a multiple pregnancy and a short cervical length. Cervical lengthPessary group Control group RR (95%CI) CxL < 38 mm12%29%0.42 (0.19-0.91) CxL >= 38 mm13%10%1.26 (0.74-2.15) (P for interaction 0.01)

18. Other Markers 1.Obstetric history (parity) Nulliparous Parous with no previous preterm birth Parous with at least one previous preterm birth 2.Chorionicity Monochorionic Dichorionic 3.Number of fetuses Twin Triplet

19. One marker at a time analysis

20. Modeling Logit P(D=1|T,Y) = β 0 + β 1 T + Σ β i Y i + Σ β j TY j D = 1 composite poor perinatal outcome Y = Markers T = Treatment 1 pessary 0 control - Internal validation by bootstrapping

21. Multi-marker model * Shrunken with an average shrinkage factor of 0.76 c-stat : 0,71 (95%CI: 0,66-0,77); optimism-corrected c-stat: 0,69 (95%CI: 0,63-0,74)

22. How can the model be used in practice? 1.Calculating Risk without pessary Using the model and setting treatment = 0 (control) 2.Calculating Risk with pessary Using the model and setting treatment = 1 (pessary) 3.Calculating the predicted absolute benefit from pessary Risk without pessary – Risk with pessary Positive: woman benefits Negative: woman does not benefit

23. Predicted benefit from pessary

24. Calibration of the predicted benefit

25. Model performance Multimarker positivity rate: 35% (31-39%) Benefit from pessary in multimarker-positives 15% (7- 23%) Benefit from no pessary in multimarker-negatives 8% (3-13%) Risk reduction by multimarker-based strategy 10% (6-15%)

26. Conclusion Common assumption for application of risk prediction models for treatment selection: “Being at higher risk of outcome implies a larger benefit from treatment” Not necessarily true Developing models using trial data and modeling the interaction between markers and treatment might be a more optimal strategy

27. Open Research Questions Optimal modeling strategy? Optimal algorithm for variable selection? Optimal method for optimism correction?

28. Thanks! Any Questions?

29. Multimarker vs. CxL only Multimarker +Multimarker - Short cervix1749 Long cervix120505

30. Two examples