Illustration of the evaluation of risk prediction models in randomized trials Examples from women’s health studies Parvin Tajik, MD PhD candidate Department of Clinical Epidemiology & Biostatistics Department of Obstetrics & Gynecology Academic Medical Center, University of Amsterdam, the Netherlands FHCRC 2014 Risk Prediction Symposium June 11, 2014

Clinical Problem I Pre-eclampsia

fullPIERS model Lancet, 2011

Development Method Patients: 2000 women admitted in hospital for pre-eclapmsia (260 event) Outcome: Maternal mortality or other serious complications of pre-eclampsia Logistic regression model with stepwise backward elimination

Final model Logit P(D) = 2.68 – (0.054 × gestational age at eligibility) + (1.23 × chest pain or dyspnoea) – (0.027 × creatinine) + (0.21 × platelets) + ( × platelets 2 ) + (0.01 × AST) – ( × AST 2 ) + ( × creatinine × platelet) – ( × platelets × AST) – ( × platelets × SpO2)

Performance of full-PIERS model Reported good risk discrimination and calibration

Online calculator

HYPITAT trial ( ) P P Women at wks of pregnancy with mild pre-eclampsia (n=750) I I Early Induction of labor (LI) C C Expectant monitoring (EM) O O Composite measure of adverse maternal outcomes

HYPTAT Results (relative risk 0.71, 95% CI 0.59–0.86, p<0·0001) Management Adverse maternal outcomes Total Labor induction117 (31 % ) 377 Expectant monitoring166 (44 % ) 379

Modeling Logit P(D=1|T,Y) = β 0 + β 1 T + β 2 Y + β 3 TY D = 1 Adverse maternal outcome Y = fullPIERS score T = Treatment 1 Labor induction 0 Expectant monitoring

FullPIERS for guiding labor induction P for interaction: 0.93 fullPIERS score

Clinical Problem II Preterm birth

Cervical pessary Medical device inserted to vagina to provide structural support to cervix

ProTWIN trial ( ) P Women with multiple pregnancy (twin or triplet) between 12 & 20 weeks pregnancy I Cervical Pessary (n = 403) C Control (n = 410) O Primary: Composite Adverse perinatal outcome

ProTWIN Results (relative risk 0.98, 95% CI 0.69–1.39) Management Composite adverse perinatal outcome Total Pessary53 (13 % ) 401 No pessary55 (14 % ) 407

Pre-specified subgroup analysis Cervical length ( = 38 mm)

Pre-specified subgroup analysis Trial Conclusion: Clinicians should consider a cervical pessary in women with a multiple pregnancy and a short cervical length. Cervical lengthPessary group Control group RR (95%CI) CxL < 38 mm12%29%0.42 ( ) CxL >= 38 mm13%10%1.26 ( ) (P for interaction 0.01)

Other Markers 1.Obstetric history (parity) Nulliparous Parous with no previous preterm birth Parous with at least one previous preterm birth 2.Chorionicity Monochorionic Dichorionic 3.Number of fetuses Twin Triplet

One marker at a time analysis

Modeling Logit P(D=1|T,Y) = β 0 + β 1 T + Σ β i Y i + Σ β j TY j D = 1 composite poor perinatal outcome Y = Markers T = Treatment 1 pessary 0 control - Internal validation by bootstrapping

Multi-marker model * Shrunken with an average shrinkage factor of 0.76 c-stat : 0,71 (95%CI: 0,66-0,77); optimism-corrected c-stat: 0,69 (95%CI: 0,63-0,74)

How can the model be used in practice? 1.Calculating Risk without pessary Using the model and setting treatment = 0 (control) 2.Calculating Risk with pessary Using the model and setting treatment = 1 (pessary) 3.Calculating the predicted absolute benefit from pessary Risk without pessary – Risk with pessary Positive: woman benefits Negative: woman does not benefit

Predicted benefit from pessary

Calibration of the predicted benefit

Model performance Multimarker positivity rate: 35% (31-39%) Benefit from pessary in multimarker-positives 15% (7- 23%) Benefit from no pessary in multimarker-negatives 8% (3-13%) Risk reduction by multimarker-based strategy 10% (6-15%)

Conclusion Common assumption for application of risk prediction models for treatment selection: “Being at higher risk of outcome implies a larger benefit from treatment” Not necessarily true Developing models using trial data and modeling the interaction between markers and treatment might be a more optimal strategy

Open Research Questions Optimal modeling strategy? Optimal algorithm for variable selection? Optimal method for optimism correction?

Thanks! Any Questions?

Multimarker vs. CxL only Multimarker +Multimarker - Short cervix1749 Long cervix120505

Two examples