Statistical issues in the validation of surrogate endpoints Stuart G. Baker, Sc.D.

Surrogate endpoint: definition used to make conclusions about the effect of intervention on true endpoint obtained sooner, at less cost, or less invasively than the true endpoint

Outline Asking the right questions Hypothesis testing  General framework for validation and application  Graphical view Estimation (meta-analytic)  General framework for validation and application  Trial-level statistics –graphical view  Predicted effect of intervention for binary surrogate and true endpoints (NEW approach) – graphical view Caveats

Asking the right questions

Validation trial: Both surrogate and true endpoints are observed  QUESTION: Are the conclusions about the effect of intervention on true endpoint the same when based on (i) only surrogate endpoint (ii) only the true endpoint ? Application trial: Surrogate but not true endpoint is observed  QUESTION: What is the effect of intervention on true endpoint?

Hypothesis testing

Validation trial New trial Surrogate endpoints True endpoints Surrogate endpoint If Prentice Criteria hold, valid hypothesis testing using surrogate endpoint Validation Application General Framework for Hypothesis Testing Test hypothesis using surrogate endpoint Valid hypothesis test: H0(T): no effect of intervention on true implies H0(S): no effect of intervention on surrogate so that reject H0(S) implies reject H0(T) Prentice Criteria; pr (true | surrogate) not depend on group + extra requirement (if binary: surrogate predicts true) (Buyse,Molenbergs, 1998)): easy to reject; hard to show they hold

Understanding hypothesis testing Graphical illustration using  Binary surrogate endpoint  Binary true endpoint

Treatment B Fraction with surrogate endpoint Fraction with true endpoint Treatment A B B A A A= B for true Fraction with a surrogate endpoint Fraction with true endpoint Validation Trial Prentice Criterion “holds” Application trial Extrapolation: Prentice Criterion “holds” (but how close is close enough?) 0 0 implies A= B for surrogate

Treatment B Fraction with surrogate endpoint Fraction with true endpoint Treatment A B B A A A= B for true Fraction with a surrogate endpoint Fraction with true endpoint Validation Trial Prentice Criterion does not hold Application trial Hypothesis testing gives incorrect conclusion Hypothesis testing gives incorrect conclusion 0 0 Extrapolation: same lines does not imply A= B for surrogate

Estimation Meta-analytic (based on multiple previous trials)

Previous trials Validation trial Surrogate endpoints True endpoints Surrogate endpoint Predicted effect of intervention on true endpoint Validation (similar confidence intervals) Observed effect of intervention on true endpoint Application General Framework for Estimation True endpoint Application trial Surrogate endpoint Predicted effect of intervention on true endpoint model

Meta-analytic methods of estimation Trial-level statistics  Buyse et al (2000); Gail et al (2000) Estimated predicted effect of intervention on true endpoint  proposal for binary surrogate and true endpoints  simple computations

Focus Binary surrogate endpoint Binary true endpoint

AT=0T=1 S=0 S=1 BT=0T=1 S=0xx S=1xx BT=0T=1 S=0 S=1 AT=0T=1 S=0xx S=1xx BT=0T=1 S=0xx S=1xx AT=0T=1 S=0xx S=1xx BT=0T=1 S=0xx S=1xx AT=0T=1 S=0xx S=1xx x x BT=0T=1 S=0xx S=1xx x x Application trial Previous trial 1 Previous trial 1 Previous trial 2 Previous trial 3 Validation trial AT=0T=1 S=0xx S=1xx DATA SCHEME

Meta-analysis of trial-level statistics Graphical overview of approach of Buyse et al (2000) and Gail et al (2000)

Fraction with surrogate endpoint Fraction with true endpoint B Trial-level meta-analysis (simplified overview) A d Previous study 1 Previous study 2 Previous study 3 Regression: using random effects and within trial data

Meta-analysis of estimated predicted effects of intervention A new approach for binary surrogate and true endpoints

Fraction with surrogate endpoint Fraction with true endpoint B Predicted effect of intervention on true endpoint based on surrogates A and B in new study and data from previous study 1 A d1d1d1d1 Note: Lines for each group need not be identical—Prentice Criterion not needed

Fraction with surrogate endpoint Fraction with true endpoint B Predicted effect of intervention on true endpoint based on surrogates A and B in new study and data from previous study 2 A d1d1d1d1 d2d2d2d2 Note: Lines for each group need not be identical—Prentice Criterion not needed

Fraction with surrogate endpoint Fraction with true endpoint B Predicted effect of intervention on true endpoint based on surrogates A and B in new study and data from previous study 3 A d1d1d1d1 d2d2d2d2 d3d3d3d3 d=(d 1 w 1 + d 2 w 2 + d 3 w 3 )/(w 1 +w 2 +w 3 )

Meta-analysis of estimated predicted treatment effects: d 1, d 2, d 3 d= d= (d 1 w 1 +d 2 w 2 + d 3 w 3 ) /(w 1 +w 2 +w 3 ), Weights w i are based on a random-effects model for d i (with variance    simpler than a random-effects for slopes w i = 1 / (sampling variance of d i +    Weights minimize variance of d if d i are not correlated  simplification since d i ‘s are correlated

Meta-analysis computation d= d= (d 1 w 1 +d 2 w 2 + d 3 w 3 ) /(w 1 +w 2 +w 3 ), where  d i ={f i0A p A +f i1A (1-p A )} - {f i0B p B +f i1B (1-p B )} Application or validation trial: fraction with surrogate endpoint= p A and p B Previous trials: fraction with true given surrogate endpoint: f i0A, f i1A, f i0B, f i1B w i =1/(V i +   ), V i = sampling variance of (d i ) To estimate    adapt method of DerSimonian and Laird for usual meta-analysis accounting for covariance among d i ’s due to share parameters p A and p B To compute variance of d To compute variance of d  Bootstrap trials and data within trials

Meta-analysis simulation d= d= (d 1 w 1 +d 2 w 2 + d 3 w 3 ) /(w 1 +w 2 +w 3 ), where  d i ={f i0A p A +f i1A (1-p A )} - {f i0B p B +f i1B (1-p B )} Simulation  Generate random f i0A, f i1A, f i0B, f i1B  Generate random data for each trial Mean squared error  Slightly smaller for meta-analysis of predicted effect of intervention than for meta-analysis of trial-level statistics (computed via method-of- moments)

Hypothetical data: Example 1

Hypothetical data: Example 2

Real Data (x 10) from multicenter trial in Gail et al (2000): surrogate is cholesterol level, true endpoint is artery diameter

Caveats

Needed even if surrogate is validated with data from many previous studies Extrapolation to a new trial  Hypothesis testing  Estimation using data from previous trials Surrogate endpoint does not predict harms that might arise after surrogate is observed

When caveats are less critical Preliminary drug development when the surrogate endpoint is used to decide on further development or definitive testing with a true endpoint Establishing dose or timing of an intervention previously shown effective using true endpoint at a different (suboptimal?) dose or timing

Summary

Types of trials Validation trial:  Both surrogate and true endpoint  Do you obtain the same conclusion about effect of intervention on true endpoint using (i) surrogate endpoint and (ii) true endpoint? Application trial:  Only surrogate endpoint  What is the effect of intervention on true endpoint?

Hypothesis testing Not validated if reject Prentice’s criteria Not clear what to conclude about surrogate if cannot reject Prentice’s criteria

Estimation (meta-analysis) Not need Prentice’s criteria Meta-analysis of trial-level statistic  Applicable to all types of endpoints Meta-analysis of estimated predicted effect of intervention on true endpoint  Binary surrogate and true endpoints  Computationally simple  Slightly smaller MSE than with meta-analysis of trial-level statistics (in simulation)