1. Point/Counterpoint – Single Phase 3 trial POINT: A single phase 3 trial is often insufficient Brian Smith, Amgen, Inc.

2. Disclaimer The views expressed herein represent personal views and do not necessarily represent the views or practices of Amgen.

3. On “significance” and replication If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent. point), or one in a hundred (the 1 per cent. point). Personally, the writer prefers to set a low standard of significance at the 5 per cent. point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance. — Sir Ronald Aylmer Fisher 'The Arrangement of Field Experiments', The Journal of the Ministry of Agriculture, 1926, 33, 504.

4. How Likely Would an Ineffective Treatment Make it to Market Given 2 Phase 3, p-value < 0.05 trials? Remember – Type 1 error (one sided) – Probability that a treatment which does not work will appear to work after the experiment is completed This probability is typically 2.5% for a Phase 3 trial – Power – For a treatment that has a particular true effect x, the probability that the treatment will appear to work after the experiment is completed This probability is typically 90% for a Phase 3 trial

5. How Likely Would an Ineffective Treatment Make it to Market Given 2 Phase 3, p-value < 0.05 trials? Simplifying assumptions for illustration – The treatment either does not work or – The treatment has true effect x – Two phase 3 studies are to be done in sequence – Historically only 15% of compounds for this therapeutic area are effective After the first Phase 3 trial, you have success. – what is the probability that the treatment is effective? – 0.15 * 0.9 / (0.15 * 0.9 + 0.85 * 0.025) = 0.864

6. How Likely Would an Ineffective Treatment Make it to Market Given 2 Phase 3, p-value < 0.05 trials? After the second Phase 3 trial, you have success. – what is the probability that the treatment is effective? – 0.864 * 0.9 / (0.864 * 0.9 + 0.136 * 0.025) = 0.996 This is just a simple application of Bayes Theorem Suppose you just wanted to do a single trial and if positive you wanted to be equally confident that the treatment worked, what would α be? – 0.15 * 0.9 / (0.15 * 0.9 + 0.85 * α) = 0.996 – α = 0.00069

7. Some other Priors Prior ProbabilityProbability Effective After 1 st Trial Probability Effective After 2 nd Trial Significance Level for Single Trial 15%86.4%99.6%0.00069 50%97.3%99.92%0.00012 80%99.3%99.98%0.00003

8. What do we Learn? Positive Results (p-value < 0.025) from 1 trial is not nearly enough evidence of a positive effect You could reduce α and increase sample size if want single trial

9. If you wanted to calculate this probability Use full Bayesian approach – Prior distribution for first Phase 3 study – After trial find posterior distribution – Use this as prior distribution for second Phase 3 study – After trial find posterior – Calculate probability of treatment difference Thus, the calculations that I performed are more for illustration purposes

10. What Else Do We Know? Phase 3 is equally about safety endpoints Really, don’t you just feel better when results replicate?

11. Conclusions Replication is an under appreciated concept in evidence generation Do you have to have 2 trials? – No, but you should generate the same amount of evidence with one Bayesian perspective really helps make sense of these sort of questions.