1. Crucial Statistical Caveats for Percutaneous Valve Trials
Chul Ahn, Ph.D. FDA/CDRH/OSB/DBS
2/22/2010

2. DISCLOSURES
I have no real or apparent conflicts of interest to report.

3. Outline Multiple secondary endpoints Data poolability - Pooling sites
- Pooling feasibility and pivotal studies
Today I’d like to talk about some of the key elements/issues we always discuss with sponsors in all of our studies. They are multiple secondary endpoints and data poolability, especially pooling data between sites and between feasibility and pivotal studies.
Might consider pooling data between two delivery systems– transfemoral/transapical.
2/22/2010

4. Potential Endpoints
Primary endpoint: mortality or composite of survival, hospitalization and/or NYHA - claim of statistical significance goes into the indications for use
Secondary endpoints: hospitalization, NYHA, MACCE, 6MWT, QOL, Procedural & Clinical Success among others - claim of statistical significance appears in labeling
Major Adverse Cardiac and Cerebro-vascular Events (MACCE)
2/22/2010

5. Multiple Secondary Endpoints
An overall significance level 0.05 is to be applied to the safety and effectiveness primary endpoints.
Only after the primary endpoint demonstrates statistical significance, the pre-specified secondary endpoints can be tested for statistical significance at an additional overall significance level 0.05.
Analysis with post-hoc selected secondary endpoints is exploratory, and may not be able to provide valid statistical evidence of safety and effectiveness in the current confirmatory pivotal trial.
2/22/2010

6. Multiple Secondary Endpoints
A claim of statistical significance from the pre-specified secondary endpoints may only be allowed when
hypothesis testing was pre-specified, and
multiple testing issues were taken into account.
Multiple testing may inflate Type I error
It may lead to positive findings purely due to chance.
With 10 endpoints (when nothing going on with each of these) and testing each at 0.05, the chance of having at least one false positive finding is as large as 40%.
2/22/2010

7. Strategies for handling multiplicity
Bonferroni
Test each of k endpoints at the same level 0.05/k
e.g. if k =10, test each endpoint at 0.005
May be too conservative!
Two sequential test procedures:
Hierarchical closed test procedure
Holm’s step-down procedure
Hierarchical closed test procedure
Order K (>1) multiple secondary endpoints based on clinical importance. If the primary endpoint(s) is statistically significant, then the first important secondary endpoint could be tested at a significance level If the first secondary endpoint is significant, then the second important secondary endpoint could be tested at a significance level The process continues until the first time a statistical significance testing is failed. Then the procedure stops and the remaining secondary endpoints won’t be tested. Claims of statistical significance could appear in the labeling for those secondary endpoints that passed the test, if allowed by clinical judgment.
Holm's step-down procedure (A step-down improvement of the classical Bonferroni method)
Order observed univariate p values, and the first secondary endpoint could be declared statistically significant, if p(1) < 0.05/k. Given that the first secondary endpoint is significant, then the second secondary endpoint could be declared statistically significant, if p(2) < 0.05/(k-1).
2/22/2010

8. Data Poolability
Data pooling occurs in many different areas: between studies, multi-centers, physicians or operators, different indications, different patient populations, different device or models, US vs. foreign studies, possibly different body locations and so on.
It may be necessary to pool study subjects across different groups in order to obtain adequate sample sizes. However, it is important to validate data pooling.
Clinical trials are almost always performed with heterogeneous samples of patients (Simon, 1980), and so,
Another consideration:
If there is a change in protocol during the study, can the patients afterwards be pooled with those before?
In surgical and operator-skilled studies, is there evidence of a learning curve?
In device trials, clinical outcomes are often influenced by physician skills.
2/22/2010

9. Validity of Data Pooling
Pooling should be checked in terms of all aspects – clinical, engineering, regulatory, and statistical perspectives.
Regulatory considerations: Whether pooling is valid is a question of whether the results under pooling will be valid and interpretable, a criterion that applies to any endpoint, primary or secondary.
Statistical concerns: If there are critical clinical, engineering or regulatory concerns on pooling, further statistical efforts on poolability assessment adds no value.
It is important to validate data pooling. Perhaps, it should be first checked from clinical and engineering perspectives, and then from regulatory perspectives, and lastly from statistical perspectives.
Pooling s/b pre-specified and post-hoc pooling s/b avoided;
If pre-specified, is there any penalty for the “interim look”? If post-hoc, study results from the pooling are not be able to provide valid scientific evidence.
2/22/2010

10. Pooling sites
Sponsor’s claim: Statistical analysis showed that there were no site effects in the 30-days MAE rates, which justifies pooling of data across institutions.
2/22/2010

11. Proposed Poolability Analysis
The sponsor only showed that the combined event rates are not different across centers.
However, they need to show homogeneity across centers regarding treatment difference (the treatment effects are the same across centers).
The combined event rates (the event rates with two treatments combined)
2/22/2010

12. Are treatment effects the same across centers? (Poolability Question)
Clearly, there is a difference in the event rates between treatment and control across sites: a site-by-treatment interaction.
2/22/2010

13. Site-by-Treatment Interaction
In device trials, site-to-site variation can be large.
- large variation between sites in physician experience and training in using or implanting the device (sites with device inventor usually perform better)
- variation between sites in patient population and patient
management
When site-by-device interactions are large or qualitative, then a scientifically credible explanation is required.
2/22/2010

14. Qualitative & Quantitative interaction
Qualitative interaction: device effect changes sign depending on the site
Quantitative interaction: device effect differ in magnitude, but are in the same direction (model dependent: sometimes, possible to remove them by a transformation of the variable) T T C C
site 1
site 2
site 1
site 2
Qualitative interaction
Quantitative interaction
2/22/2010

15. What to do if site-by-treatment interaction is significant?
If the interaction is significant, it is may be problematic to interpret the treatment effects directly.
If there is significant site-by-treatment interaction, pooling data across site is of concern.
The treatment effect should be evaluated by individual site, and any anomalies should be investigated. For example, in superiority trial, we should examine whether any reversal of treatment difference (qualitative interaction) exists.
2/22/2010

16. Pooling feasibility and pivotal studies
Pooling should be avoided as they serve different purposes.
Feasibility study is to collect information and generate hypothesis to be tested in a confirmatory study.
Pivotal study is to confirm findings obtained from the feasibility study.
2/22/2010

17. Thanks! chul.ahn@fda.hhs.gov
2/22/2010