Crucial Statistical Caveats for Percutaneous Valve Trials Chul Ahn, Ph.D. FDA/CDRH/OSB/DBS 2/22/2010

DISCLOSURES I have no real or apparent conflicts of interest to report.

Outline Multiple secondary endpoints Data poolability - Pooling sites - Pooling feasibility and pivotal studies Today I’d like to talk about some of the key elements/issues we always discuss with sponsors in all of our studies. They are multiple secondary endpoints and data poolability, especially pooling data between sites and between feasibility and pivotal studies. Might consider pooling data between two delivery systems– transfemoral/transapical. 2/22/2010

Potential Endpoints Primary endpoint: mortality or composite of survival, hospitalization and/or NYHA - claim of statistical significance goes into the indications for use Secondary endpoints: hospitalization, NYHA, MACCE, 6MWT, QOL, Procedural & Clinical Success among others - claim of statistical significance appears in labeling Major Adverse Cardiac and Cerebro-vascular Events (MACCE) 2/22/2010

Multiple Secondary Endpoints An overall significance level 0.05 is to be applied to the safety and effectiveness primary endpoints. Only after the primary endpoint demonstrates statistical significance, the pre-specified secondary endpoints can be tested for statistical significance at an additional overall significance level 0.05. Analysis with post-hoc selected secondary endpoints is exploratory, and may not be able to provide valid statistical evidence of safety and effectiveness in the current confirmatory pivotal trial. 2/22/2010

Multiple Secondary Endpoints A claim of statistical significance from the pre-specified secondary endpoints may only be allowed when hypothesis testing was pre-specified, and multiple testing issues were taken into account. Multiple testing may inflate Type I error It may lead to positive findings purely due to chance. With 10 endpoints (when nothing going on with each of these) and testing each at 0.05, the chance of having at least one false positive finding is as large as 40%. 2/22/2010

Strategies for handling multiplicity Bonferroni Test each of k endpoints at the same level 0.05/k e.g. if k =10, test each endpoint at 0.005 May be too conservative! Two sequential test procedures: Hierarchical closed test procedure Holm’s step-down procedure Hierarchical closed test procedure Order K (>1) multiple secondary endpoints based on clinical importance. If the primary endpoint(s) is statistically significant, then the first important secondary endpoint could be tested at a significance level 0.05. If the first secondary endpoint is significant, then the second important secondary endpoint could be tested at a significance level 0.05. The process continues until the first time a statistical significance testing is failed. Then the procedure stops and the remaining secondary endpoints won’t be tested. Claims of statistical significance could appear in the labeling for those secondary endpoints that passed the test, if allowed by clinical judgment. Holm's step-down procedure (A step-down improvement of the classical Bonferroni method) Order observed univariate p values, and the first secondary endpoint could be declared statistically significant, if p(1) < 0.05/k. Given that the first secondary endpoint is significant, then the second secondary endpoint could be declared statistically significant, if p(2) < 0.05/(k-1). 2/22/2010

Data Poolability Data pooling occurs in many different areas: between studies, multi-centers, physicians or operators, different indications, different patient populations, different device or models, US vs. foreign studies, possibly different body locations and so on. It may be necessary to pool study subjects across different groups in order to obtain adequate sample sizes. However, it is important to validate data pooling. Clinical trials are almost always performed with heterogeneous samples of patients (Simon, 1980), and so, Another consideration: If there is a change in protocol during the study, can the patients afterwards be pooled with those before? In surgical and operator-skilled studies, is there evidence of a learning curve? In device trials, clinical outcomes are often influenced by physician skills. 2/22/2010

Validity of Data Pooling Pooling should be checked in terms of all aspects – clinical, engineering, regulatory, and statistical perspectives. Regulatory considerations: Whether pooling is valid is a question of whether the results under pooling will be valid and interpretable, a criterion that applies to any endpoint, primary or secondary. Statistical concerns: If there are critical clinical, engineering or regulatory concerns on pooling, further statistical efforts on poolability assessment adds no value. It is important to validate data pooling. Perhaps, it should be first checked from clinical and engineering perspectives, and then from regulatory perspectives, and lastly from statistical perspectives. Pooling s/b pre-specified and post-hoc pooling s/b avoided; If pre-specified, is there any penalty for the “interim look”? If post-hoc, study results from the pooling are not be able to provide valid scientific evidence. 2/22/2010

Pooling sites Sponsor’s claim: Statistical analysis showed that there were no site effects in the 30-days MAE rates, which justifies pooling of data across institutions. 2/22/2010

Proposed Poolability Analysis The sponsor only showed that the combined event rates are not different across centers. However, they need to show homogeneity across centers regarding treatment difference (the treatment effects are the same across centers). The combined event rates (the event rates with two treatments combined) 2/22/2010

Are treatment effects the same across centers? (Poolability Question) Clearly, there is a difference in the event rates between treatment and control across sites: a site-by-treatment interaction. 2/22/2010

Site-by-Treatment Interaction In device trials, site-to-site variation can be large. - large variation between sites in physician experience and training in using or implanting the device (sites with device inventor usually perform better) - variation between sites in patient population and patient management When site-by-device interactions are large or qualitative, then a scientifically credible explanation is required. 2/22/2010

Qualitative & Quantitative interaction Qualitative interaction: device effect changes sign depending on the site Quantitative interaction: device effect differ in magnitude, but are in the same direction (model dependent: sometimes, possible to remove them by a transformation of the variable) T T C C site 1 site 2 site 1 site 2 Qualitative interaction Quantitative interaction 2/22/2010

What to do if site-by-treatment interaction is significant? If the interaction is significant, it is may be problematic to interpret the treatment effects directly. If there is significant site-by-treatment interaction, pooling data across site is of concern. The treatment effect should be evaluated by individual site, and any anomalies should be investigated. For example, in superiority trial, we should examine whether any reversal of treatment difference (qualitative interaction) exists. 2/22/2010

Pooling feasibility and pivotal studies Pooling should be avoided as they serve different purposes. Feasibility study is to collect information and generate hypothesis to be tested in a confirmatory study. Pivotal study is to confirm findings obtained from the feasibility study. 2/22/2010

Thanks! chul.ahn@fda.hhs.gov 2/22/2010