1. Measuring Effectiveness in Trials of Acute Bacterial Sinusitis Measuring Effectiveness in Trials of Acute Bacterial Sinusitis John H. Powers, MD Lead Medical Officer Antimicrobial Drug Development and Resistance Initiatives Office of Medical Policy Center for Drug Evaluation and Research U.S. Food and Drug Administration

2. 2 Introduction  Regulatory and scientific history related to noninferiority trials and demonstration of effectiveness  History of discussions regarding NI trials in antimicrobials and acute bacterial sinusitis trials  Evaluation of historical evidence of magnitude of effects of antimicrobials in placebo controlled trials in acute bacterial sinusitis  Conclusions regarding current knowledge of evaluation of effectiveness and in vitro resistance on outcomes

3. 3 Regulatory/Scientific History  1938 – pre-market requirement for safety only (based on Elixir of sulfanilamide deaths)  1962 –requirement for demonstration of effectiveness to balance any potential harms of therapy (thalidomide)  President Kennedy sent recommendations to Senate Committee: “an undefined standard of ‘substantial evidence’ of effectiveness was inadequate in terms of assuring that drugs that reach the market have been shown to be effective for the claims made for them.”  Congress defines that only source of “substantial evidence” of effectiveness is “adequate and well-controlled trials”  1970 – FDA publishes regulations that provide criteria for defining “adequate and well-controlled trials”

4. 4 Adequate and Well-Controlled 1. Clear statement of objectives 2. Study design permits valid quantitative comparison with a control 3. Select patients with disease (treatment) or at risk of disease (prevention) 4. Baseline comparability (randomization) 5. Minimize bias (blinding, etc.) 6. Appropriate methods of assessment of outcomes 7. Appropriate methods of analysis  21 CFR 314.126

5. 5 Regulatory/Scientific History  1985 – recognition of issues with trials designed to show “similarity” rather than superiority of drugs  “If the intent of the trial is to show similarity of the test and control drugs, the report of the study should assess the ability of the study to have detected a difference between treatments. Similarity of test drug and active control can mean either that both drugs were effective or that neither was effective. The analysis of the study should explain why the drugs should be considered effective in the study, for example, by reference to results in previous placebo- controlled studies of the active control drug.  21 CFR 314.126 (b)(2)(iv)

6. 6 Regulatory/Scientific History  Noninferiority trials attempt to:  rule out how much inferior a new treatment might be compared to an already proven effective treatment  AND ensure control drug effect relative to placebo is consistent under the conditions of the trial  2000 – ICH-E10 Guidance on Choice of Control Group and Related Issues in Clinical Trials  Section 1.5 describes information necessary to select noninferiority margin determined by analysis of historical evidence of sensitivity to drug effects (HESDE)  magnitude by which control drug may be reliably and reproducibly shown superior to placebo in previous superiority trials  Effect of control relative to placebo should be well-characterized and consistent from trial to trials so that effect in current trial consistent

7. 7 Regulatory/Scientific History  Well-designed experiments have positive and negative controls to determine results causally related to intervention and not conditions of experiment (internal validity)  Lack of “negative” controls in noninferiority trials means they lack intrinsic measure of internal validity  Data on effect of control relative to placebo external to NI trial  Similar potential biases to historical controlled trials  Conditions of experiment (e.g. enrolling patients who do not have disease, timing of outcomes beyond natural history of disease, etc.) may change effect of control and affect conclusions regardless of margin chosen  Demonstration of noninferiority does not necessarily mean drug has demonstrated effectiveness relative to placebo

8. 8 Regulatory/Scientific History  Selecting of appropriate margin of inferiority integral to design of noninferiority trials  “In practice, the noninferiority margin chosen usually will be smaller than that suggested by the smallest effect size of the active control because of interest in ensuring that some clinically acceptable effect size (or fraction of the control drug effect) was maintained.”  ICH E-10, section 1.5.1.1., page 11  “Effect size” refers to magnitude of benefit of active control drug relative to placebo from previous placebo controlled trials

9. 9 Quantitative Comparison with Control Three criteria before one can perform noninferiority trial: 1. Quantitative assessment of effect of control drug relative to placebo based on data from previous trials:  Reliable, well-characterized and reproducible effect from trial to trial  Based on trials that are adequate and well controlled  Evaluation of all previous superiority trials  Takes into account variability of the data ( confidence intervals rather than point estimates) THEN THEN 2. Selection of margin that is less than effect of control relative to placebo to preserve some benefit of control based on clinical judgment AND 3. Maintenance constancy of the effect of the control from trial to trial  Similar definition of disease, endpoints, timing of endpoints  Changes in medical practice, adjunctive therapies, antimicrobial resistance

10. 10 Definitions and Determining Margin  M(1) defined as magnitude of benefit of active control compared to placebo as measured in current trial but determined from data in previous superiority trials  M(2) defined loss of effect of test drug compared to active control where drug is still considered clinically useful based on clinical judgment  Results of trial demonstrate noninferiority if success rates with test drug minus success rates with active control exclude M(2); T – C < M(2)  BUT this only shows that test drug is effective relative to placebo when M(2) is less then M(1); M(2) < M(1)

11. 11 Non-Inferiority Margins Quantifying the Effects of a Control Drug 0 1020 10 20 30 30 Favors placebo Favors active drug 25 20 30 M(1) = benefit of active drug over placebo based on previous placebo controlled trials Preserved benefit over placebo Non-inferiority margin (“delta”) Difference in Point Estimates M(2) - acceptable loss of effect relative to control

12. 12 Non-Inferiority Margins Quantifying the Effects of a Control Drug 0 1020 10 20 30 30 Favors placebo Favors active drug 25 20 30 M(1) = benefit of active drug over placebo based on previous placebo controlled trials Preserved benefit over placebo Non-inferiority margin (“delta”) Difference in Point Estimates M(2) - acceptable loss of effect relative to control

13. 13 Non-Inferiority Margins Determining the Effect of a Test Drug 0 1020 10 20 30 30 Favors test drug Favors control drug Difference in Point Estimates

14. 14 Non-Inferiority Margins Determining the Effect of a Test Drug 0 1020 10 20 30 30 Favors test drug Favors control drug Difference in Point Estimates

15. 15 Non-Inferiority Margins Determining the Effect of a Test Drug current trial control test placebo past trials 0 100 80 success rates control

16. 16 History with Antimicrobials/ABS  Several meetings addressing issues in noninferiority trials in study of antimicrobials  Need to evaluate data on each indication to determine margins or whether data to support noninferiority trials  Issues with selection of patients with disease  Issues with defining outcomes and timing of outcomes  Issues with analysis (ITT and subgroup analyses)  October 2003 – AIDAC meeting on clinical trial design in ABS  No constellation of signs/symptoms predicts bacterial etiology so sinus puncture necessary at to define patients with disease  No studies correlating >ten days of symptoms with sinus punctures  Lack of evidence of specificity of radiographic findings with positive culture on sinus puncture  No evidence to support “presumed eradication” of organisms  Timing of outcome important in relation to ability of trial to evaluate effectiveness and time to resolution of symptoms may be most sensitive measure of outcome  Lack of evidence from previous placebo controlled trials to base any noninferiority margin – trials should be superiority trials

17. 17 Evaluating Data from Literature Criteria for submission of published literature reports alone as evidence of effectiveness 1. Multiple studies conducted by different investigators where each clearly has adequate design and findings consistent across studies 2. High level of detail in published reports including clear and adequate descriptions of statistical analysis plans, prospectively determined analytical methods and study endpoints, and full accounting of all enrolled patients 3. Clearly appropriate endpoints objectively assessed and not dependent on investigator judgment 4. Robust results achieved by protocol-specified analyses that yield consistent conclusions of efficacy and do not require selected post-hoc analyses, subsetting or reduced datasets (e.g. analysis of only responders) 5. Conduct of studies by groups with properly documented operating procedures Guidance for Industry: Providing Clinical Evidence of Effectiveness for Human Drugs and Biological Products p. 19.

18. 18 504030 2010 0 20304050 Favors placeboFavors drug What One Would Need to Justify a Margin of 10% in Acute Bacterial Sinusitis Similar disease definition Similar disease definition Similar endpoint definition Similar endpoint definition Similar timing of endpoint Similar timing of endpoint Constancy of effect of control Constancy of effect of control

19. 19 ABS Placebo Controlled Trials  Description of trials  21 trials compared antimicrobial to placebo in all languages but 4 not prospective, not randomized and/or no direct patient outcomes  17 randomized, prospective trials evaluated  2787 total patients (1132 ‘evaluable’ on placebo and 1331 ‘evaluable’) on various drugs/doses/durations  No quinolones  One trial in children with cefuroxime in children (no evidence of benefit)  Dates from 1964 to 2005 (41 year span)  8 trials published since 2000, 2 in 2005, showing that placebo controlled trials can and are being performed  3 published since last AIDAC meeting in 2003 including one in US, one in Europe and one in pediatric population  Demographics  Average age per trial was 37 years with age range 18-93 years  Average gender distribution per trial was 60% female in trials with adults or mixed populations

20. 20 504030 2010 0 20304050 Ganaca et al. 1973 n=50 Lindbaek et al. 1996 n=127 Haye et al. 1998 n=168 Hansen et al. 2000 n=127 Wald et al. 1986 n=93 Varonen et al. 2003 n=146 van Buchem et al. 1997 n=206 deSutter et al. 2003 n=135 Axelsson et al. 1970 n=142 Norrelund et al. 1978 1978 n=135 Kaiser et al. 2001 n=265 a (77) b Bucher et al. 2003 n=251 Garbutt et al. 2001 n=161 Stalman et al. 1997 n=186 Favors placeboFavors study drug Analysis of Efficacy in Placebo Controlled Trials in Acute Bacterial Sinusitis cefuroxime d14 amoxicillin-clavulanate d14 amox or amoxicillin-clav d14 azithromycin d8 pivampicillin d8 pencillin or lincomycin d10 amoxicillin d14 amox or doxy or penicillin d 14 amoxicillin or amox-clav d10 penicillin d7 azithromycin d14 amoxicillin or penicillin d10 cyclacillin (not specified) Kristo et al. 2005 n=82 Lindbaek et al. 1998 n=70 doxycycline d10 Merenstein et al. 2005 n=135 amoxicillin or penicillin d10 amoxicillin d10 a b

21. 21 ABS Placebo Controlled Trials 1. No reliable, consistent magnitude of benefit of antimicrobials compared to placebo [M(1)], so no evidence upon which to base any noninferiority margin  Majority of trials do not provide evidence of benefit of antimicrobials compared to placebo; most trials powered to rule out differences of 15% to 35% which is amount needed to justify 10% to 15% margin  Trial with largest point estimate of benefit had lower bound of 95% confidence interval 11.3%; most of point estimates of treatment difference <10%  Point estimates success rates in placebo groups range from 29% to 95% and point estimate success with various drugs range from 35% to 93%  No evidence of decreasing complications or prevention of chronic sinusitis 2. Cannot select loss of effect of test drug relative to control [M(2)] since magnitude of benefit of control relative to placebo [M(1)] not clear 3. No constancy of effect of control  Different definitions used for enrollment criteria  None of trials used receipt of additional antimicrobial as an endpoint  Timing of fixed endpoints vary and not longer than 14 days for primary fixed endpoint; effects of drug compared to placebo disappear with time

22. 22 504030 2010 0 20304050 Ganaca et al. 1973 n=50 Lindbaek et al. 1996 n=127 Haye et al. 1998 n=168 Hansen et al. 2000 n=127 Wald et al. 1986 n=93 Varonen et al. 2003 n=146 van Buchem et al. 1997 n=206 deSutter et al. 2003 n=135 Axelson et al. 1970 n=142 Norrelund et al. 1978 1978 n=135 Kaiser et al. 2001 n=265* (77) Bucher et al. 2003 n=251 Garbutt et al. 2001 n=161 Stalman et al. 1997 n=186 Favors placeboFavors study drug Analysis of Safety in Placebo Controlled Trials in Acute Bacterial Sinusitis Kristo et al. 2005 n=82 Lindbaek et al. 1998 n=70 Merenstein et al. 2005 n=135 odds ratio 3.89 (2.09, 7.25) p-value 0.017 GI AEs 3 drug, 0 placebo 6 excluded from analysis on drug, 2 on placebo

23. 23 504030 2010 0 20304050 Ganaca et al. 1973 n=50 Lindbaek et al. 1996 n=127 Haye et al. 1998 n=168 Hansen et al. 2000 n=127 Wald et al. 1986 n=93 Varonen et al. 2003 n=146 van Buchem et al. 1997 n=206 deSutter et al. 2003 n=135 Axelsson et al. 1970 n=142 Norrelund et al. 1978 1978 n=135 Kaiser et al. 2001 n=265 a (77) b Bucher et al. 2003 n=251 Garbutt et al. 2001 n=161 Stalman et al. 1997 n=186 Favors placeboFavors study drug Analysis of Efficacy in Placebo Controlled Trials in Acute Bacterial Sinusitis Kristo et al. 2005 n=82 Lindbaek et al. 1998 n=70 Merenstein et al. 2005 n=135 a b gemi study 010 gemi study 186 gemi 009

24. 24 In Vitro Resistance and Clinical Outcomes  Given uncertainty about magnitude of treatment effect with any drug, correlation between in vitro resistance and clinical outcomes also unclear  Even with susceptible organisms, lack of correlation between microbiological outcomes and clinical outcomes in patients  “Bacterial survival in the maxillary sinus despite a high concentration in the sinus illustrates that MIC values determined in the laboratory do not always mirror the sensitivity of the bacteria to antibiotics in vivo.”  Carenfeldt C et al. Scand J Infect Dis 1975;7:259-64.  Patient often recover clinically despite persistence of organism and differences in potency of antimicrobials does not necessarily translate into differences in clinical outcomes  Carenfeldt C et al. Acta Otolaryngol 1990;10:128-135.

25. 25 Conclusions  Need for demonstration of effect of control drug relative to placebo in noninferiority trial has been noted in regulations since 1980’s  Evaluation of previous placebo controlled trials in ABS does not show reliable and reproducible magnitude of effects of antimicrobials relative to placebo for studying new drugs in clinical trials  Demonstration of noninferiority in acute bacterial sinusitis trials still leaves uncertainty as to whether this demonstrates effectiveness of drugs relative to placebo  Demonstration of effectiveness needed to balance any potential harms of therapy

26. Back Up Slides

27. 27 Substantial Evidence  “Upjohn contends that ‘the totality of materials, which include these 54 articles, the material submitted over the years since these products were certified, and the clinical experience in totality clearly satisfy the substantial evidence’. It says the clinical experience, widespread throughout the world, used by thousands upon thousands of doctors in 750 million doses is a very significant factor.”  Upjohn v. Finch, 1970 Appendix A, p. 12

28. 28 Substantial Evidence  “The Commissioner concludes that Congress itself has described the type of evidence that is suitable to support claims of effectiveness. The claims must be supported by adequate and well controlled investigations. This means that the experimental factors must be so controlled that the effectiveness of an anti-infective drug on the disease process in patients can be compared with the effect of no treatment or of a recognized effective treatment of patients with the same disease or condition.”  Upjohn v. Finch, 1970 Appendix A, p. 12

29. 29 Substantial Evidence  “Upjohn recognizes this fact, but contends that several in vitro studies reported, the mouse study, and the uncontrolled observations reported in the literature, when added to the evidence obtained from the studies in which controls were attempted, raise the quality of the data to the level required by the law.”  Upjohn v. Finch, 1970 Appendix A, p. 12

30. 30 Substantial Evidence  “The in vitro studies are suggestive of some effectiveness…in laboratory experiments utilizing artificially cultured microorganisms as test systems, but because the studies are not at all correlated clinical [trial] experience they cannot be used as a basis for concluding the drugs will have the effectiveness claimed for them when used to treat naturally occurring clinical disease in man.”  Upjohn v. Finch, 1970 Appendix A, p. 12