1. U.S. Food and Drug Administration Notice: Archived Document The content in this document is provided on the FDA’s website for reference purposes only. It was current when produced, but is no longer maintained and may be outdated.

2. Statistical Criteria for Establishing Safety and Efficacy of Allergenic Products Tammy Massie, PhD Mathematical Statistician Team Leader Bacterial, Parasitic and Allergenic Product Team

3. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 3 Outline Goal Present and Discuss Select Relevant Statistical Concepts Statistical Concepts Applied to Allergenic Products Consideration of Standards Conclusion Note: All graphics presented are simulated for illustrative purposes only

4. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 4 Goal Introduce statistical concepts including types of data collected and associated analyses relevant in clinical studies designed to examine the safety and efficacy of allergenic products.

5. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 5 Outline Goal Select Relevant Statistical Concepts Statistical Concepts Applied to Allergenic Products Consideration of Standards Conclusion

6. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 6 Select Relevant Statistical Concepts Types of Data –Continuous –Longitudinal –Survival Important Measures –Central Tendency –Spread Bias Covariates Missing Values Hypothesis Testing/Confidence Intervals

7. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 7 Normal Distribution Curves- Continuous Response Data Same mean Different variance Smaller variance (σ=0.5) Larger variance (σ=2) Medium variance (σ=1) Observed Data Observed Response

8. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 8 Comparison of Groups Different Mean, Similar Variance Ideal Situation- Clear separation of groups Different Mean, Similar Variance Limited separation of groups Different Mean, Different Variance Limited separation of groups Inadequate sample size/lack power Improper subject selection Inappropriate time frame

9. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 9 Types of Data- Multivariate Continuous Response Data Explanatory Variable Response Variable

10. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 10 Types of Data- Continuous Response Data Explanatory Variable Response Variable

11. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 11 Types of Data- Longitudinal Time Response over Time Response

12. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 12 Types of Data- “Survival” (time until event occurs) Proportion Time in Days

13. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 13 Important Issues to Consider Impediments –Bias –Confounding –Covariates –Missing Values Some Solutions –Well Designed (pre-specified) Study Randomization: promotes balance regarding covariates Stratification Adequate sample size/Well powered Appropriate –Endpoints: clinically meaningful, practical, validated, etc. –Timeframe: consider allergy season, frequency of data collection, etc.

14. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 14 Outline Goal Select Relevant Statistical Concepts Statistical Concepts Applied to Allergenic Products Consideration of Standards Conclusion

15. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 15 Allergenic Products Data Collected: Natural Exposure or Chamber Study Safety Endpoints –Adverse Events Local Reactions Systemic Reactions Pre-Specified Efficacy Endpoints –Symptoms (*) –Use of Rescue Medication (*) –Quality of Life *can be combined

16. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 16 Illustration of Potential Data- Separated into Two Groups Combined Medication and Symptom Score Time (in weeks) Initiation of Allergy Season Key

17. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 17 Presence of Allergen and Symptom Scores Combined Medication & Symptom Score Time (in weeks) Pollen Count (grains/m 3 )

18. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 18 Illustration of Potential Data Individual Responses Combined Medication and Symptom Score Time (in weeks)

19. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 19 Illustration of Potential Data- Mean and 95% Confidence Interval of Two Groups Time (in weeks) Combined Medication and Symptom Score

20. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 20 Illustration of Potential Data- Mean and 95% Confidence Interval of Two Groups Difference between the mean of two groups Combined Medication and Symptom Score Time (in weeks)

21. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 21 Illustration of Potential Data- Mean and 95% Confidence Interval of Two Groups Combined Medication and Symptom Score Time (in weeks) WHERE THE ACTUAL MEANS OF EACH GROUP COULD BE….

22. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 22 Illustration of Potential Data- 95% Confidence Interval of the Two Groups The 95% confidence interval of the differences between two groups Time (in weeks) Combined Medication and Symptom Score

23. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 23 Comparing Groups Examples have illustrated –How data can scatter –There can be different degrees of separation between groups Ideally provides reasonable replication of real and meaningful differences –Why examining means (& standard deviations) alone may not sufficient Differences between groups include variability that must be accounted for

24. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 24 Differences between Groups with 95% Confidence Intervals (CI) - Δ 0 δ Key 95% CI Difference Mean Difference

25. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 25 Examination of 95% CI of Differences between Treatment Groups Non-inferiority Margin Note: Δ or δ must be pre- specified and depends on 1)Type of Study 2)Comparator 3)Anticipated Safety 4)Efficacy/Effectiveness 5)Benefit/Risk Profile - Δ 0 δ + Clinically Meaningful Margin

26. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 26 -Δ 0 δ Difference between Groups with 95% Confidence Intervals

27. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 27 Summary of Allergenic Example A lowerbound of the 95% CI greater than a pre- specified threshold (δ) ensures reproducible statistical significance that translates into clinically meaningful difference Example for illustrative purposes examined only single time point; however, this should be extended to an agreeable timeframe using appropriate longitudinal analysis methodologies –Selection of timeframe should consider the potential for missing values

28. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 28 Outline Goal Present and Discuss Select Relevant Statistical Concepts Discuss Statistical Concepts Applied to Allergenic Products Provide Consideration of Standards Conclusion

29. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 29 Standards for Consideration Measuring Differences –Lower (or Upper) Bound of a Confidence Interval –Pre-defined Difference between Groups based on a specific Value % Change

30. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 30 Standards for Consideration (cont) P-value –Probability of observing a result as extreme or more extreme than the one observed, given that the null hypothesis is true –May not be adequate alone Confidence Interval –Gives an estimated range of values which is likely to include the unknown population parameter, the estimated range being calculated from a given set of sample data. –Provides a range of the magnitude of effect and an estimate of its reliability

31. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 31 Outline Goal Present and Discuss Select Relevant Statistical Concepts Statistical Concepts Applied to Allergenic Products Consideration of Standards Conclusion

32. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 32 Conclusion Establishment of a meaningful difference as a target in the protocol is critical and should be agreed upon before the study is implemented.

33. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 33 Acknowledgement and Thanks A. Dale Horne, Dr.P.H. Estelle Russek-Cohen, Ph.D. Henry Hsu, Ph.D. Ronald L. Rabin, MD Jay E. Slater, MD Drusilla Burns, Ph.D. Paul Richman, Ph.D. Colleen Sweeney, CDR, USPHS Elizabeth Valenti, LCDR, USPHS

34. 13th International Paul-Ehrlich-Seminar, Sept. 15, 2011 34 Questions?