1. Data Mining AERS FDA’s (Spontaneous) Adverse Event Reporting System Division of Drug Risk Evaluation Office of Drug Safety Carolyn McCloskey, M.D., M.P.H. Drug Safety and Risk Management Advisory Committee May 18, 2005 Carolyn McCloskey, M.D., M.P.H. Drug Safety and Risk Management Advisory Committee May 18, 2005

2. 2 OutlineOutline 1.Brief History of data mining (DM) activities at the FDA 2.Current Use in the Division of Drug Risk Evaluation (DDRE) – CRADA Application Development - WebVDME Pilot - Examples and Selected Conclusions Other CRADA Activities 3.Future Directions in DDRE Pharmacovigilance 1.Brief History of data mining (DM) activities at the FDA 2.Current Use in the Division of Drug Risk Evaluation (DDRE) – CRADA Application Development - WebVDME Pilot - Examples and Selected Conclusions Other CRADA Activities 3.Future Directions in DDRE Pharmacovigilance

3. 3 Historical Overview February 1998:Office of Women’s Health Grant (Ana Szarfman) March 2003 – July 2005: Cooperative Research & Development Agreement (CRADA) Research in more advanced methodology (Drug-drug interaction & logistic regression modeling)

4. 4 CRADA - Cooperative Research and Development Agreement Web Visual Data Mining Environment (WebVDME) With Lincoln Technologies, Inc. March 2003 – July 2005 Web Visual Data Mining Environment (WebVDME) With Lincoln Technologies, Inc. March 2003 – July 2005

5. 5 CRADA Objectives User-friendly application Web-based environment Performance Evaluations by User Groups Training Continued development and refinement User-friendly application Web-based environment Performance Evaluations by User Groups Training Continued development and refinement

6. 6 Empirical Bayes Geometric Mean (EBGM) An observed/expected score Adjusts for sampling variation (e.g., sample size) No adjustment for reporting bias Allows data stratification in DM software –Standard stratification: gender, age group, year An observed/expected score Adjusts for sampling variation (e.g., sample size) No adjustment for reporting bias Allows data stratification in DM software –Standard stratification: gender, age group, year

7. 7 EB05 – EB95 Interval Interval in which the EBGM score could be found because of sampling variability EB05 is the lower bound EB95 is the upper bound 90% probability of EBGM occurring between EB05 and EB95 Interval in which the EBGM score could be found because of sampling variability EB05 is the lower bound EB95 is the upper bound 90% probability of EBGM occurring between EB05 and EB95

8. 8 Example of Sampling Variability Adjustment (for small numbers) Adverse Event (AE) Observed Count (N) Expected Count (E) Obs/Exp* N/E (RR) EBGM**EB05EB95 Myalgia1,6653344.994.974.785.18 Spinal Osteoarthritis 17 36.164.543.036.60 * Obs = Observed; Exp = expected ** EBGM = Empirical Bayes Geometric Mean - adjusts for sampling variability

9. 9 CRADA Pre-Pilot Performance Evaluations May 2003 – October 2003 WebVDME record retrieval validation with AERS case retrieval Multiple trade & ingredient nomenclature Drug assignment allocations (suspect & concomitant) Duplicate removal logic OIT system performance evaluations WebVDME record retrieval validation with AERS case retrieval Multiple trade & ingredient nomenclature Drug assignment allocations (suspect & concomitant) Duplicate removal logic OIT system performance evaluations

10. 10 CRADA Pilot Medical Safety Evaluators CRADA Pilot Medical Safety Evaluators Evaluated data mining scores for drugs and biologics Indication vs. new signal Labeled vs. unlabeled Innocent bystanders or concomitant medications Drug names Safety Evaluators’ ease of use Evaluated data mining scores for drugs and biologics Indication vs. new signal Labeled vs. unlabeled Innocent bystanders or concomitant medications Drug names Safety Evaluators’ ease of use

11. 11 CRADA Pilot Epidemiologists Evaluated Temporal trends Drug name selections Suspect & Concomitant selections Stratification Signal strengths Epidemiologists’ ease of use Evaluated Temporal trends Drug name selections Suspect & Concomitant selections Stratification Signal strengths Epidemiologists’ ease of use

12. 12 Pilot Examples New vs. Old Drug EBGM Rankings & Confidence Intervals New vs. Old Drug EBGM Rankings & Confidence Intervals

13. 13 New Drug (1 Year) EBGM (EB05-EB95) EB05 =2.0

14. 14 OLDER DRUG (>10 Years) EBGM (EB05-EB95) EB05 =2.0

15. 15 Selected Pilot Conclusions - 1 WebVDME DM - Statistical tool assists in identifying unusual patterns with AERS data but –! Patterns Need Interpretation! WebVDME DM - Statistical tool assists in identifying unusual patterns with AERS data but –! Patterns Need Interpretation!

16. 16 Selected Pilot Conclusions - 2 Knowledge of data in database imperative to interpret –Clinical & pharmacologic activities of drug –Other - reporting disproportionalities which also reflect limitations of AERS data Knowledge of data in database imperative to interpret –Clinical & pharmacologic activities of drug –Other - reporting disproportionalities which also reflect limitations of AERS data

17. 17 AERS DATABASE LIMITATIONS

18. 18 Continuing CRADA Activities - DDRE March 2004 - Present Access by interested Reviewers to WebVDME –Training –Application Refinements addressing – Technical problems identified – Customization by user needs Access by interested Reviewers to WebVDME –Training –Application Refinements addressing – Technical problems identified – Customization by user needs

19. 19 Summary – 1 DM Signals in DDRE Assist in prioritizing evaluations of case series Identify associations, NOT a cause or degree of risk Reflect limitations of data Assist in prioritizing evaluations of case series Identify associations, NOT a cause or degree of risk Reflect limitations of data

20. 20 Summary – 2 DM Signals in DDRE Threshold a compromise between sensitivity and specificity (false positives & negatives)  Absence of a DM signal ≠ absence of a drug-event association  Magnitude of DM scores ≠ magnitude of risk Always require clinical case report and reporting bias evaluation Threshold a compromise between sensitivity and specificity (false positives & negatives)  Absence of a DM signal ≠ absence of a drug-event association  Magnitude of DM scores ≠ magnitude of risk Always require clinical case report and reporting bias evaluation

21. 21 Future Directions of DM Prospective signal detection Parallel use with traditional pharmacovigilance methods in DDRE Continued research in more advanced methodology (Drug-drug interaction & logistic regression modeling) Prospective signal detection Parallel use with traditional pharmacovigilance methods in DDRE Continued research in more advanced methodology (Drug-drug interaction & logistic regression modeling)

22. 22 AcknowledgmentsAcknowledgments Division of Drug Risk Evaluation –Rita Ouellet-Hellstrom, Ph.D., M.P.H. –Mary Willy, Ph.D. –Mark Avigan, M.D., C.M. Division of Drug Risk Evaluation –Rita Ouellet-Hellstrom, Ph.D., M.P.H. –Mary Willy, Ph.D. –Mark Avigan, M.D., C.M.

23. THANK YOU