1. Data Mining in VAERS to Enhance Vaccine Safety Monitoring at the FDA
Robert Ball, MD, MPH, ScM
Dale Burwen MD, MPH
M. Miles Braun, MD, MPH
Division of Epidemiology
Office of Biostatistics and Epidemiology
DIMACS, October 18, 2002

2. What is the Vaccine Adverse Event Reporting System (VAERS)?
National system for surveillance of adverse events after vaccination initiated by National Childhood Vaccine Injury Act 1986 and established 1990
Jointly managed by FDA and CDC
Reports received from health professionals, vaccine manufacturers, and the public

3. Post-licensure Safety Monitoring
How we do it
VAERS
Potentially rapid detection of signal of new safety concern
Rarely allows determination of causality
Enhanced surveillance
Obtain standardized information on reports
Controlled studies of hypothesized causal relationships raised in surveillance
Communicate results

4. Detecting unrecognized adverse events Monitoring known reactions
Uses of VAERS
Detecting unrecognized adverse events
Monitoring known reactions
Identifying possible risk factors
Vaccine lot surveillance

5. Reported diagnoses are not verified
Limitations of VAERS
Reported diagnoses are not verified
Lack of consistent diagnostic criteria
Wide range in data quality
Underreporting
Inadequate denominator data
No unvaccinated control group
Usually not possible to assess whether a vaccine caused the reported adverse event

6. Analysis of VAERS Data
Describe characteristics and look for patterns to detect “signals” of adverse events plausibly linked to a vaccine
Signals detected through analysis of VAERS data almost always require confirmation through a controlled study

7. Fundamental Problem in Assessing Spontaneous Reports
VAERS ~10-15K reports / year
AERS ~20K reports / year (CBER)
How can a sensitive system to detect potential product problems not be overloaded and overwhelmed by information to which we have to respond?

8. “Data Mining”
Identify events reported more commonly for one product than others
Proportional Reporting Ratios (PRR)
Empirical Bayesian Geometric Mean (EBGM)
Don’t account for medical knowledge or biases in reporting
EBGM algorithm implemented by Lincoln Technologies and PPD Informatics
VAERS Data Mining Environment (VDME)
PRR algorithm implemented in standard packages (e.g. SAS, STATA) on an ad hoc basis

9. Proportional Reporting Ratio
Compares the adverse event profile of one vaccine to other vaccines
Number of reports with
Adverse Other Adverse Total
Event Y Events
Vaccine X a b (a+b)
Other vaccines c d (c+d)
PRR = [a/(a+b)] / [c/(c+d)]

10. Proportional Reporting Ratio
Compares the adverse event profile of one vaccine to other vaccines
Evans has proposed using PRR  2, n  3, and chi square  4 as criteria for selecting pairs for further evaluation

11. Background: Empirical Bayesian Data Mining
Similar to PRR in comparing one vaccine to others
Calculates observed and expected frequencies
Observed: # of reported events/vaccine
Expected: Based on overall frequency of the event for all vaccines, and the total # of reports of the vaccine of interest
Identifies cells with very small expected counts – accounts for the instability of the small number

12. Empirical Bayesian Data Mining
Ranks vaccine-event combinations by Empirical Bayesian Geometric Mean (EBGM)
Dumouchel has proposed EBGM  2 as criterion to select pairs for further evaluation
Multi-item Gamma Poisson Shrinkage (MGPS) algorithm detects multi-way combinations
V=vaccine; S=symptom
V S
V S S
V S S S

13. Rotavirus Vaccine-Intussusception
Clinical Trials Signal
Wild type RV & intussusception study
FDA - licensure
CDC - recommendations for use
Post-marketing Surveillance (VAERS)
Background rates
Population-based incidence rates
Withdrawal

14. Rotavirus Vaccine and Intussusception: Signal Emergence

15. Vaccine Profiles

16. Anthrax Vaccine: 3-Dimensional Assessment (V-S-S)

17. Effect of Stratification on EBGM: Anthrax Vaccine and Selected COSTARTS

18. Selection of “Item Sets” for Empirical Bayesian Data Mining
The choice of “Item Sets” influences the Multi-item Gamma Poisson Shrinkage (MGPS) algorithm
Currently all combinations (e.g. 2D v-v, s-s, v-s where v=vaccine; s=symptom)
If input is restricted to only v-s combinations the magnitude of the EBGM and rank for pairs with small numbers are affected
Appropriate selection of Item Sets needs systematic evaluation

19. Effect of Item Set Selection on EBGM

20. Challenges What is the best method?
Bayesian vs. PRR vs. other?
What are criteria for making this decision?
How should each method be applied and interpreted?
What level of PRR/EBGM?
How should statistics be interpreted?

21. Challenges
Should data mining methods be used for automated screening or as analytic tools?
Importance of stratification suggests need for intermediate level epi/stat sophistication in users
Users need training to properly interpret results
Computing resources
Substantial effort required for data preparation
Software needs user-friendly features to enhance end-user control over:
Defining data subsets of interest
Stratification
Combining adverse event terms
Selecting item sets prior to data mining

22. Challenges Usual method of monitoring for signals:
Physician review of individual reports as they arrive
Physician review of serious reports
Committee review of serious reports at weekly meeting
Physician review of monthly numerical summaries of selected vaccines
Periodic vaccine or disease-specific surveillance summaries
Where does data mining best fit in this process?
How can data mining results be best communicated to decision makers, health care providers, and the public?

23. Next Steps and Future Challenges
Continue using PRR and Empirical Bayesian methods in routine practice
Systematic comparison of methods
Simulation study in collaboration with CDC
Large size of AERS database, especially with 2 way and 3 way interactions
Is simpler better? e.g. PRR with chi-square
Drug dictionary in AERS

24. Summary Automated summary of a large amount of data
Potential for improving usual methods of monitoring for signals
Other methods should also be considered
Further understanding and experience is needed

25. Acknowledgments FDA CDC Others
Manette Niu, Phil Perucci, other CBER staff, Ana Szarfman and other CDER staff
CDC
Henry Rolka, Vitali Poole, Penina Haber, John Iskander, and other CDC staff
Others
Lincoln Technologies, Inc.
PPD Informatics
William DuMouchel

26. Selected References
Dumouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. American Statistician 1999;53:
Evans SW, Waller PC, Davis S. Use of proportional reporting ratios for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf 2001;10:
Niu MT, Erwin DE, Braun MM. Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. Vaccine 2001;19: