Data Mining in VAERS to Enhance Vaccine Safety Monitoring at the FDA Robert Ball, MD, MPH, ScM Dale Burwen MD, MPH M. Miles Braun, MD, MPH Division of Epidemiology Office of Biostatistics and Epidemiology DIMACS, October 18, 2002

What is the Vaccine Adverse Event Reporting System (VAERS)? National system for surveillance of adverse events after vaccination initiated by National Childhood Vaccine Injury Act 1986 and established 1990 Jointly managed by FDA and CDC Reports received from health professionals, vaccine manufacturers, and the public

Post-licensure Safety Monitoring How we do it VAERS Potentially rapid detection of signal of new safety concern Rarely allows determination of causality Enhanced surveillance Obtain standardized information on reports Controlled studies of hypothesized causal relationships raised in surveillance Communicate results

Detecting unrecognized adverse events Monitoring known reactions Uses of VAERS Detecting unrecognized adverse events Monitoring known reactions Identifying possible risk factors Vaccine lot surveillance

Reported diagnoses are not verified Limitations of VAERS Reported diagnoses are not verified Lack of consistent diagnostic criteria Wide range in data quality Underreporting Inadequate denominator data No unvaccinated control group Usually not possible to assess whether a vaccine caused the reported adverse event

Analysis of VAERS Data Describe characteristics and look for patterns to detect “signals” of adverse events plausibly linked to a vaccine Signals detected through analysis of VAERS data almost always require confirmation through a controlled study

Fundamental Problem in Assessing Spontaneous Reports VAERS ~10-15K reports / year AERS ~20K reports / year (CBER) How can a sensitive system to detect potential product problems not be overloaded and overwhelmed by information to which we have to respond?

“Data Mining” Identify events reported more commonly for one product than others Proportional Reporting Ratios (PRR) Empirical Bayesian Geometric Mean (EBGM) Don’t account for medical knowledge or biases in reporting EBGM algorithm implemented by Lincoln Technologies and PPD Informatics VAERS Data Mining Environment (VDME) PRR algorithm implemented in standard packages (e.g. SAS, STATA) on an ad hoc basis

Proportional Reporting Ratio Compares the adverse event profile of one vaccine to other vaccines Number of reports with Adverse Other Adverse Total Event Y Events Vaccine X a b (a+b) Other vaccines c d (c+d) PRR = [a/(a+b)] / [c/(c+d)]

Proportional Reporting Ratio Compares the adverse event profile of one vaccine to other vaccines Evans has proposed using PRR  2, n  3, and chi square  4 as criteria for selecting pairs for further evaluation

Background: Empirical Bayesian Data Mining Similar to PRR in comparing one vaccine to others Calculates observed and expected frequencies Observed: # of reported events/vaccine Expected: Based on overall frequency of the event for all vaccines, and the total # of reports of the vaccine of interest Identifies cells with very small expected counts – accounts for the instability of the small number

Empirical Bayesian Data Mining Ranks vaccine-event combinations by Empirical Bayesian Geometric Mean (EBGM) Dumouchel has proposed EBGM  2 as criterion to select pairs for further evaluation Multi-item Gamma Poisson Shrinkage (MGPS) algorithm detects multi-way combinations V=vaccine; S=symptom V S V S S V S S S

Rotavirus Vaccine-Intussusception Clinical Trials Signal Wild type RV & intussusception study FDA - licensure CDC - recommendations for use Post-marketing Surveillance (VAERS) Background rates Population-based incidence rates Withdrawal

Rotavirus Vaccine and Intussusception: Signal Emergence

Vaccine Profiles

Anthrax Vaccine: 3-Dimensional Assessment (V-S-S)

Effect of Stratification on EBGM: Anthrax Vaccine and Selected COSTARTS

Selection of “Item Sets” for Empirical Bayesian Data Mining The choice of “Item Sets” influences the Multi-item Gamma Poisson Shrinkage (MGPS) algorithm Currently all combinations (e.g. 2D v-v, s-s, v-s where v=vaccine; s=symptom) If input is restricted to only v-s combinations the magnitude of the EBGM and rank for pairs with small numbers are affected Appropriate selection of Item Sets needs systematic evaluation

Effect of Item Set Selection on EBGM

Challenges What is the best method? Bayesian vs. PRR vs. other? What are criteria for making this decision? How should each method be applied and interpreted? What level of PRR/EBGM? How should statistics be interpreted?

Challenges Should data mining methods be used for automated screening or as analytic tools? Importance of stratification suggests need for intermediate level epi/stat sophistication in users Users need training to properly interpret results Computing resources Substantial effort required for data preparation Software needs user-friendly features to enhance end-user control over: Defining data subsets of interest Stratification Combining adverse event terms Selecting item sets prior to data mining

Challenges Usual method of monitoring for signals: Physician review of individual reports as they arrive Physician review of serious reports Committee review of serious reports at weekly meeting Physician review of monthly numerical summaries of selected vaccines Periodic vaccine or disease-specific surveillance summaries Where does data mining best fit in this process? How can data mining results be best communicated to decision makers, health care providers, and the public?

Next Steps and Future Challenges Continue using PRR and Empirical Bayesian methods in routine practice Systematic comparison of methods Simulation study in collaboration with CDC Large size of AERS database, especially with 2 way and 3 way interactions Is simpler better? e.g. PRR with chi-square Drug dictionary in AERS

Summary Automated summary of a large amount of data Potential for improving usual methods of monitoring for signals Other methods should also be considered Further understanding and experience is needed

Acknowledgments FDA CDC Others Manette Niu, Phil Perucci, other CBER staff, Ana Szarfman and other CDER staff CDC Henry Rolka, Vitali Poole, Penina Haber, John Iskander, and other CDC staff Others Lincoln Technologies, Inc. PPD Informatics William DuMouchel

Selected References Dumouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. American Statistician 1999;53:177-190. Evans SW, Waller PC, Davis S. Use of proportional reporting ratios for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf 2001;10:483-486. Niu MT, Erwin DE, Braun MM. Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. Vaccine 2001;19:4627-4634.