1. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 1 Methodologies and Automated Applications for Post-Marketing Outcomes Surveillance of Medical Devices and Medications Michael E. Matheny, MD, MS NLM Biomedical Informatics Fellow Decision Systems Group, Department of Radiology Brigham & Women’s Hospital, Boston, MA

2. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 2 Outline Post-Marketing Surveillance Background Post-Marketing Surveillance Background Statistical Methodology Development Statistical Methodology Development Computer Application Development Computer Application Development Clinical Examples Clinical Examples Future Directions Future Directions

3. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 3 Background Surveillance Rationale Phase 3 Trials insufficient to ensure adequate safety of medications and devices Phase 3 Trials insufficient to ensure adequate safety of medications and devices –Low frequency events are not detected –Protected populations (pregnant women, children) and more ill populations not represented –Complications delayed by a number of years cannot be detected

4. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 4 Background FDA Medical Devices 1,700 types of devices 1,700 types of devices 500,000 device models 500,000 device models 23,000 manufacturers 23,000 manufacturers

5. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 5 Background FDA New Drug Applications (NDA)

6. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 6 Background Current Post-Marketing Surveillance Combination of mandatory and voluntary adverse event reporting Combination of mandatory and voluntary adverse event reporting –Mandatory reporting by manufacturers and health facilities –Voluntary MedWatch / MAUDE reports by providers and patients 2004 Drug-Related Adverse Event Reports Total422,889 Manufacturer & Facility Reports Manufacturer & Facility Reports401,396 MedWatch MedWatch21,493

7. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 7 Background MAUDE Cypher Reporting Rate 20032004 FDA WarningFDA Warning Cancelled

8. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 8 Background Current Post-Marketing Surveillance ‘Phase 4’ Trials ‘Phase 4’ Trials –Poor Compliance As of March 2006 report, 797 of 1231 (65%) agreed- upon trials had yet to be startedAs of March 2006 report, 797 of 1231 (65%) agreed- upon trials had yet to be started –Barriers Lack of manufacturer incentivesLack of manufacturer incentives –Expensive –Drug already on the market Lack of regulatory enforcementLack of regulatory enforcement

9. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 9 Background Medical Device Recalls Boston Scientific cardiac stent (1998) Boston Scientific cardiac stent (1998) –Balloon rupture at low pressures Guidant cardio-defibrillator (2005) Guidant cardio-defibrillator (2005) –Malfunction due to electrical short Vioxx (2004) Vioxx (2004) –cardiovascular complications Tequin (2006) Tequin (2006) –Hypoglycemia and hyperglycemia

10. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 10 Background FDA Response Increasing demands for Phase 4 trials Increasing demands for Phase 4 trials Legislation to increase quality of adverse event reporting Legislation to increase quality of adverse event reporting Emphasizing trial registries (clinicaltrials.gov) as way to prevent omission of results Emphasizing trial registries (clinicaltrials.gov) as way to prevent omission of results Commissioned IOM report “The Future of Drug Safety” Commissioned IOM report “The Future of Drug Safety”

11. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 11 Background Adverse Event Data Continuum Phase 3 Trials MAUDE / MedWatch Voluntary Registry Mandatory Registry Internal Validity+++--+++++ Generalizibility+/- +++++ Breadth+++++++ Immediacy---+++++ Lack of Bias+++--- ++

12. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 12 Statistical Methods Medical Outcomes Monitoring Using registry data that tracks all patients allows different types of analysis than used in the FDA’s adverse event reporting systems Using registry data that tracks all patients allows different types of analysis than used in the FDA’s adverse event reporting systems No generally accepted methods for monitoring registry data for adverse events No generally accepted methods for monitoring registry data for adverse events –Lack of sufficient discrete electronic data sources to construct registries –Some outcomes are challenging or expensive to track for an entire population

13. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 13 Objective Develop methodologies and implement an automated computer monitoring system to perform outcomes surveillance of registry data for new medical devices and medications Develop methodologies and implement an automated computer monitoring system to perform outcomes surveillance of registry data for new medical devices and medications

14. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 14 Statistical Methods Statistical Process Control

15. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 15 Statistical Methods Bayesian Updating Statistics

16. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 16 Statistical Methods Establishing Baseline Data Primary Data Sources Primary Data Sources –Phase 3 trial data –Post-Marketing data from a closely related medication/device Alternative Data Sources Alternative Data Sources

17. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 17 Statistical Methods Establishing Alerting Thresholds Use number of events and sample size to calculate proportion with confidence intervals Use number of events and sample size to calculate proportion with confidence intervals Typically, medical domains use 95% CI or 1.96 sigma from the point estimate Typically, medical domains use 95% CI or 1.96 sigma from the point estimate

18. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 18 Statistical Methods Establishing Alerting Thresholds SPC BUS

19. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 19 Statistical Methods Establishing Alerting Thresholds

20. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 20 Statistical Methods Establishing Alerting Thresholds Wilson’s method of comparison between two proportions Wilson’s method of comparison between two proportions

21. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 21 Statistical Methods Risk Stratification Allows creating subgroups for separate analyses Allows creating subgroups for separate analyses Single variable Single variable Logistic regression model with scoring thresholds Logistic regression model with scoring thresholds

22. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 22 Application Development DELTA Data Extraction and Longitudinal Time Analysis (DELTA) Data Extraction and Longitudinal Time Analysis (DELTA) Design Goals Design Goals –Generic data import format –Allow both prospective and retrospective analyses –Modular framework to allow sequential addition of statistical methodologies –Multiple alerting methods –Any number of concurrent ongoing analyses

23. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 23 Application Development DELTA Data Dictionary Statistical Modules DELTA Database Clinical Data Entry Source Database Source IT Manager SPC BUS VPN Intranet DELTA Users Web Server

24. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 24 Application Test Data Cypher Drug-Eluting Stent (DES) Setting: Setting: –Brigham & Women’s Hospital (07/2003 – 12/2004) Population: Population: –All patients receiving a drug-eluting stent (2270) Outcome: Outcome: –Post-procedural in-hospital mortality (27) Baseline: Baseline: –University of Michigan Data (1997-1999)

25. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 25 Application Development DELTA

26. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 26 Application Development DELTA

27. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 27 Application Development DELTA

28. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 28 Application Development DELTA

29. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 29 Application Development DELTA

30. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 30 Application Development DELTA

31. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 31 Application Development DELTA

32. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 32 Application Development DELTA

33. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 33 Application Development DELTA

34. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 34 Risk Stratification Potential Solution Incorporate individual risk prediction models in order to adjust for case mix and illness severity Incorporate individual risk prediction models in order to adjust for case mix and illness severity

35. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 35 Possible Risk Prediction Methods Linear / Logistic Regression Linear / Logistic Regression Artificial Neural Networks Artificial Neural Networks Bayesian Networks Bayesian Networks Support Vector Machines Support Vector Machines

36. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 36 LR External Validation Models Model DatesLocationSample NY 19921991NY5827 NY 19971991 – 1994NY62670 CC 19971993 – 1994Cleveland, OH12985 NNE 19991994 – 1996NH, ME, MA, VT15331 MI 20011999 – 2000Detroit, MI10796 BWH 20011997 – 1999Boston, MA 2804 ACC 20021998 – 2000National100253

37. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 37 LR External Validation Setting: Setting: –Brigham & Women’s Hospital (01/2002 – 09/2004) Population: Population: –All patients undergoing percutaneous coronary intervention (5216) Outcome: Outcome: –Post-procedural in-hospital mortality (71)

38. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 38 LR External Validation Results CurveDeathsAUCHL χ 2 HL (p) NY 1992 96.70.8231.1<0.001 NY 1997 61.60.8832.2<0.001 CC 199778.80.8827.8<0.001 NNE 1999 56.20.8945.9<0.001 MI 2001 61.80.8630.4<0.001 BWH 2001 136.10.8939.7<0.001 ACC 2002 49.90.9042.0<0.001 BWH 200470.50.937.610.473

39. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 39 LR External Validation Conclusions Excellent discrimination across all models Excellent discrimination across all models Calibration (Hosmer-Lemeshow) poor for all models but recent local one Calibration (Hosmer-Lemeshow) poor for all models but recent local one Addressed categorical risk stratification by keeping all records in one stratum Addressed categorical risk stratification by keeping all records in one stratum Calibration problems over time limit application, and require exploration of recalibration methods Calibration problems over time limit application, and require exploration of recalibration methods

40. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 40 OPUS (TIMI-16) Setting: Setting: –888 Hospitals in 27 Countries Intervention: Intervention: –Oral IIb-IIIa Inhibitor vs Placebo Population: Population: –Intervention Arm [Both arms identical at 30 days] (6867) Outcome: Outcome: –30 day mortality –Trial stopped early due to elevation in intervention arm Baseline: Baseline: –Control Arm (3421)

41. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 41 OPUS (TIMI-16) 30 Day Mortality Control Intervention PeriodEventsPatientsEvent Rate (%)EventsPatientsEvent Rate (%)p 1000.005 * 20170.00300.0* 30480.01951.11.000 401350.072492.80.102 512680.4115292.10.070 644630.9189511.90.173 757640.7331,5282.20.008 881,0890.7462,1612.10.003 9111,4120.8542,8151.90.004 10161,8050.9783,6102.2<0.001 11212,1731.0924,4102.1<0.001 12332,7011.21095,4472.00.011 13463,3601.41346,7562.00.031 14463,4211.31346,8672.00.031

42. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 42 OPUS (TIMI-16) Alert Summary

43. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 43 CLARITY (TIMI-28) Setting: Setting: –313 Hospitals in 23 Countries Intervention: Intervention: –Oral Anti-Platelet Agent vs Placebo Population Population –Intervention Arm (1751) Outcome: Outcome: –Major Bleeding –DSMB concerned, but trial did not stop early Baseline Baseline –Control Arm (1739)

44. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 44 CLARITY (TIMI-28) Major Bleeding ControlIntervention PeriodEventsPatientsEvent Rate (%)EventsPatientsEvent Rate (%)p 1040.001 * 20130.00120.0* 31273.70230.01.000 42405.01492.00.586 52732.72832.41.000 631162.651264.00.724 741682.471734.00.548 842141.992264.00.262 972762.5102843.50.624 1083372.4123613.30.502 11104242.4154503.30.423

45. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 45 CLARITY (TIMI-28) Major Bleeding ControlIntervention PeriodEventsPatientsEvent Rate (%)EventsPatientsEvent Rate (%)p 12115362.1165542.90.438 13 6392.0186492.80.468 14177762.2227802.80.517 15178921.9239262.50.427 16181,0411.7281,0582.60.180 17201,1921.7311,1952.60.156 18241,3141.8311,3272.30.414 19251,4571.7311,4592.10.500 20261,5841.6321,6062.00.509 21301,7391.7341,7511.90.706

46. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 46 CLARITY (TIMI-28) Alert Summary

47. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 47 OPUS /CLARITY Conclusions SPC performed well in the positive study, but did have some false positive alerts in the negative study SPC performed well in the positive study, but did have some false positive alerts in the negative study LR stratified SPC failed to alert early in the positive study, but performed well in the negative study LR stratified SPC failed to alert early in the positive study, but performed well in the negative study BUS was more specific than SPC in both studies BUS was more specific than SPC in both studies

48. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 48 Sensitivity Analysis Setting: Setting: –Brigham & Women’s Hospital (01/2002 – 12/2004) Population: Population: –All patients undergoing percutaneous coronary intervention (6175) Outcome: Outcome: –Post-procedural major adverse cardiac events (403) DeathDeath Post-Procedural Myocardial InfarctionPost-Procedural Myocardial Infarction Repeat VascularizationRepeat Vascularization Baseline: Baseline: –Arbitrarily set event rates and sample sizes

49. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 49 Sensitivity Analysis Results

50. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 50 Clinical Alert Setting: Setting: –Brigham & Women’s Hospital (01/2002 – 12/2004) Population: Population: –All patients receiving a vascular closure device after percutaneous coronary intervention (3947) Outcome: Outcome: –Retroperitoneal Hemorrhage (25) Baseline: Baseline: –Stanford University Data (2000 – 2004)

51. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 51 Event Rate Elevation

52. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 52 Manual Review Triggered root cause analysis Triggered root cause analysis Manual chart review and multivariable analsysis Manual chart review and multivariable analsysis Final Result: Not related, confounded by indication Final Result: Not related, confounded by indication

53. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 53 Future Work Methodology Address Calibration Concerns Address Calibration Concerns –Recalibration of Logistic Regression models –Development of Machine Learning Risk Prediction Models Address BUS Insensitivity Address BUS Insensitivity –Incorporate data weight decay over time

54. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 54 Future Work Application Outcomes DB DELTA Agent Brigham and Women’s Hospital DELTA reports DELTA Server Massachusetts Data Analysis Center (Mass-DAC) SMTP E-mail Server E-mail alert Outcomes DB DELTA Agent Massachusetts General Hospital Outcomes DB DELTA Agent St. Elizabeth’s Medical Center MA Claims DB MA Death Index

55. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 55 Future Work New Medication Outcomes Surveillance New Medication Outcomes Surveillance Inpatient (versus Outpatient) Inpatient (versus Outpatient) –More frequent monitoring –Higher quality source data –Outcomes easier to capture

56. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 56 Future Work Develop Data Repository Patient Demographics Local Institution DELTA Server Centralized Data Repository Computerized Order Entry Laboratory Progress Notes (NLP) Hospital Billing Medication Administration Radiology IS State Death Index Other Outcomes

57. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 57 Future Work Initial Framework New medication laboratory monitoring protocol New medication laboratory monitoring protocol Standard measures that are most commonly affected in new medications Standard measures that are most commonly affected in new medications –AST, ALT, Creatinine, WBC, Platelets Establish reasonable baselines Establish reasonable baselines –Closely Related medication lab results –Unrelated medication lab results –Expert Panel Estimation

58. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 58 Acknowledgements Mentors Mentors –Lucila Ohno-Machado, MD, PhD –Frederic S. Resnic, MD, MS Collaborators Collaborators –Nipun Arora, MD –Sharon Lise-Normand, PhD –Ewout Steyerberg, PhD Programming Team Programming Team –Richard Cope –Barry Coflan –Atul Tatke Funding Funding –NLM R01-LM-08142 –NLM T15-LM-07092

59. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 59 Michael Matheny, MD MS mmatheny@dsg.harvard.edu Brigham & Women’s Hospital Thorn 309 75 Francis Street Boston, MA 02115 Michael Matheny, MD MS mmatheny@dsg.harvard.edu Brigham & Women’s Hospital Thorn 309 75 Francis Street Boston, MA 02115 The End