Safety, Can You Paradigm? A Statistical Lament Janet Turk Wittes Statistics Collaborative

2 Harms identified late fenflurmine-phentermine (Fen Phen) Rofecoxib (Vioxx) Troglitazone (Rezulin) HRT(Premarin and PremPro) Celecoxib (Celebrex) Telithromycin (Ketek) Rosiglitazone(Avandia) Antidepressants, anti-epileptics….

3 Could we have identified these harms earlier? Troglitazone (Rezulin) -removed from market in 2000  Lots of liver abnormalities  Severe toxicities noted in 1997  Other equally effective drugs didn’t have same problems

4 Could we have identified these harms earlier? Troglitazone (Rezulin) -removed from market in 2000 Rofecoxib (Vioxx) -removed from market in 2004  Every study showed excess heart attack  Attributed to benefit of naproxen

5 Could we have identified these harms earlier? Troglitazone (Rezulin) – removed from market in 2000 Rofecoxib (Vioxx) -removed from market in 2004 HRT(Premarin/PremPro)-major label change 2006  Heart attacks in Puerto Rican girls on oral contraception -1960’s  Men on estrogens had higher event rates – 1970’s

6 Could we have identified these harms earlier? Troglitazone (Rezulin) – removed from market in 2000 Rofecoxib (Vioxx) -removed from market in 2004 HRT(Premarin and PremPro)-label change 2006 Celecoxib (Celebrex) – paper published 2005 Telithromycin (Ketek) – major label change 2007

7 “CELEBRATE :: CELEBREX” December 2004

8 How we statisticians help to save drugs We find safety boring

9 For efficacy we think hard about…  Outcomes  Population to study  Protocol  Analysis of primary outcome  Control of Type I error rate  Other outcomes  Missing data  Sensitivity analyses

10 How we statisticians save drugs Because we find safety boring….  We don’t look at preclinical and early Phase data  We don’t ask about Chemistry Biology What PK/PD studies show Safety part of analysis plan is an afterthought

11 How the statistical  -police protect drugs We test hypotheses Put events in correct body system Give precise definitions No data dredging Too many type 1 errors if we dredge

12 And we divide and… conquer obfuscate

13 e.g. Neuropathy EventTC Neuropathic pain10 Neuropathy10 Neuropathy NOS52 Neuropathy peripheral20 …21 … …

14 e.g. Neuropathy EventTC … Parathesia32 Parathesia NOS40 Parathesia other01 … Peripheral motor neuropathy60 Peripheral sensory neuropathy32

15 True(ish) data from a coxib C T Cardiac disorders4246 Respiratory3329 Vascular disorders 7 9

16 True(ish) data from a coxib C T Cardiac disorders4246 Angina2 2 Angina aggravated0 2 Angina unstable0 3 … Cardiac arrest0 1 Cardiac failure congest2 0 Coronary artery disease4 7 … Myocardial infarction510

17 True(ish) data from a coxib Respiratory 3329 Dyspnea13 Vascular disorders79 Cerebral infarction01 Pulmonary embolism02 TIA20

18 If you combined… No. of people with at least one serious thromboembolic event or evidence of heart failure PlaceboCoxib 16 27

19 Other ways to save drugs Modified Daley’s Rule: Censor early and often

20 e.g., Rofecoxib- short follow-up

21 Through 36 months

22 With denominators (Bresalier et al. NEJM :1092) (And see Adam Boyd’s poster!)

23 Known or suspected adverse events Monitor them Look at events, their (near) synonyms, labs  Are they real?  Are they too frequent?

24 Hierarchical multiplicity Think of biology Order hierarchy by decreasing  Biological plausibility  Objectivity Look for monotone decreasing hazard ratio

25 Which dose of celecoxib do you want?

26 APC Study (Placebo vs high dose) Outcomen CV death 6 +MI 19 +Stroke 26 +CHF 29 +Angina 34 +CV procedure Other CV62

27 Adenoma Prevention with Celecoxib (APC) Study HR CV death 5.1 +MI 3.8 +Stroke 3.4 +CHF 3.2 +Angina2.1 +CV procedure Other CV1.1

28 APC Study CV death 5.1 ( 0.6, 43.2) +MI 3.8 ( 1.3, 11.4) +Stroke 3.4 ( 1.4, 8.3) +CHF 3.2 ( 1.4, 7.4) +Angina2.1 ( 1.0, 4.3) +CV procedure1.7 ( 1.0, 3.1) Other CV1.1 ( 0.7, 1.8)

29 APC Study CV death 5.1 ( 0.6, 43.2) MI 3.8 ( 1.3, 11.4) Stroke 3.4 ( 1.4, 8.3) CHF 3.2 ( 1.4, 7.4) Angina2.1 ( 1.0, 4.3) CV procedure1.7 ( 1.0, 3.1) Other CV1.1 ( 0.7, 1.8)0.7 Solomon (2006). Circulation 114:1028

30 Unknown harms: usual approach Respond by  Agonizing  Checking informed consent document  Asking for more frequent looks  Asking for more thorough analyses Worry about falsely discovered harm

31 Sentinel events Identify Follow in the next patients Invent formal statistical methods

32 Single sentinel event Childhood vaccine 30 day follow-up for serious adverse events 1 death occurred DSMB: did the vaccine cause the death?

33 Women’s Health Initiative Early in the trial, DSMB noted:  Increase in stroke  Increase in pulmonary embolism  Increase in myocardial infarction Possible sentinel events  Myocardial infarction  The big meanies: stroke, PE, MI

34 Proposal 1.Identify sentinel event (or cluster or rate) 2.Monitor for subsequent occurrence(s)  Have reasonable power  Be statistically unbiased (exclude sentinel)  Type 1 error rate may be large (~0.2) Lachenbruch, Wittes: 2007

35 Safety report sample –abnormal lab values Time A B Total Point [N= 150] [N= 148] [N= 298] _______________________________________________________________ SCREENING RANDOM WEEK WEEK WEEK WEEK WEEK WEEK WEEK ___________________________________________________________________

36 But wait! You also get: Time A B Total Point [N= 150] [N= 148] [N= 298] _______________________________________________________________ SCREENING 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) RANDOM 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 2 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 3 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 4 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 5 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 6 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 7 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 8 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) _______________________________________________________________

37 And 150 pages of where’s Waldo Time A B Total Point [N= 150] [N= 148] [N= 298] _______________________________________________________________ SCREENING 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) RANDOM 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 2 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 3 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 4 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 5 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 6 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 7 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) WEEK 8 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) EARLY TERM 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) UNSCHEDULED 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) _______________________________________________________________

38 And if this isn’t enough… Change from baseline where missing is counted as zero (change in HR=64????) Values out of temporal order Lots and lots of decimal places P-values to 3 and 4 significant digits Etc., etc. etc.

39 We need to change our habits Current statistical approach  One variable at a time  Template applied to all studies  No wonder the docs don’t ask us to work with them! Simple change in attitude  Safety parameters aren’t separable  Focus first from biological insights and previous hints  Then scan the other variables  Then refocus

40 Conclusions Worry about multiplicity, but not too much  Listen to Joe Heyse’s talk this afternoon Beware the censor-happy protocol and analysis Don’t be too much the statistician But don’t forget randomness