Handling (and Preventing) Missing Data in RCTs ASENT March 7, 2009 Janet Wittes Statistics Collaborative

2 Topics Missing values: What, me worry? Methods of treatment Methods of prevention Moral: prevention is better than cure Or – “The moral of this tale is ‘care’.”

3 Topics turned into learning objectives Why are missing data important? What is the distinction between  Ignorable and non-ignorable missing?  MAR and NMAR? What are general approaches to analysis? How can we prevent (or minimize) missing?

4 Extent of missing primary outcome data Cardiovascular outcome trial: 1-2% Short-term blood pressure trial: 5-10% 12 week pain trial:20-40% 12 week antipsychotic drug:30-50% 1 year Alzheimer’s Disease:20-50% Source: informal experience

5 Why people don’t care about missing data Many people care about missing data in outcome trials Many do not care about missing data in symptom trials  Who cares about those who don’t take drug? “We know the drug won’t work if you don’t take it” “I am not interested in what happens after people stop.” No evidence that the two groups differ in Pr{missing} We are interested in what we observe – “complete cases” Too hard/expensive to bring back those who stop med

6 So let me convince you to care What many assume (explicitly or implicitly)

7 What many assume

8

9 What statisticians fear (and assume)

10 What statisticians fear (and assume) Those still in study differ from those randomized We can’t characterize who is missing Who is missing differs by group

11 How papers typically report (Hard to ferret out extent and timing of missing) “Missing values imputed using LOCF” (hard to tell that 48% have missing 12 month data)

12 The fundamental dilemma Want to know: Effect of intervention if everyone took it Can learn: Effect of intervention among randomized

13 So what question should we ask  What would have been the effect if people were forced to continue?  What is the effect among people who can tolerate it?  What would the effect have been if it were measured?

14 Language – Little/Rubin Ignorable  Missing completely at random (MCAR): random number selected the missing  Missing at random (MAR) Given observed data, missingness mechanism does not depend on unobserved. Rarely definitively determined from the data at hand Nonignorable: Not missing at random (NMAR)

15 Examples of MAR ( Probability of missing depends only on values of observed) 2 measurements of same variable made at the same time.  If they differ by more than a given amount a third is taken.  3rd measurement missing for those who do not differ by the given amount. Subject removed from trial if condition is not sufficiently controlled (criteria pre-defined)

16 Examples of NMAR ( Probability of missing depends on value of the missing data)  AD patient too demented to come for measurement  Chronic pain: VAS every two weeks Measured at Week 4 – pain still bad Patient feels better at week 5 so doesn’t come at Week 6

17 Rule of thumb: no benefit for missingness Method of imputation shouldn’t give us stronger results than what we would have seen from the complete cases Simple example: More uncertainty but results strengthen. 80 patients: 40/group 20 success in treated and 12 in control P-value (Fisher’s exact) = patients but 20 in each group missing Assume missing data share results in observed P-value now 0.040

18 Handling missing binary outcomes Just ignore the missing observations Impute missing on basis of  Proportion in own group  Best case – all pbo fail; all rx succeed  Worst case – all pbo success; all rx fail  Proportion in placebo group (“not unreasonable guess”)  Proportion in opposite group (“reasonable worst case”) Multiple imputation

19 Problems with usual approach Too many degrees of freedom Some methods overstate effect Some methods understate effect

20 Loss of 4 points in ADAS-Cog Two groups – treated and control 120 per group 40% in placebo; 20% in treated Look at relative risk (<1 is “good”) Missing % equal in both groups

21 Example: loss of 4 points on ADAS-COG

22 Continuous, longitudinal, time-to-event outcomes Just ignore the missing observations Impute missing on basis of data in:  Own group  Combined group  Placebo group  Opposite group (“worst reasonable case”) * Last Observation Carried Forward Baseline Observation Carried Forward Last rank carried forward # Carry forward trajectories Multiple imputation *Proschan et al (2001)., J Stat Planning 96: 155 # O’Brien, Zhang, Bailey (2005). Stat Med 24:34

23 Effect sizeP-value Completers (n=500/650) LOCF BOCF Placebo trajectory Effect size and p-values for pain study

24 Message Analyses produce very different results Can affect  Direction of effect  Effect size

25 Informed consent documents unclear Participation in this study is entirely voluntary. Your treatment and your doctor’s attitude toward you will not be affected should you decide not to participate in this study… You will be asked to return for follow-up visits and to provide follow-up information. If you agree to participate, you may withdraw from the study at any time without affecting any benefits to which you would otherwise be entitled.

26 Permissive protocols encourage missing data  “Drop-outs will not be replaced” Suggests that it would be ok to replace them Suggests that analysis will ignore them  “Expect 10% drop out, therefore increase sample size by 10%”  “The primary analysis will use the intent-to-treat pop” “The ITT pop is defined as all those randomized who…” The ITT pop is defined as the evaluable group

27 Language about withdrawal: an outcome trial The reason that a subject discontinues from the study will be recorded in the Case Report Form. A discontinuation occurs when an enrolled subject ceases participation in the study, regardless of the circumstances, prior to completion of the protocol. … The final evaluation required by the protocol will be performed at the time of study discontinuation.

28 Outcome: continuous measure at week 48 Subjects must be withdrawn from the study (i.e., from any further study medication or study procedure) for the following reasons:  At their own or their legally authorized representative’s request  If, in the investigator’s opinion, continuation in the study would be detrimental to the subject's well-being  Occurrence of an intolerable treatment-emergent adverse event as determined by the investigator and/or the subject  Failure of the subject to return to the study site for scheduled visits  Persistent noncompliance  Pregnancy

29 Prevention of missing values Revise informed consent forms Make protocols less permissive Define outcome measures that don’t allow “success” for missing  E.g., Define measures as success or failure and missing = failure

30 Improved informed consent document Participation in this study is entirely voluntary. Your treatment and your doctor’s attitude toward you will not be affected should you decided not to participate in this study… If you agree to participate, you may withdraw from the study at any time without affecting any benefits to which you would otherwise be entitled. You will be asked to return for follow-up visits and to provide follow-up information even if you are not taking study medication.

31 Protocols Be vigilant about permissive language Distinguish between  Stopping meds  Stopping active visits  Withdrawing consent to be followed passively Understand the importance of full follow-up  (even for those who stop study medication)`

32 Typical language about withdrawal in protocols The reason that a subject discontinues from the study medication will be recorded in the Case Report Form. A discontinuation from the study occurs when an enrolled subject ceases participation a participant in the study dies, is permanently lost to follow-up, or withdraws consent, regardless of the circumstances, prior to completion of the protocol. … An final evaluation required by the protocol will be performed at the time of study discontinuation of study medication.

33 But, if there will be missing data Choose analytic methods that  Do not add false precision  Are reasonably conservative  Are interpretable Recognize need for big increase in sample size