Download presentation
1
Missing Data in Randomized Control Trials
John W. Graham The Prevention Research Center and Department of Biobehavioral Health Penn State University IES/NCER Summer Research Training Institute, July 2008
2
Sessions in Four Parts (1) Introduction: Missing Data Theory
(2) Attrition: Bias and Lost Power (3) A brief analysis demonstration Multiple Imputation with NORM and Proc MI (4) Hands-on Intro to Multiple Imputation
3
Recent Papers Graham, J. W., Cumsille, P. E., & Elek-Fisk, E. (2003). Methods for handling missing data. In J. A. Schinka & W. F. Velicer (Eds.). Research Methods in Psychology (pp. 87_114). Volume 2 of Handbook of Psychology (I. B. Weiner, Editor-in-Chief). New York: John Wiley & Sons. Graham, J. W., (2009, in press). Missing data analysis: making it work in the real world. Annual Review of Psychology, 60. Collins, L. M., Schafer, J. L., & Kam, C. M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330_351. Schafer, J. L., & Graham, J. W. (2002). Missing data: our view of the state of the art. Psychological Methods, 7,
4
Part 1: A Brief Introduction to Analysis with Missing Data
5
Problem with Missing Data
Analysis procedures were designed for complete data
6
Solution 1 Design new model-based procedures
Missing Data + Parameter Estimation in One Step Full Information Maximum Likelihood (FIML) SEM and Other Latent Variable Programs (Amos, LISREL, Mplus, Mx, LTA)
7
Solution 2 Data based procedures Two Steps
e.g., Multiple Imputation (MI) Two Steps Step 1: Deal with the missing data (e.g., replace missing values with plausible values Produce a product Step 2: Analyze the product as if there were no missing data
8
FAQ Aren't you somehow helping yourself with imputation?
9
NO. Missing data imputation . . .
does NOT give you something for nothing DOES let you make use of all data you have . . .
10
FAQ Is the imputed value what the person would have given?
11
NO. When we impute a value . .
We do not impute for the sake of the value itself We impute to preserve important characteristics of the whole data set . . .
12
We want . . . unbiased parameter estimation
e.g., b-weights Good estimate of variability e.g., standard errors best statistical power
13
Causes of Missingness Ignorable Non-Ignorable
MCAR: Missing Completely At Random MAR: Missing At Random Non-Ignorable MNAR: Missing Not At Random
14
MCAR (Missing Completely At Random)
MCAR 1: Cause of missingness completely random process (like coin flip) MCAR 2: Cause uncorrelated with variables of interest Example: parents move No bias if cause omitted
15
MAR (Missing At Random)
Missingness may be related to measured variables But no residual relationship with unmeasured variables Example: reading speed No bias if you control for measured variables
16
MNAR (Missing Not At Random)
Even after controlling for measured variables ... Residual relationship with unmeasured variables Example: drug use reason for absence
17
MNAR Causes The recommended methods assume missingness is MAR
But what if the cause of missingness is not MAR? Should these methods be used when MAR assumptions not met? . . .
18
YES! These Methods Work! Suggested methods work better than “old” methods Multiple causes of missingness Only small part of missingness may be MNAR Suggested methods usually work very well
19
Methods: "Old" vs MAR vs MNAR
MAR methods (MI and ML) are ALWAYS at least as good as, usually better than "old" methods (e.g., listwise deletion) Methods designed to handle MNAR missingness are NOT always better than MAR methods
20
Analysis: Old and New
21
Old Procedures: Analyze Complete Cases (listwise deletion)
may produce bias you always lose some power (because you are throwing away data) reasonable if you lose only 5% of cases often lose substantial power
22
Analyze Complete Cases (listwise deletion)
very common situation only 20% (4 of 20) data points missing but discard 80% of the cases
23
Other "Old" Procedures Pairwise deletion Mean substitution
May be of occasional use for preliminary analyses Mean substitution Never use it Regression-based single imputation generally not recommended ... except ...
24
Recommended Model-Based Procedures
Multiple Group SEM (Structural Equation Modeling) Latent Transition Analysis (Collins et al.) A latent class procedure
25
Recommended Model-Based Procedures
Raw Data Maximum Likelihood SEM aka Full Information Maximum Likelihood (FIML) Amos (James Arbuckle) LISREL 8.5+ (Jöreskog & Sörbom) Mplus (Bengt Muthén) Mx (Michael Neale)
26
Amos 7, Mx, Mplus, LISREL 8.8 Structural Equation Modeling (SEM) Programs In Single Analysis ... Good Estimation Reasonable standard errors Windows Graphical Interface
27
Limitation with Model-Based Procedures
That particular model must be what you want
28
Recommended Data-Based Procedures
EM Algorithm (ML parameter estimation) Norm-Cat-Mix, EMcov, SAS, SPSS Multiple Imputation NORM, Cat, Mix, Pan (Joe Schafer) SAS Proc MI LISREL 8.5+ Amos 7
29
EM Algorithm Expectation - Maximization Alternate between
E-step: predict missing data M-step: estimate parameters Excellent (ML) parameter estimates But no standard errors must use bootstrap or multiple imputation
30
Multiple Imputation Problem with Single Imputation: Too Little Variability Because of Error Variance Because covariance matrix is only one estimate
31
Too Little Error Variance
Imputed value lies on regression line
32
Imputed Values on Regression Line
33
Restore Error . . . Add random normal residual
34
Regression Line only One Estimate
35
Covariance Matrix (Regression Line) only One Estimate
Obtain multiple plausible estimates of the covariance matrix ideally draw multiple covariance matrices from population Approximate this with Bootstrap Data Augmentation (Norm) MCMC (SAS 8.2, 9)
36
Data Augmentation stochastic version of EM EM Data Augmentation
E (expectation) step: predict missing data M (maximization) step: estimate parameters Data Augmentation I (imputation) step: simulate missing data P (posterior) step: simulate parameters
37
Data Augmentation Parameters from consecutive steps ...
too related i.e., not enough variability after 50 or 100 steps of DA ... covariance matrices are like random draws from the population
38
Multiple Imputation Allows:
Unbiased Estimation Good standard errors provided number of imputations (m) is large enough too few imputations reduced power with small effect sizes
39
ρ From Graham, J.W., Olchowski, A.E., & Gilreath, T.D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8,
40
Part 2 Attrition: Bias and Loss of Power
41
Relevant Papers Graham, J.W., (in press). Missing data analysis: making it work in the real world. Annual Review of Psychology, 60. Collins, L. M., Schafer, J. L., & Kam, C. M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330_351. Hedeker, D., & Gibbons, R.D. (1997). Application of random-effects pattern-mixture models for missing data in longitudinal studies, Psychological Methods, 2, Graham, J.W., & Collins, L.M. (2008). Using Modern Missing Data Methods with Auxiliary Variables to Mitigate the Effects of Attrition on Statistical Power. Annual Meetings of the Society for Prevention Research, San Francisco, CA. (available upon request) Graham, J.W., Palen, L.A., et al. (2008). Attrition: MAR & MNAR missingness, and estimation bias. Annual Meetings of the Society for Prevention Research, San Francisco, CA. (available upon request)
42
What if the cause of missingness is MNAR?
Problems with this statement MAR & MNAR are widely misunderstood concepts I argue that the cause of missingness is never purely MNAR The cause of missingness is virtually never purely MAR either.
43
MAR vs MNAR "Pure" MCAR, MAR, MNAR never occur in field research
Each requires untenable assumptions e.g., that all possible correlations and partial correlations are r = 0
44
MAR vs MNAR Better to think of MAR and MNAR as forming a continuum
MAR vs MNAR NOT even the dimension of interest
45
MAR vs MNAR: What IS the Dimension of Interest?
How much estimation bias? when cause of missingness cannot be included in the model
46
Bottom Line ... All missing data situations are partly MAR and partly MNAR Sometimes it matters ... bias affects statistical conclusions Often it does not matter bias has tolerably little effect on statistical conclusions (Collins, Schafer, & Kam, Psych Methods, 2001)
47
Methods: "Old" vs MAR vs MNAR
MAR methods (MI and ML) are ALWAYS at least as good as, usually better than "old" methods (e.g., listwise deletion) Methods designed to handle MNAR missingness are NOT always better than MAR methods
48
Yardstick for Measuring Bias
Standardized Bias = (average parameter est) – (population value) X 100 Standard Error (SE) |bias| < 40 considered small enough to be tolerable
49
A little background for Collins, Schafer, & Kam (2001; CSK)
Example model of interest: X Y X = Program (prog vs control) Y = Cigarette Smoking Z = Cause of missingness: say, Rebelliousness (or smoking itself) Factors to be considered: % Missing (e.g., % attrition) rYZ rZR
50
rYZ Correlation between cause of missingness (Z)
e.g., rebelliousness (or smoking itself) and the variable of interest (Y) e.g., Cigarette Smoking
51
rZR Correlation between
cause of missingness (Z) e.g., rebelliousness (or smoking itself) and missingness on variable of interest e.g., Missingness on the Smoking variable Missingness on Smoking (often designated: R or RY) Dichotomous variable: R = 1: Smoking variable not missing R = 0: Smoking variable missing
52
CSK Study Design (partial)
Simulations manipulated amount of missingness (25% vs 50%) rZY (r = .40, r = .90) rZR held constant r = .45 with 50% missing (applies to "MNAR-Linear" missingness)
53
CSK Results (partial) (MNAR Missingness)
25% missing, rYZ = no problem 25% missing, rYZ = no problem 50% missing, rYZ = no problem 50% missing, rYZ = problem * "no problem" = bias does not interfere with inference These Results apply to the regression coefficient for X Y with "MNAR-Linear" missingness (see CSK, 2001, Table 2)
54
But Even CSK Results Too Conservative
Not considered by CSK: rZR In their simulation rZR = .45 Even with 50% missing and rYZ = .90 bias can be acceptably small Graham et al. (2008): Bias acceptably small (standardized bias < 40) as long as rZR < .24
55
rZR < .24 Very Plausible
Study rZR _________ _____ HealthWise (Caldwell, Smith, et al., 2004) .106 AAPT (Hansen & Graham, 1991) .093 Botvin Botvin Botvin All of these yield standardized bias < 10 (estimated)
56
Attrition in HealthWise
Best (pretest) predictors of Attrition in HW Gender Lifetime Sex Lifetime Alcohol Use Lifetime Smoking Lifetime Dagga Use
57
CSK and Follow-up Simulations
Results very promising Suggest that even MNAR biases are often tolerably small But these simulations still too narrow
58
Beginnings of a Taxonomy of Attrition
Causes of Attrition on Y (main DV) Case 1: not Program (P), not Y, not PY interaction Case 2: P only Case 3: Y only (CSK scenario) Case 4: P and Y only
59
Beginnings of a Taxonomy of Attrition
Causes of Attrition on Y (main DV) Case 5: PY interaction only Case 6: P + PY interaction Case 7: Y + PY interaction Case 8: P, Y, and PY interaction
60
Taxonomy of Attrition Cases 1-4 Cases 5-8 often little or no problem
Jury still out (more research needed) Very likely not as much of a problem as previously though Use diagnostics to shed light
61
Use of Missing Data Diagnostics
Diagnostics based on pretest data not much help Hard to predict missing distal outcomes from differences on pretest scores Longitudinal Diagnostics can be much more helpful
62
Hedeker & Gibbons (1997) Plot main DV over time for four groups:
for Program and Control for those with and without last wave of data Much can be learned
63
Empirical Examples Hedeker & Gibbons (1997) Hansen & Graham (1991)
Drug treatment of psychiatric patients Hansen & Graham (1991) Adolescent Alcohol Prevention Trial (AAPT) Alcohol, smoking, other drug prevention among normal adolescents (7th – 11th grade)
64
Empirical Example Used by Hedeker & Gibbons (1997)
IV: Drug Treatment vs. Placebo Control DV: Inpatient Multidimensional Psychiatric Scale (IMPS) 1 = normal 2 = borderline mentally ill 3 = mildly ill 4 = moderately ill 5 = markedly ill 6 = severely ill 7 = among the most extremely ill
65
From Hedeker & Gibbons (1997)
Placebo Control IMPS low = better outcomes Drug Treatment Weeks of Treatment
66
Longitudinal Diagnostics Hedeker & Gibbons Example
Treatment droppers do BETTER than stayers Control droppers do WORSE than stayers Example of Program X DV interaction But in this case, pattern would lead to suppression bias Not as bad for internal validity in presence of significant program effect
67
AAPT (Hansen & Graham, 1991) IV: Normative Education Program vs Information Only Control DV: Cigarette Smoking (3-item scale) Measured at one-year intervals 7th grade – 11th grade
68
AAPT Control Program Control Cigarette Smoking
(high = more smoking; arbitrary scale) Program th th th th th
69
Longitudinal Diagnostics AAPT Example
Treatment droppers do WORSE than stayers little steeper increase Control Little evidence for Prog X DV interaction Very likely MAR methods allow good conclusions (CSK scenario holds)
70
Use of Auxiliary Variables
Reduces attrition bias Restores some power lost due to attrition
71
What Is an Auxiliary Variable?
A variable correlated with the variables in your model but not part of the model not necessarily related to missingness used to "help" with missing data estimation Best auxiliary variables: same variable as main DV, but measured at waves not used in analysis model
72
Model of Interest
73
Benefit of Auxiliary Variables
Example from Graham & Collins (2008) X Y Z complete cases cases missing Y X, Y variables in the model (Y sometimes missing) Z is auxiliary variable
74
Benefit of Auxiliary Variables
Effective sample size (N') Analysis involving N cases, with auxiliary variable(s) gives statistical power equivalent to N' complete cases without auxiliary variables
75
Benefit of Auxiliary Variables
It matters how highly Y and Z (the auxiliary variable) are correlated For example increase rYZ = N = 500 gives power of N' = 542 ( 8%) rYZ = N = 500 gives power of N' = (22%) rYZ = N = 500 gives power of N' = 733 (47%) rYZ = N = 500 gives power of N' = 839 (68%)
76
Effective Sample Size by rYZ
77
Conclusions Attrition CAN be bad for internal validity
But often it's NOT nearly as bad as often feared Don't rush to conclusions, even with rather substantial attrition Examine evidence (especially longitudinal diagnostics) before drawing conclusions Use MI and ML missing data procedures! Use good auxiliary variables to minimize impact of attrition
78
Part 3: Illustration of Missing Data Analysis: Multiple Imputation with NORM and Proc MI
79
Multiple Imputation: Basic Steps
Impute Analyze Combine results
80
Imputation and Analysis
Impute 40 datasets a missing value gets a different imputed value in each dataset Analyze each data set with USUAL procedures e.g., SAS, SPSS, LISREL, EQS, STATA Save parameter estimates and SE’s
81
Combine the Results Parameter Estimates to Report
Average of estimate (b-weight) over 40 imputed datasets
82
Combine the Results Standard Errors to Report
Weighted sum of: “within imputation” variance average squared standard error usual kind of variability “between imputation” variance sample variance of parameter estimates over 40 datasets variability due to missing data
83
Materials for SPSS Regression
Starting place downloads (you will need to get a free user ID to download all our free software) missing data software Joe Schafer's Missing Data Programs John Graham's Additional NORM Utilities
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.