1. Considerations for the use of multiple imputation in a noninferiority trial setting
Kimberly Walters, Jie Zhou, Janet Wittes, Lisa Weissfeld Joint Statistical Meetings Denver, Colorado July 30, 2019

2. Outline
Inference in clinical trials: noninferiority versus superiority
choice of estimand
conservative (favoring null) assumptions for missing data
Multiple imputation (MI) of missing data
fully conditional specification (FCS) methods
imputation size, i.e., number of imputed datasets
choice of covariates
Simulation study to assess MI choices: the trade-off
bias, variance, power
computational burden

3. A word about estimands Wait! What’s an estimand?
From ICH E9 (R1) addendum on estimands:
The population
The response variable (Y)
Rules to handle observations after intercurrent events
Population-level summary of treatment effect, e.g., θT − θC
Examples: Treatment policy, While on treatment, Composite, Hypothetical, Principal stratum

4. A word about estimands From ICH E9 (R1) addendum on estimands:
“The choice of estimands for studies with objectives to demonstrate non-inferiority or equivalence requires careful reflection.”
“[S]uch trials are not conservative in nature and the importance of minimizing the number of protocol violations and deviations, non-adherence and withdrawals is indicated.”

5. Superiority Null Alternative Comparator Investigational drug
measure of
efficacy
time

6. Noninferiority Null Alternative Comparator Investigational drug
measure of
efficacy
<NIM
>NIM
time
NIM=noninferiority margin (depends on known efficacy of comparator)

7. Control-based assumptions about missing data
measure of
efficacy
Comparator
Investigational drug
SUPERIORITY
more
conservative
assumptions
NONINFERIORITY
less
conservative
assumptions
time

8. Single vs. multiple imputation (MI)
Single imputation
each missing value is replaced by a single value
examples: LOCF, WOCF, group-mean
can underestimate the variability of the outcome
Multiple imputation
each missing value is replaced by multiple (M) values
M complete datasets generated
more accurately reflects the variability of the outcome
estimates from each dataset are “rolled up” using Rubin’s rules

9. Multiple imputation design
Model for generating multiple values?
How many and which covariates?
Imputation size M?

10. Multiple imputation model

11. Multiple imputation model
SAS PROC MI and PROC MIANALYZE
Fully conditional specification (FCS)
SAS: “With an FCS statement, the variables are imputed sequentially in the order specified in the VAR statement.”
conditions on previous variables (observed and imputed values)
Predictive mean matching (PMM)
Default FCS models
SAS: “By default, a regression method is used for a continuous variable, and a discriminant function method is used for a classification variable.”

12. Covariates for MI Choice of variables
Substantive model – response (Y) and predictors (X)
Auxiliary variables (Z)
highly correlated to substantive model variables
predict the pattern of missingness
Caution from Hardt, et al. 2012: “inclusion of too many variables leads to a considerable increase in computation time or even causes programs to fail…”
They recommend this rule of thumb: “the number of cases with complete data should be at least three times the number of variables”

13. Imputation size
Hardt et al., 2012: “There was a recommendation by Little and Rubin that creating three to five data sets is usually enough. In more recent articles, probably due to increasing computational power, this recommendation has changed, suggesting the use of a greater number of imputed datasets in real data analyses, i.e. 20 – 100”
O’Kelly and Ratitch, 2014: “In our experience, using the number of imputations even on the order of several thousand does not present any practical challenges with the current technology and produces results that are quite stable with respect to different random seeds used, as well as close to the results of the maximum likelihood analysis with an equivalent model.”
The MI Procedure (SAS): “For other statistics (such as standard error and p-value) to be reliable, the rule of thumb is to use the percentages of cases with missing values as the number of imputations.”

14. Simulation study: design parameters
Response features of interventions in population
θ = true treatment effect (unknown)
σ = SD(Y) = standard deviation of endpoint measure Y
Study design
n = number of subjects total (1:1)
choice of estimand
noninferiority margin (NIM)
Data quality
p = percentage of missing data (varies by variable)
mechanism of missing data – MAR or MNAR
ρ = correlation between covariates and outcome
-0.2, 0
1, 1.5
300, 1000
change score, treatment policy, difference in means
-0.25, -0.5
10%, 40%
MAR
0.2, 0.8

15. Simulation study: MI parameters
M = number of imputations
K = number of covariates
N = number of simulations
mix of covariate types
all continuous
60% continuous and 40% binary
10, 50, 100, 500
10, 20, 30, 40, 50
10, 100, 1000

16. Simulation study: assessment criteria
Statistical operating characteristics
Bias and variance of estimate
Bias and variance of lower bound
power to reject null (inferiority)
nearness to asymptote (as M increases)
invariance to random number seed
Computational burden
run time for multiple imputation (including analysis and rollup)
number of burn-in iterations fixed (nbiter at 100; default=20)

17. Results I: imputation size
N=100; n=1000; K=7 (normal); SD=1.5; rho=0.8; p=40%; delta=-0.2
Computation time (seconds) M
Mean
Std Dev
Minimum
Maximum 10 1
0.2 2 50 5
0.4 4 6
100
0.5 9
500 49
2.1 47 61

18. Results I: imputation size
N=100; n=1000; K=7 (normal); SD=1.5; rho=0.8; p=40%; delta=-0.2 M
Statistics
Mean
Std Dev
Min
Median
Max
>-0.5
>-0.25 10
ESTIMATE
-0.223
0.096
-0.393
-0.228
-0.013 -
STDERR
0.099
0.004
0.089
0.110
LCLMEAN
-0.418
-0.580
-0.425
-0.209
77% 4%
500
-0.224
0.394
0.231
-0.006
0.098
0.002
0.092
0.105
-0.585
-0.426
-0.198

19. Results II: number of covariates
N=10; n=1000; M=10; SD=1; rho=0.2; p=10%; delta=1; *switch 20 to binary
Computation time (minutes) K
Mean
Std Dev
Minimum
Maximum
10 
0.3
0.0
0.4
20 
2.4
1.0
1.2
3.7
30 
3.5
0.9
2.7
4.9
40 
8.0
4.8
4.1
17.7
50 
9.0
3.1
6.4
16.2
50*
26.5
7.1
15.9
33.2

20. Results II: number of covariates
N=10; n=1000; M=10; SD=1; rho=0.2; p=10%; delta=1 M
Statistics
Mean
Std Dev
Min
Median
Max 20
ESTIMATE
0.980
0.125
0.856
0.944
1.213
STDERR
0.090
0.010
0.074
0.105
LCLMEAN
0.797
0.124
0.666
0.761
1.019 30
1.016
0.069
0.901
1.023
1.135
0.095
0.009
0.082
0.114
0.823
0.078
0.703
0.831
0.940 40
1.001
0.089
0.854
1.025
1.126
0.094
0.013
0.072
0.096
0.108
0.811
0.086
0.633
0.826
0.954 50
1.029
0.087
0.923
1.012
1.194
0.100
0.012
0.097
0.116
0.825
0.073
0.723
0.796
0.960

21. Results III: “realistic” scenario
N=10; n=1000; K=72 (mix); SD=1.5; rho=0.8 (Y) and 0.2 (X); p=10%; delta=0
Computation time (hours) M
Mean
Std Dev
Minimum
Maximum 10
0.6
0.2
0.5
0.9 50
4.2
1.6
2.8
7.9
100
7.5
6.2
9.2

22. Results III: “realistic” scenario
N=10; n=1000; K=72 (mix); SD=1.5; rho=0.8 (Y) and 0.2 (X); p=10%; delta=0 M
Statistics
Mean
Std Dev
Min
Median
Max 10
ESTIMATE
-0.014
0.097
-0.117
-0.044
0.162
STDERR
0.112
0.002
0.107
0.111
0.115
LCLMEAN
-0.233
0.100
-0.334
-0.265
-0.054 50
-0.001
0.114
-0.180
-0.007
0.159
0.110
-0.217
0.117
-0.397
-0.226
-0.055
100
-0.181
0.158
-0.218
0.118
-0.398
-0.057

23. Conclusions Imputation size depends on percentage missing
adds to computational burden
use M=50
Imputation covariates
well chosen covariates can reduce bias and approach MAR assumptions
too many covariates can make analysis impossible to run
type of covariates and correlation to response
K~50

24. Author contact information