1. Oops, I D-I-D It Again!
Advanced Difference-in-Differences Models in SAS® E Margaret Warton and Melissa M Parker September 6, 2018
Kaiser Permanente Research

2. Overview Brief Review of Difference-in-Differences
Power and Sample Size Calculations for D-I-D
D-I-D and Binary Outcomes
Propensity Scores and D-I-D
Difference-in-Difference-in-Differences
January 1, 2019

3. D-I-D Basics: A Brief Review
Outcome measured at two time points
D-I-D is the difference in slopes between two groups
Interaction term between exposure and measurement period (time)
How much more (or less) did the exposed group’s outcome change between time 1 and time 2 compared to the unexposed group’s outcome?
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

4. Dataset for Examples
Value-Based Insurance Designs and Medication Non-Adherence
Kaiser Permanente members in 2013
All on non-deductible insurance plan
All changed to deductible insurance plan on 1/1/2014
Exposed: VBID benefit provides free medications for chronic conditions
Unexposed: no VBID pharmacy benefit
Outcome of interest: non-adherence to medications for chronic conditions
D-I-D answers the question: Did the VBID plan mitigate the effect of the increased deductible cost on medication non-adherence?
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

5. D-I-D Sample Size and Power Calculations
Basic Steps
Specify your model
Estimate outcomes and group ratios
Estimate standard deviations and correlation
Create an "exemplary dataset"
Use PROC GLMPOWER to calculate sample size or power
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

6. D-I-D Sample Size and Power Calculations
Assumptions
1. Mixed model w/random intercept
2. Unexposed: 40% to 45%
Exposed: % to 42%
Unexposed: Exposed = 3:2
3. Std Dev: 0.03 (or 3%)
Correlation: ρ = 0.80
4. Exemplary dataset code
proc mixed data=vbid method=reml;
class study_id exposed;
model non_adh = post exposed post*exposed/solution;
random int/subject=study_id type=vc ;
run;
data adh;
Input exposed $ adh_pre adh_post wt;
datalines;
unexposed
exposed ;
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

7. PROC GLMPOWER Code Example
Sample Size
Title "VBID D-I-D Required Sample Size: PROC GLMPOWER";
proc glmpower data = adh;
class exposed;
weight wt;
model adh_pre adh_post = exposed;
repeated time;
power
effects=(exposed)
mtest = hlt
alpha = 0.05
power = 0.9
ntotal = .
stddev = 0.03
matrix("adhcorr") = lear (0.8, 1, 2, 1 2)
corrmat = "adhcorr";
run;
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

8. PROC GLMPOWER Output
Output 1: Output from PROC GLMPOWER for D-I-D Required Sample Size at 90% Power
Total sample size of 1,755 is adequate to detect a 3% difference in change in non-adherence
3:2 ratio requires 1053 unexposed and 702 exposed
Study cohort has 1458 unexposed and 1024 exposed
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

9. D-I-D and Binary Outcomes
Analysis Methods
Only two outcome measures, so D-I-D models are population-level marginal models
Different modeling solutions available in SAS
PROC GENMOD: ANOVA, repeated measures
PROC MIXED, PROC GLIMMIX : Mixed models
PROC GENMOD: Choice of link and distribution
Rarity of outcome
Preference for absolute risk differences vs relative risks
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

10. D-I-D and PROC GENMOD Examples
Absolute Risk Differences
title "Population-Level Marginal D-I-D Model: PROC GENMOD with dist=Poisson and link=identity"; proc genmod data=vbid descending; class study_id exposed (order=formatted ref='non-vbid') / param=ref; model non_adh=post|exposed /dist=poisson link=identity type3; repeated subject=study_id /type=un; format exposed exposed.; run;
1. VBID = Long format dataset 2.Model statement uses pipe for fully saturated 3. Poisson distribution 4. Identity link 5. Study_id is our repeated measures unit
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

11. D-I-D and PROC GENMOD Examples
Relative Risks and Relative Risk Ratios
title "Population-Level Marginal Model(RRR):PROC GENMOD with dist=binomial and link=log"; proc genmod data=vbid descending; class study_id exposed (order=formatted ref='Non-VBID') / param=ref; model non_adh= post|exposed /dist=binomial link=log type3; estimate 'RR:EXPOSED' exposed 1 /exp; estimate 'RR:POST' post 1 /exp; estimate 'RRR:POST*EXPOSED' post*exposed 1 /exp; repeated subject=study_id /type=un; format exposed exposed.; run;
1. VBID = Long format dataset 2.Model statement uses pipe for fully saturated 3. Poisson distribution 4. Identity link 5. Study_id is our repeated measures unit
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

12. D-I-D and PROC GENMOD Results
Marginal
Population
Model
Binomial Distribution
Identity Link
Absolute Risks
Poisson Distribution
Log Link
Relative Risk Ratios
(Exponentiated)
Non-Adherence
Estimate
95% CI
Intercept
0.406
(0.381, 0.431)
(0.382, 0.432)
Post
-0.012
(-0.033, 0.009)
0.970
0.920, 1.021
Exposed
0.013
(-0.026, 0.052)
1.038
(0.938,1.135)
Exposed*
-0.017
(-0.050, 0.016)
0.959
(0.885, 1.040)
D-I-D is -1.7%  prevalence of non-adherence declined more in VBID group (NS)
Poisson and binomial identity link are identical (binomial is more efficient)
Binomial log exponentiated=relative risk
No quick intuition on prevalence of non-adherence
0.96  Post/Pre relative risk of poor adherence was lower in VBID than in non-VIBD group (NS).
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

13. Propensity Scores and D-I-D
Balancing Case-Mix Differences
Recent method used to balance case-mix differences so that exposed and reference groups are as similar as possible
Plan carefully in the design stage to eliminate imbalances
In observational research residual imbalances are likely
Reed ME et al., Health Aff, 2017.
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

14. Propensity Scores and D-I-D
Use the conditional probability of being exposed given a set of observed covariates to balance case mix differences in the analysis.
Stage 1: Development of the propensity score (PS) model
Check for acceptable level of balance
Determine how PS will be used in the analysis
Weights, matching, stratified sample, covariate
Long, iterative process
Ends when satisfactory covariate balance has been achieved
Stage 2: Run D-I-D in PS weighted, matched or stratified sample
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

15. Propensity Scores and D-I-D
Stage 1 Tackled Using PROC PSMATCH
Run logistic regression model or import pre-calculated PS
Assess covariate balance
Variable information (mean, STD, prevalence)
Standardized difference
difference in means (prevalence) between the exposed and reference groups divided by the pooled standard deviation
0 means no difference; ≤ d ≤ 0.10 indicates a negligible difference
Ratio of variances ideally equal to 1
Graphical exploration with boxplots, CDF plots, quantile-quantile plots
Create and output matched sample or PS weights
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

16. Propensity Scores and D-I-D
Stage 1 Code Using PROC PSMATCH
title "VBID D-I-D: PROC PSMATCH for Propensity Scores";
proc psmatch data=VBID;
class exposed;
psmodel exposed (treated=last)= age_at_index female low_ses hispanic
black asian other race_miss english comorbid1-comorbid3;
assess ps lps var=(female white age_at_index low_ses hispanic black
asian other english comorb0-comorb3) /
plots(nodetails)=(cdfplot boxplot(display=(ps age_at_index))
stddiff(ref=0.10) )
weight=atewgt(stabilize=no) ;
output out (obs=all)=outx
lps=lps
ps=ps
atewgt(stabilize=no)=iptw_u ;
run;
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

17. Propensity Scores and D-I-D
Stage 1 OUTPUT using PROC PSMATCH
Visual inspection of the plot shows how covariate imbalances are removed after PS weighting.
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

18. Propensity Scores and D-I-D
Stage 1 OUTPUT using PROC PSMATCH
Characteristic
All
ATE Weighted
VBID
N=1024
No VBID
N=1458
Std Diff
Variance Ratio
VBID N=1024
No VBID N=1458
Female
42%
48%
-0.12
0.978
45%
46%
-0.01
0.998
Age, mean (SD)
50.4 (8.3)
52.0 (8.0)
-0.19 
1.076 
51.4 (7.9)
51.3 (8.4)
0.01 
0.883 
Race
Black
White
Asian
Other
14%
64% 5%
12%
57%
23% 7%
0.08
0.14
-0.23
-0.04
1.193
0.943
0.690
0.849
13%
61%
19% 6%
60%
20%
0.001
0.02
-0.02
-0.007
1.002
0.993
0.972
0.974
Hispanic ethnicity
18%
15%
0.09
1.179
17%
16%
0.01
1.027
English Spkr
99%
94%
0.32
0.100
97%
96%
0.05
0.778
Assessment of covariate balance after weighting
All standardized differences between and 0.10
Variance ratios closer to 1, with the exception of age
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

19. Propensity Scores and D-I-D
Stage 1 OUTPUT using PROC PSMATCH
Distributional balance of continuous covariates is similar and aligned to the overall sample after weighting.
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

20. Propensity Scores and D-I-D
Stage 2 ATE Weighted D-I-D
title "D-I-D model for change in medication adherence, Main effects model in ATE weighted cohort";
proc genmod data=xwt descending;
class study_id exposed (order=formatted ref='Non-VBID') / param=ref;
model non_adh = post|exposed
/dist=binomial link=identity type3;
repeated subject=study_id /type=un;
weight iptw_u ;
format exposed exposed.;
run;
Unadjusted
ATE Weighted
Non-Adherence
Estimate
95% CI
Intercept
0.406
(0.381, 0.431)
0.408
(0.382, 0.434)
Post
-0.012
(-0.033, 0.009)
-0.014
(-0.035, 0.008)
Exposed (VBID)
0.013
(-0.026, 0.052)
0.002
(-0.044, 0.044)
Exposed* Post (DID)
-0.017
(-0.050, 0.016)
-0.021
(-0.057, 0.015)
Poor adherence decreased by 2.1% more in VBID compared to non-VBID (NS)
Exposed: 41.0% (pre) to 37.5% (post)
Reference: 40.8% (pre) to 39.4% (post)
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

21. Difference in Difference-in-Differences (DDD)
Assessing Heterogeneity of Treatment Effects
Formal comparison of 2 or more D-I-D estimates
Null hypothesis
The D-I-D among patients living in low SES neighborhoods is equal to the D-I-D among patients who do not live in low SES neighborhoods.
DDD estimator is the parameter estimate of the low_ses*exposure*post interaction term
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

22. Difference in Difference-in-Differences (DDD)
SAS Code for DDD model
title "D-I-D model for change in medication adherence, Testing for EM by SES in ATE weighted cohort";
proc genmod data=xwt descending ;
class study_id;
model non_adh = post exposed post*exposed low_ses low_ses*post*exposed
/dist=binomial link=identity type3;
repeated subject =study_id /type=un;
estimate 'low_ses=1, pre, VBID' intercept 1 exposed 1 low_ses 1 ;
estimate 'low_ses=1, post, VBID' intercept 1 post 1 exposed 1 post*exposed 1
low_ses 1 low_ses*post*exposed 1 ;
estimate 'low_ses=1, pre, NON-VBID' intercept 1 low_ses 1 ;
estimate 'low_ses=1, post, NON-VBID' intercept 1 post 1 low_ses 1 ;
estimate 'low_ses=0, pre, VBID' intercept 1 exposed 1 ;
estimate 'low_ses=0, post, VBID' intercept 1 post 1 exposed 1 post*exposed 1 ;
estimate 'low_ses=0, pre, NON-VBID' intercept 1 ;
estimate 'low_ses=0, post, NON-VBID' intercept 1 post 1 ;
estimate 'low_ses=1, DID' post*exposed 1 low_ses*post*exposed 1 ;
estimate 'low_ses=0, DID' post*exposed 1 ;
weight iptw_u ;
run;
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

23. Difference in Difference-in-Differences (DDD)
DDD Output
Contrast Estimate Results
Label
Mean Estimate
Mean
L'Beta Estimate
Standard Error
Alpha
L'Beta
Chi- Square
Pr > 
ChiSq
Confidence Limits
low_ses=1, pre, VBID
0.4703
0.4133
0.5274
0.0291
0.05
261.43
<.0001
low_ses=1, post, VBID
0.4091
0.3297
0.4885
0.0405
101.99
low_ses=1, pre, NON-VBID
0.4680
0.4186
0.5174
0.0252
344.65
low_ses=1, post, NON-VBID
0.4544
0.4050
0.5038
325.22
low_ses=0, pre, VBID
0.3956
0.3627
0.4285
0.0168
554.63
low_ses=0, post, VBID
0.3672
0.3338
0.4006
0.0171
463.67
low_ses=0, pre, NON-VBID
0.3933
0.3655
0.4211
0.0142
769.23
low_ses=0, post, NON-VBID
0.3797
0.3520
0.4073
0.0141
724.76
low_ses=1, DID
0.0438
0.0467
1.04
0.3074
low_ses=0, DID
0.0201
0.0178
0.69
0.4064
Among patients living in low SES neighborhoods, the prevalence of poor adherence decreased by 4.76% (NS) more in VBID compared to reference group.
Among patients not living in low SES neighborhoods, the prevalence of poor adherence decreased by 1.48% (NS) more in VBID compared to reference group.
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

24. Difference in Difference-in-Differences (DDD)
DDD Output
Analysis Of GEE Parameter Estimates
Empirical Standard Error Estimates
Parameter
Estimate
Standard Error
95% Confidence Limits Z
Pr > |Z|
Intercept
0.3933
0.0142
0.3655
0.4211
27.74
<.0001
Post
0.0108
0.0076
-1.26
0.2085
Exposed
0.0023
0.0212
0.0438
0.11
0.9125
post*exposed
0.0178
0.0201
-0.83
0.4064
LOW_SES
0.0747
0.0270
0.0219
0.1276
2.77
0.0056
post*exposed*LOW_SES
0.0465
0.0584
-0.71
0.4804
The DDD= -3.28%.
This means the DID (post – pre) among patients living in low SES neighborhoods was 3.28% more extreme than the DID (post - pre) among patients not living in low SES neighborhoods.
The difference did not reach statistical significance.
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

25. Concluding Remarks
D-I-D has several methodological benefits and results have a causal interpretation as long as time ordering is preserved.
Sample size and power calculations for D-I-D models are easy to do in PROC GLMPOWER.
Use PROC GENMOD for population level marginal effect estimates of pre-post change in a binary outcome.
Log link for relative risk, Identity link for absolute risk
PROC PSMATCH is your one-stop shop for developing a propensity score model, assessing covariate balance and creating a matched cohort or propensity score weights.
Use DDD to assess heterogeneity of treatment effects.
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

26. Acknowledgements Mary E. Reed, PhD Andrew J. Karter, PhD
Howard H. Moffet, MPH
Jennifer Y. Liu, MPH
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

27. Questions?
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

28. Contact Information
E. Margaret Warton, MPH
Kaiser Permanente Division of Research; Oakland, CA
Melissa M. Parker, MS
Kaiser Permanente Division of Research; Oakland, CA
January 1, 2019
| © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.