Download presentation
Presentation is loading. Please wait.
Published byIvan Lesmana Modified over 6 years ago
1
Oops, I D-I-D It Again! Advanced Difference-in-Differences Models in SAS® E Margaret Warton and Melissa M Parker September 6, 2018 Kaiser Permanente Research
2
Overview Brief Review of Difference-in-Differences
Power and Sample Size Calculations for D-I-D D-I-D and Binary Outcomes Propensity Scores and D-I-D Difference-in-Difference-in-Differences January 1, 2019
3
D-I-D Basics: A Brief Review
Outcome measured at two time points D-I-D is the difference in slopes between two groups Interaction term between exposure and measurement period (time) How much more (or less) did the exposed group’s outcome change between time 1 and time 2 compared to the unexposed group’s outcome? January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
4
Dataset for Examples Value-Based Insurance Designs and Medication Non-Adherence Kaiser Permanente members in 2013 All on non-deductible insurance plan All changed to deductible insurance plan on 1/1/2014 Exposed: VBID benefit provides free medications for chronic conditions Unexposed: no VBID pharmacy benefit Outcome of interest: non-adherence to medications for chronic conditions D-I-D answers the question: Did the VBID plan mitigate the effect of the increased deductible cost on medication non-adherence? January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
5
D-I-D Sample Size and Power Calculations
Basic Steps Specify your model Estimate outcomes and group ratios Estimate standard deviations and correlation Create an "exemplary dataset" Use PROC GLMPOWER to calculate sample size or power January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
6
D-I-D Sample Size and Power Calculations
Assumptions 1. Mixed model w/random intercept 2. Unexposed: 40% to 45% Exposed: % to 42% Unexposed: Exposed = 3:2 3. Std Dev: 0.03 (or 3%) Correlation: ρ = 0.80 4. Exemplary dataset code proc mixed data=vbid method=reml; class study_id exposed; model non_adh = post exposed post*exposed/solution; random int/subject=study_id type=vc ; run; data adh; Input exposed $ adh_pre adh_post wt; datalines; unexposed exposed ; January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
7
PROC GLMPOWER Code Example
Sample Size Title "VBID D-I-D Required Sample Size: PROC GLMPOWER"; proc glmpower data = adh; class exposed; weight wt; model adh_pre adh_post = exposed; repeated time; power effects=(exposed) mtest = hlt alpha = 0.05 power = 0.9 ntotal = . stddev = 0.03 matrix("adhcorr") = lear (0.8, 1, 2, 1 2) corrmat = "adhcorr"; run; January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
8
PROC GLMPOWER Output Output 1: Output from PROC GLMPOWER for D-I-D Required Sample Size at 90% Power Total sample size of 1,755 is adequate to detect a 3% difference in change in non-adherence 3:2 ratio requires 1053 unexposed and 702 exposed Study cohort has 1458 unexposed and 1024 exposed January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
9
D-I-D and Binary Outcomes
Analysis Methods Only two outcome measures, so D-I-D models are population-level marginal models Different modeling solutions available in SAS PROC GENMOD: ANOVA, repeated measures PROC MIXED, PROC GLIMMIX : Mixed models PROC GENMOD: Choice of link and distribution Rarity of outcome Preference for absolute risk differences vs relative risks January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
10
D-I-D and PROC GENMOD Examples
Absolute Risk Differences title "Population-Level Marginal D-I-D Model: PROC GENMOD with dist=Poisson and link=identity"; proc genmod data=vbid descending; class study_id exposed (order=formatted ref='non-vbid') / param=ref; model non_adh=post|exposed /dist=poisson link=identity type3; repeated subject=study_id /type=un; format exposed exposed.; run; 1. VBID = Long format dataset 2.Model statement uses pipe for fully saturated 3. Poisson distribution 4. Identity link 5. Study_id is our repeated measures unit January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
11
D-I-D and PROC GENMOD Examples
Relative Risks and Relative Risk Ratios title "Population-Level Marginal Model(RRR):PROC GENMOD with dist=binomial and link=log"; proc genmod data=vbid descending; class study_id exposed (order=formatted ref='Non-VBID') / param=ref; model non_adh= post|exposed /dist=binomial link=log type3; estimate 'RR:EXPOSED' exposed 1 /exp; estimate 'RR:POST' post 1 /exp; estimate 'RRR:POST*EXPOSED' post*exposed 1 /exp; repeated subject=study_id /type=un; format exposed exposed.; run; 1. VBID = Long format dataset 2.Model statement uses pipe for fully saturated 3. Poisson distribution 4. Identity link 5. Study_id is our repeated measures unit January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
12
D-I-D and PROC GENMOD Results
Marginal Population Model Binomial Distribution Identity Link Absolute Risks Poisson Distribution Log Link Relative Risk Ratios (Exponentiated) Non-Adherence Estimate 95% CI Intercept 0.406 (0.381, 0.431) (0.382, 0.432) Post -0.012 (-0.033, 0.009) 0.970 0.920, 1.021 Exposed 0.013 (-0.026, 0.052) 1.038 (0.938,1.135) Exposed* -0.017 (-0.050, 0.016) 0.959 (0.885, 1.040) D-I-D is -1.7% prevalence of non-adherence declined more in VBID group (NS) Poisson and binomial identity link are identical (binomial is more efficient) Binomial log exponentiated=relative risk No quick intuition on prevalence of non-adherence 0.96 Post/Pre relative risk of poor adherence was lower in VBID than in non-VIBD group (NS). January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
13
Propensity Scores and D-I-D
Balancing Case-Mix Differences Recent method used to balance case-mix differences so that exposed and reference groups are as similar as possible Plan carefully in the design stage to eliminate imbalances In observational research residual imbalances are likely Reed ME et al., Health Aff, 2017. January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
14
Propensity Scores and D-I-D
Use the conditional probability of being exposed given a set of observed covariates to balance case mix differences in the analysis. Stage 1: Development of the propensity score (PS) model Check for acceptable level of balance Determine how PS will be used in the analysis Weights, matching, stratified sample, covariate Long, iterative process Ends when satisfactory covariate balance has been achieved Stage 2: Run D-I-D in PS weighted, matched or stratified sample January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
15
Propensity Scores and D-I-D
Stage 1 Tackled Using PROC PSMATCH Run logistic regression model or import pre-calculated PS Assess covariate balance Variable information (mean, STD, prevalence) Standardized difference difference in means (prevalence) between the exposed and reference groups divided by the pooled standard deviation 0 means no difference; ≤ d ≤ 0.10 indicates a negligible difference Ratio of variances ideally equal to 1 Graphical exploration with boxplots, CDF plots, quantile-quantile plots Create and output matched sample or PS weights January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
16
Propensity Scores and D-I-D
Stage 1 Code Using PROC PSMATCH title "VBID D-I-D: PROC PSMATCH for Propensity Scores"; proc psmatch data=VBID; class exposed; psmodel exposed (treated=last)= age_at_index female low_ses hispanic black asian other race_miss english comorbid1-comorbid3; assess ps lps var=(female white age_at_index low_ses hispanic black asian other english comorb0-comorb3) / plots(nodetails)=(cdfplot boxplot(display=(ps age_at_index)) stddiff(ref=0.10) ) weight=atewgt(stabilize=no) ; output out (obs=all)=outx lps=lps ps=ps atewgt(stabilize=no)=iptw_u ; run; January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
17
Propensity Scores and D-I-D
Stage 1 OUTPUT using PROC PSMATCH Visual inspection of the plot shows how covariate imbalances are removed after PS weighting. January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
18
Propensity Scores and D-I-D
Stage 1 OUTPUT using PROC PSMATCH Characteristic All ATE Weighted VBID N=1024 No VBID N=1458 Std Diff Variance Ratio VBID N=1024 No VBID N=1458 Female 42% 48% -0.12 0.978 45% 46% -0.01 0.998 Age, mean (SD) 50.4 (8.3) 52.0 (8.0) -0.19 1.076 51.4 (7.9) 51.3 (8.4) 0.01 0.883 Race Black White Asian Other 14% 64% 5% 12% 57% 23% 7% 0.08 0.14 -0.23 -0.04 1.193 0.943 0.690 0.849 13% 61% 19% 6% 60% 20% 0.001 0.02 -0.02 -0.007 1.002 0.993 0.972 0.974 Hispanic ethnicity 18% 15% 0.09 1.179 17% 16% 0.01 1.027 English Spkr 99% 94% 0.32 0.100 97% 96% 0.05 0.778 Assessment of covariate balance after weighting All standardized differences between and 0.10 Variance ratios closer to 1, with the exception of age January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
19
Propensity Scores and D-I-D
Stage 1 OUTPUT using PROC PSMATCH Distributional balance of continuous covariates is similar and aligned to the overall sample after weighting. January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
20
Propensity Scores and D-I-D
Stage 2 ATE Weighted D-I-D title "D-I-D model for change in medication adherence, Main effects model in ATE weighted cohort"; proc genmod data=xwt descending; class study_id exposed (order=formatted ref='Non-VBID') / param=ref; model non_adh = post|exposed /dist=binomial link=identity type3; repeated subject=study_id /type=un; weight iptw_u ; format exposed exposed.; run; Unadjusted ATE Weighted Non-Adherence Estimate 95% CI Intercept 0.406 (0.381, 0.431) 0.408 (0.382, 0.434) Post -0.012 (-0.033, 0.009) -0.014 (-0.035, 0.008) Exposed (VBID) 0.013 (-0.026, 0.052) 0.002 (-0.044, 0.044) Exposed* Post (DID) -0.017 (-0.050, 0.016) -0.021 (-0.057, 0.015) Poor adherence decreased by 2.1% more in VBID compared to non-VBID (NS) Exposed: 41.0% (pre) to 37.5% (post) Reference: 40.8% (pre) to 39.4% (post) January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
21
Difference in Difference-in-Differences (DDD)
Assessing Heterogeneity of Treatment Effects Formal comparison of 2 or more D-I-D estimates Null hypothesis The D-I-D among patients living in low SES neighborhoods is equal to the D-I-D among patients who do not live in low SES neighborhoods. DDD estimator is the parameter estimate of the low_ses*exposure*post interaction term January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
22
Difference in Difference-in-Differences (DDD)
SAS Code for DDD model title "D-I-D model for change in medication adherence, Testing for EM by SES in ATE weighted cohort"; proc genmod data=xwt descending ; class study_id; model non_adh = post exposed post*exposed low_ses low_ses*post*exposed /dist=binomial link=identity type3; repeated subject =study_id /type=un; estimate 'low_ses=1, pre, VBID' intercept 1 exposed 1 low_ses 1 ; estimate 'low_ses=1, post, VBID' intercept 1 post 1 exposed 1 post*exposed 1 low_ses 1 low_ses*post*exposed 1 ; estimate 'low_ses=1, pre, NON-VBID' intercept 1 low_ses 1 ; estimate 'low_ses=1, post, NON-VBID' intercept 1 post 1 low_ses 1 ; estimate 'low_ses=0, pre, VBID' intercept 1 exposed 1 ; estimate 'low_ses=0, post, VBID' intercept 1 post 1 exposed 1 post*exposed 1 ; estimate 'low_ses=0, pre, NON-VBID' intercept 1 ; estimate 'low_ses=0, post, NON-VBID' intercept 1 post 1 ; estimate 'low_ses=1, DID' post*exposed 1 low_ses*post*exposed 1 ; estimate 'low_ses=0, DID' post*exposed 1 ; weight iptw_u ; run; January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
23
Difference in Difference-in-Differences (DDD)
DDD Output Contrast Estimate Results Label Mean Estimate Mean L'Beta Estimate Standard Error Alpha L'Beta Chi- Square Pr > ChiSq Confidence Limits low_ses=1, pre, VBID 0.4703 0.4133 0.5274 0.0291 0.05 261.43 <.0001 low_ses=1, post, VBID 0.4091 0.3297 0.4885 0.0405 101.99 low_ses=1, pre, NON-VBID 0.4680 0.4186 0.5174 0.0252 344.65 low_ses=1, post, NON-VBID 0.4544 0.4050 0.5038 325.22 low_ses=0, pre, VBID 0.3956 0.3627 0.4285 0.0168 554.63 low_ses=0, post, VBID 0.3672 0.3338 0.4006 0.0171 463.67 low_ses=0, pre, NON-VBID 0.3933 0.3655 0.4211 0.0142 769.23 low_ses=0, post, NON-VBID 0.3797 0.3520 0.4073 0.0141 724.76 low_ses=1, DID 0.0438 0.0467 1.04 0.3074 low_ses=0, DID 0.0201 0.0178 0.69 0.4064 Among patients living in low SES neighborhoods, the prevalence of poor adherence decreased by 4.76% (NS) more in VBID compared to reference group. Among patients not living in low SES neighborhoods, the prevalence of poor adherence decreased by 1.48% (NS) more in VBID compared to reference group. January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
24
Difference in Difference-in-Differences (DDD)
DDD Output Analysis Of GEE Parameter Estimates Empirical Standard Error Estimates Parameter Estimate Standard Error 95% Confidence Limits Z Pr > |Z| Intercept 0.3933 0.0142 0.3655 0.4211 27.74 <.0001 Post 0.0108 0.0076 -1.26 0.2085 Exposed 0.0023 0.0212 0.0438 0.11 0.9125 post*exposed 0.0178 0.0201 -0.83 0.4064 LOW_SES 0.0747 0.0270 0.0219 0.1276 2.77 0.0056 post*exposed*LOW_SES 0.0465 0.0584 -0.71 0.4804 The DDD= -3.28%. This means the DID (post – pre) among patients living in low SES neighborhoods was 3.28% more extreme than the DID (post - pre) among patients not living in low SES neighborhoods. The difference did not reach statistical significance. January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
25
Concluding Remarks D-I-D has several methodological benefits and results have a causal interpretation as long as time ordering is preserved. Sample size and power calculations for D-I-D models are easy to do in PROC GLMPOWER. Use PROC GENMOD for population level marginal effect estimates of pre-post change in a binary outcome. Log link for relative risk, Identity link for absolute risk PROC PSMATCH is your one-stop shop for developing a propensity score model, assessing covariate balance and creating a matched cohort or propensity score weights. Use DDD to assess heterogeneity of treatment effects. January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
26
Acknowledgements Mary E. Reed, PhD Andrew J. Karter, PhD
Howard H. Moffet, MPH Jennifer Y. Liu, MPH January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
27
Questions? January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
28
Contact Information E. Margaret Warton, MPH Kaiser Permanente Division of Research; Oakland, CA Melissa M. Parker, MS Kaiser Permanente Division of Research; Oakland, CA January 1, 2019 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.