Difference-in-Differences Models Gustavo Angeles MEASURE Evaluation University of North Carolina at Chapel Hill Workshop on Impact Evaluation of Population, Health and Nutrition Programs Accra, Ghana July 18-29, 2016

I. Difference-in-differences: Basic set-up 2 groups: Program group (“with program”) Comparison group (“without program”) 2 points in time: Baseline survey Follow-up survey Recommended: Follow-up survey is longitudinal at the individual, household or locality level

Difference-in-Differences Outcome B Program Group A Baseline Follow-up Time

Difference-in-Differences B Outcome Program Group B-A A Baseline Follow-up Time

Difference-in-Differences Outcome B Program Group B-A A D Comparison Group D-C C Baseline Follow-up Time

Difference-in-Differences Outcome B Program Group B-A D-C A D Comparison Group D-C C Baseline Follow-up Time

Difference-in-Differences Impact = (B-A)-(D-C) Outcome B Program Group B-A D-C A D Comparison Group D-C C Baseline Follow-up Time

Difference-in-Differences Impact = (B-A)-(D-C) B Outcome Program Group B-A D-C A D Comparison Group D-C C Follow-up Time Baseline Key condition: “Parallel trends assumption.” Program group would have had the same change as the Comparison group in absence of the program.

Difference-in-Differences Impact = (B-A)-(D-C) Outcome B Program Group B-A A True change; diff-in-diff under-estimates program impact D Comparison Group D-C C Baseline Follow-up Time Key condition: “Parallel trends assumption.” Program group would’ve had the same change as the Comparison group in absence of the program. Limitations: - Strong assumption; “true change” could’ve been different.

Difference-in-Differences Impact = (B-A)-(D-C) Outcome B Program Group B-A A True change; diff-in-diff under-estimates program impact D Comparison Group D-C C Baseline Follow-up Time Key condition: “Parallel trends assumption.” Program group would’ve had the same change as the Comparison group in absence of the program. Limitation: - Strong assumption; true change could’ve been different - It requires “short” time interval, but it reduces magnitude of impact to estimate.

Difference-in-Differences Impact = (B-A)-(D-C) Outcome B Program Group B-A D-C A D Comparison Group D-C C Baseline Follow-up Time Key Issue: Selection of Comparison Group Question: What is the best way to select a program and comparison group, so the two groups will behave similarly and will have the same change, in the absence of the program?

Difference-in-Differences: Testing the “Parallel trends assumption” Impact = (B-A)-(D-C) Outcome B Program Group A D E Comparison Group C F Pre-Baseline Baseline Follow-up Time One way: You need Pre-Baseline data!

Difference-in-Differences: Testing the “Parallel trends assumption” Impact = (B-A)-(D-C) B Outcome Program Group A D E A-E Comparison Group C F C-F Pre-Baseline Baseline Follow-up Time One way: You need Pre-Baseline data! In this example, “Parallel trends assumption” holds if: (A-E)=(C-F) Problems: - Pre-baseline data rarely available - Past behavior is only an indication of future behavior.

Difference-in-Differences: Not good if different true changes Impact = (B-A)-(D-C) Outcome B Program Group A E D True change Comparison Group C F Pre-Baseline Baseline Follow-up Time In this case Diff-in-diff provides and incorrect estimate of program impact. It underestimates program impact.

Difference-in-Difference: Not good if different trends Impact = (B-A)-(D-C) Outcome B Program Group True Impact A E D True change Comparison Group C F Pre-Baseline Baseline Follow-up Time In this case Diff-in-diff provides and incorrect estimate of program impact. It underestimates program impact.

Difference-in-Differences: Extensions (3 points in time) Outcome B Impact 1 Program Group A D Comparison Group C Baseline Follow-up 1 Follow-up 2 Time Key condition: “Parallel trends assumption” holds for each time period.

Difference-in-Differences: Extensions (3 points in time) Outcome G Impact 2 B Impact 1 Program Group A D H Comparison Group C Baseline Follow-up 1 Follow-up 2 Time Key condition: “Parallel trends assumption” holds for each time period.

The DID model

Figure 1. DID Model – Structure of the pooled data set Variable names Iid Cluster P T PxT Y x1 x2 x3 … ID Iid: Individual identifier number 1 1 1 0 0 0 … … … 2 1 1 0 0 1 … … … 3 1 1 0 0 1 … … … 4 1 1 0 0 0 … … … 1 2 0 0 0 1 … … … 2 2 0 0 0 0 … … … 3 2 0 0 0 1 … … … … … … … … … … … … 1 200 1 0 0 0 … … … 2 200 1 0 0 1 … … … 3 200 1 0 0 0 … … … Cluster ID: Cluster identifier number P: Program summy T: Time dummy PxT: The interaction dummy Y: The dependent variable x1, x2, x3 : cluster, household, or individual characteristics Baseline 1 1 1 1 1 1 … … … 2 1 1 1 1 0 … … … 3 1 1 1 1 1 … … … 4 1 1 1 1 0 … … … 1 2 0 1 0 0 … … … 2 2 0 1 0 1 … … … 3 2 0 1 0 1 … … … … … … … … … … … … 1 200 1 1 1 1 … … … 2 200 1 1 1 0 … … … 3 200 1 1 1 1 … … … Follow- Up

Difference-in-differences Method widely used in program evaluation Key is the “Parallel trends assumption.” It works better if the program was randomly allocated between the program group and the comparison group. The two groups will be “similar” in observed and unobserved characteristics. An alternative is to use matching procedures to find a matched Comparison Group, before you implement the baseline. You can match on community-characteristics. It works better if there is a “short” time interval between baseline and follow-up, but, how “short” to still measure impact? It depends on the outcome and selection of comparison group. It could control for fixed unobserved characteristics that could be the source of biased estimates of program impact (endogeneity). It is better to have longitudinal surveys.

Thank you!

This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14-00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www.measureevaluation.org