1. Difference-in-Differences Models
Gustavo Angeles MEASURE Evaluation
University of North Carolina at Chapel Hill
Workshop on Impact Evaluation of Population, Health and Nutrition Programs
Accra, Ghana July 18-29, 2016

2. I. Difference-in-differences: Basic set-up
2 groups:
Program group (“with program”)
Comparison group (“without program”)
2 points in time:
Baseline survey
Follow-up survey
Recommended: Follow-up survey is longitudinal at the individual, household or locality level

3. Difference-in-Differences
Outcome B
Program Group A
Baseline
Follow-up
Time

4. Difference-in-DifferencesB
Outcome
Program Group
B-A A
Baseline
Follow-up
Time

5. Difference-in-Differences
Outcome B
Program Group
B-A A D
Comparison Group
D-C C
Baseline
Follow-up
Time

6. Difference-in-Differences
Outcome B
Program Group
B-A
D-C A D
Comparison Group
D-C C
Baseline
Follow-up
Time

7. Difference-in-Differences
Impact =
(B-A)-(D-C)
Outcome B
Program Group
B-A
D-C A D
Comparison Group
D-C C
Baseline
Follow-up
Time

8. Difference-in-Differences
Impact =
(B-A)-(D-C) B
Outcome
Program Group
B-A
D-C A D
Comparison Group
D-C C
Follow-up
Time
Baseline
Key condition: “Parallel trends assumption.” Program group would have had the same change as the Comparison group in absence of the program.

9. Difference-in-Differences
Impact =
(B-A)-(D-C)
Outcome B
Program Group
B-A A
True change;
diff-in-diff
under-estimates
program impact D
Comparison Group
D-C C
Baseline
Follow-up
Time
Key condition: “Parallel trends assumption.” Program group would’ve had the same change as the Comparison group in absence of the program.
Limitations: - Strong assumption; “true change” could’ve been different.

10. Difference-in-Differences
Impact =
(B-A)-(D-C)
Outcome B
Program Group
B-A A
True change;
diff-in-diff
under-estimates
program impact D
Comparison Group
D-C C
Baseline
Follow-up
Time
Key condition: “Parallel trends assumption.” Program group would’ve had the same change as the Comparison group in absence of the program.
Limitation: - Strong assumption; true change could’ve been different
- It requires “short” time interval, but it reduces magnitude of impact to estimate.

11. Difference-in-Differences
Impact =
(B-A)-(D-C)
Outcome B
Program Group
B-A
D-C A D
Comparison Group
D-C C
Baseline
Follow-up
Time
Key Issue: Selection of Comparison Group
Question: What is the best way to select a program and comparison group, so the two groups will behave similarly and will have the same change, in the absence of the program?

12. Difference-in-Differences: Testing the “Parallel trends assumption”
Impact =
(B-A)-(D-C)
Outcome B
Program Group A D E
Comparison Group C F
Pre-Baseline
Baseline
Follow-up
Time
One way: You need Pre-Baseline data!

13. Difference-in-Differences: Testing the “Parallel trends assumption”
Impact =
(B-A)-(D-C) B
Outcome
Program Group A D E
A-E
Comparison Group C F
C-F
Pre-Baseline
Baseline
Follow-up
Time
One way: You need Pre-Baseline data!
In this example, “Parallel trends assumption” holds if: (A-E)=(C-F)
Problems: - Pre-baseline data rarely available
- Past behavior is only an indication of future behavior.

14. Difference-in-Differences: Not good if different true changes
Impact =
(B-A)-(D-C)
Outcome B
Program Group A E D
True change
Comparison Group C F
Pre-Baseline
Baseline
Follow-up
Time
In this case Diff-in-diff provides and incorrect estimate of program impact. It underestimates program impact.

15. Difference-in-Difference: Not good if different trends
Impact =
(B-A)-(D-C)
Outcome B
Program Group
True Impact A E D
True change
Comparison Group C F
Pre-Baseline
Baseline
Follow-up
Time
In this case Diff-in-diff provides and incorrect estimate of program impact. It underestimates program impact.

16. Difference-in-Differences: Extensions (3 points in time)
Outcome B
Impact 1
Program Group A D
Comparison Group C
Baseline
Follow-up 1
Follow-up 2
Time
Key condition: “Parallel trends assumption” holds for each time period.

17. Difference-in-Differences: Extensions (3 points in time)
Outcome G
Impact 2 B
Impact 1
Program Group A D H
Comparison Group C
Baseline
Follow-up 1
Follow-up 2
Time
Key condition: “Parallel trends assumption” holds for each time period.

18. The DID model

19. Figure 1. DID Model – Structure of the pooled data set
Variable names
Iid Cluster P T PxT Y x1 x2 x3 … ID
Iid: Individual identifier number
… … …
… … …
… … …
… … …
… … …
… … …
… … …
… … … … … … … … …
… … …
… … …
… … …
Cluster ID: Cluster identifier number
P: Program summy
T: Time dummy
PxT: The interaction dummy
Y: The dependent variable
x1, x2, x3 : cluster, household, or individual characteristics
Baseline
… … …
… … …
… … …
… … …
… … …
… … …
… … …
… … … … … … … … …
… … …
… … …
… … …
Follow- Up

20. Difference-in-differences
Method widely used in program evaluation
Key is the “Parallel trends assumption.” It works better if the program was randomly allocated between the program group and the comparison group. The two groups will be “similar” in observed and unobserved characteristics.
An alternative is to use matching procedures to find a matched Comparison Group, before you implement the baseline. You can match on community-characteristics.
It works better if there is a “short” time interval between baseline and follow-up, but, how “short” to still measure impact? It depends on the outcome and selection of comparison group.
It could control for fixed unobserved characteristics that could be the source of biased estimates of program impact (endogeneity). It is better to have longitudinal surveys.

21. Thank you!

22. This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government.