Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences Slides by Paul J. Gertler & Sebastian Martinez

2 Measuring Impact Experimental design/randomization Quasi-experiments –Regression Discontinuity –Double differences (diff in diff) –Other options

3 Case 4: Regression Discontinuity Assignment to treatment is based on a clearly defined index or parameter with a known cutoff for eligibility RD is possible when units can be ordered along a quantifiable dimension which is systematically related to the assignment of treatment The effect is measured at the discontinuity – estimated impact around the cutoff may not generalize to entire population

4 Anti-poverty programs  targeted to households below a given poverty index Pension programs  targeted to population above a certain age Scholarships  targeted to students with high scores on standardized test CDD Programs  awarded to NGOs that achieve highest scores Indexes are common in targeting of social programs

5 Target transfer to poorest households Construct poverty index from 1 to 100 with pre-intervention characteristics Households with a score <=50 are poor Households with a score >50 are non-poor Cash transfer to poor households Measure outcomes (i.e. consumption) before and after transfer Example: Effect of Cash Transfer on Consumption

6

7 Non-Poor Poor

8

9 Treatment Effect

10 Oportunidades assigned benefits based on a poverty index Where Treatment = 1 if score <=750 Treatment = 0 if score >750 Case 4: Regression Discontinuity

11 Case 4: Regression Discontinuity 2 Baseline – No treatment

12 Treatment Period Case 4: Regression Discontinuity

13 Potential Disadvantages of RD Local average treatment effects – not always generalizable Power: effect is estimated at the discontinuity, so we generally have fewer observations than in a randomized experiment with the same sample size Specification can be sensitive to functional form: make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: –Nonlinear Relationships –Interactions

14 Advantages of RD for Evaluation RD yields an unbiased estimate of treatment effect at the discontinuity Can many times take advantage of a known rule for assigning the benefit that are common in the designs of social policy –No need to “exclude” a group of eligible households/individuals from treatment

15 Measuring Impact Experimental design/randomization Quasi-experiments –Regression Discontinuity –Double differences (Diff in diff) –Other options

16 Case 5: Diff in diff Compare change in outcomes between treatments and non-treatment –Impact is the difference in the change in outcomes Impact = (Y t1 -Y t0 ) - (Y c1 -Y c0 )

17 Time Treatment Outcome Treatment Group Control Group Average Treatment Effect

18 Time Treatment Outcome Treatment Group Control Group Estimated Average Treatment Effect Average Treatment Effect

19 Diff in Diff Fundamental assumption that trends (slopes) are the same in treatments and controls Need a minimum of three points in time to verify this and estimate treatment (two pre- intervention)

20 Case 5: Diff in Diff

21 Impact Evaluation Example – Summary of Results