1. Impact Evaluation Methods

2. Randomized Trials Regression Discontinuity Matching Difference in Differences

3. The Goal Causality We did program X, and because of it, Y happened.

4. The Goal Causal Inference Y happened because of X, not for some other reason. Thus it makes sense to think that if we did X again in a similar setting, Y would happen again.

5. Getting to Causality In a more research-friendly universe, we’d be able to observe a single person (call him Fred) after we both gave and didn’t give him the treatment. Y treated Fred -Y untreated Fred

6. Getting to Causality In the reality-based community, finding this Y treated Fred -Y untreated Fred “counterfactual” is impossible. Is the solution to get more people?

7. Getting to Causality With more people, we can calculate Average (treated)-Average(untreated). But what if there’s an underlying difference between the treated and untreated?

8. Getting to Causality Confounding Factors/Selection Bias/Omitted Variable Bias Textbook Example: If textbooks were deliberately given to the most needy schools, the simple difference is incorrect. If textbooks were already present in the schools where parents cared a lot about education, the simple difference is incorrect.

9. Problem Solved If we randomize the treatment, on average, treatment and control groups should be the same in all respects, and there won’t be selection bias. Check that it’s true for all observables. Hope that it’s therefore true for all unobservables.

10. Math You’d Rather Not See See Clair’s slides from September 15 -omitted variable bias Very accessible reading from same week by Duflo, Glennerster & Kremer. -selection bias

11. Randomization Randomize who gets treated. Check if it came out OK. Basically, that’s it.

12. Randomization Examples: Progresa-Cash if kids go to school Moving to Opportunity-voucher to move to better neighborhood Fertilizer & Hybrid Seed Loan maturity & Interest rate Deworming

13. Regression Discontinuity Being involved in a program is clearly not random. Smarter kids get get scholarships. Kids in smaller classes learn better. Big firms are more likely to unionize.

14. Regression Discontinuity Being involved in a program is clearly not random. Or is it? Scholarship cutoff +1 girl vs. scholarship cutoff-1 girl Isreali 41 kid school vs. Isreali 40 kid school Union-yes 50%+1 school vs. Union-yes 50% -1 school

15. Regression Discontinuity Being involved in a program is clearly not random. Or is it? Scholarship cutoff +1 girl vs. scholarship cutoff-1 girl Isreali 41 kid school vs. Isreali 40 kid school Union-yes 50%+1 school vs. Union-yes 50% -1 school

16. So how do we actually do this? 1.Draw two pretty pictures 1.Eligibility criterion (test score, income, or whatever) vs. Program Enrollment 2.Eligibility criterion vs. Outcome

17. So how do we actually do this? 2. Run a simple regression. (Yes, this is basically all we ever do, and the stats programs we use can run the calculation in almost any situation, but before we do it, it’s necessary to make sure the situation is appropriate and draw the graphs so that we can have confidence that our estimates are actually causal.) Outcome as a function of test score (or whatever), with a binary (1 if yes, 0 if no) variable for program enrollment.

18. As Good As Random, Sort Of Randomize who gets treated (within a bandwidth). Check if it came out OK (within a bandwidth). (within a bandwidth) Basically, that’s it (within a bandwidth).

19. Difference in Differences Change for the treated - Change for the control (t1-t0)-(c1-c0) t1-t0-c1+c0 t1-c1-t0+c0 t1-c1-(t0-c0) Which is the same as…

21. Examples Malaria Bleakley, Hoyt. Malaria Eradication in the Americas: A Retrospective Analysis of Childhood Exposure. Working paper. Land Reform Besley, Timothy and Robin Burgess. Land Reform, Poverty Reduction, and Growth: Evidence from India. Quarterly Journal of Economics. May 2000, 389-430.

22. Matching Match each treated participant to one or more untreated participant based on observable characteristics. Assumes no selection on unobservables Condense all observables into one “propensity score,” match on that score.

23. Matching After matching treated to most similar untreated, subtract the means, calculate average difference

24. Matching Examples: Does piped water reduce diarrhea? Jalan, Jyotsna and Martin Ravallion. Does Piped Water Reduce Diarrhea for Children in Rural India? Journal of Econometrics. January 2003, 153-173. Anti-poverty program in Argentina Jalan, Jyotsna and Martin Ravallion. Estimating the Benefit Incidence of an Antipoverty Program by Propensity Score Matching. Journal of Business and Economic Statistics. January 2003, 19-30.

25. Matching Matching algorithm can be performed in many ways. Guido Imbens’ webpage http://elsa.berkeley.edu/~imbens/estimators.shtml

26. Summary The weakest (easiest) assumption is the best assumption. Randomization wins. Real scientists use it too.

27. Proof by One Example LaLonde, Robert. Evaluating the Econometric Evaluations of Training Programs with Experimental Data. American Economic Review, September 1986. Run a randomization and analyze it well. Then pretend you don’t have all the data that you do, construct fake comparison groups using the census, and show that none of your crazy methods get you right answer.