Impact Evaluation Methods

Randomized Trials Regression Discontinuity Matching Difference in Differences

The Goal Causality We did program X, and because of it, Y happened.

The Goal Causal Inference Y happened because of X, not for some other reason. Thus it makes sense to think that if we did X again in a similar setting, Y would happen again.

Getting to Causality In a more research-friendly universe, we’d be able to observe a single person (call him Fred) after we both gave and didn’t give him the treatment. Y treated Fred -Y untreated Fred

Getting to Causality In the reality-based community, finding this Y treated Fred -Y untreated Fred “counterfactual” is impossible. Is the solution to get more people?

Getting to Causality With more people, we can calculate Average (treated)-Average(untreated). But what if there’s an underlying difference between the treated and untreated?

Getting to Causality Confounding Factors/Selection Bias/Omitted Variable Bias Textbook Example: If textbooks were deliberately given to the most needy schools, the simple difference is incorrect. If textbooks were already present in the schools where parents cared a lot about education, the simple difference is incorrect.

Problem Solved If we randomize the treatment, on average, treatment and control groups should be the same in all respects, and there won’t be selection bias. Check that it’s true for all observables. Hope that it’s therefore true for all unobservables.

Math You’d Rather Not See See Clair’s slides from September 15 -omitted variable bias Very accessible reading from same week by Duflo, Glennerster & Kremer. -selection bias

Randomization Randomize who gets treated. Check if it came out OK. Basically, that’s it.

Randomization Examples: Progresa-Cash if kids go to school Moving to Opportunity-voucher to move to better neighborhood Fertilizer & Hybrid Seed Loan maturity & Interest rate Deworming

Regression Discontinuity Being involved in a program is clearly not random. Smarter kids get get scholarships. Kids in smaller classes learn better. Big firms are more likely to unionize.

Regression Discontinuity Being involved in a program is clearly not random. Or is it? Scholarship cutoff +1 girl vs. scholarship cutoff-1 girl Isreali 41 kid school vs. Isreali 40 kid school Union-yes 50%+1 school vs. Union-yes 50% -1 school

Regression Discontinuity Being involved in a program is clearly not random. Or is it? Scholarship cutoff +1 girl vs. scholarship cutoff-1 girl Isreali 41 kid school vs. Isreali 40 kid school Union-yes 50%+1 school vs. Union-yes 50% -1 school

So how do we actually do this? 1.Draw two pretty pictures 1.Eligibility criterion (test score, income, or whatever) vs. Program Enrollment 2.Eligibility criterion vs. Outcome

So how do we actually do this? 2. Run a simple regression. (Yes, this is basically all we ever do, and the stats programs we use can run the calculation in almost any situation, but before we do it, it’s necessary to make sure the situation is appropriate and draw the graphs so that we can have confidence that our estimates are actually causal.) Outcome as a function of test score (or whatever), with a binary (1 if yes, 0 if no) variable for program enrollment.

As Good As Random, Sort Of Randomize who gets treated (within a bandwidth). Check if it came out OK (within a bandwidth). (within a bandwidth) Basically, that’s it (within a bandwidth).

Difference in Differences Change for the treated - Change for the control (t1-t0)-(c1-c0) t1-t0-c1+c0 t1-c1-t0+c0 t1-c1-(t0-c0) Which is the same as…

Examples Malaria Bleakley, Hoyt. Malaria Eradication in the Americas: A Retrospective Analysis of Childhood Exposure. Working paper. Land Reform Besley, Timothy and Robin Burgess. Land Reform, Poverty Reduction, and Growth: Evidence from India. Quarterly Journal of Economics. May 2000,

Matching Match each treated participant to one or more untreated participant based on observable characteristics. Assumes no selection on unobservables Condense all observables into one “propensity score,” match on that score.

Matching After matching treated to most similar untreated, subtract the means, calculate average difference

Matching Examples: Does piped water reduce diarrhea? Jalan, Jyotsna and Martin Ravallion. Does Piped Water Reduce Diarrhea for Children in Rural India? Journal of Econometrics. January 2003, Anti-poverty program in Argentina Jalan, Jyotsna and Martin Ravallion. Estimating the Benefit Incidence of an Antipoverty Program by Propensity Score Matching. Journal of Business and Economic Statistics. January 2003,

Matching Matching algorithm can be performed in many ways. Guido Imbens’ webpage

Summary The weakest (easiest) assumption is the best assumption. Randomization wins. Real scientists use it too.

Proof by One Example LaLonde, Robert. Evaluating the Econometric Evaluations of Training Programs with Experimental Data. American Economic Review, September Run a randomization and analyze it well. Then pretend you don’t have all the data that you do, construct fake comparison groups using the census, and show that none of your crazy methods get you right answer.