1. Different Methods of Impact Evaluation

2. How to measure impact? Assessing causality Event 2 (Effects) Event 1
Impact of program = outcome 1 – outcome 2
In practice, we compare two groups, one of which benefited from the program, the other one did not
Event 2 (Effects)
Outcome 1
Event 1
e.g. Education program/ treatment
Causes
Event 2 occurs if and only if Event 1 occurred before
Event 1
No Education Program
Event 2
Outcome 2
Causes

3. Constructing the counterfactual
what would have happened in the absence of the program? …
… for the people who benefitted from the program: we don’t observe it
thus, impact evaluation will have to mimic it
Counterfactual is constructed by selecting a group not affected by the program – this is the main challenge of impact evaluation
Methods differ by the way the counterfactual is constructed

4. Non-Experimental Methods
Differences between outcomes for treated and non-treated (in our ex: reached / not) are the sum of
Inherent differences: “SELECTION” BIAS
TREATMENT EFFECT: that we want to isolate from 1
The essence of non-experimental methods is to find a way to redress the selection bias ex-post.
The critical objective of impact evaluation is to establish a credible comparison group – a group of individuals who in the absence of the program would have had outcomes similar to those who were exposed to the program.
However, in reality it is generally the case that individuals who participate in a program and those who were not are different:
programs are placed in specific areas (for example, poorer or richer areas)
individuals are screened for participation in the program (for example, on the basis of poverty or on the basis of their motivation)
and, in addition, the decision to participate is often voluntary.
For all of these reasons, those who were not exposed to a program are often not a good comparison group for those who were, and any differences between the groups can be attributed to two factors: pre-existing differences (selection bias) and the impact of the program.
Since we have no reliable way to estimate the size of the selection bias, we typically cannot decompose the overall difference into a treatment effect and a bias term.
Selection bias disappears in the case of randomization

5. Non-Experimental Methods
Simple Difference
Multivariate (Multiple) Regression
“Difference in difference” (Multiple Regression with Panel Data)
Matching
Randomization/RCT’s (advanced topic covered separately)
The critical objective of impact evaluation is to establish a credible comparison group – a group of individuals who in the absence of the program would have had outcomes similar to those who were exposed to the program.
However, in reality it is generally the case that individuals who participate in a program and those who were not are different:
programs are placed in specific areas (for example, poorer or richer areas)
individuals are screened for participation in the program (for example, on the basis of poverty or on the basis of their motivation)
and, in addition, the decision to participate is often voluntary.
For all of these reasons, those who were not exposed to a program are often not a good comparison group for those who were, and any differences between the groups can be attributed to two factors: pre-existing differences (selection bias) and the impact of the program.
Since we have no reliable way to estimate the size of the selection bias, we typically cannot decompose the overall difference into a treatment effect and a bias term.
Selection bias disappears in the case of randomization

6. Simple difference
Simple difference is a first measure – why is it not sufficient?
There may be differences between the two groups (age, location, gender, initial endowment, bargaining power)
So we may want to control for these differences

7. Multi-variate regression
Suppose we can observe these differences.
Age composition of the group, initial infrastructure in the school, level of education, …
We can include all these variables in a regression: Y = a.T + b.Age + c.Infr + d.Edu +…
A regression provides the linear combination of observable variables that best “mimics” the outcome
Each coefficient represents the effect of each variable
Will give us the effect of the treatment everything else being equal, or more exactly every other observable characteristics being equal

8. Multi-variate regression
Problems of the regression
You may want to include many many variables, to control for as many characteristics as possible
Problem of sample size (degrees of freedom)
More important: do you have measures of everything?
Bargaining power, Pro-activeness, Intrinsic motivation, hopelessness
There are unobservables

9. Panel data Simple difference: before/after
Counterfactual = same group before the program
Can we trust this? Assumption = would have remained the same
Ex: Police project
Double difference
Control for the situation before the program
Ex: Group 1 = Treatment Group; Group 2 = Control Group
2006: Group 1 : Group 2 : 60
2009: Group 1 : 50 (+66%) Group 2 : 90 (+50%)
Effect = +16%
Assumption: they would have grown at the same pace
Not sure…

10. Matching
We compare pairs of 2 individuals for which the values taken by ALL variables are the same

11. Matching
Variation: Propensity Score Matching: all the variables do not need to be exactly the same, but you look for individuals which have the same “profile”
Problems
This matching method requires a big dataset: find pairs on a sufficient number of variables
What about unobservables?

12. An example Case study: US Congress elections, 2002
phone calls to potential voters to encourage them to vote; reached
Outcome: did they actually go vote?
1st method: compare the (reached) vs. the (not reached)
2nd method: introduce co-variates in a regression
3rd method: introduce baseline data (vote in 1998)
4th method: do a matching

13. 1st comparison: we suspect a selection bias
Reached
Not Reached
Difference
Female
56.2%
53.8%
2.4 pp*
Newly Regist.
7.3%
9.6%
-2.3 pp*
From Iowa
54.7%
46.7%
8.0 pp*
Voted in 2000
71.7%
63.3%
8.3 pp*
Voted in 1998
46.6%
37.6%
9.0 pp*
Selection Bias: differences in observable / unobservable characteristics → differences in outcome not due to the treatment 13 13

14. Non-Experimental Methods
Estimated Impact
1 – Simple Difference
10.8 pp *
2 – Multiple regression
6.1 pp *
3 – Multiple regression with panel data (diff-in-diff)
4.5 pp *
4 – Matching
2.8 pp *
5 – Randomized Experiment
0.4 pp

15. For more details, please read Case Study: “Get out the vote
For more details, please read Case Study: “Get out the vote? Do phone calls encourage voting” under References