1. Sampling for Impact Evaluation -theory and application-
Carlo Azzarri, IFPRI
November 27, 2018
University of Rome ”La Sapienza”

2. Outline Introduction Targeting analysis Objective Theory of change
Impact Evaluation (IE)
Evaluation design
Sample size
Randomization
Type I and II errors
Panel data
11. Conclusions

3. Introduction
This is a basic introduction to sampling for impact evaluation.
Focuses mainly on Impact Evaluation and simple random sample design, but basic ideas are transferrable to more complex randomized designs and non-random sample designs.
Main Question: how do we construct a sample to credibly detect a given effect size within our evaluation budget constraints?

4. What we are not going to talk about…
Being an expert in sample design for IE
Non-random designs (matching, Regression Discontinuity Designs, etc.)

5. Targeting Analysis
Do we need to select the poorest/hungriest farmers to benefit from the project?
2. Do we need to select the most promising farmers?
If so, how would we select them under 1 and 2?

6. Targeting Analysis: selection of farmers
Is there any system already in place to target the poorest/hungriest vs. promising farmers?
Collect basic information on demographics and, possibly, land assets (listing ALL farmers in project areas)
Assign farmers to 2 levels
Subsistence farmers (poorest)?
Market-driven and most advanced (promising) farmers?
Which farmers are expected to benefit more from the project?

7. Targeting Analysis: Community validation
Lists of eligible farmers are brought to the community
Only farmers whose status is confirmed by the community can be beneficiaries
Target group meet jointly the two conditions:
Being subsistence farmers
AND poor according to the community

8. Objective
Determine the causal effect of the project on outcomes (not only on outputs):
Farmers’ wellbeing?
Land productivity?
Input supply, labor productivity, environment, women’s conditions, health and nutrition,…?
…all of the above plus-> for whom? For which development domain? For which type of households? For which livelihood?
What would be the impact with a different intervention?
Is the intervention sustainable, especially at scale?

9. EVALUATE EFFECTIVENESS
Objective (cont’d)
Monitoring to track implementation efficiency (input - output)
Evaluation to estimate causal effectiveness on outcomes (output - outcome)
INPUTS
OUTCOMES/IMPACTS
OUTPUTS
MONITOR EFFICIENCY
EVALUATE EFFECTIVENESS
$$$
BEHAVIOUR
Note: Diagram from World Bank training material produced by Arianna Legovini, Lead Economist - AIEI

10. Theory of Change/1
Impact evaluation must be based on a set of hypotheses on the change that can be achieved as a consequence of the intervention
How would you think the project can affect the life of the beneficiaries?

11. Theory of Change/2 ACTIVITIES OUTPUTS FIRST ORDER OUTCOMES
Livelihood Strategies /Coping Strategies /Vulnerability to Shocks
Project implementation
Production increase
Income and Expenditure
Saving/
Investment
General Household Expenditure
School Enrolment and Attendance
Expenditure on Health and Education for children
School Progression
Health Status
Food Intake
Dietary Diversity
Food Security
Psychological well being *
Other Expenditures for children: Food, Clothing, Recreation
Targeting
ACTIVITIES
OUTPUTS
FIRST ORDER OUTCOMES
SECOND ORDER OUTCOMES
Asset Building
THIRD ORDER OUTCOMES
Labour Participation
Child Labour
Remittances
Access to Credit
Time Allocation of Children
Time Allocation/ Caring arrangements/ Migration of Adults and Caregivers *
Intra-household decision making *
Utilization of Health Services
Time and risk preferences

12. Impact Evaluation
How would you go about measuring the causal impact of a project on …
-productivity?

13. Impact Evaluation - MethodY
Bens
Impact? t
Before
After 13

14. Impact Evaluation - MethodY
Bens
Non Bens
RCTs t
Before
After 14

15. Impact Evaluation - Method
Diff in Diff
Impact Y
Pre-existing Difference
Bens
Non Bens
When panel data are not available or when there is additional need to account for baseline differences between treatment and control groups, propensity score matching or propensity score weighting can be applied. The details for applying these techniques are developed in the next section. t
Before
After 15

16. Impact Evaluation - Method
Diff in Diff
Impact Y
with
Propensity Score Matching
Bens
Non Bens
When panel data are not available or when there is additional need to account for baseline differences between treatment and control groups, propensity score matching or propensity score weighting can be applied. The details for applying these techniques are developed in the next section.
Gary King: Why Propensity Scores Should Not Be Used for Matching t
Before
After 16

17. Impact Evaluation
What about if we have a sharp eligibility cut-off point?
-assume the project targets only farmers with <.3 ha

18. Regression Discontinuity Design

19. Impact Evaluation - Method
Causal effect: change that is due to our project and not to other actors or factors (confounders)
… taking into account any other factors also changing during the program period
… taking into account any systematic differences between beneficiaries and non- beneficiaries of AR intervention
It is crucial that the “control group” is comparable to the “treatment group”

20. Evaluation Design
How can we ensure that treatment and control villages are comparable?
Treatment villages
Control villages

21. Evaluation Design
How can we ensure that treatment and control villages are comparable?

22. Random Treatment Assignment
Evaluation Design
Random Treatment Assignment

23. Random Treatment Assignment
Evaluation Design
Random Treatment Assignment

24. Evaluation Design A B C D Treatment villages Control villages
BENEFICIARY FARMERS
WOULD BE BENEFICIARY FARMERS A B
NON BENEFICIARY FARMERS
WOULD BE NON BENEFICIARY FARMERS C D

25. X Randomized assignment with multiple interventions
4. Randomize intervention 2
3. Randomize intervention 1
1. Eligible Population
2. Evaluation sample
= Ineligible X
= Eligible

26. Spillovers: How to know about direct and indirect impacts?
Treatment Group A
Non-Treatment Group C
Affected by Spillovers
Pure Control Group B
E.g. Sharing food. Even when the comparison group is not directly provided with the program, it may indirectly be affected by spillovers from the treatment group. Spillovers themselves are often of policy interest because they constitute indirect program impacts.
Direct program impact = A – B
Spillover/indirect impact = C - B