1. AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J. Gertler & Sebastian Martinez Impact Evaluation Methods: Impact Evaluation Methods: Randomization, IV, Regression Discontinuity

2. Measuring Impact Randomized Experiments Quasi-experiments –Randomized Promotion-Instrumental Variables –Regression Discontinuity –Difference in difference – panel data –Matching

3. Randomization The gold standard in evaluating the effects of interventions It allows us to form a treatment and control groups –identical characteristics –differ only by intervention Closest approximation to counterfactual

4. Random Assignment Each eligible unit has the same chance of receiving the intervention – Mimics chocolate experiment Allows us to compare the treatment and control group

5. Random Assignment vs. Random Sample Random Assignment –Are the observed results due to the intervention rather than other confounding factors? (internal validity) Random Sample –Do the results found in the sample apply to the general population/are they generalizable? (external validity)

6. Randomization Random Sample (external validity) Random Assignment (internal validity)

7. Example of Randomization What is the impact of providing free books to students on test scores? Randomly assign a group of school children to either: - Treatment Group – receives free books - Control Group – does not receive free books

8. Randomization Random Assignment

9. How Do You Randomize? 1) At what level? –Individual –Group School Community District

10. When would you use randomization? Universe of eligible individuals typically larger than available resources at a single point in time –Fair and transparent way to assign benefits –Gives an equal chance to everyone in the sample Good times to randomize: –Pilot programs –Programs with budget/capacity constraints –Phase in programs

11. Oportunidades Example Randomized treatment/controls –Community level randomization 320 treatment communities 186 control communities Pre-intervention characteristics well balanced

12. Oportunidades Example

13. Difference in Means

14. Impact Evaluation Example: Randomization vs. Other Methods

15. Other Analyses Often Lack Internal Validity Enrolled vs non-enrolled –The baseline characteristics will be different because people have chosen which group they want to be in Before and after –There are often other interventions going on at the same time

16. Measuring Impact Randomized Experiments Quasi-experiments –Randomized Promotion-Instrumental Variables –Regression Discontinuity –Difference in difference – panel data –Matching

17. When Do We Use Random Promotion? Common scenario: –National Program with universal eligibility –Voluntary enrollment in program Can not control who enrolls and who does not

18. Randomized Promotion Possible solution: random promotion or incentives into the program –Information –Money –Other help/Incentives Also called –Encouragement designs –Incentive schemes

19. Study Components Intervention –Chocolate Randomized Promotion –Encouragement to take chocolate

20. Example of Promotion Design for SATs What is the impact of supplementary learning material on student test scores? –Outcome Student test scores –Intervention Supplementary learning materials (all teachers can access these) –Randomized promotion Letters encouraging students to use materials (sent to randomly assigned teachers)

21. What Information Does Randomized Promotion Give Us? How effective is the treatment? –We can analyze the effect the treatment had on the outcome in the sub-group of subjects who would not have used the intervention unless encouragement was present

22. How Effective is the Treatment? Local Average Treatment Effect –Effect of the intervention on those who would not have enrolled unless encouraged

23. Encouragement Design ENCOURAGED Take-up = 80% Mean outcome = 100 NOT ENCOURAGED Take-up = 30% Mean outcome = 90 Change Take-up = 50% Change Y=10 Impact = 20 Never Takeup Takeup if Encouraged Always Takeup

24. Example: Community Based School Management What is the effect of decentralization of school management on learning outcomes? –All communities are eligible –Community management of hiring, budgeting, oversight –Need to write proposal

25. Community Based School Management 1500 schools in the evaluation Each community chooses whether to participate in program

26. Promotion Design Community based school management –Provision of technical assistance and training by NGOs for submission of grant application –Random selection of communities with NGO support

27. Community Based School Management Outcome – learning outcome Intervention –decentralization of management to community –1500 schools Promotion –NGO support –schools randomized to receive this support

28. Examples – Randomized Promotion Maternal Child Health Insurance in Argentina –Intensive information campaigns Employment Program in Argentina –Transport voucher Community Based School Management in Nepal –Assistance from NGO Health Risk Funds in India –Assistance from Community Resource Teams

29. Randomized Promotion Just an example of an Instrumental Variable A variable correlated with treatment but nothing else (i.e. random promotion) Again, we really just need to understand how the benefits are assigned –Dont have to exclude anyone

30. Measuring Impact Randomized Experiments Quasi-experiments –Randomized Promotion-Instrumental Variables –Regression Discontinuity –Difference in difference – panel data –Matching

31. Introduction What is the impact of monetary incentives on test scores to schools below some ranking? Compare the schools right above the ranking point to schools below the cutoff point

32. Indexes are Common in Targeting of Social Programs Anti-poverty programs –targeted to households below a given poverty index Pension programs –targeted to population above a certain age Scholarships –targeted to students with high scores on standardized test

33. Example: What is the Effect of Cash Transfer on Consumption? Intervention: –Cash transfer to poor households Evaluation: –Measure outcomes (i.e. consumption, school attendance rates) before and after transfer, comparing households just above and below the cut-off point.

34. Example: What is the Effect of Cash Transfer on Consumption? Target poorest households for cash transfer Method: –construct poverty index from 1 to 100 with pre-intervention characteristics households with a score <=50 are poor households with a score >50 are non-poor

35. Non-poor Poor

37. Treatment effect

38. Regression Discontinuity When to use this method? –the beneficiaries/non-beneficiaries can be ordered –there is a cut-off point for eligibility. –cut-off determines the assignment of a potential beneficiary to the treatment or no- treatment

39. Regression Discontinuity 2 Baseline – No treatment

40. Regression Discontinuity Treatment Period

41. Sharp and Fuzzy Discontinuity Sharp discontinuity –the discontinuity precisely determines treatment –equivalent to random assignment in a neighborhood e.g. social security payment depend directly and immediately on a persons age Fuzzy discontinuity –discontinuity is highly correlated with treatment. –use the assignment as an IV for program participation. e.g. rules determine eligibility but there is a margin of administrative error.

42. Examples Effect of class size on scholastic achievement (Angrist and Lavy, 1999) Effect of transfers on labor supply (Lemieux and Milligan, 2005) Effect of old age pensions on consumption -BONOSOL in Bolivia (Martinez, 2005)

43. Potential Disadvantages of RD We estimate the effect of the program around the cut-off point. –the effect is estimated at the discontinuity –fewer observations than in a randomized experiment with the same sample size –limits generalizability Make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: –nonlinear relationships –interactions

44. Advantages of RD for Evaluation RD allows one to estimate the effect of an intervention at the discontinuity Can take advantage of a known rule for assigning the benefit that are common in the designs of social policy –No need to exclude a group of eligible households/individuals from treatment

45. Summary DesignWhen to useAdvantagesDisadvantages Randomization Whenever possible When an intervention will not be universally implemented Gold standard Most powerful Not always feasible Not always ethical Random Promotion When an intervention is universally implemented Learn and intervention Only looks at sub- group of sample Regression Discontinuity If an intervention is assigned based on rank Assignment based on rank is common Only look at sub- group of sample