# AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J.

## Presentation on theme: "AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J."— Presentation transcript:

AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J. Gertler & Sebastian Martinez Impact Evaluation Methods: Impact Evaluation Methods: Randomization, IV, Regression Discontinuity

Measuring Impact Randomized Experiments Quasi-experiments –Randomized Promotion-Instrumental Variables –Regression Discontinuity –Difference in difference – panel data –Matching

Randomization The gold standard in evaluating the effects of interventions It allows us to form a treatment and control groups –identical characteristics –differ only by intervention Closest approximation to counterfactual

Random Assignment Each eligible unit has the same chance of receiving the intervention – Mimics chocolate experiment Allows us to compare the treatment and control group

Random Assignment vs. Random Sample Random Assignment –Are the observed results due to the intervention rather than other confounding factors? (internal validity) Random Sample –Do the results found in the sample apply to the general population/are they generalizable? (external validity)

Randomization Random Sample (external validity) Random Assignment (internal validity)

Example of Randomization What is the impact of providing free books to students on test scores? Randomly assign a group of school children to either: - Treatment Group – receives free books - Control Group – does not receive free books

Randomization Random Assignment

How Do You Randomize? 1) At what level? –Individual –Group School Community District

When would you use randomization? Universe of eligible individuals typically larger than available resources at a single point in time –Fair and transparent way to assign benefits –Gives an equal chance to everyone in the sample Good times to randomize: –Pilot programs –Programs with budget/capacity constraints –Phase in programs

Oportunidades Example Randomized treatment/controls –Community level randomization 320 treatment communities 186 control communities Pre-intervention characteristics well balanced

Difference in Means

Impact Evaluation Example: Randomization vs. Other Methods

Other Analyses Often Lack Internal Validity Enrolled vs non-enrolled –The baseline characteristics will be different because people have chosen which group they want to be in Before and after –There are often other interventions going on at the same time

Measuring Impact Randomized Experiments Quasi-experiments –Randomized Promotion-Instrumental Variables –Regression Discontinuity –Difference in difference – panel data –Matching

When Do We Use Random Promotion? Common scenario: –National Program with universal eligibility –Voluntary enrollment in program Can not control who enrolls and who does not

Randomized Promotion Possible solution: random promotion or incentives into the program –Information –Money –Other help/Incentives Also called –Encouragement designs –Incentive schemes

Study Components Intervention –Chocolate Randomized Promotion –Encouragement to take chocolate

Example of Promotion Design for SATs What is the impact of supplementary learning material on student test scores? –Outcome Student test scores –Intervention Supplementary learning materials (all teachers can access these) –Randomized promotion Letters encouraging students to use materials (sent to randomly assigned teachers)

What Information Does Randomized Promotion Give Us? How effective is the treatment? –We can analyze the effect the treatment had on the outcome in the sub-group of subjects who would not have used the intervention unless encouragement was present

How Effective is the Treatment? Local Average Treatment Effect –Effect of the intervention on those who would not have enrolled unless encouraged

Encouragement Design ENCOURAGED Take-up = 80% Mean outcome = 100 NOT ENCOURAGED Take-up = 30% Mean outcome = 90 Change Take-up = 50% Change Y=10 Impact = 20 Never Takeup Takeup if Encouraged Always Takeup

Example: Community Based School Management What is the effect of decentralization of school management on learning outcomes? –All communities are eligible –Community management of hiring, budgeting, oversight –Need to write proposal

Community Based School Management 1500 schools in the evaluation Each community chooses whether to participate in program

Promotion Design Community based school management –Provision of technical assistance and training by NGOs for submission of grant application –Random selection of communities with NGO support

Community Based School Management Outcome – learning outcome Intervention –decentralization of management to community –1500 schools Promotion –NGO support –schools randomized to receive this support

Examples – Randomized Promotion Maternal Child Health Insurance in Argentina –Intensive information campaigns Employment Program in Argentina –Transport voucher Community Based School Management in Nepal –Assistance from NGO Health Risk Funds in India –Assistance from Community Resource Teams

Randomized Promotion Just an example of an Instrumental Variable A variable correlated with treatment but nothing else (i.e. random promotion) Again, we really just need to understand how the benefits are assigned –Dont have to exclude anyone

Measuring Impact Randomized Experiments Quasi-experiments –Randomized Promotion-Instrumental Variables –Regression Discontinuity –Difference in difference – panel data –Matching

Introduction What is the impact of monetary incentives on test scores to schools below some ranking? Compare the schools right above the ranking point to schools below the cutoff point

Indexes are Common in Targeting of Social Programs Anti-poverty programs –targeted to households below a given poverty index Pension programs –targeted to population above a certain age Scholarships –targeted to students with high scores on standardized test

Example: What is the Effect of Cash Transfer on Consumption? Intervention: –Cash transfer to poor households Evaluation: –Measure outcomes (i.e. consumption, school attendance rates) before and after transfer, comparing households just above and below the cut-off point.

Example: What is the Effect of Cash Transfer on Consumption? Target poorest households for cash transfer Method: –construct poverty index from 1 to 100 with pre-intervention characteristics households with a score <=50 are poor households with a score >50 are non-poor

Non-poor Poor

Treatment effect

Regression Discontinuity When to use this method? –the beneficiaries/non-beneficiaries can be ordered –there is a cut-off point for eligibility. –cut-off determines the assignment of a potential beneficiary to the treatment or no- treatment

Regression Discontinuity 2 Baseline – No treatment

Regression Discontinuity Treatment Period

Sharp and Fuzzy Discontinuity Sharp discontinuity –the discontinuity precisely determines treatment –equivalent to random assignment in a neighborhood e.g. social security payment depend directly and immediately on a persons age Fuzzy discontinuity –discontinuity is highly correlated with treatment. –use the assignment as an IV for program participation. e.g. rules determine eligibility but there is a margin of administrative error.

Examples Effect of class size on scholastic achievement (Angrist and Lavy, 1999) Effect of transfers on labor supply (Lemieux and Milligan, 2005) Effect of old age pensions on consumption -BONOSOL in Bolivia (Martinez, 2005)

Potential Disadvantages of RD We estimate the effect of the program around the cut-off point. –the effect is estimated at the discontinuity –fewer observations than in a randomized experiment with the same sample size –limits generalizability Make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: –nonlinear relationships –interactions

Advantages of RD for Evaluation RD allows one to estimate the effect of an intervention at the discontinuity Can take advantage of a known rule for assigning the benefit that are common in the designs of social policy –No need to exclude a group of eligible households/individuals from treatment

Summary DesignWhen to useAdvantagesDisadvantages Randomization Whenever possible When an intervention will not be universally implemented Gold standard Most powerful Not always feasible Not always ethical Random Promotion When an intervention is universally implemented Learn and intervention Only looks at sub- group of sample Regression Discontinuity If an intervention is assigned based on rank Assignment based on rank is common Only look at sub- group of sample

Download ppt "AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J."

Similar presentations