Designing Impact Evaluations: What are the Appropriate Research Questions and Methods? Mywish K. Maredia Michigan State University Workshop for Managers of Impact Evaluation May 13, 2013 InterAction, 1400 16th St. NW, Suite 210 Washington, DC

IMPACT

Your Role in Explaining the ‘Miracle’ As Impact Evaluation Managers, your role is to ensure that a plan is in place to bridge the knowledge gap between how project outputs result in impacts; and Doing so based on evidence generated using credible methodology

Focus of this presentation Discuss the development of appropriate impact evaluation questions What types of impact evaluation designs are most appropriate in different contexts and given the evaluation questions? Purpose: To share some preliminary thoughts; present a framework; and To facilitate discussion and exchange of ideas / experience

Clarifying the term: Impact evaluation What it is: It is concerned with establishing a causal link between realized impacts (the effect) and an intervention (the ‘cause’) which could be a program, activity, policy change, etc. The goal of the analysis is to ‘rule out’ other possibilities / explanations for the observed effects

What do we mean by evaluation design? Every evaluation is essentially a research or a discovery project/activity If your results are to be reliable, you have to give the evaluation a structure that will tell you what you want to know That structure – the arrangement of discovery- is the evaluation’s design The design depends on what kinds of questions your evaluation is meant to answer

Development of Impact Evaluation Questions Characteristics of ‘appropriate’ IE questions: They should be narrow/specific Focus on small number of questions (~5) Focused on ‘summative’ evaluation of a project/intervention They should reflect the input of program staff and sponsors

Examples of common impact evaluation (research) questions Overall impact (effectiveness) Did it work? Did the intervention produce the intended impacts in the short, medium and long term? For whom, in what ways and in what circumstances did the intervention work? What unintended impacts (positive and negative) did the intervention produce? Source: Rogers (2012) Introduction to Impact Evaluation. Impact Evaluation Notes No.1

Examples of common impact evaluation (research) questions (cont’d) Nature of impacts and their distribution Are impacts likely to be sustainable? Did these impacts reach all intended beneficiaries? Influence of other factors on the impacts How did the intervention work in conjunction with other interventions, programs or services to achieve outcomes? What helped or hindered the intervention to achieve these impacts?

Examples of common impact evaluation (research) questions (cont’d) How it works How did the intervention contribute to intended impacts? What were the particular features of the intervention that made a difference? To what extent are differences in impact explained by variations in implementation? Matching intended impacts to needs To what extent did the impacts match the needs of the intended beneficiaries?

Common impact evaluation designs (focused on causal analysis) Methods for examining the factual. For e.g.: Comparative case studies Beneficiary/expert attribution Methods for creating counterfactual Experimental designs or RCTs (based on the principle of random assignment) Pipeline comparisons Other methods/approaches (using statistical techniques to form credible comparison groups). For e.g.,: Propensity score matching (PSM) Instrumental variables (IV) Regression discontinuity (RD) Difference in difference (DD)

IE Methods In theory, there are multiplicity of methods and approaches that can be used to assess impacts Each have problems and limitations There is no ‘one size fits all’ method/approach

Choosing an appropriate impact evaluation design Depends on… 1. The nature of the research questions Research Questions Appropriate Methods What is the effectiveness of a program Observational and correlational methods Whether observed effects can reasonably be attributed to the intervention and not to other sources Experimental and quasi-experimental methods What is the net impact of the program Cost effectiveness; cost-benefit analysis with qualitative methods to summarize the full range of impacts

Choosing an appropriate impact evaluation design (cont’d) Depends on… 2. The nature of your program Nature of your program Methods to consider Will you roll out your program over time? Pipeline design What is the unit of intervention--individuals, groups, communities? Those that give enough statistical power based on the number of ‘units of observations’ Is program assigned to participants or do they self-select? RCT, quasi-experimental There is no credible reason for other influencing factors (e.g., water pump) Before/after comparison

Choosing an appropriate impact evaluation design (cont’d) Depends on… 3. what participants / stakeholders will consent to? 4. What are your resources and time constraints?

Practical Considerations in Designing Impact Evaluation Establishing the program theory (the logic behind a program and the causal chain from inputs to outcome to impact) Understanding the program setting How participants are selected (to mitigate selection bias) Decision tree on which method is applicable and should be explored

Practical Considerations in Designing Impact Evaluation (Cont’d) Sample size Power calculation – Does the setting allow for enough numbers of units of intervention and units of observation for a robust design? Tradeoff between power and cost Time frame Is there enough time to observe the impact? Flexibility Strive for rigor, but be flexible…

Thank you