1. Prof. (FH) Dr. Alexandra Caspari caspari@fb4.fh-frankfurt.de www.fh-frankfurt.de/caspari Rigorous Impact Evaluation What It Is About and How It Can Be Done In Practice Alexandra Caspari, Frankfurt/Main Germany Conference »Perspectives on Impact Evaluation: Approaches to Assessing Development Effectiveness« 31 st March – 2 nd April 2009, Cairo

2. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 1 University of Applied Sciences Historical Review – The Evaluation Gap  MDGs (2000), ‘Paris Declaration on Aid Effectiveness’ (2005), and ‘Agenda for Action’ (Accra, 2008):  Increasing attention to Impact Evaluations  Lack of knowledge about effectiveness of projects and programs  2006: Report “When will we ever learn?” of the CGD ‘Evaluation Gap Working Group’  gap in quantity and quality of impact evaluations: -too few impact evaluations are being carried out and -those conducted often unable to properly assess impact because of methodological shortcomings  Recommendation: ‘Collective Action’  International Initiatives (NONIE, 3IE, …)

3. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 2 University of Applied Sciences What is Impact Evaluation?  OECD/DAC (2002): “positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”  emphasises on ‘produced by’: -measures impact with clear causation (causal attribution) -considers the counterfactual, i.e. the question “What difference did this program make?” “What would have happened without the intervention?” Rigorous Impact Evaluation (RIE):  Distinction against more “usual evaluations” by adding “rigorous”  focus on clear causation  use of adequate methods (to meet methodological shortcomings)  most important point: selection of the evaluation design to consider the counterfactual

4. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 3 University of Applied Sciences The Counterfactual  Causal effect: An actual effect δ i caused by a treatment T (a program) is the difference between the outcome Y i1 under a treatment T (T=1), i.e. program participant, minus the alternative outcome Y i0 that would have happened without the treatment T (T=0), i.e. non-participant :  Impact is not directly observable: -one can observe any given individual either as a treated person (participant) or untreated person (non-participant) but not both states -if individual i is participating in a program (T=1), then the outcome Y i0 is unobservable -this unobservable outcome Y i0 is called counterfactual  Analyzing the difference between the observed outcome and the unobserved potential outcome by choosing the best evaluation design

5. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 4 University of Applied Sciences Considering the Counterfactual  often used non-experimental designs: ● : observation, P: participants (treated), t: time (first, second observation), X: project intervention one-group pre-test post-test design (a) time t2t2 t1t1 P measured impact impact indicator  measured impact =  the counterfactual is not considered!  with non-experimental designs causal attribution is not possible!

6. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 5 University of Applied Sciences Considering the Counterfactual  necessary: experimental or quasi-experimental designs  adequate comparison group (‘with-and without comparison’)  „Real“ Experiments / Randomized Controlled Trials (RCTs): (Laboratory)Experiments: -random assignment of individuals to treatment (P) and control group (C)  groups differ solely due to chance -treatment and conditions are known/checkable Field experiments: -take place in real-world settings -anyhow treatment and control groups are assigned at random  Quasi-Experiments: -no random assignment -has a source of randomization that is “as if” randomly assigned -control group is often reconstructed ex-post

7. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 6 University of Applied Sciences Considering the Counterfactual ● : observation, P: participants (treaded), C: control group (non-treated), D: difference, t: time (first, second observation), X: project intervention over- estimated impact one-group pre-test post-test design (a) time t2t2 t1t1 P measured impact impact indicator static group comparison (4) time impact indicator t2t2 t1t1 C P measured impact = D t2 (single difference) time impact indicator t2t2 t1t1 C P D t1 pre-test post-test control group design (1)/(2) (double difference) measured impact = D t2 – D t1 D t2

8. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 7 University of Applied Sciences Approaches to Impact Evaluation  appropriate impact evaluation designs are often reject as unnecessarily sophisticated or because of ethical concerns  various realistic ways in which quasi-experimental designs can be introduced in an ethically and politically acceptable manner: -Matching on Observables -Regression Discontinuity -Propensity Score Matching (PSM) -Pipeline Approach -Multiple Comparison Group Design

9. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 8 University of Applied Sciences Possible Approaches in Practice  Matching on Observables: -characteristics (access tor services, economic level, type of housing, etc.) on which the comparison group should match the program group (individuals, households or areas) are identified carefully -often easily observable or identifiable characteristics -unobservable differences has to be kept in mind -control group is build out of those individuals, households or areas which match best -quasi-experimental design “pretest-posttest-comparison with post- test non-equivalent control group” (3) or at least “static group comparison” (4) is possible  single-difference (SD) possible

10. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 9 University of Applied Sciences Possible Approaches in Practice  Regression Discontinuity: -if a program is assigned using a clear threshold for eligibility comprised for one ore more criteria (age, income less than…) -control group is built out of those just above the threshold and hence not eligible for the program -those individuals will have comparable characteristics -quasi-experimental design “pre-test post-test non-equivalent control group design” (2) possible!  double-difference (DD) possible!

11. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 10 University of Applied Sciences Possible Approaches in Practice  Pipeline Approach: -if large programs (housing or community infrastructure, immunization, …) are introduced in phases over several years -when there are no major differences between the characteristics of families, communities scheduled for each phase and -when there is no selection criteria for participants of the first phase (the poorest families, communities, …)  participants of phase 2 & 3 = control group for participants phase 1  quasi-experimental design “pre-test post-test non-equivalent control group design” (2) possible!  double-difference (DD) possible

12. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 11 University of Applied Sciences Important Remarks  The international discussion about RIE refers just to a small aspect of evaluation: the causal attribution of impact  Impact is measured at the level of target groups/participants  because target groups are typically large, for this evaluation step quantitative methods are necessary (representativeness vs. profundity)  other evaluation methods are not condemned!  causal attribution is necessary but not sufficient  ‘black box’ remains: why does a program have impact (or does not)  comprehensive meaningful and reliable impact evaluations need the use of mixed method, i.e. use of quantitative and qualitative methods

13. Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 12 University of Applied Sciences  http://www.fh-frankfurt.de/de/.media/~caspari/2008bmzwpwirkungsevaluation.pdf Reference: Caspari, Alexandra/Barbu, Ragnhild (2008): Wirkungsevaluierungen Zum Stand der internationalen Diskussion und dessen Relevanz für die Evaluierung der deutschen Entwicklungszusammenarbeit