Current practices in impact evaluation Howard White Independent Evaluation Group World Bank

Impact evaluation Defining characteristics: counterfactual analysis outcomes This presentation briefly overviews approaches to rigorous impact evaluation, using examples from various development agencies So not concerned here with other uses of word ‘impact’, such as Environmental impact assessment Participatory impact analysis

Impact evaluation in official development agencies Recent claims (e.g. CGD) that there is none: Not independent Not rigorous Our review showed Evaluation departments do a wide range of evaluations, many of which tackle impact through deductive means But there is a significant body of IE using rigorous methods But support claim for ‘more and better impact evaluation’

Before versus after The simplest comparison is to see how an indicator has changed during the intervention Normally this is monitoring, not evaluation – it tells the factual not the counterfactual Before is not the counterfactual as other things may also have changed However, sometimes before versus after is a valid measure of impact, e.g. for water supply reducing time collecting water (Finnish study) and school rehabilitation.

Simple comparison group Compare indicators amongst beneficiaries (treatment group) and non-beneficiaries (comparison group) This is a single difference comparison and is the most commonly found approach It is flawed (biased) if the way in which beneficiaries are selected has some correlation with the outcome indicators of interest. It is the failure to address this bias which is exciting such concern about lack of rigour

Examples of selection bias School facilities and learning outcomes Social funds and social capital Microfinance and SME development

How to address selection bias Random assignment (examples will come from DFID) Pipeline approach (e.g. UNCDF, DFID, and IDB) If selection based on observables then can use a variety of quasi-experimental means Propensity score matching (examples from IDB) Regression-based approaches, including regression discontinuity (also IDB) If unobservables are time invariant then can use panel data (or recall in single survey, e.g. IFAD) to remove them (double-differencing) Try to measure unobservables

But there’s more to impact evaluation than worrying about selection bias Open the black box: the importance of context and a theory-based approach Policy relevance Triangulation (Danida)

Doing an impact evaluation The importance of baseline data The time and cost of conducting a survey The potential of secondary data (IOB) The right skills mix

Challenges for development agencies Scale up rigorous impact evaluation Application of rigorous impact evaluation to new aid instruments Assessing impact in other evaluation studies, such as country evaluations

Meeting the challenges Scaling up Support initiatives (CGD and World Bank) IE Guidelines for own use, promote internally Training and mutual support Common or coordinated program New instruments and incorporating in other studies IE Guidelines to tackle these issues?

Main messages There is a case of doing more and better IE, meaning address selection bias Many agencies are already doing such studies, showing its feasibility Need to strengthen both technical rigour and use of theory-based approach Need to think of how to do IE beyond ‘projects’