Food and Agriculture Organization of the United Nations Office of Evaluation March 2015 UNEG EPE 2015 FAO Impact Assessments in Recovery Context

Fragile, insecure and volatile context FAO cash-for-work programme in Somalia FAO livelihood recovery programme post-floods in Pakistan. FAO South Sudan response programme. Field impact assessment in fragile, insecure and volatile context Under the restoring livelihoods/recovery in food security and nutrition sector. High demand for evaluation and FIA, Large interventions, complexity of many community level interventions Crowded operating environment, Weak M&E data (no baseline, high output delivery…est.) FAO Office of Evaluation

Traditional methods and its limitations Randomized Control Trial (RCT) or statistical counterfactuals. Quasi-experimental designs with a pretest/posttest comparison and a matched control group. Non-experimental options (observation, interactions, surveys or case studies). For RCTs = no available data for social accounting matrixes, market data..est. very expensive For statistical counterfactuals and posttest/pretest = no comparison group, control groups can not be identified (IDPs), no baselines, no access volatile environment…est. Non-experimental options = Systematic positive biased (we rely on direct beneficiaries and implementing partners). FAO Office of Evaluation

FAO Office of Evaluation The alternatives Realist evaluations. Process tracing. Participatory Rural Appraisal (PRA) time-related and relation techniques. Creative identification of comparison groups (Judgmental Matching). Pipeline design. Creative uses of secondary data. Realist evaluations: The approach focuses on the specific context in which a programme is implemented and addresses the questions: “What works?”, “For whom?”, “When?” and “Why?” Whereas the conventional counterfactual focuses on the comparison group and often pays very little attention to the “factual”, realist evaluation seeks to understand in detail the mechanisms through which an intervention operates and how different sectors of the target population are affected. Process tracing: Looks at the little steps along the way and asks “Could there have been another way of doing each step? And what difference would this have made?” – breaking up the intervention into compounents, very useful but needs a sound ToC. PRA time-related and relation techniques: Social Mapping, Pairwise Ranking & Problem Analysis, Well-being analysis…est. Judgmental Matching: involves creating a comparison group by finding a match for each site in the treatment group based on the evaluation team judgements about what variables are important Rather than comparisons between areas with and without projects, and can be used to compare their project with alternative interventions. Pipeline design: When programme is implemented in phases over a period of time (pilot areas then scaled up), the segments of the population only affected by the latter phases can be used as comparison groups for the earlier phases. Creative uses of secondary data: The object is to find secondary data that describes in sufficient detail what change occurred in reasonably comparable communities during the time-frame of the programme. Though this can be done qualitatively, adding a IPPC Map FAO Office of Evaluation

FAO Office of Evaluation Examples FAO cash-for-work programme in Somalia: Participatory Rural Appraisal (PRA) FAO livelihood recovery programme in Pakistan Pipeline design FAO South Sudan response programme Judgmental Matching PRA time-related and relation techniques: Social Mapping, Pairwise Ranking & Problem Analysis, Well-being analysis…est. Pipeline design: When programme is implemented in phases over a period of time (pilot areas then scaled up), the segments of the population only affected by the latter phases can be used as comparison groups for the earlier phases. Judgmental Matching: involves creating a comparison group by finding a match for each site in the treatment group based on the evaluation team judgements about what variables are important Rather than comparisons between areas with and without projects, and can be used to compare their project with alternative interventions. FAO Office of Evaluation

FAO Office of Evaluation Lessons learned - 1 Highly useful - reveals a range of programmatic issues that feeds into future project designs, corrective actions and better targeting. Quality secondary data can be used to measure quantitative outcome and impact indicators. Attributing causality remains a challenge for generalization, thought provides details on detailed dynamics to paint the big picture . Timing of the impact assessment is key. Important to assess the adequacy of the secondary data in terms of sample coverage, when it was collected, and how the indicators were measured. Often secondary data is not available to control for alternative explanations of the observed changes (such as special characteristics of the project group making them more likely to succeed). FAO Office of Evaluation

FAO Office of Evaluation Lessons learned - 2 Selecting the best available comparison group on which data can be collected within tight budget and time frames. Comparison with other programs that use different approaches for achieving similar objectives. In order to obtain a counterfactual, similar PRA exercises could be conducted in comparable communities. Sometimes these variations were built into the programme design, in other cases they occur because of unforeseen circumstances or because each project is given considerable autonomy in terms of how programmes are organized. In order to obtain a counterfactual, similar PRA exercises could be conducted in comparable communities, asking them to identify changes that occurred in their lives relative to the kinds of changes the programme was promoting in the target communities. FAO Office of Evaluation

FAO Office of Evaluation Thank you FAO Office of Evaluation