1. UKES Annual Conference
Climate adaptation and resilience impact assessments: Identifying ways to improve usefulness and feasibility
UKES Annual Conference
11th May 2017

2. Context
Growing interest in knowing what impact climate adaptation and resilience building interventions have
But challenging to conduct impact assessments that are useful, practical and ultimately good value for money
Objective: Identify ways of doing this
using experiences from the Malawian
Enhancing Climate Resilience Programme

3. Overview Types of resilience and adaptation interventions
Challenges of doing impact assessment
Potential ways of addressing the challenges: missing baselines and making assessments useful

4. Types of interventions
Resilience interventions includes anticipatory, absorptive and adaptive approaches
Examples of interventions (focussed on Household and community levels):
Agricultural practices and technologies
Weather information, early warning systems
Credit access: insurance, micro finance
Irrigation supply

5. Feasibility and usefulness challenges
Climate
shock
Intervention
Welfare
Impact
Flooding
- The disaster is wide-spread (a significant drought, very large flood or storm etc.).
- Your project is only being rolled out in a portion of the area affected by the disaster, or has a phased geographic rollout.
- The disaster is relatively predictable (based on annual weather patterns), so a counterfactual can be planned for.
A suitable comparison region can be identified within the affected area, with similar environmental and/or demographic profile.
- Disaster preparedness and response projects are often not suitable for a quasi-experimental approach to counterfactual analysis because:
Any disaster preparedness or response project will naturally aim to include all vulnerable groups in its delivery, making it difficult (or possibly unethical) to exclude certain groups for the sake of comparison.
- Disasters are often localised, making it hard to compare across regions. Even when a disaster strikes a large region, there will always be areas that are closer/further from the epicentre, making it difficult to find affected comparison groups outside of your treatment group.
- The extent of a disaster’s impacts on people or infrastructure is specific to the unique characteristics of that location (for example their level of vulnerability) adding further characteristics that have to be accounted for when selecting a comparison region.
Baseline
End-line
Time
Drought exposure

6. Using existing data: missing baselines
Sensitivity
Exposure
Adaptive
capacity
Interviews with project staff about
their targeting process
Malawi hazards and vulnerability mapping tool:

7. Using recall: missing baselines
All survey data is recall: matter of degree
Using the baseline data to cross-check recall responses
Being selective about which things you ask people to recall
Ask them to recall well-known events that coincide with the baseline date of interest
Bamberger Reconstructing Baseline Data for Impact Evaluation and Results Management. The World Bank Report 4.

8. Bound analysis: comparison group issues
Potential impact: increased food security
-20%
40%
80%
80% Households
secure
40% Households
Food secure
Endline
40% Households
Food secure
50% Households
Food secure
Baseline
Intervention area
Non-intervention area
Manski (2013) Public policy in an uncertain world
Manski (2007) Identification for Prediction and Decision

9. Combining counterfactual with theory-based approaches: improving usefulness
Judea Pearl (2009) Causality: Models, reasoning and inference

10. Other ways to improve usefulness and feasibility
Asking project stakeholders what intervention impacts they are interested in (used to develop a protocol)
Power tests informed by pilot study data
Ensuring that that cohorts of interest for the second phase are sufficiently powered
Mitchell, S. et al Causal Inference with Small Samples and
Incomplete Baseline for the Millennium Villages Project. Working paper.

11. Questions and discussion points
Tracking long-term impacts
What to do when no non-beneficiaries
How to use big data (mobile phones, social media)
Multiple overlapping interventions
Spillovers
Small sample sizes (synthetic controls)
Expert or modelling informed counterfactuals
Other theory-based impact assessment methods that could be applied