1. Challenges in Measuring the Impact of Global Climate Change Interventions to Increase Resilience to Rare and Unpredictable Events David Hemson, Ph.D., Senior Evaluator Global Climate Change Monitoring and Evaluation Project Development and Training Services (dTS) Contact: dhemson@OnlineDTS.com

2. Rare and unpredicted but not a result of climate change… Gare Montparnasse, 22 October 1895 2

3. Introduction The prospects for an impact evaluation of three challenging USAID projects are scoped. The objective of all three is to improve resiliency to events which are intrinsically rare and unpredictable. While USAID is committed to undertaking impact evaluations of CC projects there are severe challenges IRT these projects. What evaluation methods are appropriate in these conditions? USAID defines resilience as the ability of people, households, communities, countries, and systems to mitigate, adapt to and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth. http://www.usaid.gov/resilience http://www.usaid.gov/resilience 3

4. The Three Projects Climate resilience in the Kazakhstan wheat sector has the goal of improving seasonal forecasts, using demonstration plots to present results, and advising adaptation among farmers. –Kazakhstan is the 6 th largest exporter of wheat and droughts sharply reduce yields and threaten regional food security. The Angola project aims to improve the early warning system to flooding in the Cunene province, map most vulnerable areas and improve responses under the Contingency Plan. –Floods in this province have drowned hundreds and severely disrupted lives of communities. 4

5. The Three Projects Cont. In Ethiopia the intervention will strengthen traditional conciliation to reduce drought-related conflict. –There has been an increase in conflicts between clans over grazing and other issues in the south of the country. All develop resilience to extreme events such as droughts, floods, and the resulting social effects. 5

6. Challenges to Impact Evaluation (IE) IE is based on establishing double-differences in variables of key means pre/post (P/P) and between treatment and control (T/C). Clear separation of pre/post and T/C is needed but this is always difficult as this may require changes to implementation plans. IE’s are intrinsically difficult to undertake especially in design, funding, endlines after project conclusion. More so in these projects in which impacts are of uncertain, long term and designed for discontinuous and stochastic rare events. Some impacts immediate, others (as in these cases) may only be realized fully well beyond project end. 6

7. Acute challenges to IE The precipitating CC events (droughts, floods, storm surges) are extreme but differ in: –Rarity –Magnitude (maxima/minima) –Impact: severe < devastating Extreme events may also be single or compound; posing very changing dimensions in scale and impact. Proxies (e.g. heavy rains, “near-catastrophes”…) could be employed but their evaluation also cannot be planned. 7

8. Acute Challenges: Cont. If an extreme event happens well after project completion there will be confounding factors at play e.g. changes in housing structures, new political structures, etc. Research activities are difficult and possibly unwelcome in a disaster situation. Endline unknown, problematic to fund, and possibly more uncertain than the events being studied. In short is a double-difference impact evaluation possible in these circumstances? 8

9. Explorations in Method: PE+ Possibly an augmented Performance Evaluation (“PE+”) is a solution. PE of activities would be undertaken and supplemented with expert opinion (meteorologists, beneficiaries, policy makers, NGOs) on probable outcome and impact. Semi-structured interviews would include survey of expert belief in defined impact being realized at baseline and conclusion of the project. These will measure priors at baseline and posteriors at PE endline. The two measures establish changes (pre/post) brought about through the intervention to the probability of improved resilience. Probability of realized Impact = pT 2 - pT 1 9

10. A Potential PE+ Causal Network 10 Scientific information CC events of different magnitudes Meteorological Data is improved Planning for improved early warning systems CDev, planning Improved output: Demonstration Plots, activated early warning, advice and support Unimproved output: Demonstration Plots, activated early warning, advice and support Improved HH resilience Improved HH resilience Unimproved HH resilience Unimproved HH resilience e1e1 c1c1 c2c2 c2c2 c1c1 e2e2

11. Conclusions IE very difficult if not impossible with P-P and T-C Kazakstan somewhat different with long term trends in CC with more continuous data. Backdata critical to understanding phenomenon and modeling social impact. Simulation offers way of examining both CC variances and probable social outcomes from CC events of varying dimensions. 11

12. The interventions will reduce crises such as these: rescue of woman who spent days in a tree, Mozambique, March 2000 12