Presentation on theme: "Re-thinking Humanitarian Impact Assessment: theory and practice OCHA Joint Review of Inter- Agency Evaluations, Geneva 12 th June, 2009."— Presentation transcript:
Re-thinking Humanitarian Impact Assessment: theory and practice OCHA Joint Review of Inter- Agency Evaluations, Geneva 12 th June, 2009.
Main Aim of Study To provide an overview of current experiences and thinking in order to develop and test a conceptual framework to be used for understanding, planning and implementing impact assessment 2 OCHA Workshop on Inter-Agency Evaluations, June 2009.
3 Main parts of the report Five-part conceptual framework 4 Case studies on impact assessment Conclusions and recommendations
4 OCHA Workshop on Inter-Agency Evaluations, June 2009. Methodology Research conducted Sep 2008 – Apr 2009 Literature review Key informant interviews Discussions at 24th ALNAP biannual meeting Case studies Impact assessment survey (ALNAP full and observer members)
5 OCHA Workshop on Inter-Agency Evaluations, June 2009. Five-part framework 1.Understanding and Balancing Stakeholder interests 2.Understanding and defining impact 3.Methodological approaches and challenges 4.Engaging local actors and affected populations 5.Capacities and incentives for improved impact assessment
6 OCHA Workshop on Inter-Agency Evaluations, June 2009. 1. Understanding and Balancing Stakeholder Interests Impact assessments are more likely to be used if they meet the interests of stakeholders Decisions about purpose and scope are political Difference and tension between ‘proving impact’ (accountability) and ‘improving practice’ (learning) Allow enough time to negotiate and ensure adeqaute participation
7 OCHA Workshop on Inter-Agency Evaluations, June 2009. 2. Understanding and defining humanitarian impact and theories of change A widely recognised definition: Lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions (Roche, 2000) Theory of change must be clear, realistic and understood by all stakeholders.
8 OCHA Workshop on Inter-Agency Evaluations, June 2009. 3. Methodological Approaches and challenges Methodological appropriateness could be considered the “gold standard” for impact evaluation (NONIE, SG2, 2008) Key issues include: Indicators: moving beyond outputs Overcoming the attribution problem with appropriate approaches and methods Baselines, monitoring and data collection Timing and amount of time
9 OCHA Workshop on Inter-Agency Evaluations, June 2009. 4) Engaging local actors and affected populations throughout Participation by affected populations is not a key feature of impact assessments. Attempts to improve this include: The Listening Project ECB ‘Good Enough Guide’ Feinstein International Center Participatory Impact Assessments (PIA) Affected populations, national and local actors should be involved at all stages ‘Learning partnerships’ between donors, implementing partners, communities, national actors and other stakeholders are needed.
10 OCHA Workshop on Inter-Agency Evaluations, June 2009. 5. Capacities and incentives Lack of individual and organisational capacity to do good impact assessments Institutional incentives can override humanitarian ones; too few incentives to conduct good impact assessments; results-based approaches can create perverse incentives A number of cultural barriers and biases that hinder good quality humanitarian impact assessment Scope for sector-wide initiative to strengthen capacity and address disincentives?
11 OCHA Workshop on Inter-Agency Evaluations, June 2009. Case studies 1.Impact evaluation of Community-Driven Reconstruction, Northern Liberia (IRC) 2.Participatory impact assessment in pastoral communities, Niger (LWR/Tufts) 3.Impact study of FAO’s emergency programme in DRC 4.Tsunami Recovery Impact Assessment and Monitoring System (TRIAMS)
12 OCHA Workshop on Inter-Agency Evaluations, June 2009. Agency IRC, Colombia, Stanford LWR, FICFAOIFRC, WHO, UNDP, National Governments affected countries Countries LiberiaNigerDRCIndonesia, Maldives, Sri Lanka, Thailand Scope Programme level; 42 communities in 2 Districts Programme level: 10 communities Country-wideRegional, four countries Timing Prospective; integrated into project from start Retrospective; carried out 18 months after the project began. Retrospective; began 1 year after programme ended. Ongoing; began 1 year after recovery began Cost ($) 200,0005,000100,000542,000 Overview of Case Studies
13 OCHA Workshop on Inter-Agency Evaluations, June 2009. Balancing stakeholder interests Accountability vs. learning needs to be clear at the outset: negotiate, don’t avoid the debate! Understanding effects vs. field-testing methods Collective international action vs. national buy-in and ownership
14 OCHA Workshop on Inter-Agency Evaluations, June 2009. Definitions of Impact 1.The net difference that IRC’s work makes in people’s lives 2.Those benefits and changes to people’s livelihoods, as defined by the project participants, and brought about as a direct result of the project. 3.Positive and/or negative changes induced (more or less directly) by FAO emergency interventions on target groups, their households, organisations, communities or on the environment in which they live. 4.Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended
15 OCHA Workshop on Inter-Agency Evaluations, June 2009. Methodologies – quantitative and qualitative combined Case studyQualitative methodsQuantitative methods 1. IRC impact evaluation of CDR, Liberia Semi-structured interviews, behavioural goods measurement Randomised evaluation, ‘before and after’ household survey 2. PIA of support to pastoralists, Niger Semi-structured household interviews, focus group discussions; ‘before and after’ ranking and scoring (Not applicable) 3. Impact study of FAO emergency and rehabilitation work in DRC ‘Before and after’ recall methods, life histories, focus-group discussions at village level, qualitative survey of NGOs Household survey with ‘control’ group 4. TRIAMS in Indonesia, Maldives, Sri Lanka and Thailand. Encouraged qualitative approaches, including beneficiary perception surveys, in order to triangulate with quantitative routine and survey data Routine data-collection, household surveys
16 OCHA Workshop on Inter-Agency Evaluations, June 2009. Methodology – indicators need to be flexible and robust Adapting indicators to changing contexts is key (PIA example) ‘Proxy’ indicators of impact may be useful (PiA) or inadequate (FAO example)
17 OCHA Workshop on Inter-Agency Evaluations, June 2009. Attribution – strengths IRC Eliminated selection bias and some ‘confounding’ factors. Perceived as more transparent by communities Mixture of quantitative and qualitative methods enabled triangulation of data PIA Participatory Can be done ex-post No ‘control’ group required. Enables statistical analysis of qualitative data Low cost (approx $US 5, 000) IRC Both ‘before and after’ and ‘with and without’ comparisons Can be done ex-post Mixed methods enabled Rich data derived from story-telling TRIAMS Tracked recovery outputs and outcomes over time
18 OCHA Workshop on Inter-Agency Evaluations, June 2009. Attribution – weaknesses IRC Not all confounding factors eliminated e.g. ‘treatment’ communities more rural than urban. Large sample size required. Logistical challenges ‘Before and after’ survey results reflect changes in survey responses, rather than changes in behaviour. Costly (approx US$200,000) PIA Did not eliminate confounding factors e.g. contextual factors posed challenge to attribution Qualitative data only Reliant on memory of situation months or years previously. Measures relative change rather than actual change Prone to selection bias Qualitative data less rich IRC Reliant on memory of what happened several months/years previously ‘Control’ group prone to leakage Selection bias Relatively costly US$100,000 TRIAMS No attribution To date, focus on monitoring performance rather than impact
19 OCHA Workshop on Inter-Agency Evaluations, June 2009. Analysis and Use of data XXXX
20 OCHA Workshop on Inter-Agency Evaluations, June 2009. Capacities and incentives XXXX
21 OCHA Workshop on Inter-Agency Evaluations, June 2009. Conclusions Humanitarian impact assessment is not only desirable but possible Each case study has specific strengths and weakness Together, the five key areas form a conceptual framework which could be used as a starting point for developing and improving impact assessment in the humanitarian sector Work already underway with Save Alliance, framework being used to shape Tsunami impact assessment Advising OCHA work
22 OCHA Workshop on Inter-Agency Evaluations, June 2009. Recommendations (1) The humanitarian sector should develop and institutionalise sustainable approaches to impact assessment Identify appropriate stakeholder analysis tools for use in discussions of impact assessment, which help to make interests explicit and identify common ground Initiate a cross-agency discussion on the feasibility and desirability of a clear definition of humanitarian impacts and outcomes Develop relationships with academic partners and other experts in the field to to design and deliver a toolkit outlining the key methods of impact assessment for use in the humanitarian sector. This could include practical examples of mixed-method approaches. Develop a shared database of impact indicators that could potentially be used in humanitarian evaluations Undertake further research on the mix of impact-assessment methods most appropriate in the different emergency phases of relief, recovery and reconstruction
23 OCHA Workshop on Inter-Agency Evaluations, June 2009. Recommendations (2) The humanitarian sector should develop and institutionalise sustainable approaches to impact assessment Ensure the views of affected people are centre-stage to ensure credibility Promote the use of livelihoods approaches as a framework for analysis Invest in and build long-term, national and international partnerships for impact assessment between affected populations, academics, donors, governments, civil society and the private sector Review existing programming and funding approaches across the sector in terms of how they currently enable or inhibit effective and timely impact assessments Work towards improved project and programme, organisational and sector- wide performance frameworks which explicitly define impact and embed impact orientation in all stages of the project cycle Consider how donors, agencies and the sector as a whole can better reward individuals and organisations for doing effective impact assessments. `
24 OCHA Workshop on Inter-Agency Evaluations, June 2009. Pointers for the present discussion Who are the stakeholders for IA impact assessments? What interests? Do you have a sufficiently clear understanding of impact and theories of change? What methodologies would be most appropriate given your stakeholder needs and understanding of impacts? How will you engage affected people and national / local stakeholders? What capacities do you have to undertake and use an impact assessment, and what incentives are in place?
25 OCHA Workshop on Inter-Agency Evaluations, June 2009. 3.1 Indicators: Moving Beyond Outputs Identifying impact indicators involves value judgements about what kinds of changes are significant for whom (Roche, C. 2000)
26 OCHA Workshop on Inter-Agency Evaluations, June 2009. 3.2 The attribution problem Comparative vs. theory-based approaches Quantitative vs. qualitative methods A mixed approach can provide the best information
27 OCHA Workshop on Inter-Agency Evaluations, June 2009. 3.3 Baseline and other data “Reports were so consistent in their criticism of agency monitoring and evaluation practices that a standard sentence could almost be inserted into all reports along the lines of: It was not possible to assess the impact of this intervention because of the lack of adequate indicators, clear objectives, baseline data and monitoring.” (ALNAP, 2003) Key issues include: Weak or non-existent baselines Data is often unavailable or unreliable Data collected is mainly quantitative Monitoring systems focus on process and outputs Lack of collective and coordinated approaches to data collection
28 OCHA Workshop on Inter-Agency Evaluations, June 2009. 3.4 Time and Timing IA should be carried out when impacts are likely to be visible and measurable (depends on the specific goals and indicators). Insufficient time can result in inadequate monitoring and data collection