Impact evaluation: External and internal stakes Impact evaluation seminar - 2 to 6 December, Phnom Penh

International context and stakes Uncertainty on aid, Paris Declaration 2005, achievement of MDG (Millennium Development Goals) and after-MDG  Highlights the Results-Oriented Management (ROM)  Focuses on results/effects/impact  Insists on participative approach Since the 90s, increasing initiatives to improving quality and impact of humanitarian action  Sphere, HAP Standards in Accountability and Quality, Code of behaviour, Don not harm, etc.  Increasing concern for understanding changes induced by humanitarian aid Impact evaluation seminar - 2 to 6 December, Phnom Penh

Stake of accountability: Stakeholders have a moral obligation to allocate necessary means to the purpose of their mission and to prove the changes made by their action; Stake of visibility: Need for demonstrating one’s added- value among an increasing number of stakeholders in the humanitarian sector and international solidarity; Stake of financial funding: donors prescriptions to prove impact and/or changes along with the development of new tools and approaches (Value for money, Theory of change, etc. ) International context and stakes Impact evaluation seminar - 2 to 6 December, Phnom Penh

How we keep busy How we make a difference OutputsOutcomeImpactInputs Results Chain Crossing the red line....

Définitions Outputs/Extrants/Produits: The immediate results of the organisation’s activities – the processes, goods and services that it produces. The organisation controls its outputs. → 40 health professionals are trained on inclusive approach, reception and follow-up of people with disabilities Outcomes/Effets: The likely or achieved short-term and medium-term effects of an intervention’s outputs. They contribute to the improvement of people’s life. The organisation only influences outcomes, besides external factors. → Improved access to health services for persons with disabilities → Improved health conditions of persons with disabilities Impact: Long-term, sustainable changes in the conditions of people and the state of the environment that structurally reduce poverty, improve human well-being. Your organisation contributes partially and indirectly to these enduring results in society or the environment. → Improved quality of life of persons with disabilities Impact evaluation seminar - 2 to 6 December, Phnom Penh

1950/60s1980/90s1970/80s Impact analysis ex ante to anticipate the economic effects of interventions Taking account of unexpected and negative effects, one-dimensional models (Cost-profit analysis, social impact measurement, environmental impact measurement) Evolution towards a multidimensional comprehension of impact, development of ex- post evaluations Linear approach of change, target populations as objects of studies and not actors of change Rural assessment participative approach, populations as actors of change A short story of impact 2000 → Elaboration of new approaches; theory of change, outcome mapping

Two main approaches  Usual methods of impact evaluation: different types of studies and evaluation, sometimes requiring important resources;  Approaches measuring effects by being integrated within the intervention with the use of tools monitoring changes at different levels (short and mid-term effects, sometimes impact) E.g.: The planning, Monitoring and Evaluation tool of ACTIONAID, highlights learning and ownership (ALPS for Accountability, learning and planning system) Impact evaluation seminar - 2 to 6 December, Phnom Penh

Why measuring impact? CREDIBILITY LEARNING ACCOUNTABILITY ADVOCACY IMPACT

 What has changed?  For whom?  How significant was it?  Will it last?  In what ways did we contribute to these changes? Key questions on impact measurement Impact evaluation seminar - 2 to 6 December, Phnom Penh

Responsibility of impact Impact evaluation seminar - 2 to 6 December, Phnom Penh

« The systematic analysis of lasting or significant change –positive or negative, intended or not – in people’s lives brought about by an action or a series of actions » (Roche, OXFAM) « Measuring the changes taking place as the result of an emergency project (The Good Enough Guide Impact Measurement and Accountability in Emergencies) One or several definitions of impact? Impact evaluation seminar - 2 to 6 December, Phnom Penh

Common elements of definition: concept of change/effects, changes are positive or negative, changes are expected/desired or not Elements of definition adapted to each intervention type and environment: the object of change, the sustainable, significant or structuring aspects of change Towards an open definition? Impact evaluation seminar - 2 to 6 December, Phnom Penh

Impact measurement challenges  When initiating an impact evaluation?  Which approach?  What level of investigation?  What resources?  What participation of the different stakeholders?  What responsibility to changes? Impact evaluation seminar - 2 to 6 December, Phnom Penh

“Handicap International reaffirms the importance of taking effective action, generating positive and measurable effects and introducing changes that bring about tangible improvements for the beneficiaries, and for people with disabilities in particular. For Handicap International, the development of services and facilitating access to these services are essential to achieving a positive impact on the living conditions of beneficiaries» Federal Strategy Chapter 2 - An approach centred on achieving a measurable impact

HI and partners’stakes in Impact measurement  Sharing common references and terminology  Proposing and implementing rigorous methods and tools adapted to projects and intervention contexts  Asking the good questions to each effect or impact measurement initiative: What, when, how, why?  Integrating effect or impact measurement within or Monitoring- Evaluation tools and systems  Supporting a balance between the quantitative and qualitative analysis  Analysing and taking into account negative effects of projects Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh

Seminar objectives  Identify and analyse HI and partners’ practices on effect and impact measurement, share experiences, draw recommendations for improving tools and methods;  Identify, by technical domain and by intervention context, the areas of change expected, the different levels of change and the related indicators;  Evaluate the possibility of integrating and implementing new approaches of change or impact measurement; criteria of choice and identification of resources;  Identify the ways of integrating change follow-up and measurement into Monitoring-Evaluation tools Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh

LET’S GO TO WORK! Impact evaluation seminar - 2 to 6 December, Phnom Penh