1. Impact Assessment Training
10/11th September 2013
Oslo

2. Workshop Outcomes
By the end of these two days you will have:
Developed a common understanding of what is meant by impact assessment; and how it differs from and complements the processes of monitoring and evaluation.
Identified key challenges that you face in conducting impact assessments
Worked with pre-selected case studies  to design an impact assessment process
Considered how the results of impact assessments might be used in learning, reporting and being more accountable  to both donors and stakeholders
Identified ways in which you can take forward learning from this workshop.

3. The vicious circle

4. So what do we mean when we talk about Impact?
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

5. Impact
Think about an event or a person who made a significant impact on your life (a birth or death of a family member or friend, a relationship, a marriage, an accident)
What does impact mean in relation to this?
What concepts or ideas does it include?
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

6. How do you “assess” this impact?
... You try to understand the nature of the change that has taken place in you and to determine its significance in your life.
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

7. Impact Assessment
“The systematic analysis of significant and/or lasting change – positive or negative, intended or not – in the lives of target groups, brought about by a given action or a series of actions”
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

8. Its not that easy... consider Danny..
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

9. Why assess impact? To understand the implications of our work
To become more accountable to those we work with (stakeholders)
To support institutional learning and decision making and improve future work
To contribute to policy development and effective advocacy
To help demonstrate organisational performance
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

10. Key areas of enquiry for impact
Five essential questions:
What has changed?
For whom?
How significant was it?
Will it last?
In what ways did we contribute to these changes?
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

11. The challenge for this workshop
To be able to develop Impact Assessment processes which are:
Simple and user-friendly
Build on existing structures and systems
Are useful for accountability both upwards and downwards
Are useful for your organisational learning
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

12. Session 2
Relationship between M&E and Impact Assessment

13. M&E or Impact Assessment?
How is the assessment of impact different from the processes of monitoring and evaluation?
What to do?
When to do it?
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

14. Example
In a project to build social housing for a local community, for example:
Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective etc
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

15. But...
It’s possible to have well-built cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine “settlements”).
Thousands of “successful projects and programmes” which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
Impact Assessment addresses the “So what?” question: as a result of our efforts, what’s changed for whom; and how significant is this for them?
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

16. Impact Assessment addresses the
Impact – so what?!
Thousands of “successful” projects and programmes make no lasting difference to people’s lives.
Impact Assessment addresses the
“So What?” Question
As a result of our efforts, what has changed for whom; and how significant is this for them?
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

17. Differences in Brief
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

18. Log Frame Approach e. g. Skills training programme - a results chain
Log Frame Approach e.g. Skills training programme - a results chain... Does it always work like this???
Inputs
Activities
Outputs
Outcomes
Impact
Funds, expertise etc
Training
Improved skills
Improved livelihoods
Less poverty

19. Small groups’ task
Small UK based NGO, working through partners in Ethiopia in Education
Goal: All children have access to free basic education
What high level outcomes would ensure that this goal is achieved?
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

20. Small groups’ task Select one outcome
Discuss what activities/outputs would lead to the achievement of this outcome
Develop this into a mini project
Develop indicators at each level
Use different colour post its!
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

21. Small groups’ task Level Detail Indicators Impact
All children have access to free basic education
Outcome 1
Outputs
....
.....

22. Analysing the results
To what extent are you able to assess the key questions for impact assessment?
What has changed?
For whom?
How significant/lasting are these changes for different target groups?
What -if anything - did our programme (project) contribute?
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

23. Measuring impact based on results chain
Tends to be limited to assessing what you expect/hope will change... Good for demonstrating results to donors..
Positive impact
Negative impact
Expected
xxxx ?
Unexpected
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

24. Answers this afternoon!
So...
So how else to assess impact?
What are the challenges?
What different approaches could we use?
Answers this afternoon!

25. Session 3
Focus on change and how to assess it

26. Thinking about Change How does change happen?
How do/can development organisations conceptualise this?
What is their realistic sphere of influence on changes that do take place?

28. Change is..
Complex in that many different changes can take place simultaneously in people’s lives
• Continuous in that nothing in society or the environment is ever static
• Variable in pace, scale and/or over the course of any intervention
• Not necessarily lasting or sustainable.

29. What do we need to know to assess impact?
Who or what was involved in the change? (e.g. individual actors or state institutions)
What strategies were used to bring about the change? (e.g. reform, mass mobilization)
What were the contexts that affected how the change happened? (e.g. urbanization, power relationships)
What was the process or pathway of change? (e.g. demonstration effects, cumulative progress)
How were our efforts connected to this?

30. Organisational response to this?
Many organisations are developing Theories of Change to better support their understanding of how change happens and their role in the process

31. Four Key Elements
Understand how change happens and your role in this
Develop Organisation/Programme change pathway
Assess the impact of your efforts
Critically reflect on your pathway and how you thought change happened

32. How do they complement other planning and M&E processes?
Strategic plans?
Log frames?
M&E systems?
Learning loops?

33. 3. ToC doesn’t replace results frameworks - they make them stronger
How change happens
Organisation or Programme change pathway
Impact assessment
Reflection and adaptation of ToC
ToC = critical analysis process
Logic Model = Accountability Framework which is informed by ToC

34. Summary of some of the differences
Logic Models
Theories of Change
Describe component parts of a projects
Make linear links between activities and results
Are used as a management tool
Situate programme efforts in bigger picture of change
Surface and articulate assumptions
Explain causal pathways of Change
Are used as a critical analysis process

35. Examples of Impact Assessment Frameworks:

36. Theory of Change – Women's Empowerment
Contributes
To change in:
Contributes to:
Contributes to:
Marginalised
women are empowered and access rights, opportunities and services
including:
Equal access to property and resources
Reduction in harmful traditional practises
Increased representation in local, regional and national leadership positions
Increased economic empowerment
Reduction in violence and exploitation against women?
Capacity Building to partners
Power holders at local and regional level ensure that all women access rights, opportunities and services
MRDF
Increased partner capacity
Awareness of rights
Funds for Capacity Building
Marginalised women have the confidence, knowledge and skills to access rights, opportunities and services
Funds for partners
Access to information
Activities proposed by partners
Access to support
Marginalised women are organised, active and influential and play an active role in society
Access to credit
Direct sphere of influence
Access to resources
Marginalised women are supported and empowered by their families, society and culture
Long Term Impact
Access to training
Theory of Change – Women's Empowerment
(MRDF as a stone making ripples in the water)

37. Session 4

38. Approaches and strategies
Three approaches:
Post programme: Testing logic of log frame ( impact “evaluation”)
Participatory ToC approach which is used to design monitor and assess efforts
Research: looking back sometime later and assessing changes and their relation to programme efforts
Four strategies:
Build into existing M&E
Tracer and tracker studies
Ensure key moments of critical reflection
Commission a retrospective study
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective? But the questions cannot stop there. It’s possible to have well-built, cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine ‘settlements’). In development, there are thousands of ‘successful projects and programmes’ which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts.
• Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?
The table below clarifies the difference between monitoring

39. Plans for tomorrow
Case Study Task

40. Day 2

41. Session 1
Case Study Task 1

42. Reflection on yesterday

43. Main steps in designing an Impact Assessment
Define purpose, approach and scope of the assessment
Develop/confirm theory of change and/or dimensions of change
Develop a list of areas of enquiry which will help you to explore impact
Select tools and methods for gathering and analysing relevant information
Decide on reporting strategies
Make concrete plans and timelines (who, when, how....)

44. The challenges To develop a process which is: Simple and user-friendly
Build on existing structures and systems
Are useful for accountability both upwards and downwards
Are useful for your organisational learning
And the challenges that we identified yesteday

45. ACCOUNTABILITY TO THE POOR
It starts with belief that poor people and their own organisations are both capable of – and should be involved in managing – their own development processes. ALPS supports the view that poor people and their organisations should be given the space to negotiate their own position and be given the opportunity to do so at local, national and international levels. It requires a belief that the INGO’s role is to support development initiatives, initiated and led by poor people (and their organisations) themselves.
While many INGOs have responded to the demands of greater accountability by increasing reporting to central offices, ALPS tried to do the opposite. It tried to reduce the number of reports (written in English) emphasising instead greater accountability to the community groups and partners with whom we work. More decision-making was decentralised from ActionAid’s then HQ in London and greater decision-making power given to national teams running their own programmes in countries across the world. The power balance had significantly shifted.

46. Case Study Task 1 In small groups:
Share the details of the programme you are working on (you need to be clear about context, overall goal and top level outcomes)

47. Case Study Task 1
What will be the main purpose of doing this impact assessment? Organisational learning? To meet donor demands? Accountability to stakeholders? For advocacy
Which approach (or combination of approaches?) to Impact Assessment would be most appropriate in this Why?
What will be the scope and scale of this assessment?

48. Session 2
Case Study Task 2

49. Case Study Task 2:
What is your realistic “scope of influence”? Which what areas of impact will you realistically be able to “assess”, and which areas of change will you be able to “illustrate contributions to change”?
Based on this, what “Dimensions of Change” will you be looking to assess?
Develop a menu of areas of enquiry which will enable you to set baselines and track progress in relation to impact

50. Being clear about your scope of influence
Sphere of indirect influence
(there are other factors/actors
which influence changes
you want to see)
Assessment process less
rigorous –
illustrations of impact
Sphere of direct influence
(you are working directly with target groups on specific changes)
Need to assess impact rigorously

51. Tools/ Sources of Information
Dimension
Task 1
Areas of Enquiry
Task 2
Tools/ Sources of Information
Task 3
Direct scope of influence
e.g.
Capacity of partners
Ability to plan and deliver
Levels of technical capacity
Approaches to networking with others
Level of sustainability
Shifts in confidence
Indirect scope of influence

52. Areas of Enquiry and indicators
Written neutrally to encourage expected/ unexpected
Designed to test logic of project and expected changes
Can track levels of xx, shifts in xxx, trends, perceptions,
Requires open-ended questions

53. Session 3
Case Study Task 3

54. Case Study Task 3
Propose a range of appropriate methods that you could use to gather relevant data for both monitoring and assessing impact (including building on or adapting existing tools and mechanisms).

55. Making the process manageable and robust
Baselines
Sample size
Triangulation
Selecting the right tools and ways collecting useful information
Ensuring that you have asked the right questions in the right way

56. Baselines The problems? The solutions?
Dedicated desk research and ask others
Plan from the start and use rolling baselines
Reconstructing them

57. Sample size
A good sample is one that is sufficiently large and is unbiased.
How to select?
Mathematical?
Random?
Pragmatic?

58. Triangulation
You need three perspectives to ensure validity of information gathered. Could include:
Three ways of checking same info
Asking the same question of three target groups

59. A word about tools and collecting information
Less is more..
Be aware of time resources and capacity
Stick to the “old faithfuls” as the basis
Mix of methods is good
Be led by areas of enquiry
And perhaps the hardest thing of all.. Try to enable people to tell the real story...

60. Checklist – use for areas of enquiry and selection of tools
In order to assess impact effectively, your areas of enquiry combined with data gathering methods needs to capture this information
Have there been any changes?
How many people were affected (which target groups)?
How were they affected (and were they affected differently)?
Were these changes intended?
How do they compare to baselines (have you got evidence)?
What can be attributed to your organisational efforts?
How confident are you in reporting these findings?

61. Session 4
Wrap up and the way forward