1. An introduction to Monitoring and Evaluation in Outcome Mapping Applied and adapted use
Bern, 15 April 2010

4. Inspired by Jeff Conklin, cognexus.org
Applied Monitoring
Vision and Mission
Intentional design
IMPACT
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
Time
Inspired by Jeff Conklin, cognexus.org

5. Inspired by Jeff Conklin, cognexus.org
Applied Monitoring
Vision and Mission
Intentional design
IMPACT
Boundary Partners
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
CONTROL
INFLUENCE
Time
Inspired by Jeff Conklin, cognexus.org

6. Applied Monitoring Time Vision and Mission IMPACT OUTCOMES OUTPUTS
Intentional design
IMPACT
OUTCOMES
OUTPUTS
Performance
Your programme’s functioning as an organizational unit
ACTIVITIES
INPUTS
Time
Inspired by Jeff Conklin, cognexus.org

7. Inspired by Jeff Conklin, cognexus.org
Applied Monitoring
Vision and Mission
Intentional design
IMPACT
OUTCOMES
Strategy
The effectiveness of how you are attempting to influence boundary partners
OUTPUTS
ACTIVITIES
INPUTS
Time
Inspired by Jeff Conklin, cognexus.org

8. Applied Monitoring Time Vision and Mission IMPACT OUTCOMES Outcomes
Intentional design
IMPACT
OUTCOMES
OUTPUTS
Outcomes
Change you influence in the boundary partners
ACTIVITIES
INPUTS
Time
Inspired by Jeff Conklin, cognexus.org

9. Adapted Monitoring For example… You have no baseline
Are in complex situations where relations of cause and effect are unknown
Are a non-hierarchal organisation
You may fruitfully use Outcome Mapping to monitor your results if you...
Have not done an intentional design and therefore do not have a baseline
Are primarily interested in understanding the process through which you are influencing change
Operate in complex situations in which cause and effect are unknown and therefore unexpected effects are common
Are a non-hierarchal organisation in which success depends on units and individuals operating with autonomy and little control

10. Situations in which this…
Vision and Mission
Intentional design
IMPACT
OUTCOMES
OUTPUTS
ACTIVITIES
INPUTS
Time
Inspired by Jeff Conklin, cognexus.org

11. Looks like this

12. Vision and Mission OUTCOME OUTPUT OUTCOME OUTPUT OUTCOME OUTCOME
ACTIVITY
OUTCOME
OUTCOME
OUTPUT
OUTCOME
ACTIVITY
OUTPUT
OUTPUT
INPUTS
OUTCOME
ACTIVITY
OUTCOME
INPUTS
ACTIVITY
OUTPUT
INPUTS
INPUTS

13. You monitor by
Watching social actors for changes in their behaviour, relationships, actions, policies or practices
Focusing on the outcomes you are achieving
Working backwards from outcomes to determine how you contributed to that change, rather than attempt to attribute the change to your activities
And then deciding how to improve strategy and organisational performance in order to achieve outcomes
Focus on what you achieve rather than what you do
You ascertain how you contributed rather than attempt to attribute the change to your activities
Monitor boundary actors. You monitor social actors for changes in their behaviour, relationships, actions, policies or practices. The focus of monitoring should be on the social actors you are attempting to influence and not on the activities you have carried out. This is another distinction compared to conventional monitoring.
Monitor from outcomes to activities. When you have identified a change, you then work backwards to determine how you contributed to that change. Conventional evaluation (and by implication conventional monitoring) attempts to understand if what was planned is what resulted, or if this logic bears fruit: inputs -> activities -> outputs -> outcomes -> impact. In complex and dynamic situations where what results, and even what was done, was often not pre-planned, this logic is not applicable. Instead, using the approach of the coroner examining the cause of death, the criminal investigator the author of a crime, or the epidemiologist the source of a disease, the outcome mapper first identifies the outcome and then works back to figure out what activities contributed to it.
You map the process of changes to which you are contributing
And then make decisions on how to improve your strategy and organisational performance

15. Evaluation is the least developed of the three stages of Outcome Mapping, formally at least.
Limited to designing the Evaluation Plan.

16. Elements of an Outcome Mapping Evaluation Plan
EXAMPLE EVALUATION PLAN
Evaluation Issue: Results Achieved by Recipient Research Institutions
Who Will Use the Evaluation? How? When?
Questions
Information Sources
Evaluation Methods
Who Will Conduct and Manage the Evaluation?
Date (Start & Finish)
Cost
Executive director to use findings to inform program proposal
What influence has the institution had on research users and other researchers?
Progress Markers & Outcome Journal
Trip reports
Strategy Journal
Key informant interviews with program staff
Senior Consultant with regional and sectoral expertise
Six months
35K

17. But as with monitoring, the outcome concept is wonderfully adaptable.
It can potentially serve the three principal evaluation modes.

18. Michigan Association for Evaluation Annual Conference
May 3, 2000
Formative evaluation
“When the cook tastes the soup, that’s formative evaluation.” (Bob Stake)
Provides information for program:
improvement
modification
documentation
management
Provides feedback to improve program effectiveness before the end of the project.
- Michigan Association for Evaluation, provided by John Seeley
Michigan Association for Evaluation Annual Conference

19. Michigan Association for Evaluation Annual Conference
May 3, 2000
Summative evaluation
When the guest tastes it, that’s summative evaluation.” (Bob Stake)
Conducted after the project is completed.
Provide evidence to support judgment of the project’s worth.
Is used in decisions about the merits of continuing the project or program.
- Michigan Association for Evaluation
Michigan Association for Evaluation Annual Conference

20. Developmental evaluation
“When a guest and a cook concoct a soup together, that co-creation is developmental evaluation.” (Michael Quinn Patton)
Supports innovation to develop a programme that is adapting to emergent and dynamic realities in complex environments.
Infuse the development team discussions and decision-making with evaluative questions, thinking and data.

21. Examples of formal evaluations
Treatment response for problematic use of ecstasy, ketamine and gamma-hydroxybutyrate in Australia
Six civil society projects in Bosnia and Herzegovina
Community development program with Roma and non-Roma communities in Bosnia and Herzegovina
OM applied as a planning, monitoring and evaluation tool to the Ceja Andina project in Carchi, Ecuador
A working group on 'values’
Use of OM to recommend a treatment response for problematic use of ecstasy, ketamine and gamma-hydroxybutyrate in Australia, by Josephine Norman
Outcome Mapping evaluation of six civil society projects in Bosnia and Herzegovina, by Steve Powell
Evaluation of community development program with Roma and non-Roma communities in Bosnia and Herzegovina, by Steve Powell
OM applied as a planning, monitoring and evaluation tool to the Ceja Andina project in Carchi, Ecuador, by Kaia Ambrose
Evaluation of a working group on 'values’, by Daniel Roduner, Switzerland
All available through the The OM learning community website:

22. Examples of adaption of Outcome Mapping in evaluation
7 international networks – 2 based in Latin America, 2 in Europe, 2 in Asia, and I in the USA.
8 programmes of five grant makers – Oxfam Novib, IDRC, Ford Foundation, Hivos, and the Open Society Institute
Their 130+ social change and development grantees
Outcomes not predefined
Since 2003 I have used the Outcome Mapping concept of outcomes in evaluations of:
7 international networks – 2 based in Latin America, 2 in Europe, 2 in Asia, and I in the USA.
8 programme s of five grant makers – Oxfam Novib, IDRC, Ford Foundation, Hivos, and the Open Society Institute assessing the outcomes of 131 social change and development grantees (NGOs, community based organisations, networks, research centres working on a wide range of issues – environment, human rights, agriculture, health, ICT, democracy)
No one had predefined the vast majority of outcomes that we harvested.

23. Evaluation
■ Internal outcomes ■ External outcomes ■ Their significance ■ The organisation’s contribution
■ Internal Outcomes – The changes you are influencing in the key internal actors of your non-hierarchal, autonomously managed organisation, alliance, partnership or network.
■ External outcomes – The changes you are influencing – from inspiring to pressuring, passing through supporting, assisting, funding or in any way aiding – social actors in the world outside.
In both cases you can identify as well:
■ Significance – Why is each change important? How does it complement counterbalance or undermine other outcomes?

24. In these evaluations we “harvested” descriptions of who changed what, when and where, its significance and the contribution of the organisation.
In addition, we substantiate the outcomes with independent third parties who have a working knowledge of the change and how it came about.
We mapped processes of change to understand how outcomes reinforce or undermine each other.
In evaluation we “harvest” the same information as when monitoring (description of who changed what, when and where, its significance and how the organisation had contributed)
In addition, we have substantiated the outcomes with independent third parties who have a working knowledge of the change and how it came about
And in some cases of mapped processes of change to understand how outcomes reinforce or undermine each other.

25. Thank you! For more information, examples of use and to share your experiences