1. Complexity and Outcome Harvesting

2. Most recently
In June the World Bank published 10 case studies and a toolkit for using Outcome Harvesting

3. e Harvesting
Origins

4. Inspired and informed by the Outcome Mapping methodology.
e Harvesting
Inspired and informed by the Outcome Mapping methodology.
A tool for practitioners operating in dynamic, uncertain situations to monitor and evaluate the social change results they are achieving.

5. Outcome Harvesting Six steps:
Design the harvest — users => uses => evaluation questions => data to be collected
Review documentation and draft outcomes => intervention’s contribution
Engage with informants — generally within the intervention
Substantiate with independent but knowledgeable third parties
Analyse, interpret — What? + So What? but not Now What?
Support use of findings
TRIANGULATION

6. International social change networks
GLOBAL PARTNERSHIP FOR THE PREVENTION OF ARMED CONFLICT

7. International development funders

8. What did all of these organisations have in common?
COMPLEXITY!
What did all of these organisations have in common?

9. Relationships of cause and effect are KNOWN
Simple
Relationships of cause and effect are KNOWN
Now I will link the cartoons with the fish soup story with complexity. To explain what I mean by complexity, I find it helpful to consider it’s opposite.
A SIMPLE situation is when you want to plan to reach the island and you can calculate with considerable certainty the distance, the difficulty and therefore energy and time it will take you to swim or row or sail or tool over in a motor boat. It is simple because the relationships of cause and effect between what you will do and what you will achieve are KNOWN 9

10. M&E of a simple intervention
Vision
Annual polio vaccination campaign
IMPACT
OUTCOMES
Plan
OUTPUTS
ACTIVITIES
For example, the monitoring and evaluation of a simple development situation such as the annual polio vaccination campaign…EXPLAIN
INPUTS
Time
Inspired by Jeff Conklin, cognexus.org

11. Relationships of cause and effect are UNKNOWN
Complex
Relationships of cause and effect are UNKNOWN
A COMPLEX situation, in contrast, is his scenario. You want to get to the island but you cannot calculate when you will arrive or even if swimming or rowing or sailing or motor boating will be most effective. Relationships of cause and effect are UNKNOWN 11

12. M&E of a complex intervention
Strengthening nonviolent responses to communal conflict in the Horn of Africa
OUTCOME
Vision
OUTPUT
ACTIVITY
OUTCOME
Time
OUTCOME
OUTPUT
OUTCOME
ACTIVITY
OUTPUT
OUTPUT
INPUTS
OUTCOME
You may have a plan for which you have obtained resources but as soon as you begin to implement it, the activities change and their outputs too. Some lead to outcomes and other do not. And outcomes emerge in the most unexpected places. So in these situations, one or more of the five types of Developmental Evaluation will be appropriate.
ACTIVITY
OUTCOME
INPUTS
ACTIVITY
OUTPUT
INPUTS
INPUTS
Plan

13. Dimensions in which the relationships of cause and effect are known
NOT EITHER OR
OUTPUT
OUTCOME
INPUTS
ACTIVITY
Time
SIMPLE
Dimensions in which the relationships of cause and effect are known
Importantly, not everything is complex. There are dimensions in which the relationships of cause and effect are known. 13 13

14. NOT EITHER OR COMPLEX Unknown relations of cause and effect dominate.
OUTPUT
OUTCOME
INPUTS
ACTIVITY
Time
COMPLEX
Unknown relations of cause and effect dominate.
Results are substantially unforeseeable.
But in complex situations the most important relationships of cause and effect are unknown and therefore the results substantially unforeseeable.
And if you do not distinguish correctly the difference between simple and complex dimensions…. 14 14

15. Outcome Harvesting and complexity
High
The greater the: 1. Disagreement about what is the development challenge 2. Disagreement about what is its solution 3. Uncertainty about what will be the results of your actions to solve the development challenge The more Outcome Harvesting may be useful
COMPLEXIMETER
What I find helpful for monitoring and evaluation from Snowden’s and Zimmerman’s complexity models is they identify three factors of complexity:
1. The higher the disagreement about a) what is the development challenge or b) its solution, the greater the complexity.
2. And c) the higher the uncertainty about what will be the results of your actions to solve the development challenge, also the more complexity you face.
Degrees of disagreement about what to do and why, and what will be the results, underline that there are both subjective and objective dimensions to complexity because because it depends on what you know at a given moment of time and how true is that knowledge.
Low

16. Michigan Association for Evaluation Annual Conference
Formative
evaluation
Michigan Association for Evaluation Annual Conference
May 3, 2000
Summative evaluation
Developmental
evaluation
Developmental Evaluation is not, however, the same as evaluation of development.
- “When the cook tastes the soup, that’s formative evaluation.” (Bob Stake)
-When the guest tastes it, that’s summative evaluation.” (Bob Stake)
“When a guest and a cook concoct a soup together, that co-creation is Developmental Evaluation.” (Michael Quinn Patton)
- Developmental Evaluation is easily confused with international development evaluation. They are not
the same, though Developmental Evaluation can be used in the evaluation of development. Developmental Evaluation supports the development of innovation, which can include an economic, political, social, cultural, technological, environmental or other type of development intervention. Development evaluation is about evaluating programmes, usually in developing countries, sponsored or funded through the different mechanisms of international development co-operation.
Michigan Association for Evaluation Annual Conference

17. When do you evaluate? Developmental Formative Summative Time
Progress of intervention
Or instead would face this situation of uncertainty that they were going to have the intended results.
Time

18. Long-term Goal (Impact) Objectives or Outcomes
RESULTS BASED M&E
INTENTIONAL DESIGN
Long-term Goal (Impact)
STEP 1: Vision
Objectives or Outcomes
STEP 2: Mission
Indicators
STEP 3: Boundary Partners
Outputs
Indicators
STEP 4: Outcome Challenges
Activities
STEP 5: Progress Markers
Inputs
STEP 6: Strategy Maps

19. Long-term Goal (Impact) STEP 1: Vision
GLOBAL PARTNERSHIP FOR THE PREVENTION OF ARMED CONFLICT
Long-term Goal (Impact)
STEP 1: Vision
Objectives or Outcomes
STEP 2: Mission
Indicators
STEP 3: Boundary Partners
Outputs
STEP 4: Outcome Challenges
Indicators
They tried but found up to 15 progress markers per International networks in which time is at a premium
Activities
STEP 5: Progress Markers
Inputs
STEP 6: Strategy Maps

20. Long-term Goal (Impact) Objectives or Outcomes
ICT4D in Asia PAN and Africa ACACIA
Nigeria Evidence‐based Health System Initiative
EcoHealth Fieldbuilding Leadership Initiative
Consorcio por la Salud, Ambiente y Desarrollo
Long-term Goal (Impact)
STEP 1: Vision
Objectives or Outcomes
STEP 2: Mission
Indicators
STEP 3: Boundary Partners
Outputs
STEP 4: Outcome Challenges
Indicators
They tried but found up to 15 progress markers per International networks in which time is at a premium
Activities
STEP 5: Progress Markers
Inputs
Regional Peacebuilding Programme in the Horn of Africa
STEP 6: Strategy Maps

21. Long-term Goal (Impact) Objectives or Outcomes
Art and Culture Programme in Central America
Long-term Goal (Impact)
STEP 1: Vision
Objectives or Outcomes
STEP 2: Mission
Indicators
STEP 3: Boundary Partners
Outputs
STEP 4: Outcome Challenges
Indicators
They tried but found up to 15 progress markers per International networks in which time is at a premium
Activities
STEP 5: Progress Markers
Inputs
STEP 6: Strategy Maps

22. https://undp.unteamworks.org/node/370238
In sum
Outcome Harvesting is “an evaluation approach that does not measure progress towards predetermined outcomes, but rather collects evidence of what has been achieved, and works backward to determine whether and how the project or intervention contributed to the change.” - UNDP

23. Discussion
Have any of you found other ways to apply the principles of Outcome Mapping to evaluate projects, programmes or organisations when the elements of intentional design have been missing?