1. Assessing Contribution and Enhancing Social Learning: An Introduction to Outcome Mapping (OM) 16 December 2004 Kaia Ambrose

2. Presentation Outline OM in Carchi, Ecuador. OM in Carchi, Ecuador. Underlying logic of OM. Underlying logic of OM. Brief outline of stages of OM. Brief outline of stages of OM. Some key concepts - boundary partners, their changes and how we measure that change. Some key concepts - boundary partners, their changes and how we measure that change. OM and learning organizations. OM and learning organizations. The OM toolbook. The OM toolbook. Conclusions and ideas for the afternoon session. Conclusions and ideas for the afternoon session.

3. First, a little context… Ecopar Ecopar Ecuadorian NGO focused on research, training and capacity building in tropical ecosystems/life zones.Ecuadorian NGO focused on research, training and capacity building in tropical ecosystems/life zones. The Ceja Andina Project The Ceja Andina Project IDRC-funded.IDRC-funded. Sustainable use of agriculture and forest biodiversity in the ceja andina (cloud forest) region of the northern Ecuadorian Andes.Sustainable use of agriculture and forest biodiversity in the ceja andina (cloud forest) region of the northern Ecuadorian Andes. Research-oriented (traditional and participatory), social learning processes, policy development and strengthening of local government.Research-oriented (traditional and participatory), social learning processes, policy development and strengthening of local government. Developing OM since March 2002 – “walking it through and making it make sense in our context”.Developing OM since March 2002 – “walking it through and making it make sense in our context”.

4. M&E concerns in the Ceja Andina Project As a team and as an NGO, how can we learn from the project in order to improve the way we “do” development (in a research context)? How can we become a learning organization while still being accountable? As a team and as an NGO, how can we learn from the project in order to improve the way we “do” development (in a research context)? How can we become a learning organization while still being accountable? How can we learn (iterative processes) together with our local partners? How can we learn (iterative processes) together with our local partners? How can we bring creativity and real interest into the M&E process? (create culture of M&E) How can we bring creativity and real interest into the M&E process? (create culture of M&E) What is the balance between rigor and utility in our M&E objectives? What is the balance between rigor and utility in our M&E objectives? Assess need for OM (complementary, bits of). Assess need for OM (complementary, bits of).

6. What is Outcome Mapping? an approach for designing M&E in relation to the broader development context but assessment is within your sphere of influence. an approach for designing M&E in relation to the broader development context but assessment is within your sphere of influence. an integrated and participatory M&E approach; also for planning. an integrated and participatory M&E approach; also for planning. an approach that views outcomes as changes in the behaviour, relationships, or actions of partners. Development is accomplished an approach that views outcomes as changes in the behaviour, relationships, or actions of partners. Development is accomplished through changes in the behaviour of people. a methodology that characterizes and assesses the program or project’s contributions and influence to the achievement of outcomes. a methodology that characterizes and assesses the program or project’s contributions and influence to the achievement of outcomes.

7. Program / project’s Sphere of Influence PROGRAM (performance and strategies) = Program’s partners (behavioural change)

8. Outcome Mapping does not: Focus on impact (we’re not in the impact business, we’re in the business of change). Focus on impact (we’re not in the impact business, we’re in the business of change). Look for attribution (we look for contribution). Look for attribution (we look for contribution). Isolate contributions of program / project in achievement of results. Isolate contributions of program / project in achievement of results. Promote linear, cause-effect thinking in a “sterile” context. Promote linear, cause-effect thinking in a “sterile” context. Focus solely on actions promoted by program / project. Focus solely on actions promoted by program / project.

9. Influencing Outcomes Partners Program Assessing Program Influence Assessing Changes in Behaviour Assessing Internal Performance

10. Outcome Mapping

11. OM can help us: Besides designing or clarifying programme logic, helps record and assess monitoring data: Besides designing or clarifying programme logic, helps record and assess monitoring data: How far have our boundary partners progressed towards achieving outcomes?How far have our boundary partners progressed towards achieving outcomes? What are we doing to support the achievement of outcomes?What are we doing to support the achievement of outcomes? Indicate cases of positive performance and areas of improvement; front stage and back stage. Indicate cases of positive performance and areas of improvement; front stage and back stage. Evaluate intended and unexpected results; positive deviance. Evaluate intended and unexpected results; positive deviance. Gather data on the contribution that a program made to bringing about changes in its partners (strategies - new tools, aptitudes, knowledge). Gather data on the contribution that a program made to bringing about changes in its partners (strategies - new tools, aptitudes, knowledge). Establish evaluation priorities and evaluation plan. Establish evaluation priorities and evaluation plan.

12. Boundary Partners – Core ideas Boundary partners – individuals, groups, organizations with whom the program interacts directly and with whom the program anticipates opportunities for influence. Boundary partners – individuals, groups, organizations with whom the program interacts directly and with whom the program anticipates opportunities for influence. Outcome challenges – description of the ideal changes in the behaviour, relationships, activities and/or actions of a BP that contribute to vision. Outcome challenges – description of the ideal changes in the behaviour, relationships, activities and/or actions of a BP that contribute to vision. Progress markers – a set of graduated indicators of changed behaviours for a BP that focus on the depth and quality of change. Progress markers – a set of graduated indicators of changed behaviours for a BP that focus on the depth and quality of change.

13. Boundary Partners Social change is about relationships. Social change is about relationships. We’re all interconnected. We’re all interconnected. Social learning – create consensus, among multiple cognitive beings, regarding direction of program in terms of influencing outcomes, in order to arrive at concerted action. Social learning – create consensus, among multiple cognitive beings, regarding direction of program in terms of influencing outcomes, in order to arrive at concerted action. Boundary partners participate in process of OM (in M&E). Boundary partners participate in process of OM (in M&E).

14. Boundary Partners (have their own BPs) Programme Programme’s Boundary Partners Boundary Partners’ Boundary Partners Ceja Andina Project Municipality Town population Rural Agriculture Association Pesticide salesmen

15. Ask yourselves … « In which individuals, groups, or organisations is our programme trying to encourage change so that they can contribute to the vision? With whom will we work directly? » « Are we choosing X boundary partner because we want to influence their behaviour and actions, or because they will influence others? Or both?» « What behavioural changes do we (collectively, between project and boundary partners) want to see in BP that will contribute to the vision (Outcome Challenges)?.»

16. Progress Markers Describes progression of changed behaviours in the boundary partner (is a set, as opposed to a single indicator in order to demonstrate the depth and complexity of change process). Describes progression of changed behaviours in the boundary partner (is a set, as opposed to a single indicator in order to demonstrate the depth and complexity of change process). Changes in actions, activities, & relationships leading up to the ideal outcome challenge statement. Changes in actions, activities, & relationships leading up to the ideal outcome challenge statement. Expect to see, like to see, love to see. Expect to see, like to see, love to see.

17. Progress Markers = Change Ladder Expect to see PMs Love to see PMs Like to see PMs Early response to program’s basic activities. More active learning, engagement. Truly transformative. Set quite high.

18. Example Progress Markers The project Expects to See local communities: 1. Participating in field day demonstrations. 2. Establishing a structure for cooperation in the partnership. The project would Like to see local communities: 3. Articulating a vision for the ceja andina forest that is locally relevant. 4. Requesting new opportunities for training. The project would Love to see local communities: 5. Influencing national policy and debates on CBNRM.

19. Progress Marker Checklist Each Progress Marker: Describes a changed behaviour by the boundary partner Describes a changed behaviour by the boundary partner Can be monitored & observed Can be monitored & observed Limit the number of PMs. Limit the number of PMs. As a set, Progress Markers: Are graduated from easier to more difficult to achieve changes in behaviour Are graduated from easier to more difficult to achieve changes in behaviour Describe the change process of a single boundary partner Describe the change process of a single boundary partner Remember: progress markers are changeable!!

20. 6 Types of Strategies Aimed at the Boundary Partner Aimed at the Boundary Partner`s Environment StrategySupportivePersuasiveCausal I-1 Direct Output E-1 Alter physical or regulatory environment I-2 Arouse New Skills/ Thinking I-3 Supporter who guides change over time E-2 Modify the information system E-3 Create / Strengthen a Peer Network The program facilitates new tools, techniques, resources in order to contribute to outcomes and development process.

21. Creating your strategy map

22. Organizational Practices Help us to define our role as an NGO according to changing contexts and shifting paradigms. Help us to define our role as an NGO according to changing contexts and shifting paradigms. Important to how the program is going to function to effectively fulfill its mission. Important to how the program is going to function to effectively fulfill its mission. Supporting change in its boundary partners requires that the program be able to change and adapt as well. Supporting change in its boundary partners requires that the program be able to change and adapt as well. The things that we do as an organization to: The things that we do as an organization to: foster creativity & innovationfoster creativity & innovation seek the best ways to assist partnersseek the best ways to assist partners maintain our nichemaintain our niche

23. Eight Organizational Practices 1. Prospecting for new ideas, opportunities, & resources. 2. Seeking feedback from key informants. 3. Obtaining the support of your next highest power 4. Assessing & (re)designing products, services, systems, and procedures. 5. Checking up on those already served to add value. 6. Sharing your best wisdom with the world. 7. Experimenting to remain innovative. 8. Engaging in organizational reflection.

24. Journals Set overall intentions and strategies; design and articulate program’s logic (how it will contribute to change within a complex system). Record internal and external data: How we are progressing towards outcomes. How we are contributing to change. Indicate cases of positive performance /improvement. Assess intended and unexpected results.

25. Outcome and Performance Monitoring Boundary Partners Program Strategies and Activities Performance Journal Organizational Practices Outcome Journal Strategy Journal Outcome Challenges and Progress Markers

26. Conclusions Conclusions OM can help a program to be more strategic about the actors it targets, the changes it expects to see, and the means it employs to support and facilitate those changes. OM can help a program to be more strategic about the actors it targets, the changes it expects to see, and the means it employs to support and facilitate those changes. OM is not based on a cause-effect framework; it recognizes that multiple, nonlinear events lead to change. It sees interrelationships, not a snapshot. OM is not based on a cause-effect framework; it recognizes that multiple, nonlinear events lead to change. It sees interrelationships, not a snapshot. Attribution becomes contribution. Attribution becomes contribution.

27. Conclusions Focus is on behavioural change – monitors and evaluates whether a program has contributed to changes in behaviours in a way that would be logically consistent with supporting development changes in the future. Focus is on behavioural change – monitors and evaluates whether a program has contributed to changes in behaviours in a way that would be logically consistent with supporting development changes in the future. Program must also learn and change, reconsider and adjust its goals, methods, interventions. Program must also learn and change, reconsider and adjust its goals, methods, interventions. Self-assessment, systematically collected. Self-assessment, systematically collected.

28. Conclusions Improving rather than proving. Improving rather than proving. Understanding rather than simply reporting. Understanding rather than simply reporting. Creativity and knowledge, rather than just taking credit. Creativity and knowledge, rather than just taking credit.

29. For more information, case studies, materials, articles and presentations on Outcome Mapping: www.idrc.ca/evaluation

30. Afternoon session (what I propose) Fun dynamic! Identify your boundary partner! Fun dynamic! Identify your boundary partner! Continued, in-depth presentation on: Continued, in-depth presentation on: Boundary partners.Boundary partners. Outcome challenges.Outcome challenges. Progress markers.Progress markers. Practical exercise on progress markers. Practical exercise on progress markers. Challenges / successes of the methodology. Challenges / successes of the methodology. Open discussion. Open discussion. Postcard exercise (key messages). Postcard exercise (key messages).

31. Not everything that counts can be counted. And not everything that can be counted, counts. Albert Einstein

32. OM challenges Workload / paperwork– we made too much work for ourselves and didn’t prioritize (too many progress markers, tried to monitor everything). Workload / paperwork– we made too much work for ourselves and didn’t prioritize (too many progress markers, tried to monitor everything). Trial-and-error reporting system. Trial-and-error reporting system. Scaling-up within NGO. Scaling-up within NGO. New boundary partners. New boundary partners. Different boundary partners have different experiences with the projects (example: municipalities). Different boundary partners have different experiences with the projects (example: municipalities). Project team had no formal training on the methodology. Project team had no formal training on the methodology.

33. OM challenges Weak monitoring plan (no one to lead OM from the beginning). Weak monitoring plan (no one to lead OM from the beginning). How to combine with more quantitative data collection. How to combine with more quantitative data collection. Encouraging on-going boundary partner participation in OM M&E activities. Encouraging on-going boundary partner participation in OM M&E activities.

34. OM successes Build cohesion with team and with partners. Build cohesion with team and with partners. Identify and develop interconnectedness among boundary partners through dialogue, consensus and collaborative action; social learning space. Identify and develop interconnectedness among boundary partners through dialogue, consensus and collaborative action; social learning space. How to connect with people becomes part of the methodology. How to connect with people becomes part of the methodology. Highly participatory. Highly participatory. Donor is flexible – encourages learning, creativity, adjustments. Makes space for dialogue and interaction (and really listens!) Donor is flexible – encourages learning, creativity, adjustments. Makes space for dialogue and interaction (and really listens!) Self-assessment “all around” (of team, of boundary partners). Self-assessment “all around” (of team, of boundary partners).

35. OM successes Innovation of tools and application of tools. Innovation of tools and application of tools. Build culture of M&E within NGO / team and boundary partners. Build culture of M&E within NGO / team and boundary partners. Concrete data for developing evaluation plan. Concrete data for developing evaluation plan. Meetings for discussion / self-assessment = learning organization (analyze our learning). Permanent system of reflection. Meetings for discussion / self-assessment = learning organization (analyze our learning). Permanent system of reflection. Interest to learn more about OM by other NGOs in Ecuador (request for information, presentations, etc.) = good promotion of methodology (and project). Interest to learn more about OM by other NGOs in Ecuador (request for information, presentations, etc.) = good promotion of methodology (and project).