2. Day 1:

3. Introduction

4. Course Facilitators

5. Expectations of the Workshop

6. Agenda for Day 1

7. Workshop Process Start & end on time
Give everyone a chance to speak & ask questions
« Issues and Concerns»
Questions & comments at any time
Questions et commentaires en français

8. Introductory Exercise - Each person, share an achievement you`re very proud of & the role you played - Look for patterns in the stories - Be prepared to share the patterns in plenary
To get to know people at tables.
Tell the people at your table about an achievement you’re very proud of and the role you played.
Once you’ve heard about all the achievements, look for patterns: commonalities/contrasts in the « stories ».
Share the patterns with the rest of the room in plenary.
[We should be looking for examples that indicate that achievements usually depend on other people as well]
[TIME: 30 Minutes]

9. Conceptual Overview

10. Evaluation Challenges
Establishing cause & effect in an open system
Timing
Encouraging iterative learning
Clarifying values
1st assessment challenges (measurement problems)
1 attributing specific changes to an external program, project or organization`s activities
- particularly difficult with measuring « impact » ( using the definition from a classical results chain -- long term effects on large group of intended beneficiaries)
2 when to assess?
2nd methodological challenges in evaluation (fostering learning oriented, facilitated, OD type evaluation)
3 engaging groups in a learning process
4The lens through which you see things
Discussion Points and Questions Are these challenges you are grappling with? How?
Create a list of anything else they are facing.

11. Challenge 1: Establishing Cause/Effect in an Open System
Multiple actors & factors contribute
Causality is mutual, therefore usually attribution is not possible
Unintended results often ignored
We work in complex, open systems (many things happen at once, we aren`t the only people involved, relationships among events and actors creates a changing contex)
Linear logics models (if/then, so what?) aren`t adequate to capture this
Emphasis on attribution usually tries to credit a single actor whereas, in reality, multiple actors create results and need credit
Focus on positive intended results is insufficient as unexpected positive and negative results occur and should be accounted for and learned from
We were faced with the challenge of being honest & realistic in crediting the multiple actors & factors that contribute to development
Is this a problem you are grappling with? How?

12. Shifting Influence Over Time
Distinguish between Om and a traditional results chain. Last time people thought that OM was using a results chain because of this slide.
Time

13. Challenge 2: Timing Tracking incremental, cumulative change
Not assessing results as though final
Problem of « when to assess »?
Wait for impact and it`s too late, look too early and you get activities
Focus on ultimate « impact » neglects the importance of more upstream changes
Cumulatively these small changes add up to important development results
If you assess at a point in time and then at another you can lose the details
Focus on impact implies that the story ends when it occurs but, in reality, change is an onngoing process
Donors don`t do development, people do: people make choices, we move on, we don`t live the results.
How do you react? Are these challenges you are facing?

14. Are we having an impact yet ?

15. Challenge 3: Encouraging Iterative Learning
Emphasis on development results
Easy to measure vs. important to know
Keeping results within their context
Making time for reflection
To build on the past we need to capture it if we are to have more than anecdotal learning
Emphasis on long term impact. Can`t just look at results but need to look at what the program is doing as well in order to get better at it. Avoiding « clueless feedback »
Overly simplistic logic models and indicators (especially quantiative) have made us forget why we`re assessing (to understand the effect we have and get better at supporting development processes)
Context is important in order to learn (from situations, not generic lessons) – ask the why? How? Questions
The one challenge no methodologically can answer – giving people time (takes incentives, they have to thinkthe time is worth it, that`s why OM isn`t a requirement,)

16. Accountability & Learning: A Balancing Act

17. Accountability & Learning: A Balancing Act

18. Challenge 4: Clarifying Values
Assessment is not value neutral
Making explicit the implicit
Building the capacity to engage
Process of developing a collaborative value system
Not a universal but a perspective of what happened
Self and partners
Not hollow participation but engagement
What is relevant and meaningful participation?

19. Challenges and Outcome Mapping
Cause & effect in open system
2. Timing
Iterative Learning
Clarifying Values
OM Tools
Vision & Outcome Challenges
Outcome challenge & Progress Markers
Org. Practices &
Monitoring Journals
4. Mission, Boundary Partners, Progress Markers, Org. Practices

20. What is Outcome Mapping?
an integrated PM&E tool
an approach that focuses on changes in the behaviour, relationships,or actions of partners (as outcomes)
a methodology that characterizes and assesses the program’s contributions to the achievement of outcomes
an approach for designing in relation to the broader development context but assessing within your sphere of influence

21. Encouraging evaluative thinking throughout the P,M &E phases is a key to success
Assumptions about P, M &E in OM
It`s collaborative
It`s sharing credit
OM designed around facilitated group planning and learning processes
3 stages; 12 steps (of a workshop)
can be used as a whole system or separately with other tools and methods (examples of how its been used in 2 programms)
Planning Stage
- answers 4 questions: why? (vision); who? (who will the program work with directly to encourage change?); what? (what changes are being sought?); How? (What is the program to do to encourage change in its partners and in itself so that it remains relevant and effective?)
Monitoring
1 – set priorities
2 – OM offers 3 data collection tools (journals) for capturing info on elements from the planning stage
Based largely on self-assessment
Eval
Planning a useful eval process

22. At What Level? Course is focusing at the program level
Definition of Program: A group of related projects and activities with a specific set of resources (human, financial, capital) directed to the achievement of a set of goals within a specified period of time
Can be used by projects, organizations, and communities too
Talk about using OM at project, program, and organizational levels. In this course, we`re talking about using it at program level. Not appropriate for small projects because too much detail.

23. Key Concepts of Outcome Mapping
Sphere of influence
Outcomes as behavioural change

24. Sphere of Influence Planning & Design Context Assessing

25. The Focus of Outcome Mapping
Use reach term, participants last time thought we were advocating the results chain
Behavioural Changes

26. What is an « Outcome » in OM?
Change(s) in the behaviour, relationships, activities, and/or actions of an individual, group, or organization that the program was helpful in bringing about.
These changes are aimed at encouraging human and ecological well-being.

27. Changes Sought Changes in State Ecosystem Human System
Changes in Behaviour

28. Why Behavioural Change?
To stress that development is done by, and for, people
To illustrate that although a program can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected

29. How Can Outcome Mapping Be Used?
For a program to tell its performance story in outcome terms by:
articulating its goals and designing its activities
designing a monitoring system for assessing internal performance and outcomes of partners
setting a use-oriented evaluation plan

30. Why Use Outcome Mapping?
Focussing on changes in partners’ behaviour, relationships, or actions allows a program to:
measure results within its sphere of influence
obtain feedback about its efforts in order to improve its performance
take credit for its contributions to the achievement of outcomes
show progress towards outcomes

31. Stage 1: Intentional Design

32. « I have a dream! » Martin Luther King

35. Step 1 : Vision Statement
Reflects the broad human, social, & environmental betterment in which the program is engaged and to which it is contributing

36. Example Vision Statement
Local authorities, communities, and international organizations in developing countries in Africa recognize the value of HIV/AIDS intervention as an integral part of social & economic development. Municipal, regional, and national governments actively support HIV/AIDS prevention activities by formulating and implementing effective public health policies. Using research findings, they have developed a comprehensive public health strategy to slow down the infection rate. Formerly marginalized groups (e.g. women and youth) are organized into advocacy groups that can effectively formulate their needs to policy makers. All groups have access to reliable and relevant technical information about HIV/AIDS prevention and are able to make informed choices. In essence, there are healthier, happier, and wealthier communities.

37. Facilitation Questions?
“Imagine that in 3-5 years the program has been extremely successful. What changes will you have helped bring about? What are your partners doing differently? What have they achieved? In essence, what would total success look like?”

39. VISION STATEMENT
The Mission is that “bite” of the Vision Statement on which the program is going to focus

40. Step 2: Mission Statement
Describes how the program intends to:
Operationalize its role in support of the vision by determining the areas in which it will work
Support the achievement of outcomes by its boundary partners

41. Facilitation Questions?
“How can the program best support or contribute to the achievement of the vision? What areas do you need to work in? What do you need to do? ”

42. Example Mission Statement
In support of this vision and on behalf of its donors, the program will work in the areas of research, dissemination, capacity building, & coordination. It will contribute to the production, synthesis, & dissemination of research data, position papers, & other information that will sensitize local & international actors to HIV/AIDS prevention. The program will seek to expand the range of disciplines involved in HIV/AIDS research. It will enhance HIV/AIDS research capacity in order to produce credible information for local, national, & international policy-making & program development. It will promote an interest in HIV/AIDS research among new researchers by providing research fellowships, mentorship, & training opportunities. The program will contribute to the development of linkages between Northern & Southern researchers & encourage partnerships between research organizations, advocates, & decision makers. It will increase its visibility & credibility among the donor community & will convince them of the utility of supporting HIV/AIDS prevention.

43. Developing a Vision or Mission
Warm-up – general conversation
Individual write up
Discuss & Post
Facilitator and/or participant draft
Present & discuss
Purge the jargon
Verify with partners (if required) & Finalize

44. Developing a Vision with the Community in Nagaland

45. Suggestions and Tips!
If time is a limiting factor you can carry out an survey in advance to create a draft vision & mission before the workshop starts. Use the draft version as a starting point for discussion in order to save time.

46. Exercise 1:
Correct Errors
in Vision & Mission
Statements

48. Step 3: Boundary Partners
Those individuals, groups, & organizations with whom the program interacts directly to effect change & with whom the program can anticipate some opportunities for influence.

49. Program`s Sphere of Influence
The Real World
Program
= Program`s Boundary Partners

50. Boundary Partners (have boundary partners)
Program’s Boundary Partners
Boundary Partners’ Boundary Partners
Program

51. Typology of IDRC Boundary Partners
Applied Research Community
e.g. researchers, institutions, NGOs, networks
IDRC Program
System Gatekeepers / Dev`t Agents e.g. business, policy makers, NGOs, gov`t
Beneficiaries
e.g. communities, farmers, women, youth

52. Facilitation Questions?
« In which individuals, groups, or organizations is your program trying to encourage change so that they can contribute to the vision?  With whom will you work directly? »

53. Suggestions and Tips!
A program can choose its boundary partners because it wants to influence them or because they will influence others
Example: a rural development NGO chooses to work directly with farmer organizations who in turn, lobby the Minister of Agriculture to effect change.

54. Exercise 2: Identify the Program`s Boundary Partners

55. Working on Your Own Case

56. Guiding Questions In your context:
How would the ideas of Outcome Mapping resonate?
Is a focus on behavioural change appropriate?
How would you foster the imagination of your group to come up with a vision?
Are you clear about your contributions to the vision? (i.e., your mission)
Who are your boundary partners?
How could you include them in the process?

57. Conclusion & Evaluation of Day 1 What’s Working? What’s Not?
At your tables, talk about what’s working, what isn’t and elect one person to stay behind for 10 minutes to provide us feedback.

58. Day 2:

59. Review of Day 1 & Feedback from Evaluation

60. Agenda for Day 2

62. Step 4: Outcome Challenge
Description of:
The changed behaviours (relationships, activities, and/or actions) of a boundary partner
How they could contribute ideally to the vision

63. Example Outcome Challenge Statement
The program intends to see researchers in developing countries that have gained the expertise required to carry out HIV/AIDS research in a multidisciplinary fashion using quantitative, qualitative, and/or participatory methodologies. They are conducting research on the legislative, economic, social, and health aspects of HIV/AIDS with special emphasis on vulnerable groups including youth and women. They are providing credible evidence that allows research findings to be translated into effective HIV/AIDS policies and programs. They are skillfully disseminating research results in the appropriate format for the audience (e.g., to the scientific community, the general public, NGOs, government officials). They are successfully obtaining funding from multiple sources to conduct HIV/AIDS research. More experienced researchers are mentoring those new to the field and HIV/AIDS researchers globally are engaged in productive communications using various means especially electronic networks.

64. « Those Who Dream Make a Difference! » Ms. Kalpana Pant, Chaitanya

65. Phrasing of an Outcome Challenge
“The program intends to see [boundary partner] who [description of behaviours in the active present tense]”

66. Facilitation Questions?
“Ideally, in order to contribute to the vision, how would the boundary partner be behaving?”
“What would your boundary partner be doing to contribute maximally to the vision?”

67. Suggestions and Tips!
Similarity of behaviours sought among boundary partners (policy makers and community members) in international model forests was unexpected by program staff

68. Outcome Challenge Checklist
Outcome Challenge Statements:
Describe behavioural changes
About a single boundary partner
Set out the ideal behavioural changes
Describe the boundary partner`s contributions to the vision

69. Exercise 3: Identify Errors in Outcome Challenge Statements

70. Program Framework Vision: Mission: Boundary Partner:
Outcome Challenge:
Boundary Partner:
Outcome Challenge:
Boundary Partner:
Outcome Challenge:
Boundary Partner:
Outcome Challenge:

71. Overview Exercise: Put Together a Program Framework

73. Step 5: Progress Markers
A graduated set of statements describing a progression of changed behaviours in the boundary partner
Changes in actions, activities, & relationships leading up to the ideal outcome challenge statement

74. Levels of Progress Markers
The program sets out what it would:
Expect to see its boundary partners doing?
Like to see its boundary partners doing?
Love to see its boundary partners doing?

75. How Many Progress Markers?
Limit to a manageable amount of data to be collected:
4 “expect to see”
8 “like to see”
3 “love to see”

76. Why Graduated Progress Markers?
Articulate the complexity of the change process
Permit on-going assessment of partners’ progress (including unintended results)
Encourages the program to think about how it can intentionally contribute to the most profound transformation possible
Make mid-course corrections & improvement easier

77. Example Progress Markers
The program Expects to See local communities:
Participating in regular model forest (MF) partnership meetings
2. Establishing a structure for cooperation in the partnership
3. Acquiring new skills for involvement in the MF
4. Contributing resources necessary to get the MF operational (minimum)

78. Example Progress Markers
The program would Like to See local communities:
Articulating a vision for the MF that is locally relevant
Promoting their involvement with the MF nationally
Expanding the partnership
Calling upon external experts when necessary
Requesting new opportunities for training
Disseminating concrete examples of benefits arising from MF activities
Seeking out new partners for the MF
Obtaining funding from different national sources

79. Example Progress Markers
The program would Love to See
local communities:
13. Helping other MFs establish themselves
14. Sharing lessons-learned internationally
15. Influencing national policy debates & formulation on resource use and management

80. Facilitation Questions?
« How can the program know the boundary partner is moving toward the outcome? What would they be doing? »
« What milestones would be reached as the boundary partner moves towards their intended role in contributing to the vision? »

81. Suggestions and Tips!
If a program has multiple boundary partners but the primary result to be achieved is changing their relationships with one another then it is possible to create a single outcome challenge & set of progress markers for the whole group

82. Progress Marker Checklist
Each Progress Marker:
Describes a changed behaviour by the boundary partner
Can be monitored & observed
As a set, Progress Markers:
Are graduated from easier to more difficult to achieve changes in behaviour
Describe the change process of a single boundary partner

83. Exercise 4: Identify Correct Progress Markers & Categorize as Expect, Like, or Love to See
At the end, ask one group to explain their order and then ask the others to talk about any differences in what they included.

85. Step 6: Strategy Map
Outlines the program`s approach in working with the boundary partner
Indicates the relative influence the program is likely to have on boundary partner
Helps pinpoint strategic gaps in the approach or if the program is overextended
Suggests the type of evaluation method appropriate to track and assess the performance of the program

86. 6 Types of Strategies Strategy Causal Persuasive Supportive I-1
Direct Output
I-2
Arouse New Skills/ Thinking
I-3
Supporter who guides change over time
Aimed at the Boundary Parnter
E-1
Alter physical or regulatory environment
E-2
Modify the information system
E-3
Create / Strengthen a Peer Network
Aimed at the Boundary Partner`s Environment

87. 6 Types of Strategies

88. Facilitation Questions?

89. Suggestions and Tips!
Facilitator asks the group questions about their strategies
While they are discussing them (s)he quietly plots them on a strategy map
After being shown the strategy map, the group is asked to analyze it for gaps, overextension, and the influence that can be expected

90. Exercise 5: Plot Activities on a Strategy Map

92. « The oft repeated and echoing question in Outcome Mapping « what or who needs to change? » raised us from being providers of development, achieving outputs, to actually believing we were agents of change. » Raj Verma, NEPED

93. Step 7: Organizational Practices
The things that you do as an organization to:
foster creativity & innovation
seek the best ways to assist your partners
maintain your niche

94. Why Organizational Practices?
Important to how the program is going to function to effectively fulfill its mission
Supporting change in its boundary partners requires that the program be able to change and adapt as well

95. 8 Organizational Practices
Prospecting for new ideas, opportunities, & resources
Seeking feedback from key informants
Obtaining the support of your next highest power
Assessing & (re)designing products, services, systems, and procedures

96. 8 Organizational Practices
Checking up on those already served to add value
Sharing your best wisdom with the world
Experimenting to remain innovative
Engaging in organizational reflection

97. Facilitation Question?
“What do you have to be doing in order to remain effective?”

98. Suggestions and Tips!
This can be a difficult set of concepts to get across; frame it in terms of changes in behaviour internal to the organization
Have the group think of the program as an organization in its own right

99. Exercise 6: Identify the Organizational Practices

100. Exercise 6a: Draw the 7 steps of Outcome Mapping

101. Organizational Practices
Overview of Stage 1
Evaluation Question: Are you with us?
Organizational Practices

102. Working on Your Own Case

103. Guiding Questions In your own context:
How would the idea of planning beyond what can be achieved be accepted? (i.e. outcome challenge and progress markers)
What kinds of strategies do you use – causal, persuasive, supportive?
How will you get your group to talk about your own organizational practices?
After tables have worked through these questions themselves, they can pose any questions that emerged in trying to think more practically about how Outcome Mapping would work in their context.
We walk from table to table and answer questions.
Feel free to look through the manual for ideas on process and facilitation

104. Evaluation of Day 2 Feedback from « critical friends »
Do you understand? Are you with us?

105. Day 3:

106. Review of Day 2 & Feedback from Evaluation

107. Agenda for Day 3

108. Stage 2: Outcome & Performance Monitoring

109. Outcome Mapping Monitoring
Systematic collection of data
A regular learning & improvement cycle
Encourages the program to challenge itself
Collects & organizes, does not analyze data

111. Thierry Barreto-Fernandez West Africa Rural Foundation, Senegal
« Being busy creates a mindset that is not conducive to innovation and creativity. Without interaction, there is no innovation. Time to discuss, reflect, and generate new ideas is the ransom that outcome mapping demands for innovation. »
Thierry Barreto-Fernandez
West Africa Rural Foundation, Senegal

112. Step 8: Monitoring Priorities
Tempting to gather information on a whole range of issues but this is not practical
Base priorities on intended use of intended users:
Use(s) for findings and from process
Improve performance through learning
Help meet accountability/reporting requirements
Supply information for planned evaluation
Inform publicity documents, communication activities, or case-study materials

113. Monitoring Plan

114. Boundary Partner
Program

115. Outcome Challenge & Progress Markers
Boundary Partner
Program
Outcome Challenge & Progress Markers

116. Strategies & Activities
Boundary Partner
Program
Strategies & Activities

117. Organizational Practices
Boundary Partner
Program
Organizational Practices

118. Boundary Partner Program Outcome Journal Strategy Journal Performance

119. Organizing Monitoring Meetings
Regular face-to-face meetings to record observed changes
Filling out electronic data sheets on an on-going basis, then aggregating changes for entire program
Interviews and/or focus groups with boundary partners

121. Step 9: Outcome Journal
Uses progress markers to chart boundary partner`s progress
Sets starting point or benchmark
Captures details on changes in the boundary partner, contributing factors & actors, and sources of evidence

122. Outcome Journal Work Dating From/To:
Contributors to Monitoring Update:
Outcome Challenge:
Low =
Medium =
High =
Expect to see:
Who?
LMH 1 2 3

123. Set Values for Low, Medium & High
Can capture either quantity or quality of change:
Quantity (number of boundary partners)
Quality (depth of change)

124. Baseline
multiple individuals, groups or organizations in each boundary partner set
identify each boundary partner’s starting point in terms of progress markers
boundary partners may be at different stages of development
individual change process can be captured over time

126. Step 10: Strategy Journal
Captures data on the strategies being employed to encourage change in the boundary partner
Helps the program gauge its contributions to the achievement of outcomes and modify its actions accordingly
Feeds into work planning & (modifications) or new activities

127. (How did it help the boundary partner?)
Strategy Journal
Working Dating From/To:
Contributors to Monitoring Update:
Strategy to be Monitored:
Effectiveness?
(How did it help the boundary partner?)
Outputs

128. Step 11: Performance Journal
Records data on how the program is operating as an organization to fulfill its mission
Includes information on the program’s practices
Feeds into work planning

129. Performance Journal Working Dating From/To:
Contributors to Monitoring Update:
Practice 1: Prospecting for New Ideas, Opportunities, & Resources
Example or Indicators:
Sources of Evidence:
Lessons:
Practice 2: Seeking Feedback from Key Informants
Example or Indicators:
Sources of Evidence:
Lessons:

130. Exercise 7 Identify your monitoring priorities (e. g
Exercise 7 Identify your monitoring priorities (e.g. work on your real case)

131. Stage 3: Evaluation Planning

133. « Evaluative thinking is a learned behaviour that you have to practice and improve at on an on-going basis! » Michael Quinn Patton

134. Step 12: Evaluation Plan
A short description of the main planning elements of an evaluation study
For Whom? What? Where? How? By Whom? When? How Much?

135. Elements of an Evaluation Plan

136. Setting Evaluation Priorities
Criteria with which to select include:
Learning needs
Reporting requirements
Partners’ information needs

137. Guiding Principles for Evaluation
Balancing learning & accountability
Action oriented
Each study is unique
Process not only product
Build capacity at all levels through participation

138. « Not everything that counts can be counted
« Not everything that counts can be counted. And not everything that can be counted, counts. » Albert Einstein

139. An Evaluation of IMFNS Assessment of 1st phase
Wanted to measure human, community, & partnership elements of model forests
Client acceptance of methodology essential
OM provided a framework for the study

140. Evaluation Questions Who has IMFNS reached?
How have the behaviours of IMFNS`boundary partners changed?
What has been the nature of IMFNS` interactions with its partners?

141. Process of the IMFNS Evaluation
Recreated planning framework in 1 day with IMFNS staff
Developed progress markers for communities & goverment officials across the 5 sites
Conducted a document review & key informant interviews
Comparison of boundary partners across MF sites for analysis

142. Comparing Boundary Partners

143. Lessons from the IMFNS Study
Difficult to link what the Secretariat did with outcomes achieved without using cause-effect
Boundary partners should have been involved in the development of the framework & analysis of data
Should have included more analysis of organizational practices
Did not study the relevance of boundary partners within their environment

144. A Self-Assessment of NEPED
Preparation for external evaluation
Process
Fred knew people and context prior to workshop
Retrospectively built the framework (took 1.5 days including HML values)
« Ah-ha » moment happened at end for some participants
Organizing data collection (preliminary identification, collection and verification, analysis and writing report took another 1.5 days)

145. Lessons from the NEPED Study
Group dynamics (worked together well,willing to trust Fred, cooperative)
Data (they had the data but it wasn`t organized)
Preparation for external review (intended use)
Relevance of work with state government (beyond trees, number of women involved, number of villages worked with)
Could have had greater external participation (Gov`t of Nagaland, etc.)

146. Conclusion

147. Encouraging evaluative thinking throughout the P,M &E phases is a key to success
Assumptions about P, M &E in OM
It`s collaborative
It`s sharing credit
OM designed around facilitated group planning and learning processes
3 stages; 12 steps (of a workshop)
can be used as a whole system or separately with other tools and methods (examples of how its been used in 2 programms)
Planning Stage
- answers 4 questions: why? (vision); who? (who will the program work with directly to encourage change?); what? (what changes are being sought?); How? (What is the program to do to encourage change in its partners and in itself so that it remains relevant and effective?)
Monitoring
1 – set priorities
2 – OM offers 3 data collection tools (journals) for capturing info on elements from the planning stage
Based largely on self-assessment
Eval
Planning a useful eval process

148. Conclusion
A system to think holistically & strategically about how to contribute maximally to development
A method to articulate contributions to the development process through boundary partners
Outcomes are changes in the behaviour, actions, and/or relationships of boundary partners
An approach that encourages on-going learning, challenges, & improvement through M&E activities

149. Back to Work… How are you going to put your training in Outcome Mapping to use?

150. Were Your Expectations Met?

151. Evaluation of the Workshop In terms of the content and process, what should we keep, add, change?
Evaluation sheet is in the binder

152. Thank You! http://www.idrc.ca/evaluation
Evaluation sheet is in the binder