Challenging Evaluation: An Introduction to Outcome Mapping Amy Etherington & Rebecca Lee Mini-training for IDRC Interns & PDAs December 7 & 8, 2005

Session overview: Evaluation context – where does OM fit? Introduction to OM methodology – main concepts and tools OM book – its useful! How is OM being used – when is it appropriate?

Warm-up question: What words come to your mind when you think of evaluation?

IDRCs Evaluation System Promotes ownership & use of findings at all levels Decentralized Focus on evaluation processes Monitors quality Accountability for results at program & project levels

IDRCs Evaluation Unit Strategic evaluations Cross-cutting issues Corporate reporting To BoG on performance and results Information systems Capacity building With Southern partners Promoting M&E with programs Tools & methods Organizational Assessment Outcome Mapping

Monitoring & evaluation challenges: 1.Establishing cause & effect in open systems 2.Sharing ownership & participation 3.Recognizing the contributions of others 4.Tracking progress 5.Encouraging iterative learning 6.Measuring development results of research 7.Timing

Shifting influence over time Influence Time

Are we having an impact yet ?

A framework that allows researchers to plot human behaviour and actions and assess their contribution to the aims of research projects and programs Outcomes as changes in behaviour, relationships, activities or actions of the people, groups and organizations with whom a program works directly IDRCs Evaluation Unit ( )

OM bumper-stickers: Be prepared for surprises along the way The map is not the territory! Being attentive along the journey… This is as important as the destination Keep your eyes wide open!

OM t-shirts ATTRIBUTION CONTRIBUTION

OM t-shirts IMPACT INFLUENCE

OM recognizes that in a development context, change is: Complex (multiple actors and factors) Continuous (not limited to the life of the project) Non-linear (unexpected results occur) Beyond the control of the project (but subject to its influence) Two-way (program also changes)

Principles of use Flexible: modular - use adapted to circumstances Participatory: seeks dialogue and collaboration with boundary partners in P,M,&E Evaluative thinking; culture of reflection: promotes social and organizational learning

Step 1: Vision

I have a dream! Martin Luther King, Jr. August 28, 1963

Vision is a guide: North Light house Distant star

Vision statement: Reflects the broad human, social, and environmental betterment in which the program is engaged and to which it is contributing Written in present tense - as if the vision were already a reality

Those who dream make a difference! Ms. Kalpana Pant -Chaitanya

Women and girls in rural India enjoy full access to appropriate health care, education, food and water security and freedom from violence. They earn their own livelihoods with free access to markets and to banking and municipal services. Women knowledgeably use drudgery reduction technologies and agricultural practices that support human and ecological well- being. Villages are served by public transport, are well lit at night and have police forces that enforce laws fully and equitably. Girls attend school full time and, with their families, have the information and resources to make informed decisions regarding their personal health, safety and social needs. Gender equity is evident in the allocation of household labor and household decision-making; and men in the community support gender- responsive initiatives. Schools reinforce the role of women as educated, valuable and empowered community members. Through community-based organizations, women participate in and influence decisions, policies and programs affecting their well-being and share their experiences with others.

What is the human, social and/or environmental condition that you hope to see in the region in which the program is being carried out? Who is contributing to this situation - and how? Vision: facilitation questions Dream boldly !

Step 2: Mission

The mission is that bite of the vision statement on which the program is going to focus. Written in future tense - as something the program will do

Your mission is your business What do you do? Who do you do it with? Why do you do it?

How can the program best support or contribute to the achievement of the vision? What areas do you need to work in? Where do you have credibility? Who can you work with? Mission: facilitation questions

Step 3: Boundary Partners

Boundary Partners: definition Those individuals, groups, and organizations with whom the program: interacts directly to effect change anticipates opportunities for influence engages in mutual learning

Program The rest of the world = partners

Sphere of influence Program The rest of the world = partners

Boundary Partners have Boundary Partners programprograms bpbps bp

CIDA IDRC BAIF State NGO State NGO State NGO State NGO State NGO State NGO SHGPolice Community Leaders FamiliesBanksPHCs Swayamsiddha

In which individuals, groups, or organizations is our program trying to encourage change so that they can better contribute to the vision? With whom will we work directly? Are we choosing X BP because we want to influence their behaviour and actions, or because they will influence others? Or both? What behavioural changes do we (the project and BPs, collectively) want to see in the BP that will contribute to the vision? Boundary Partners: Facilitation Questions

Step 4 & 5: Outcome Challenges and Progress Markers

One OC is about a single boundary partner Describes the ideal behavioural changes, relationships, actions and interactions in this partner Describes how these changes will contribute to the vision. Outcome Challenge: definition

One for each boundary partner Does NOT describe program strategies Written like this: The program intends to see [boundary partner] who [description of behaviours in the active present tense] Outcome Challenge: characteristics

Ideally, how would your boundary partner be acting? With whom would they be interacting? What would they be doing? Outcome Challenges: facilitation questions

Progress Markers: description A graduated set of statements describing a progression of changed behaviours in the boundary partner Describe changes in actions, activities and relationships leading to the ideal outcome Articulate the complexity of the change process Can be monitored & observed Permit on-going assessment of partners progress (including unintended results)

3 Levels of Progress Markers The program sets out what it would: Expect to see the boundary partner doing? Like to see the boundary partner doing? Love to see the boundary partner doing?

Progress markers = ladder of change Truly transformative Set quite high More active learning, engagement Early response to programs basic activities Love to see Like to see Expect to see

X

Why graduated progress markers? Taken as a set, the progress markers: are graduated from easier to more difficult to achieve changes in behaviour describe the change process of a single boundary partner are more complete than a single indicator help the program think about how it can intentionally contribute to the most profound transformation possible facilitate mid-course corrections and improvement

Influencing Outcomes Influencing Outcomes Boundary Partner Program relevance & viability Org. Practices Program Results Progress Markers Program Delivery Strategies Program

Step 6: Strategy Map

Strategy Map Outlines the program`s approach in working with the boundary partner Indicates the relative influence the program is likely to have on boundary partner Helps pinpoint strategic gaps in the approach or if the program is overextended

6 Types of Strategies Aimed at the Boundary Parnter Aimed at the Boundary Partner`s Environment StrategySupportivePersuasiveCausal I-1 Direct Output E-1 Alter physical or regulatory environment I-2 Arouse New Skills/ Thinking I-3 Supporter who guides change over time E-2 Modify the information system E-3 Create / Strengthen a Peer Network

Facilitation Questions:

Strategy images

Step 7: Organizational Practices

What the program does to: look within stay fresh, sharp, effective, healthy better serve its partners learn and change Organizational Practices: definition

Why Organizational Practices? Important to how the program is going to function to effectively fulfill its mission Supporting change in its boundary partners requires that the program be able to change and adapt

8 Organizational Practices 1.Prospecting for new ideas, opportunities, and resources 2.Seeking feedback from key informants 3.Obtaining the support of your next highest power 4.Assessing and (re)designing products, services, systems, and procedures

8 Organizational Practices 5.Checking up on those already served to add value 6.Sharing your best wisdom with the world 7.Experimenting to remain innovative 8.Engaging in organizational reflection

Step 8-11: Monitoring priorities and Journals

Count every " F " in the following text: FINISHED FILES ARE THE RE SULT OF YEARS OF SCIENTI FIC STUDY COMBINED WITH THE EXPERIENCE OF YEARS...

M & E involves making choices You cant always get what you want… But if you try sometimes, you might find You get what you need… ahhhhhhh, yeah… The Rolling Stones

Choosing WHAT to monitor Who will use the monitoring information? What will it be used for? When is it needed?

Choosing HOW to monitor Which components will be monitored? How and when will data be collected? Who will collect it? Who will analyze, collate, package data? Where and when will it be discussed and used?

Monitoring needs to be: Useful Affordable Light in work Understandable

Outcome Journal Work Dating From/To: Contributors to Monitoring Update: Low = Medium = High = Outcome Challenge: LMH Expect to see:Who? 3 1 2

Strategy Journal Working Dating From/To: Contributors to Monitoring Update: Strategy to be Monitored: Effectiveness? (How did it help the boundary partner?) Outputs

Performance Journal Working Dating From/To: Contributors to Monitoring Update: Practice 1: Prospecting for New Ideas, Opportunities, & Resources Practice 2: Seeking Feedback from Key Informants Example or Indicators: Sources of Evidence: Lessons: Example or Indicators: Sources of Evidence: Lessons:

Step 12: Evaluation Plan

Evaluation Plan: A short description of the main planning elements of an evaluation study For Whom? What? Where? How? By Whom? When? How Much?

Elements of an Evaluation Plan

Setting Evaluation Priorities Criteria with which to prioritize include: Learning needs Reporting requirements Partners information needs

How can OM be used? Designing and articulating the programs logic Recording internal and external monitoring data Indicating cases of positive performance and areas for improvement Evaluating intended and unexpected results Gathering data on the contributions that a program made to bringing about changes in its partners Establishing evaluation priorities and an evaluation plan

When is OM best used? Once strategic direction or primary program areas are established Particularly effective for larger projects Best used at the start, but can also be used as a midway or final assessment tool Activities must be sufficiently specific to identify key groups who will be influenced

Is OM appropriate? Not always! May require changes that are not possible Able to focus on OM outcomes Commitment to change and self-assessment Not intended for technical evaluations (assessing relevance of a programming area or a cost- effectiveness comparison) Credible, and compatible with donor reporting requirements Team consensus Resource commitment