1. Plan:Net Limited (www.plannet.ca)
How to plan and manage an environmental education program – AND evaluate its success Philip Cox,
Plan:Net Limited (
Alberta Council for Environmental Education (ACEE)
October 18th Webinar

2. A craft that comes by different names…
Outcomes Measurement
Performance Measurement and Evaluation
Outcomes Mapping
Managing by Results
Results Based Management

3. A craft that comes by different names…
Outcomes Measurement
Performance Measurement and Evaluation
Outcomes Mapping
Managing by Results
Results Based Management

4. Introducing Three Outcomes Measurement Tools…
Logic Model
Risk Analysis and Management Table
Monitoring and Evaluation Plan

5. Monitoring / Evaluation
The basis for workplans and job descriptions
A management reference during team meetings
Start Up
Implementation
A guide for monitoring and reporting on progress & adjusting strategies
Donor Appraisal
Monitoring / Evaluation
Proposal Development
Project Initiation
An aid in designing an evaluation
The centrepiece of your proposal or plan
1. The basis for an introductory conversation with stakeholders
Phase Out
Logic Model, Risk Analysis and Management Table, and Monitoring and Evaluation Plan are relevant to all parts of the program cycle…

6. What is Outcomes Measurement?
It is a way of thinking and a set of tools to help us:
Convert a project idea into a set of activities and anticipated changes in a way that is:
Mindful of context
In keeping with the time and resources available
Keep track of progress against the plan and use this information to:
Make management decisions,
Service accountability relationships and
To be a learning organization.

7. Outcome Measurement is Part of a Global Trend
A response to new realities...
Need to demonstrate value to citizens and taxpayers…
A response by funding bodies around the globe who are under pressure to provide evidence that their programs achieve results
EEC, World Bank, USAID, UN, WFP and others need to provide credible evidence of results they achieve
By mid 90’s, almost all OECD governments had adopted RBM approaches
In Canada - federal government and all provinces
Increasingly philanthropic organizations such as Alberta Ecotrust Foundation, EcoAction, etc.

8. What does this mean in practical terms?
When making a plan, or reporting on progress, we are challenged like never before to:
Describe the connection between the work we do and the difference we make

9. If you are planning…
A new web-based information portal for educators on climate change
You might focus attention on…
The quantity and quality of information loaded on to the portal
The ‘user-friendliness’ of the web site
The number of users and how they are accessing the portal
And also…
What teachers are looking for on the portal
How they are using the information to inform their teaching
How the teaching-learning experience in the classroom is altered

10. Exercise Think about a project that you know well
Craft two outcome statements that reflect the change you want to see
Share your statement with your neighbour and then in plenary

11. Where can Outcomes Measurement be used in your organization?
Organization as a whole
Donor funded programs
Partner organizations
Individual Partner Initiatives/Projects

12. Three Outcomes Measurement Tools
Logic Model 
- “Measuring Success…” pages
Risk Analysis and Management Table
Monitoring and Evaluation Plan

13. The Logic Model
Outcomes Measurement practice revolves around the logic model

14. Another way of thinking about Activities and Outcomes
Splash and Ripple !
Another way of thinking about Activities and Outcomes
Splash and Ripple!

15. Situation Assessment Before beginning a results based plan..
1. Looking Inward
- Mandate
Competencies
Experience
2. Looking Outward
Issues & Opportunities
3. Clarifying Priorities

16. Situation Assessment - Stakeholders
Influence
HIGH
High Interest/ Importance, High Influence
High Interest/ Importance, Low Influence
Interest or Importance
Consider ways to strengthen or protect interests
Build coalition of support
Low Interest/ Importance, High Influence
Low Interest/ Importance, Low Influence
Interpretation of Headings
Vertical Axis
-’Interest’ refers to the willingness of groups and individual to invest time and other resources into the project/activity.
-’Importance’ refers to the centrality of individuals/groups to the success of the project (it might be that groups/individuals are very important to the success of an project/activity, even if they don’t perceive that to be the case.
Horizontal Axis
-’Influence’ refers to the extent to which groups/individuals can sway the results of project work either positively or negatively by virtue of their status or economic power.
LOW
Spend more time on the other Stakeholders
Caution - be aware of risks or obstacles
HIGH
LOW

17. Discussion
Generate a list of groups with a stake in the success of your project
Place them on the stakeholder matrix according to their current
Interest/importance
Influence

18. Logic Model Format - One Version
Inputs
Activities
Outputs
Outcomes
Impact
Introduce here the “How” - “Why” test…
Use this test when you have a good draft logic model
Left to right - ask the question WHY?
i.e. take each activity and ask WHY? The answer should reflect your related outputs and short term outcomes. Ask the same question for the short term outcomes and the answer should be reflected in the collection of intermediate outcomes. Ask WHY for the collection of intermediate outcomes and the answer should be reflected in the long term outcome statement.
Right to left - ask the question HOW?
Follow the same logic.

19. Logic Model Format - Another Version
Ultimate Outcomes
Why
What we want
Intermediate Outcomes
Immediate
Outcomes
Outputs
REF: pp of guide (link splash and ripple to logic model, etc.)
How
Activities
Inputs

20. Inputs…
Are the human and physical resources that make a program or project possible.
Links outcomes logic to budgeting

21. Intermediate Outcomes
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Outputs
Activities
Inputs
Trainers, Materials, Equipment, Communications…

22. Activities
Describe the essential work of the program or project – what you DO
Should be summarized in no more than 5 statements.

23. Intermediate Outcomes
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Outputs
Activities
Assess training needs
Design training materials
Deliver training
Provide follow-up coaching
Inputs
Trainers, Materials, Equipment, Communications…

24. Outputs… Short term Each output relates to one activity.
…a change that is observable as you complete the activity.
Each output relates to one activity.
You have substantial control.
Involves
a specific group of people - those directly involved in the activity, or
the creation of a product
The change helps make outcomes possible.
Often considered to be deliverables
One set of outputs per activity set
- have a lot of control
- usually stated as numbers - OFTEN OUTPUTS ARE EXPRESSED AS STATEMENTS THAT RESEMBLE INDICATORS

25. Intermediate Outcomes
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Outputs
Defensible report issued
Relevant, user friendly training materials in place
Participants complete training
On-the-job queries addressed
Activities
Assess training needs
Design training materials
Deliver training
Provide follow-up coaching
Inputs
Trainers, Materials, Equipment, Communications…

26. Immediate Outcomes…
Observable within the project cycle, following activities
Describes the potential created by the outputs
You still have considerable control
Centres on a very specific group of people - those directly involved in the activity or using the product.

27. Intermediate Outcomes
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Stakeholder commitment to a training strategy
Trained managers apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned
Outputs
Defensible report issued
Relevant, user friendly training materials in place
Participants complete training
On-the-job queries addressed
Activities
Assess training needs
Design training materials
Deliver training
Provide follow-up coaching
Inputs
Trainers, Materials, Equipment, Communications…

28. Intermediate Outcomes…
Describe the key changes you want to make in the program/project
Observable at or shortly after the completion of the program/project
You have influence, but not control
Usually involves a larger group of people than those directly involved in the program/project
Are fewer in number; each flows naturally from several Immediate Outcomes

29. Intermediate Outcomes
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcome
Intermediate Outcomes
Agency leaders make evidence- based, results- focused planning and management decisions
Donors and community stakeholders recognize the ‘value-added’ by the agency’s work
Immediate Outcomes
Stakeholder commitment to a training strategy
Trained managers apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned
Outputs
Defensible report issued
Relevant, user friendly training materials in place
Participants complete training
On-the-job queries addressed
Activities
Assess training needs
Design training materials
Deliver training
Provide follow-up coaching
Inputs
Trainers, Materials, Equipment, Communications…

30. Ultimate Outcome Long term
…a picture of a preferred future; the reason why the program or project is important.
You can only make a contribution and have an indirect influence.
The change usually involves an even wider group of people
Is observable well after the program or project is complete.

31. Intermediate Outcomes
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
People/groups identified in environmental education projects engage with the subject matter in a way that helps them become active stewards of the environment
Intermediate Outcomes
Group leaders make evidence- based, results- focused planning and management decisions
Donors and community stakeholders recognize the ‘value-added’ by the group’s work
Immediate Outcomes
Stakeholder commitment to a training strategy
Trained educators apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned
Outputs
Defensible report issued
Relevant, user friendly training materials in place
Participants complete training
On-the-job queries addressed
Activities
Assess training needs
Design training materials
Deliver training
Provide follow-up coaching
Inputs
Trainers, Materials, Equipment, Communications…

32. A Progression of Results - Typical Terms
Ultimate Outcomes
Conditions - Social Economic
Civic Cultural Environmental
Intermediate Outcomes
Performance Improved - Lifestyle shifts
Institutional behaviour changes - policy & programming
Social mobilization
Immediate Outcomes
Capacities Developed - Institutional exchanges/collaborations
Applied knowledge Functioning spaces/infrastructure
System improvements/adaptations New/Improved leadership
Aligned Decision-making
Outputs
What we Produce - Networks Courses Trained participants Assessments Publications Infrastructure
Activities
What we Do - Research Build Train/Mentor Promote Procure Convene
Inputs
What we Need - Staff Equipment Supplies Office Space Services

33. “Organize a Provincial Climate Change Summit”
Logic Model Quiz
“Organize a Provincial Climate Change Summit”

34. Making a Logic Model - Some Tips
Vantage Point
Logic Models can be written from many different points of view -
it is important to be clear who is doing the activities - spending implementation funds
there is normally only one vantage point in a framework, it could be
a single entity, or
shared across two or more entities that are working in partnership

35. When building an logic model consider three variables…
Time
Outputs - Immediate
Immediate Outcomes - within time-frame following related activities
Intermediate Outcomes - by the end of the project or shortly afterward
Ultimate Outcomes - beyond the project
Control
Outputs - substantial control
Immediate Outcomes - still considerable control
Intermediate Outcomes - direct influence
Ultimate Outcome - indirect influence
Diffusion
Outputs - persons directly involved in the activity
Immediate Outcomes - same, possibly early interaction with additional people closely connected (e.g. users, colleagues, family members)
Intermediate Outcomes - larger teams, organizations, population segments
Ultimate Outcome - systems (communities, networks, societies)

36. Proportion…
Outputs, outcomes and impacts vary in their magnitude according to how big (in money terms) and how long the project is.
If you have a three week, $5,000 project, your outcomes should reflect what is reasonable to expect at three weeks and after $5,000 has been spent
If you have a three year, $500,000 project, your outcomes should reflect what is reasonable to expect at three years and after $500,000 has been spent

37. Scale - macro to micro
Outcome Measurement can be used at any scale - but which is best for you?

38. Writing Activity and Results Statements - Tips
Separating the Action from the Change
Activities - where you describe what you, as the implementing group, are DOING
Results (output and outcome - immediate, intermediate and ultimate) - where you describe what is DIFFERENT
For example…
Activity: Train 25 participants in Outcomes Measurement
Result: Participants apply Outcomes Measurement knowledge in when making project plans

39. - An impact statement for a governance related project
Avoid ‘run-on’ statements - for example:
“Rural population retained in rural areas THROUGH improving their living conditions BY MEANS OF enhancing rural income, better access to health, social services and infrastructure.”
An impact statement for a rural development project
“Improved political freedoms of local communities TO access, participate and CONTRIBUTE TOWARDS sustainable regional development.”
- An impact statement for a governance related project
Difficult to locate the actual change you seek

40. “increased awareness”
Do not lose sight of the people in your outcomes statements.
Sometimes we write statements without naming the people or groups who are experiencing the change. For example…
“increased awareness”
Try to include the subject in the statement at all times

41. Avoid tentative wording like: “able to”, “have increased capacity to…”, “contribute toward”. These are vague in meaning and very hard to measure.

42. Now your turn….. Describe your group project in a results logic model?
1 Ultimate Outcome
2 Intermediate Outcomes
3-4 Immediate Outcomes
4-6 Activities and Outputs

43. Sample Logic Model Worksheet
SITUATION (I.e. context, problem, identified needs, mandate, objectives)
Organization :
Name of project :
Duration :
How ?
What do we want ?
Why ?
Inputs
Activities
Outputs
Immediate outcomes
Intermediate outcomes
Ultimate outcome

44. Three Outcomes Measurement Tools
Results Logic Model
Risk Analysis and Management Table 
Monitoring and Evaluation Plan

45. Enablers and Constraints…
Are forces affecting any part of your project both positively and negatively

46. Risk Mitigation Required
Risk Analysis Matrix
Project Example…
Climate Change Conference
Immediate Outcome:
Conference participants deliberate across stakeholder lines on conference topics; they identify climate change strategies around which there is common agreement, as well as strategy areas requiring further discussion
1. What could hamper progress?
2. What is the likelihood of a problem?
3. What effect would the problem have on the Outcome?
Effects
Significant (3)
Moderate (2)
Minor (1)
Low (1)
Medium (2)
High (3)
Likelihood
Monitor Risks
Acceptable Risks
Risk Mitigation Required

47. Risk Mitigation & Monitoring
Risk mitigation strategies should reduce the likelihood of an undesirable event, or minimize the effect the event could have on the project.
Risk areas that remain above the level of acceptable risk should be monitored using risk indicators.

48. A Typical Risk Analysis and Management Table
Description of Risk
Likelihood of Occurrence
Effect on Planned Results (outputs/ outcomes/ impact)
Rank in Importance
1 = highest
Owner of the Risk
Risk Mitigation Strategy
Risk Indicators
Intermediate outcome level
Immediate Outcome level
Output level

49. Three Outcomes Measurement Tools
Logic Model
Risk Analysis and Management Table
Monitoring and Evaluation Plan  - “Measuring Success…” Appendices, pages

50. Monitoring and Evaluation - The Distinction
A continuous (or regular) self-assessment of progress -collecting data and comparing current performance with planned activities and outcomes.
Evaluation
Independent (often external), periodic, strategically focused assessment of a program/project’s continuing relevance, management, governance, results, coverage, external relationships, sustainability, etc.

51. Conventional vs Participatory Monitoring & Evaluation - Contrasts
Why
When
How
What
Who
External experts
Community members, project staff, facilitator(s)
Predetermined indicators of success
People identify their own indicators of success
Focus on scientific objectivity, evaluator distanced from other participants; delayed and limited access to results
Self-evaluation, simple methods adapted to local conditions; open, immediate sharing of results through local involvement
Usually Mid and End of Project
More frequent, small scale evaluations
Accountability - to address donor requirements
Empowerment - to help people initiate, navigate, control
Adapted from Narayan-Parker, 1993: 12 (taken from Estrella and Gaventa, 1998)

52. Different kinds of Evaluation
Formative - mid stream, where inquiry focused on areas where the project can improve
Summative - at the end, where inquiry focused on end results and lessons for future practice
What is your experience with these two kinds of evaluation?

53. Indicators…
Evidence that the program is producing expected results at the output and outcome levels.
They can be expressed quantitatively and qualitatively.
Information collected should help managers make wise decisions.

54. What indicators tell us about the wind…
Who has seen the wind?
Neither you nor I.
But where the trees bow down their heads,
The wind is passing by.
-The Wind, Christina Rosetti,

55. Quantitative Indicators
Expressed as a percentage, ratio, proportion, or number that can be analyzed statistically
Examples?
Number of conferences/public engagement activities held
Ratio of males to females attending public meetings
Number and type of research grants distributed
Proportion of targeted schools making requests for assistance with educational resources

56. Quantitative Indicators - Your Turn
Climate Change Conference Project
Immediate Outcome
Major media outlets cover the Climate Change Summit with ample in-depth news analysis pieces ?

57. Qualitative Indicators
Expressed as a change or comparison between two states of understanding or experience that can be analyzed for patterns
Relies on people’s judgment or perceptions
Examples:
Trends in the way community newspapers cover citizen actions to reduce environmental impacts
Before-after comparison in the way the largest five land developers are handling wetland areas zoned for development

58. Qualitative Indicators - Your Turn
Climate Change Conference Project
Immediate Outcome
Major media outlets cover the Climate Change Summit with ample in-depth news analysis pieces ?

59. Qualitative Indicators - Setting out Markers of Progress
Sometimes you cannot meaningfully track progress by counting “units” at the beginning (baseline), middle and end
Indicators are not always as easy as “number of people trained”…
You may need to set out…
Stages of completion, or
Markers of progress

60. Qualitative Indicators - Setting out Markers of Progress
Consider “policy formulation”…
Indicator is ‘Progress in updating ‘x’ policy’
You could say…
Baseline - ‘0’ updated policy
Target - ‘1’ updated policy
But how useful is that from a management point of view?

61. Qualitative Indicators - Setting out Markers of Progress
Would be more useful to identify progress markers. For example:
Agreed need for updated policy
Policy research underway
Policy options formulated
Discussion and approvals underway
Updated policy approved
Updated policy enacted
Baseline might be ‘a’
Yr 1 target might be ‘c’
Yr 3 target might ‘e’

62. Indicator Checklist
Climate Change Conference -
Key stakeholder groups commit financial and human resources to implement the Action Plan
Possible Indicators - How well do they score?
Trends in the level of financial commitment, by action plan strategy, by source
Number and type of organizations publicly endorsing the Climate Change Action Plan
Number and type of initiatives underway in the Province, by action plan strategy
Validity - Does it measure the result?
Yes No
Is it cost-effective to collect the information?
Does it help understand how the program or project is affecting men and women, and/or specific sub-groups of people differently?
Does it provide useful information for management decisions?
Does it communicate well to stakeholders (public, policy makers, donors)?

63. Indicators, Baseline, and Target
Meaning
The information to be collected
What the indicator would tell you at the beginning
What you want the indicator to tell you at a specific point in the future
Example
The number of visits to the organization’s new website
720
2,000

64. Sources of Information
Data Sources are the people or documents that provide data, not the method of collecting data
Sources of performance information include: people, individual beneficiaries, groups of beneficiaries, organizations, partners, and documents, etc.
To ensure reliability, try not to change data sources over time.

65. Methods of Gathering Data
See “Measuring Success…” Appendix I, Pros and Cons, Pages ; Appendix IV, Samples ,Pages

66. Collecting the Data
Conventional Methods - surveys, focus groups, observation, document reviews, counts, etc.
REFER TO: Table 7 (pp 47-9)

67. Collecting the Data
Participatory Reflection and Action Methods - modeling/mapping, ranking exercises, calendars, walks, historical profiles

68. Frequency of Collecting Data
For each indicator, determine how often data will be collected.
Indicators for outputs and immediate outcomes are more likely to be collected on a quarterly or six-monthly time period, for intermediate outcomes - less frequently

69. Responsibility - who does it?
Spell out clearly who will do the collection and analysis of the data for each indicator
Ensure local capacity is built and is responsible to ensure sustainability.
Decentralize responsibility as much as possible.

70. Report Frequency& Resp.
A Typical Monitoring and Evaluation Plan - Completed example on page 30 “Measuring Success…”
Results Levels
Measurement
Indicators
Base-Line
Target
Data Source/ Method
Report Frequency& Resp.
Intermediate Outcome
Immediate
Outcome
Output

71. Now your turn….. Take one outcome statement, brainstorm indicators
Use the indicator checklist to refine your best indicators
Identify the source, means of data gathering (a.k.a. ‘instrument’), and frequency
Repeat, if time permits
Share with a neighbour