1. What Difference do We Want to Make; Developing a Strategic Framework and Measurement Strategy for Sport Manitoba’s PSO Funding Unit June 5, 2008 Jennifer Birch-Jones, Senior Associate Performance Management Network Inc. Tel: (613) 236-2320 Fax: (613) 236-8644 Email: jennifer.birch-jones@pmn.net

2. 2 Today’s Objectives Increased awareness and understanding of: performance based planning, measurement and evaluation what difference the PSO Funding Unit is trying to make the interplay btw the PSO Funding Unit’s outcomes and the outcomes of other Sport Manitoba programs measurement considerations evaluation considerations learning considerations, including ways in which Sport Manitoba can help build capacity internally and in PSOs Consensus on a strategic framework / logic model Consensus on Next Steps

3. 3 Today’s Approach Learn from yesterday evening’s session Confirm key concepts and terms Build on the good work that already has been done Opportunity to discuss / explore issues as they relate to success Interactive – we will build the strategic framework together Ask lots of questions, especially if any of the terms / jargon not clear “Why” questions not meant to challenge, but to help get to an outcome / key result Parking Lot Logistics

4. 4 Strategic Analysis Where are we now? Monitoring & Evaluation Are we getting there? Strategic Framework Where do you want to be? Implementation How do we get there? Strategic Management Cycle Source: Kent and Wilkinson, Applied Strategic Planning, 1991.

5. 5 Key Terms and Concepts Performance-based planning combines performance measurement and evaluation concepts and approaches with strategic planning Move away from a focus on how busy we are (activity-based) to what difference we are making (outcome-based) Uses an integrated approach to planning, measurement and reporting Can strengthen the planning process by Clearly articulating and agreeing on what success looks like and how to measure and report on progress Focusing scarce resources on measuring progress for only the most important aspects Adapting the level of detail to meet the information needs of the users, e.g., Board, Minister, CEO, Director, etc.

6. 6 Key Terms and Concepts Managing for results entails: Knowing what difference you are trying to make (strategic framework and targets) Knowing what progress / success you are having (monitoring and evaluation) Finding out why something is not working (monitoring and evaluation) Making adjustments in your work to improve performance (continuous improvement / learning)

7. 7 Overview of Process 1. Develop strategic framework 2. Develop measurement / ongoing monitoring strategy Priority performance areas for monitoring Measurement strategy – indicators, data sources, data collection methods, timing, responsibility, resources, whether baseline data is needed 3. Develop evaluation framework Type of evaluation Evaluation questions Timing of evaluation Evaluation strategy - indicators, data sources, data collection methods, timing, responsibility, resources, whether baseline data is needed

8. 8 Overview of Process 4. Develop learning strategy – ways in which PSOs can better use and learn from monitoring and evaluation – their own and Sport Manitoba’s 5. Develop / refine existing processes and tools 6. Implement, learn, refine and share Engage PSOs at every opportunity! Logic model / strategic framework is the basis of both monitoring and evaluation You can adapt monitoring and evaluation to meet your needs (not vice versa)!

9. 9 Key Terms and Concepts A Strategic Framework includes: Inputs  Activities  Outputs  Goals / Outcomes Goals / outcomes can be Short-Term  Medium-Term  and Long-Term Goals / Outcomes Also known as a results chain, logic model, performance framework diagram showing the links from the activities through the sequence of outcomes to the final outcomes (or results commitments)

10. 10 Key Terms and Concepts HOW? WHO? WHERE? WHAT do we want? WHY? inputsactivitiesoutputs direct and intermediate outcomes final outcomes “Operational”“Behavioural Change”“State” users / clients / co-deliverers / beneficiaries

11. 11 REACH Primary targets or clients Partners or Co-deliverers Stakeholders Reach is defined as the group, or groups, which are reached by program / service outputs. The reach category can be segmented into different groups, e.g., SHORT- AND MEDIUM-TERM OUTCOMES LONG-TERM OUTCOMES (State level) OUTPUTS and ACTIVITIES Action / AdoptionAbility / Competence (Capacity Building) Awareness / Perception / UnderstandingActive Partner Support (Commitment) Long-term outcomes should relate to the mission, mandate and key results of the program/service provider. Direct Outcomes are the initial effect or consequence of an output. Direct outcomes include early behavioural changes in the target group or clients, reflected in changes in awareness, client satisfaction, acceptance and support. These lead to intermediate outcomes, such as new knowledge, improved capability, improved behaviour (compliance) and decision making. (Operational level) (Behavioural Change level) 11 Strategic Framework

12. 12 Key Terms and Concepts Characteristics of a Strategic Framework Sequence of variable length extending from inputs to final outcomes (results commitments) Outputs mark transition between internal and external results (outcomes) Partnerships externalize delivery process and lengthen sequence of outcomes What is achievable is directly related to: Resources Timeframe

13. 13 HOW? (Operational) Your operational environment You have direct control over the behaviors within this sphere WHAT do we want by WHOM? (Behavioral Change) Your environment of direct influence e.g., people, groups and organizations in direct contact with your operations WHY? (State) Your environment of indirect influence e.g., individuals and/or communities of interest where you do not make direct contact Performance needs to be considered in terms of its differing spheres of influence. Actions in the operational sphere should directly lead to changes in targeted groups which should in turn affect the desired ‘state’. Sources: Van Der Heijden (1996), Montague (2000) Spheres of Influence End Outcomes Immediate Outcomes Intermediate Outcomes Outputs

14. 14 Strategic Framework WHY HOW WHAT BY WHOM REACH

15. Office of the Auditor General – Basic Results Chain END OUTCOMES Office contributes to better managed government programs and better accountability to Parliament and the public. INTERMEDIATE OUTCOMES Audits and studies are relevant. Critical issued and problems are addressed by audits and agendas are changed. Audits result in better informed legislature, government, entities, and public. IMMEDIATE OUTCOMES Office support for its role and its work maintained. Clients and stakeholders engaged in audit process. OUTPUTS AND OPERATIONAL PROCESSES Source: Office of The Auditor General, pg. 12, Departmental Performance Report 2001 15

16. 16 Determine the priority / important performance areas for monitoring and why Identify what should be the key indicators for these: What would you see or hear if the expected results / outcomes you have described are being achieved? Determine how, how often and who will collect this information Determine the means by which to implement Measurement Considerations

17. 17 Measurement Considerations Sample Measurement Strategy Perfor- mance Area / Outcome Perf. Indicator Data Source Collection Method ResponsibilityTiming Increased awareness of the Health Check (HC) program by consumers The percentage change in the no. of consumers aware of the HC program btw 2008 and 2011 Canadian consumers Consumer survey HC Program staffBaseline in April 2008, Follow-up in March 2011

18. 18 Measurement Considerations  Performance indicators (or measures) are the specific items of information that track a program’s success.  PIs describe observable, measurable characteristics or changes that represent achievement of an outcome, e.g., number / percentage of Canadian consumers who are aware of the Health Check program

19. 19 Results can be operationally defined as follows: Such As: # Proportion / Percentage Rate Ratio Amount ‘Level ’ Population Target Group Program Participant Client Individual Organization Agency Community Awareness Attitude Perception / Satisfaction Knowledge Skill Behaviour / Practice Example: The percentage of clientsWho express positive satisfaction with the services / advice rendered The Measure of Change By WhomIn What A Basic Template for Defining Indicators

20. 20 Evaluation Considerations Over and above the PSO Funding Units ongoing monitoring, is there a need for periodic evaluation? If so, should the focus be: Formative? Summative? Issue-based? Some combination of the above? What would you like to learn most from an evaluation of the PSO Funding program?

21. 21 Evaluation Considerations

22. 22 Evaluation Considerations  Formative evaluations: Are associated primarily with analysis of program implementation and early results / progress Provide feedback so that implementation can be improved and barriers to improved performance can be identified and removed  Summative evaluations: Deal with whether the program has achieved its’ intended outcomes. Focuses on; Relevance / ongoing need Success Cost-effectiveness and Alternatives

23. 23 Evaluation Considerations Sample Evaluation Strategy Evaluation Question Perf. Indicator Data Source Collection Method ResponsibilityTiming How successful has PSO funding been in providing a stable environment within which to deliver sport in Manitoba? Fluctuations in annual funding amounts (overall budget and to PSOs) Extent PSOs view funding as providing a stable environment PSO Funding Unit Files PSOs File Review Interviews (Sample of 20) PSO Funding unit staff / Evaluator Evaluator Summative evaluation in 2012 Summative Evaluation in 2012

24. Evaluation Vs Monitoring Resources are usually a part of the program Targeted resources are needed for each program evaluation Attribution is generally assumedAttribution of observed outcomes is usually a key question Measures are developed & data are usually gathered through routinized process Measures are usually customized for each program evaluation Broad issuesIssue Specific OngoingEpisodic MonitoringProgram Evaluation Program managers play a key role in developing and using performance measurement / monitoring system Program evaluators are not program managers Source: Adapted from McDavid & Hawthorn, 2006.

25. 25 Learning Considerations Considering how the PSO Funding Unit plans to use monitoring and evaluation: What are the capacity implications for PSOs? Are we being realistic? What can we do to make it easier for PSOs to meet our PSO Funding Unit accountability requirements? What can we do to help PSOs being more intentional about using evaluation as a tool for learning? How can we use / share the learnings with our PSOs? Learnings about how to do monitoring and / or evaluation effectively? Learnings from the results of the information gathered and analysed from our monitoring and evaluation?

26. Logic Model Worksheet Def’ns adapted from the Treasury Board Secretariat Results-based Management Lexicon. 26

27. 27 Building the Strategic Framework What is the time frame for the Strategic Framework? What level of resources are allocated for the PSO Funding Unit? Existing / New? O&M? / Other? Staff / FTEs? What is the reach of the PSO Funding Unit? 27

28. 28 Building the Strategic Framework Over the longer-term, what is the “why” of the PSO Funding Unit? What are the key activity areas / outputs? For each of the key activity areas, what are the immediate outcomes? What are the intermediate outcomes? 28

29. 29 Building the Strategic Framework Logic check - Is the logic of the strategic framework clear and robust: Any logic leaps? Any key gaps? Any of the “reach” missing? 29

30. 30 Next Steps A good performance-based planning and reporting system takes time, patience and persistence. Strategic discussions along the way are as important as the end product. Next Steps and Timing

31. Additional Definitions and Examples

32. 32 Key Definitions and Examples Activities: An operation or work process internal to an organisation, intended to produce specific outputs (e.g. products or services). Activities are the primary link in the chain through which outcomes are achieved. Examples: Development of … Preparation of … Coordination of … Identification of …

33. 33 Key Definitions and Examples Outputs: Direct products or services stemming from the activities of a policy, program, or initiative, and delivered to a target group or population. Usually things you can count. Examples: Plans for … Tools for … Workshops / training sessions / etc. Websites Documents Etc.

34. 34 Key Definitions and Examples Reach: The individuals and organisations targeted and directly affected by a policy, program or initiative. Examples: Farm organizations Producers Landowners Etc.

35. 35 Key Definitions and Examples Outcomes: An external consequence attributed to an organisation, policy, program or initiative that is considered significant in relation to its commitments. Outcomes may be described as: immediate, intermediate or final, direct or indirect, intended or unintended. Examples of immediate outcomes: Better understanding of … Easy access to more information on … Change in attitudes regarding... Enhanced participation of … in … Increased capability of … Opportunities for employment for … in... Etc.

36. 36 Key Definitions and Examples Examples of intermediate outcomes: Continued employment (after end of project) Use of information / knowledge gained / etc. Improved decision-making Sustained change in (behaviour or practice) Etc. Examples of final outcomes: Improved socio-economic conditions Increase in exports Improved environment / health Etc.