Presentation on theme: "Performance measurement: Finding our way from outputs to outcomes."— Presentation transcript:
Performance measurement: Finding our way from outputs to outcomes
Finding our way from outputs to outcomes How do we know we are having an impact on children, youth and their families? How can we come together to ensure that the services we offer are appropriate and effective?
Outline Overview Discussion: Knowing our audience Example: Using CAFAS to track program and organizational outcomes Discussion: Building capacity to do performance measurement
Overview: Performance measurement WHAT is it? WHY is it necessary? WHO benefits? HOW is it done?
What is performance measurement? The regular collection of information for monitoring how a policy, program or initiative is doing at any point in time. It can be used to report on the level of attainment of planned results and on performance trends over time. - Treasury Board Secretariat of Canada
Why is it necessary? Serves as a descriptive tool on how a project, policy or program is doing Serves as an early warning if the direction of a program, policy or project is not going as planned
Who is the audience? End users: clients, families caregivers Service providers, educators, program staff Organization or network Health system Public at large
How is it different from evaluation? PM: provides regular snapshots of how a program or policy is doing; focuses on what the outcome is Client satisfaction surveys are often a key indicator of PM Evaluation: can provide insight into how and why an outcome is occurring Client satisfaction is part of the process evaluation and can influence outcomes
Performance measurement and evaluation SITUATION ACTIVITIES OUTCOMES INPUTS OUTPUTS Resources of a program Quantity of work, products or participants Change in target audience
How is it done? Program logic models Balanced score cards Strategy maps
Balanced scorecard approach on health promotion Source: ICES, 2004
Improve healthy behaviours, health promotion and disease prevention Improve linkages and transitions between addiction, mental health, health, education, social and justice systems Improve client focus of addiction services Improve access to appropriate addiction treatment Increase productive use & appropriate allocation of resources across system Ensure the continuum of interventions includes prevention, health promotion, early intervention, harm reduction and treatment services. Reducing Risk through influencing the broader determinants of health Improve health outcomes at the individual and population level Increase sustainability and equity of the addiction & health systems Increase availability and retention of the qualified human resources Ensure evidence informed practices are developed, implemented and maintained across province Further develop & increase equitable resources and capacity Ensure quality assurance within the addiction system Addiction System Strategy Map Ontario Federation of Community Mental Health & Addictions Programs Increase linkages, transition & integration within addiction services
In small groups Divide into 5 groups In each group, discuss What types of information your group/constituency are interested in (focus on two or three) How frequently you want the information How you want the information presented What concerns, issues or challenges exist on these types of information What solutions, ideas or strengths we can build on
Challenges and opportunities Selecting measures Valid and reliable Relevant, feasible, sensitive to changes Developmentally and culturally appropriate Measures database under development Searchable database on measures used for evaluating CYMH programs
Challenges and opportunities Information management capacity within agencies and in the government Collaboration and staff buy-in Organizational culture that fosters learning
Example: Use of CAFAS The Child and Adolescent Functional Assessment Scale (Hodges, 2002) measures outcomes among 6-17 year old children and youth receiving mental health services in Ontario CAFAS identifies level of functioning in 7 subscales: Home, School, Community, Behavior towards others, Moods and emotions, Substance use, Thinking Led by Dr. Melanie Barwick, Hospital for Sick Kids
22 (c) Barwick Reliabi lity Trainin g Softwar e Training Train- the- Trainer Booste rs Guideli nes for Specia l Pops Tech Suppo rt (IT) Site Visits Comm unities of Practic e Data Report s Data Audits Websit e Wiki Implementation Supports Source: Barwick, 2009
Severe Impairment on CAFAS Subscales at Entry to Treatment – years 2005 to 2008 (c) Barwick Source: Barwick, 2009
Severity at Entry to Treatment for Ontario and Regions (2005: N=9,065; N= 2006: N=18,255; 2007: N=23,566; 2008: N=26,974) (c) Barwick 0-30 Some need for service 40-70 Outpatient needs 80-100 Outpatient plus extra supports 2005200620072008200520062007 2008 2005200620072008 22.022.822.9 22.6 33.936.938.2 38.3 19.420.320.2 CE20.119.720.821.634.636.938.240.021.62220.127.116.11 CW19.619.220.418.735.936.738.136.821.821.120.421.9 E25.022.220.922.232.938.439.940.118.921.122.021.4 HN19.820.120.621.036.037.337.939.620.221.921.620.8 NE33.229.429.729.339.441.441.841.418.104.22.1688.2 N26.625.024.223.731.137.940.541.118.819.218.819.0 SE22.214.171.124.337.739.140.837.814.3 14.616.5 SW19.821.320.318.330.534.434.833.819.121.422.421.5 TO13.819.723.521.731.233.936.5 21.120.319.419.9 Source: Barwick, 2009
Using CAFAS data for a local agency Pull key variables & performance indicators from the CAFAS Quarterly Report Enable comparison over time Present data in a simple spreadsheet Append charts to aid interpretation Source: Roger Rolfe, 2009. Utilizing CAFAS exports and reports at CTYS.
Solution: Agency CAFAS Performance since 2006.xls Source: Rolfe, 2009.
In small groups What practices or strategies are successful in your organization/community: For getting buy-in from all staff and/or senior management? For getting consistent data and reporting? For doing evaluation?
Summary Performance measurement is a useful tool for measuring improvements in mental health outcomes for children and youth The usefulness of performance measures vary by audience needs and quality of the information Strategies that enhance organizational learning are needed to build our capacity to do performance measurement
Questions? Evangeline Danseco, PhD Director, Evaluation and Research 613.737.7600 Ext 3319 firstname.lastname@example.org Susan Kasprzak, MA Research Associate 613.737.7600 Ext 3320 email@example.com www.onthepoint.ca