1. From Outcomes to Impact How to Create Change in Family Violence Dr. Robbie Babins-Wagner CEO, Calgary Counselling Centre Adjunct Professor, Faculty of Social Work, University of Calgary

2. For at least the past 10 years funders of non- profit organizations have required outcome data as a measure of accountability. Agencies are increasingly reporting on program outcomes on an annual basis. FCSAA Power of Prevention Conference, November 29. 2013

3. The outcome agenda has brought greater accountability to the sector but there exists a gap in moving from outcomes to impact How do we improve the overall outcomes for the clients that we serve. FCSAA Power of Prevention Conference, November 29. 2013

4. My interest outcome and evaluation began almost 30 years ago when I worked in health care and extended to my work at Calgary Counselling Centre. It lead to my completing my doctorate in outcomes and now is a prime focus of my work at CCC and the collaborative work I am doing with colleagues through the Center for Clinical Excellence in Chicago FCSAA Power of Prevention Conference, November 29. 2013

5. It is also a focus of my work through the United Way of Calgary’s Leading Boldly Network The Network is focused on – Building new capacity in Calgary for Collaborative Social Innovation (CSI), and – Making progress on complex problems that are beyond the mandate of any individual organization. FCSAA Power of Prevention Conference, November 29. 2013

6. Research suggests that most agencies have some outcome data But even with having the data, we have not been successful in achieving significantly better outcomes at a magnitude that matches the need in critical areas such as mental health, child outcomes, school completion and more. FCSAA Power of Prevention Conference, November 29. 2013

7. Its suggested that – we have not marshalled the full extent of available knowledge and applied it to complex problems… – which would in turn generate new knowledge to address some of our toughest social problems. FCSAA Power of Prevention Conference, November 29. 2013

8. Frustrations 1.On the part of non-profit leaders who have not been able to find support (financial and non-financial) from funders for collecting and using information to manage to outcomes. 2.On the part of funders who have become frustrated with non-profits who have no means of determining whether they re doing what they say they do. FCSAA Power of Prevention Conference, November 29. 2013

9. The research is clear, it take a lot on the part of both agency leaders and funders to support the changes being required to solve complex problems. Having a strong outcome platform is the first step for agencies. FCSAA Power of Prevention Conference, November 29. 2013

10. The focus on having a strong outcome platform requires a difficult cultural transition from simply having a genuine interest in improvement to truly infusing outcome thinking into the way we manage our organizations FCSAA Power of Prevention Conference, November 29. 2013

11. Some funders suggest that to do this properly, requires a full 10% of their available funds for direct investment (on their parts) while at the same time supporting agencies to alter their culture and develop the human and IT systems necessary to manage to outcomes (Morino, 2010). Where this was initially tested, only 5 of 12 (42%) agencies achieved this kind of transformative change FCSAA Power of Prevention Conference, November 29. 2013

12. There are funders (in the US) who are investing in non-profit leaders who embrace the value of great information, even it they haven't had the external support to build the systems they need for collecting and using information. FCSAA Power of Prevention Conference, November 29. 2013

13. Funders are increasingly focused on outcomes that lead to larger impact: They are: – asking grant applicants to demonstrate a predisposition for using information to guide operations – providing longer term funding, with clear expectations that it will take at least 2-3 years of intensive work to create a true outcome focused culture – learning to be more flexible in how to support this work FCSAA Power of Prevention Conference, November 29. 2013

14. Funders have also learned: – They cant impose this type of change – They need to give the agencies the time and space to do it their own way, if agencies choose to do it at all – The results aren’t going to sustain if the agencies don’t own the process and resulting systems – This work isn’t for everyone. FCSAA Power of Prevention Conference, November 29. 2013

15. And when non-profit agencies elect to do this work they: – do a better job of meeting their missions – become more focused and disciplined in managing their organizations – Are enabled towards a path toward managing to outcomes – strengthen internal culture of the agency FCSAA Power of Prevention Conference, November 29. 2013

16. Moving from outcomes to impact will bring forth the following results: staff will develop better skills in their core capacity (i.e. counselling) program managers will develop better people management skills and have the tools they need board members will have the information to provide more effective oversight All of which lead to better results for clients FCSAA Power of Prevention Conference, November 29. 2013

17. Steps to get the process started: 1.Determine who you will get feedback from clients (service users) 2.Learning must be the primary goal of the phase one where you – Collect information about the problem – Collect information about possible solutions 3.Apply rigor within reason 4.Be practical 5.Create a learning culture FCSAA Power of Prevention Conference, November 29. 2013

18. The Problem Despite all the right intentions many non-profits do not have the benefit of; – good information – tools to determine where they are headed – mapping a logical course towards an objective, and – course-correct when required. FCSAA Power of Prevention Conference, November 29. 2013

19. – While these are not new problems, current economic conditions locally, provincially federally and globally suggest that we are likely to see an increased focus on results to support decreased funds for social and human services – The research suggests that organizations which demonstrate meaningful, lasting impact will be the focus of increased investment over the next decade. FCSAA Power of Prevention Conference, November 29. 2013

20. There is also the suggestion that individual organizations who can demonstrate impact will be brought together with other like minded organizations to work together in disciplined ways toward collective impact (Kania & Kramer, 2011). An additional focus will be a focus on working across silo’s - within organizations, across organizations in the community and within government FCSAA Power of Prevention Conference, November 29. 2013

21. As non-profits focus on improved client benefit, we need to: – clarify the results (outcomes) we are trying to achieve – collect the information that can best help us navigate towards these outcomes – differentiate between operational performance (overhead costs) and organizational effectiveness (results or outcomes) – ensure we have the culture to support our work FCSAA Power of Prevention Conference, November 29. 2013

22. In Sum Many non-profits have no reliable way to know whether they re on track to deliver on their promise We need to invest in continuous collection and use of information (data) to guide program/agency decisions and operations This change requires a significant culture shift within an organization – it is primarily about culture and people (not numbers) FCSAA Power of Prevention Conference, November 29. 2013

23. In Sum (cont.) We must focus on why measure and on what to measure – not just on how to measure The non-profit needs to drive the change and be the primary beneficiary of it Reasonableness and common sense must guide both agency and funder investment decisions FCSAA Power of Prevention Conference, November 29. 2013

24. Tris Lumley, head of strategy for New Philanthropy Capital (London) suggests that “Great organizations …. are build around great data. Data that [allow] then to understand the needs they address, what activities are likely to best address these needs, what actually happens as a result of these activities, and how to allocate resources and tweak what they do for even greater impact. Too often, funders set the agenda with their own requirements [and]cripple the organizations they are trying to help. FCSAA Power of Prevention Conference, November 29. 2013

25. So how do we do this? How do we manage to outcomes? FCSAA Power of Prevention Conference, November 29. 2013

26. 5 Steps Develop clarity through reflection and analysis on what change you are trying to create Gain specificity on how you will accomplish this change Determine what information (hard and soft) will be most helpful for gauging whether you are on or off track to achieve that change Collect and use this information to plan, make important decisions, track, course-correct and improve Combine the steps above with good practice judgement and keen discernment, which are more important than any single metric. FCSAA Power of Prevention Conference, November 29. 2013

27. Of most importance Gain clarity on what change you are trying to achieve Develop specificity on how you will accomplish this change Determine what information will be most helpful for gauging whether you are on or off course Collect and use this information as the basis for understanding what’s working, planning, decision making and improvement Lastly, the technology behind the systems is imporntat and necessary but is not nearly as important as the mindset of the leaders who put these systems in place FCSAA Power of Prevention Conference, November 29. 2013

28. Organizational Culture Recruit culture leaders Walk the talk Know what you stand for (core beliefs & guiding principles) Be able to answer the question “To what end” – improved client results → community impact Ensure that everyone is moving in the same direction Be clear and direct about expectations Encourage self-improvement and personal growth FCSAA Power of Prevention Conference, November 29. 2013

29. A Case Example FCSAA Power of Prevention Conference, November 29. 2013

30. The OQ 45 a 45-item self-report measure takes about five minutes to complete grade 6 reading level provides a total score, based on all 45 items, as well as three subscales: symptoms of distress, interpersonal relationships and social role functioning scores range from 0 – 180, with a clinical cut-off of 63 FCSAA Power of Prevention Conference, November 29. 2013

32. Clinical Significance Criteria for clinically significant change (Jacobson & Truax, 1991) are used to make judgements of client outcomes Clients who change in a positive or negative direction by at least 14 points are regarded as having made “reliable change” The clinical cut-off of 63 differentiates clients in the dysfunctional range from those of a functional population Very high clinical standard to meet (very unlikely to happen by chance) Most clinicians are satisfied with a smaller point change in outcome suggesting positive or negative movement. FCSAA Power of Prevention Conference, November 29. 2013

33. Four Categories of Change for the OQ CategoryDefinition No ChangeAny score that sows neither reliable (14 point) improvement or deterioration. DeterioratedThe client’s score is at least 4 points worse than the first OQ ( reliable negative change) ImprovedThe client’s final OQ score is at least 14 points less than their first session OQ Score RecoveredThe client’s final OQ score is at least 14 points less than their first session score and the client’s final OQ score is at or below the clinical cut-off of 63. FCSAA Power of Prevention Conference, November 29. 2013

34. N Deteriorated % No Change % Improved % Recovered % Total Improved & Recovered Baseline 60728.25620.914.135 CCC 2009 20521477.2102149.847123.041320.143.1 CCC 2010 22841737.6108547.551922.750922.345 CCC 2011 24691606.5115246.761224.854522.146.9 CCC 2012 22881486.5102344.759726.152022.748.8 Total 1668012377.4790947.4377122.6339020.342.9 FCSAA Power of Prevention Conference, November 29. 2013

35. 2004 - 2012Average OQ at First Session Average OQ at Last Session Average Change Full Time n=2759 68.74 S=25.13 59.42 S= 26.61 -12.62 S=20.76 Associate n=1528 69.73 S=25.96 59.94 S=27.45 -12.62 S=20.63 Resident N=5377 75.47 S=25.69 64.99 S=27.70 -13.25 S=21.73 Intern N=19153 75.95 S=26.46 65.51 S=28.66 -13.41 S=21.46 Total N=18,817 74.07 S=26.14 63.81 S=28.04 -13.32 S=21.35 FCSAA Power of Prevention Conference, November 29. 2013

36. Why Family Violence Everyone is vulnerable to family violence It is a significant social problem (Johnson, 2006) The financial cost to Canadian Society – In 2009, the total economic impact of spousal violence was estimated at $7.4 Billion (Zhang, Hoddenbagh, McDonald& Scrim, 2012) FCSAA Power of Prevention Conference, November 29. 2013

37. The 2013 Stats Can report (based on 2009 data). Only 22% of victims reported violence to the police In Canada, in 2008, 334,000 individuals were victims of self reported violence at the hands of an intimate partner – 6% of the adult population Alberta has the second highest rate of self reported abuse, followed by Saskatchewan. The consequences are significant for victims & children as well as the abuser FCSAA Power of Prevention Conference, November 29. 2013

38. Demographic Predictors: Younger Living in a common-law relationship Living in a step family Income levels and education levels were not predictors of family violence FCSAA Power of Prevention Conference, November 29. 2013

39. CCC Male Family Violence Clients (2008 – 2012) On average 35 years old, range of 18 – 76 Median income: $25,000-35,000 Marital status: Single (41.1%); Married/Common-Law (38.9%); Separated/Divorced (19.9%) FCSAA Power of Prevention Conference, November 29. 2013

40. Calgary Counselling Centre’s Responsible Choices for Men Program  The primary goal is to assist men to become violent free  Major objectives include: o decreasing all forms of abusive behaviour, o accepting responsibility for one’s behavior, o increasing self-esteem, o improving family relations, o decreasing stress, o increasing empathy towards the victim, and o stopping abuse towards children FCSAA Power of Prevention Conference, November 29. 2013

41. Program Elements: Counselling to engage in process – Specialized 14 week (30 hour) Group – Counselling for any outstanding issues Outcome measures OQ data is collected on a session by session basis during counselling, at the beginning of group, at the end of group and at the end of counselling FCSAA Power of Prevention Conference, November 29. 2013

42. Responsible Choice for Men Outcomes (2008-2012): 1545 clients (individual + group) Of these clients, 607 went on to participate in a specialized family violence group whereas 938 proceeded with counselling Average number of counselling sessions for those who participated in counselling and group = 6 – The range of counselling sessions for those who did not participate in group was 1-41. – The range of counselling sessions for those who participated in group was 1-23. FCSAA Power of Prevention Conference, November 29. 2013

43. RCM Counselling only Results 2008 - 2012 Average First Session OQ Average Last Session OQ Change Score Results n=60653.441.7-11.7 t(602) = 13.31, p <.001* Range of OQ scores 2-1441-153 FCSAA Power of Prevention Conference, November 29. 2013

44. Combined Programming RCM N=299 Unadjusted Score N=292 Adjusted score Change Score Unadjusted Change Score Adjusted Results – Unadjusted scores Results – Adjusted scores First Counselling Session 45.285.7 First Session of Group 33.177.9-11.78 Last Session of Group 28.671.4-4.5-6.5 From first session of counselling to last session of group ---------------------16.6-14.3 t(299) = 12.79, p <.001* t(291)=10.6, p<0.001* FCSAA Power of Prevention Conference, November 29. 2013

45. Combined Programming N=299Change Score Results (Unadjusted) First Counselling Session 45.2 First Session of Group 33.1-12.1 Last Session of Group 28.6-4.5 From first session of counselling to last session of group -16.6 t(299) = 12.79, p <.001* FCSAA Power of Prevention Conference, November 29. 2013

46. Combined Programming N=292Change Score Results – adjusted for social desirability First Counselling Session 85.7 First Session of Group 77.9-7.8 Last Session of Group 71.4-6.5 From first session of counselling to last session of group -14.3 t(291) = 10.63 p <.001* FCSAA Power of Prevention Conference, November 29. 2013

47. Category Unadjusted Individual Counselling - RCM CCC CasesUS Baseline Study Deteriorated43 (7%)648 (7.6%)8.2% No Change299 (50%)4055 (47.4%) 56% Improved183(30%)1933 (22.6%) 20.9% Recovered78(13%)1915 (22.4%) 14.1% Total Improved or Recovered 261(43)3848 (45%) 35% Total6378,5516,072 FCSAA Power of Prevention Conference, November 29. 2013

48. CategoryFrom first RCM Counselling session to first group session CCC CasesUS Baseline Study Deteriorated31 (5.7%)648 (7.6%)8.2% No Change305 (56.1%)4055 (47.4%)56% Improved168 (30.9%)1933 (22.6%)20.9% Recovered40 (7.4%)1915 (22.4%)14.1% Total Improved or Recovered 208 (38.8)3848 (45%) 35% Total5448,5516,072 FCSAA Power of Prevention Conference, November 29. 2013

49. Category Unadjusted From first Family Violence counselling session to last group session CCC CasesUS Baseline Study Deteriorated11 (7.2.%)648 (7.6%)8.2% No Change62 (40.8%)4055 (47.4%) 56% Improved58 (38.2%)1933 (22.6%) 20.9% Recovered21 (13.8%)1915 (22.4%) 14.1% Improved or Recovered 79 ( 52%)45%35% Total1528,5516,072 FCSAA Power of Prevention Conference, November 29. 2013

50. Category Adjusted for social desirability From first Family Violence Counselling session to last group session CCC CasesUS Baseline Study Deteriorated11 (7.2%)648 (7.6%)8.2% No Change62 (40.8%)4055 (47.4%) 56% Improved58 (38.2%)1933 (22.6%) 20.9% Recovered21 (13.8%)1915 (22.4%) 14.1% Improved or Recovered 79(52%)45%35% Total1528,5516,072 FCSAA Power of Prevention Conference, November 29. 2013

51. Combined Programming to counselling follow-up N=152 (unadjusted) Change Score Results First Counselling Session 45.3 First Session of Group33.1-12.2 Last Session of Group30.1-3 Last Counselling Session 28.8-16.5 T(151)=9.4, p<0.001* FCSAA Power of Prevention Conference, November 29. 2013

52. Conclusion – Overall results support continued use of this framework of individual and group – Decreasing the number of clients doing individual counselling only – Will work with the program to support best results for clients FCSAA Power of Prevention Conference, November 29. 2013

53. robbie.wagner@calgarycounselling.com www.calgarycounselling.com FCSAA Power of Prevention Conference, November 29. 2013