1. Evaluation and Policy: How Are They Connected? Geoff Nelson, Psychology, Wilfrid Laurier University Tim Aubry, Psychology, University of Ottawa Aimee Watson, Mental Health Commission of Canada

2. Agenda 2:45-3:00 – introductions and expectations for the workshop 3:00-3:10 – policy 3:10-3:20 – evaluation 3:20-3:30 – knowledge transfer 3:30-3:50 – small group discussion 3:50-4:00 – reporting back on small groups

3. Policy The renewal and repurposing of HPS March 21, 2013, federal budget announcement, Economic Action Plan 2013 proposes $119 million per year over five years, nearly $600 million in total, to the Homelessness Partnering Strategy using a “Housing First” approach. Building on the successful At Home/Chez Soi project, the Government will continue to work in partnership with provinces and territories, communities, the private sector and other stakeholders, such as the Federation of Canadian Municipalities, to reduce homelessness. Economic Action Plan 2013 proposes $119 million per year over five years for the Homelessness Partnering Strategy using a Housing First approach. Over the coming months, the Minister of Human Resources and Skills Development will work with Government partners at the community level to confirm the implementation details of this approach.

4. Policy The renewal and repurposing of HPS The evidence shows that the Housing First approach: can be implemented across Canada; improves the lives of those who are homeless and have a mental illness; and makes better use of public dollars, especially for those who are high service users—as demonstrated by a reduction in the use of hospital in-patient care, emergency rooms, police detentions and justice services. The project will develop a body of evidence to help provinces and territories provide services to homeless people living with a mental illness. In addition, funding arrangements are being established to ensure that people participating in the project will have supports in place once the project is completed.

5. Policy How did this renewal and repurposing of HPS happen? Evaluation and trusted messengers At Home/Chez Soi research and MHCC knowledge translation The conservative heartland – the Alberta experience and the emergence of the Canadian Alliance to End Homelessness led by Tim Richter Looking south of the border – US research on Housing First, support from the G.W. Bush administration, development of 10-year plans to end homelessness in many US cities

6. Policy How did this renewal and repurposing of HPS happen? Political will Dissatisfaction with progress in reducing homelessness in Canada Review of HPS provided an opportunity to refocus its approach Desire to do a better job of reducing homelessness in Canada An existing funding mechanism and program (HPS)

7. Program Evaluation (Rossi et al., 2004) Definition “ The systematic collection of information about the activities, characteristics, and outcomes of programs” Why Evaluate? 1.to make judgements about the program, 2.to improve program effectiveness, 3.to inform decisions about future programming”

8. PROGRAM MISSIONPROGRAM PROCESSES PROGRAM OUTCOMES NEEDSINPUTSINITIAL OUTCOMES GOALS & OBJECTIVES VALUES & PRINCIPLES ACTIVITIES OUTPUTS INTERMEDIATE OUTCOMES LONGER-TERM OUTCOMES RELATIONSHIP TO THE THREE MAIN TYPES OF PROGRAM EVALUATION NEEDS ASSESSMENT IMPLEMENTATION / PROCESS EVALUATION IMPACT / OUTCOME EVALUATION Make-Up of a Program

10. Housing First Program Mission Program Processes Outcomes Individual: - Reduce HOUSING + Housing Stability Homelessness FIRST + Functioning + Quality of life Community: - Chronic & episodic homelessness

11. Housing First: Assessment of Fidelity

12. Housing First: Assessment of Fidelity Options 1.Self-asessment of fidelity (Gilmer, Stefanic, Sklar, & Tsemberis, 2013) 2.External assessment of fidelity (Mental Health Commisssion of Canada, 2014)

13. Outcome Monitoring “The ongoing collection and analysis of information on indicators of how well programs are performing with regard to the achievement of results” (Rossi, Freeman, & Lipsey, 1999).

14.  Standardized self-report measures  Simple counts of client status  Collateral rating scales  Consumer satisfaction measures Types of Outcome Measures

15. Housing First: Outcome Indicators & Potential Measures 1.Housing status of clients (Simple count) 2.Community functioning (Multnomah Community Ability Scale – Collateral rating scale) 3.Quality of Life (Lehman’s Quality of Life Interview – Standardized self-report measure) 4.Consumer Satisfaction Measure – (Service Satisfaction Scale)

16. A PIT is a snapshot of sheltered and unsheltered homeless people in a community on a single night.  Provides a baseline  Identifies needs of the populatioin  Enables you to measure community level outcomes  Helps identify priority populations Community-Level Outcomes Point-in-Time Counts Why do a count?

17. Canadian Point-in-Time Count Methodology and Toolkit Piloted in Seven Cities in Alberta October 16, 2014

18. Who to Count?

19. Mandatory questions Screening Consent Gender Age Ethnicity Migration Immigration Homelessness History Veteran Status

20. Using Program Evaluation to Develop and Improve Housing First Programs 1.Conduct regular fidelity assessments 2.Integrate outcome monitoring into service delivery 3.Conduct regular point-in-time counts 4.Provide training of service providers on data collection 5.Provide regular feedback on collected data 6.Keep analyses simple 7.Make program improvements based on collected data

21. Knowledge Exchange “Evidence of effectiveness alone is rarely enough to ensure adoption of interventions” (Leff & Mulkern, 2002) What is Knowledge Exchange? The Canadian Institutes of Health Research defines KE as: a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the health care system www.cihr‐irsc.gc.ca/e/29418.html

22. Knowledge Exchange How do you do it? Moving Knowledge to Action: What? Who? How? Synthesis Dissemination Exchange - 1 and 3 pagers, summaries, early findings - media, community events, social media - Engagement, training, collaboration

23. Knowledge Exchange Influencing Policy Lessons from the At Home/ Chez Soi experience: Throughout At Home/Chez Soi, the project team engaged in an ongoing process to work with government. Strategies included: Worked with all levels of government (municipal, provincial, federal) Regularly held in-person meetings and briefings throughout course of project Developed briefing material- PowerPoint presentations, Early Findings reports Used key messages to ensure consistent and focused messaging Resulted in strong government engagement in At Home project sites and ongoing policy change

24. Focusing the Frame - 1 and 3 pagers, summaries, early findings - media, community events, social media - Advisory panels, collaboration, meetings A Participant Photography Project My Kitchen __________________________ I love to cook. On the street it's hard to get a decent meal. Now my fridge is full and I can choose what and when I eat. I'm so much healthier now, and I get so much satisfaction in doing something I love to do. I can even invite people over for a meal. Beats the hell out of a food kitchen. View from my Window ________________________ I am so grateful for this view. I am also grateful to not have anyone able to knock on my window at all hours of the night. I have some sense of safe.

25. Focusing the Frame - 1 and 3 pagers, summaries, early findings - media, community events, social media - Advisory panels, collaboration, meetings A Participant Photography Project Strings ____________________ The name “chaos theory” comes from the fact that the systems that the theory describes are apparently disordered. But Chaos Theory is really about finding the underlying order in seemingly random data. The “butterfly effect” is a sensitive dependence on initial conditions. Just a small change in the initial conditions can drastically change the long term behaviour of a system... I am the system

26. Small Group Discussion Questions How can your communities incorporate evaluation into your Housing First programs? What knowledge transfer strategies could you use with policy-makers in your jurisdictions to promote the adoption of Housing First