1. Engaging stakeholders

2. What is the Centre? Knowledge Bringing people and knowledge together to promote the best mental health and well-being for every child and youth CapacityPartnerships Full continuum of effective and accessible mental health services for children and youth.

3. Webinar overview This webinar will be recorded, so that it can be available on the Centres website as an educational resource The slides and any supplementary materials will be sent to participants at the end of the session The session will take approximately 45 minutes including time for questions You may ask questions or make comments at the end of the session by: Typing your question or comment

4. Engaging stakeholders Facilitators Ilana Smyth, Research Associate, Centre of Excellence Paula Cloutier, Research Associate, CHEO Who are the stakeholders of an evaluation and how do they influence evaluation? This webinar will discuss the importance of identifying stakeholders and how to engage them in planning and conducting the evaluation.

5. Outline Stages in engaging stakeholders: Identifying stakeholders and their needs Techniques to identify stakeholders Managing expectations and taking action Shared experience from recent evaluation activities Discussions and questions

6. Communication with Stakeholders Communicating with stakeholders is important throughout an evaluation Intended users of the evaluation are more likely to use the results if they have been involved in the process

7. Identify stakeholders Stakeholders are those who can affect or are affected by an evaluation process and/or its findings Individuals Groups Organizations Stakeholders are individuals Individuals within organizations Individuals within groups

8. Identify stakeholders Consider stakeholders who may have a negative influence on the evaluation Including those who oppose using resources for evaluation purposes Including those who may feel threatened or experience anxiety around certain aspects of the evaluation

9. Identify stakeholders Keep the definition of stakeholders broad at first, however its difficult for an evaluation to answer all questions for all of those interested. Necessary to then narrow down the evaluation questions and focus on a more manageable list of potential stakeholders Primary intended users or key evaluation stakeholders

10. Techniques List evaluation stakeholders Brainstorm a list of individuals or groups

11. Techniques Stakeholder influence diagram Stakeholder: Stake or Interest in the Program: What do they want to get out of the program, and/or What do they want the program to produce? Stake or Interest in the Evaluation What do they want to get out of the evaluation, and/or What do they want the evaluation to produce?

12. Techniques Power versus interest grid Subjects – Have a significant interest but little power Players – Have a significant interest and substantial power Crowd – Have little interest and not much power Context Setters – Have substantial power, but little direct interest Power Interest

13. Document stakeholder needs Advance planning is important Establish as early as possible the role and function of stakeholders Establish ground rules and to what extent stakeholders are involved in the evaluation Establish timelines for involvement

14. Manage stakeholder expectations Stakeholder management – ways of working with positive as well as negative stakeholders Better referred to as stakeholder engagement Begins in the early stages and continues through development and implementation phases

15. Engaging stakeholders after the evaluation Tell your stakeholders how things went, both with the program and with the evaluation Follow up an evaluation report with a phone call or an in-person meeting, especially with primary stakeholders Focus discussion on what the program has learned and how it intends to grow and improve, rather than just reporting on outcomes

16. Engaging stakeholders after the evaluation Invite stakeholders to brainstorm ideas on how to strengthen your program based on the evaluation findings Send out a final evaluation report to all levels of stakeholders upon the program's completion

17. Specialized Psychiatric & Mental Health Outreach Services for Children and Youth in Eastern Ontario

18. SPMHS Outreach Services Services designed for children and youth who have complex mental health disorders which result in serious psychiatric, emotional and/or behavioural disturbance Collaborative care approach Role is to reinforce, support, provide consultation and training to community based MH services Deliver services on a planned basis or by request either through travel to the community or by way of telehealth and telepsychiatry

19. Who are the Stakeholders? Funder: MCYS Service Providers: CHEO (age 0-15), ROMHC (age 16-18) -Clinicians (psychologist, psychiatrist, psychological associate, psychiatric nurse, occupational therapist) Rural Community Agencies: Providing MH services Rural Physicians: referring Consumers: Children, Youth, Caregivers

20. Discharge to Referring Physicians and/or Community SPMHS Centralized Intake Assess and refer to Appropriate Service/Program: Outreach Outpatient Inpatient Emergency Department Appropriate Community Services Consultation Assessment Clinical Care Plan Treatment Recommendations Renfrew Prescott/Russell SD&G North Lanark Ottawa Referring Physicians Not appropriate for SPMHS Services Outreach Teams North Grenville

21. Engaging the Stakeholders Consult with funders, service providers and consumers regarding outreach services (prior to program development) Clarify needs, expectations and capacity on part of both specialized and community mental health care partners (prior to program development) Develop services in response to identified local needs and local culture (be adaptive, enhance existing community services, address issues of role definition, responsibility….) Evaluate outcomes leading to ongoing refinement and improvement of service (ideally build evaluation in right from the start)

22. Getting Service Providers On board Staff meeting(s) to discuss program and evaluation with plenty of time for Q & A It is essential that those involved see the evaluation as meaningful and purposeful Appropriate training be offered as necessary Focus groups to get feedback from clinicians Willingness to make changes along the way (evaluation is not research)

23. Getting Service Providers On board with the Evaluation: Success? County Accepted to Outreach Compliance with evaluation Stormont, Dundas & Glengarry 14622 (15%) North Lanark1328 (1%) Renfrew7451 (69%) Prescott-Russell471 (2%) North Grenville1912 (63%) Ottawa55 (100%)

24. Getting Clients On board with the Evaluation: Success? Satisfaction Questionnaires (17% completion) Consider more than one way to get feedback (e.g., consumer focus groups) Getting Agencies On board with the Evaluation: Success? Share evaluation outcomes with them (2/5 who wanted to hear anything)

25. Keeping your Stakeholders Engaged Communication is key Continually demonstrate the relevance of the evaluation (one size doesnt fit all) Keeping the Evaluators Engaged Dont have an all or none attitude (view it as a process) Identify and celebrate the successes along the way (the smaller achievements add up) Learn from the failures (be willing to modify and try something else) Plan your next steps (focus groups with referring physicians)

26. Summary Identify key stakeholders at the beginning of evaluation activities as much as possible Engage stakeholders in evaluation activities from the beginning Document stakeholders involvement Ongoing communication is key Flexibility is a necessity

27. For more information Ilana Smyth Research Associate, Centre of Excellence 613.737.7600 Ext. 3325 ilsmyth@cheo.on.ca Paula Cloutier Research Associate, CHEO 613.737.7600 Ext. 3672 cloutier@cheo.on.ca