1. Jill Sherman, MPH Wayne Warry, PhD Centre for Rural and Northern Health Research Janice Willett, MD Northern Ontario School of Medicine Northern Health Research Conference Timmins, June 4-6, 2015 Developing an Evaluation Framework for the LEGs Initiative: Key Findings

2. Conflict of Interest Declaration: Nothing to Disclose Presenter: Jill E. Sherman Title of Presentation: Developing an Evaluation Framework for the LEGs Initiative – Key Findings I have no financial or personal relationships to disclose This project was supported by the Northern Ontario Academic Medicine Association (NOAMA) AHSC AFP Innovation Fund award.

3. Learning Objectives Learn about the progress of the LEGs Initiative, from the perspectives of LEG members Understand why and how an evaluation framework was developed.

4. Background: Alternate Funding Plans (AFPs) International challenge: A declining supply of physicians who are willing and able to train medical students and residents Ontario AFPs: Funded by the MOHLTC under negotiated agreements with the OMA and individual Academic Health Science Centres and their faculty. AFPs: Supplement the income of physician preceptors, to: – Increase physicians’ willingness to supervise learners – Offset (some of the) opportunity costs of teaching (“clinical repair”) – Encourage the development of academic culture at AHSCs Ensure delivery of high quality, evidence-based care Strengthen medical education Increase research and innovation

5. Local Education Groups (LEGs) Different from other AFPs Other Ontario AFPs Centrally administered and implemented through existing academic departments Include academic and clinical faculty Include funds for “clinical repair” LEGs Initiative Independent association (NOAMA) created to administer the AFP Implemented through 40+ LEGs at distributed learning sites across Northern Ontario All NOSM faculty are clinical faculty No funds for clinical repair

6. About LEGs Local Education Groups (LEGs) are self-organized groups of physician clinical faculty with group responsibility for academic deliverables and clinical teaching. – The LEG provides an organizational structure and support to clinical faculty working in distributed learning sites – Eligible for extra funds (administrative support, professional development funds, research development funds) – LEG membership is optional (at this time); 2/3 of AFP members currently belong to a LEG. – Group must submit a proposal, have an approved governance agreement, before operating as a LEG – 42 LEGs have been approved (2015); 23 at start of project (Fall 2013). Diversity of LEGs Size: Range of 5 to 100 physician members Organizational type: Community (multispecialty), Hospital Department; Family Health Team; Pan-Northern Governance: Each LEG determines how funds are allocated among members

7. Research Highlights Method: Semi-structured interviews (January–April 2014) – 8 Physician Leads, 1 Administrator (2 declined) – Purposively selected from established LEGs – Mix of hospital departments, small/medium/large community LEGs Results: Major themes identified Benefits of becoming a LEG Activities and Innovations Factors influencing LEG development Challenges and suggestions for improvement Goals and ideas of success

8. Perceived Benefits of Becoming a LEG Improved organization & delivery of medical education – Better organized – Funding for administrative support – Development of a collective approach to medical education – Local ownership – Greater consistency & accountability in delivering curriculum Increases in academic activity – Increases in learner placements – New modules/activities developed – Increases in locally delivered CEPD Greater flexibility and fairness in use of funds Encouragement & support for research and innovation Enhanced status of clinical teaching in the community Community benefits – Direct and indirect clinical recruitment – Special projects – Community acceptance of learners

9. “Now that this is more structured and organized, and now that the LEG handles a lot of what would otherwise become a headache or nuisance for individual faculty members, I don’t think they’re experiencing any difficulty in placing all the clerks, in addition to elective students and residents that come through the community.” (Medium Community LEG)

10. Some Challenges First wave of LEGs (early adoptors) – Inventing the LEG from the ground up – no models, templates Organizational and administrative challenges – LEG activity is “extra” – lower priority – Burn out of leaders Constraints to increasing academic activity – Physician shortages / Lack of spare capacity for innovation – Need to build capacity for research, KTE Inter-organizational challenges – Perception of frequent rule changes, lack of clarity on rules – NOSM systems had not adapted to LEGs

11. Towards an Evaluation Framework: Developing a Program Logic Model 5 Domains of LEG Activity A.Medical Education & Program Development B.Professional Development C.Scholarship, research, and innovation D.Recruitment, retention, and community engagement E.Leg Administration

12. Recommendations for Evaluation Planning Allow 3-5 years to prepare for evaluation – Too early to evaluate, need time to develop data collection and reporting systems Include preconditions as explanatory indicators – Factors outside of LEG control affect LEG performance Align indicators with other data requirements – Minimize administrative burden on LEGs The evaluation framework must be flexible – Evaluation priorities of LEGs vary; data should be useful to the LEGs themselves The evaluation framework must be dynamic – Goals will change over time Qualitative evaluation will remain important

13. What’s next? NOAMA Working Group is addressing recommendations – Barriers, challenges, and participant recommendations Continuation of the Evaluation Project – – Will develop and a test small number of indicators – Focus on academic domains Professional Development, Scholarship, Research and Innovation – Collect baseline data

14. For more information... Jill Sherman, Research Associate Centre for Rural and Northern Health Research, Laurentian University Sudbury, ON P3E 2C6 Tel: 705-675-1151 x4359 Email: jsherman@laurentian.cajsherman@laurentian.ca www.cranhr.ca Dorothy Wright, Executive Director NOAMA Balmoral Street Centre, Room 1007 955 Oliver Road Thunder Bay, ON P7B 5E1 Tel: 807-766-7470 Email: dwright@noama.ca www.noama.ca