1. From Program Theory to Systems Theory: Using Logic Analysis to Re- conceptualize an Evaluation Lori L. Bakken, PhD; Jonathan M. Ross, MD; Curtis A. Olson, PhD Presented at the 28 th Annual Conference of the American Evaluation Association, Denver, CO, October 16, 2014

2. Purpose & overview Purpose To demonstrate how we used a logic analysis to expand a program theory to a systems theory of change Overview Description of MM&I Logic Analysis Systems Model 1

3. Learning objectives Critically examine an educational program theory through a systems lens Use logic analysis as a tool to convert program theory to systems theory Think about ways that programs contribute to change within a larger system 2

4. Educate for quality Generate cross-disciplinary dialogue to improve clinical practice and the quality of care MM&I purpose 3

5. Create an organizational culture for learning that promotes constructive dialogue in a safe environment Identify areas for practice improvement to be acted upon by work teams Identify and describe outcomes that include – Programmatic initiatives – Team based improvement – Individual behavior change – Patient-centered care MM&I goals 4

6. MM&I Facts Weekly conference Multidisciplinary Inter-professional Multigenerational Varied learner levels Well-attended (100/week) Participants are there to learn 5

7. Audience Residents Attendings/faculty Medical students Community physicians Retired physicians Nurses Trainees from various programs (eg, physician assistants) 6

8. How is MM&I planned? Case-based Explicit learning objectives Planners reflect together -What do we want the audience members to learn, feel, change? -What could have been done better? -What systems issues need to be addressed? 7

9. How is MM&I delivered? Large conference room (100-120 participants) Progressive presentation of case Interaction with audience, direct and through audience response system – What are you thinking now? – What would you do at this point? – What issues come up for you? Summary of case Learning objectives 8

10. How often do you conduct a Goals of Care conversation as part of Code Status discussion? A.Most of the time B.Often C.Rarely D.These are two different conversations E.I’m not sure 9

11. Going forward, my efforts to include a Goals of Care conversation as part of a Code Status discussion are: A.Likely to improve B.Likely to stay the same C.Going to require some study on my part D.A and C 10

12. How was MM&I being evaluated? Single evaluation form – Check marks, Likert scale Resident presenter – Matrix of evaluations ACGME competencies + – Narrative reflection Faculty planning “committee” 11

13. But what I really wanted to know.. Are we making a difference? – In physician behavior – In our learning environment – In our cultural environment – In patient outcomes 12

14. A confluence of interests Associate Dean for Continuing Medical Education Director of the Center for Continuing Education in the Health Sciences Aligning Education for Quality—American Association of Medical Colleges Dartmouth-Hitchcock Value Institute for Quality and Safety Evaluation expertise Engaged Vice Chair for Education 13

15. Purposes of the evaluation 14 The overall purposes of this evaluation are to: Program improvement (including evaluation and assessment) Assess the impact of MM&I on patient care

16. Use of findings improve medical education at DHMC, improve the conference’s effects on clinical practices that can be directly or indirectly related to patient outcomes; and inform the discipline about continuing medical education’s role and contributions to patient care. 15

17. Steps in logic analysis Brousselle & Champagne, 2011 1.Create a logic model of the program “in action” 2.Consult the literature and experts to fill gaps and build a conceptual framework 3.Compare the logic model with the information gleaned from Step 2 to the logic model created in Step 1 to design a more accurate theory of change. 16

18. 17

19. Activity Please take a few minutes to reflect on this logic model from an evaluator’s perspective – What are your observations about the current program theory? – What theories/bodies of literature might you consider to fill gaps and develop a richer conceptual model for the conference? 18

20. Targeted literature review Studies of MM&I Conferences (7) Diagnostic Reasoning (1) Inter-professional Education (2) Human Factors Sciences and Systems Thinking (6) Organizational Learning (2) 19

21. Meetings with local experts Subset of Key Stakeholders Dartmouth Aligning Education for Quality (Dae4Q) Committee Department of Medicine’s Advisory Council for Education 20

22. Conceptual framework Formal mechanisms Systems thinking and approaches for MM&I Critical thinking Interprofessional teamwork Environmental elements and processes Training as a “fix” vs a “trigger” Psychologically safe environment Local mechanisms existed Multiple QI initiatives at DHMC 21

23. Systems model for MM&I 22

24. Conclusions & Insights The MM&I Conference is a complex educational intervention situated in a complex environment Both “operative” and “aspirational” elements in the program theory Aspects of the process unclear 23

25. The evaluation design Phase I Developmental Phase II Formative/Process Phase III Impact Learning Outcomes & QI Issues and Actions Processes of QI Activities that lead to Practice Changes MM&I contributions to improving pt care 24

26. Design/systems model alignment 25

27. Reflections on the process Value to – Program Director – Evaluators – Other Stakeholders Lessons learned