1. Session SiE5: Using Systems Evaluation in Healthcare 28 th Annual Conference of the American Evaluation Association October 16, 2014

2. Using systems thinking and sequential mixed methods to study a complex medical education program in a complex medical system dedicated to improving patient care Lori L. Bakken, PhD; Curtis A Olson, PhD; Jonathan Ross, MD; Mary G. Turco, EdD and Lisa M. Jackson, MPH

3. Plan for the session Background – Program description Preliminary steps – Situational analysis – Logic analysis Methods Results Discussion

4. What is the MM&I Conference? Traditional Model – Case based – Mistake identified – Personal responsibility emphasized – Judgmental – Stressful – Faculty dominated Newer Model – Case-based – Linked to ACGME competencies – Complex preparation – Safe learning environment – Interaction – Constant reflection – Intergenerational – Multidisciplinary – QI opportunities

5. Who attends MM&I? A.DHMC/VA Active Staff MD/DO B.Community Based Provider C.Student D.RN/PA/NP E.Resident Staff F.Retired MD G.Other

6. Preliminary Analyses

7. The evaluation challenge A complex intervention in a complex organizational environment Evaluators need to “tailor their approach to fit the complexity of the circumstances they face” (Patton, 2011, p. 109)

8. Situational complexity Each situation is unique High uncertainty about how to produce desired results Actions produce a surplus of results, many unintended Non-linear interactions within a dynamic system Multiple, uncoordinated actions aimed at same goals Patton MQ. Developmental evaluation: applying complexity concepts to enhance innovation and use. New York, Guilford Press, 2011

9. Interventional complexity-MM&I Variability in topics, presenters, and audience Multiple active ingredients in variable proportions Influenced by changing local conditions Educational approach is adaptive, emergent

10. Steps in Logic Analysis (Brousselle & Champagne, 2011) 1.Create a logic model of the program “in action” 2.Consult the literature and experts to fill gaps and build a conceptual framework 3.Compare the logic model with the information gleaned from Step 2 to the logic model created in Step 1 to design a more accurate theory of change.

11. 10

12. Building a conceptual framework Targeted Literature Review – Prior studies of MM&I Conferences (7) – Diagnostic reasoning (1) – Inter-professional education (2) – Human factors sciences and systems thinking (6) – Organizational Learning (2) Local Experts – Subset of key stakeholders – Dartmouth Aligning Education for Quality (Dae4Q) Committee – Department of Medicine’s Advisory Council for Education (DoM-ACE)

13. Systems model for MM&I 12

14. Study Design Does MM&I Contribute to Improved Patient Care?

15. Design/systems model alignment

16. The evaluation design Phase I Developmental Phase II Formative/Process Phase III Impact Learning Outcomes & QI Issues and Actions Processes of QI Activities that lead to Practice Changes Demonstrate the contributions of an educational intervention to practice changes and patient care

17. Phase 1 evaluation questions 1.What types of cases have been discussed and why were they selected? 2.What major problems have been identified through the conference? 3.What values do MM&I discussion espouse and how do participants’ perceive them? 4.What do residents learn through conference preparation and planning? 5.How does evidence get used during MM&I? 6.In what ways do participants’ knowledge or practice change regarding treatment and management?

18. Methods

19. Participant recruitment Physician participants (Internal Medicine, Specialists, Residents, Fellows) Resident presenters Non-MD health care professionals Medical Students

20. Data sources & collection methods Data Source Semi- structured Interviews Focus Groups Guided Observation Question- naire Existing Data Participants Mixed (values) Medical Students (values) Non-MD HCPs (values) Resident Presenters Video- recordings “Matrix”* *resident’s self-assessed ACGME competencies, case descriptions, QI issues, tabulated dialogue, learning objectives, participant’s evaluation

21. Data analysis Data SourceQualitativeQuantitative Transcripts of Interviews & Focus Groups Recorded Observations (critical thinking) Values Survey Matrix (case descriptions and residents’ comments) (ACGME competencies)

22. Findings

23. Cases selected and why Types – Complex, complicated and chronic – Cancer, diabetes, heart disease or psychological issues – Often involve co-morbidities – Primarily inpatient Reasons Selected – Demonstrate what went well – Highlight circumstances surrounding mismanaged care – Learn more about a specific topic – Improve communications among providers, patients, and others – Avoid a future misdiagnosis

24. Major problems identified Practice-related, e.g. delayed treatment Communications, e.g. transferred care Systems, e.g. outdated sedation procedures

25. Perceived valuesCollaborationCommunication Perspectives Respect Sharing Reflection

26. Resident presenter learning In-depth knowledge of a specific topic Improve presentation skills Reduce personal biases and seek out colleagues for help Establish a differential diagnosis as early as possible How to gain closure on emotional issues Improve ways to relate to patients Knowledge of cases that become future references (expands cognitive networks)

27. How evidence gets used (during MM&I)

28. Changes in knowledge/practice Heightened awareness of and appreciation for inter-professional perspectives Seek alternate opinions when diagnosing or treating a patient’s illness Better patient care Note: Medical Students – a special case

29. Conclusions (thus far) MM&I fosters critical thinking that is necessary for organizational learning and change Much learning occurs in the affective domain; cognitive networks are expanded and used in future practice. Repeated attendance contributes to learning through pattern recognition Practice changes do occur but they are highly dependent on the individual and the case presented Communication role-modeled in MM&I supports positive interactions among colleagues MM&I primarily addresses practice-level and communication problems and does not sufficiently address systems-level issues, although they are identified through the conference’s dialogue.

30. Interim Recommendations Create stronger mechanisms to facilitate changes that address systems-level problems Continue teaching approaches and strategies that foster organizational learning Assess the extent to which medical students learn and apply the knowledge acquired through the observations. Use the wisdom of the conference’s interdisciplinary participants to establish potential solutions to identified QI issues (that could be addressed through the VI or TDI students). Redistribute the emphasis on learning from residents to participants more broadly

31. Implications for future work Determine the frequency of reported outcomes Acquire more evidence on if and how outcomes contribute to patient care Pay attention to gender Encourage and assess ways that senior providers learn from colleagues Give more attention to assessing observational learning and its role in influencing change