1. Difficult to Assess Milestones

2. ... An Open Collaboration In Teaching and Assessment Saadia Akhtar, MD, FACEP Program Director Beth Israel Medical Center Albert Einstein College of Medicine Rodney Omron, MD, FACEP Assistant Program Director John Hopkins University Lynn Roppolo, MD, FACEP Associate Program Director University of Texas Southwestern Parkland Health and Hospital System

3. None to report Conflict of Interest

4. Identify the challenges and obstacles to assessing these milestones Create a more effective educational program to effectively teach these milestones Develop assessment tools to appropriately evaluate residents in these milestones Understand how to navigate the EM milestones wiki site

5. The Problem...

7. A simple but less defined assessment tool......

8. ... are now expanded into 23 sub- competencies!!!

9. Most faculty are convinced that the RRC-EM Milestones are a solution for which there is no known problem. -anonymous

15. Developing a roadmap...

16. 15-question survey, validated by 7 former PDs Sent to CORDEM listserver in the Fall of 2013 62% (99/160) of EM residency programs responded IRB approved study Difficult Milestones Survey

19. Objectives (revised)

21. Begin with the end in mind...

22. Sub-competency 20: Problem Based Learning and Improvement (PBLI)

23. Sub-competency 20 (PBLI): Participates in performance improvement to optimize ED function, self-learning, and patient care. Level 1Level 2Level 3Level 4 Describes basic principles of evidence- based medicine Performs patient follow-up Performs self-assessment to identify areas for continued self-improvement and implements learning plans Continually assesses performance by evaluating feedback and assessment Demonstrates the ability to critically appraise scientific literature and apply evidence-based medicine to improve one’s individual performance Applies performance improvement methodologies Demonstrates evidence-based clinical practice and information retrieval mastery Participates in a process improvement plan to optimize ED practice

24. 98/157 CORD attendees surveyed 71% do not use formal critical appraisal instrument 75% do not have established EBM curriculum 78% do not use any extramural sources (ACP, Best Evidence, EM Abstracts, or Annals of EM) Carpenter CR. Incorporating evidence-based medicine into resident education: a CORD survey of faculty and resident expectations. Acad Emerg Med 2010 EBM Practices and Expectations from EM Physician Educators

26. Sub-competency 20: Most Common EBM Assessments Methods Percent Journal Club 83.3 Direct observation by supervising faculty 47.1 Critical appraisal of the literature using EBM worksheets 26.5 Quality improvement process using EBM to create a process change 23.5 Other conference presentation by EM resident 21.6

27. Learning Basic EBM Principles Journals: ACP Journal Club, EM Abstracts, Annals Books – Emergency Medicine Decision Making by S. Weingart, 2006 – User’s Guide to the Medical Literature by G Guyatt, 2008 – Evidence-Based Diagnosis (Cambridge Medicine), TB Newman, 2009 – EBM: How to Practice and Teach EBM, 4 th Edition, SE Straus, 2010 – Evidence-Based Physical Diagnosis: Expert Consult, S McGee, 2012 – Evidence-Based Emergency Care: Diagnostic Testing and Clinical Decision Rules by JM Pines and CR Carpenter, 2013

28. Learning Basic EBM Principles Websites and podcasts – JAMAevidence: Using Evidence to Improve Care – http://www.dartmouth.edu/~library/biomed/guides/resear ch/ebm-resources-materials.html http://www.dartmouth.edu/~library/biomed/guides/resear ch/ebm-resources-materials.html – http://guides.mclibrary.duke.edu/ebm http://guides.mclibrary.duke.edu/ebm – https://www.med.emory.edu/EMAC/curriculum/diagnosis/ maindiagnostic.html https://www.med.emory.edu/EMAC/curriculum/diagnosis/ maindiagnostic.html – http://www.annemergmed.com/content/ebemresources http://www.annemergmed.com/content/ebemresources – http://adc.bmj.com/content/90/8/845.full.pdf http://adc.bmj.com/content/90/8/845.full.pdf – http://www.cebm.net/?o=1023 http://www.cebm.net/?o=1023 – http://bestbets.org/teaching/current.php http://bestbets.org/teaching/current.php – http://ktclearinghouse.ca/ http://ktclearinghouse.ca/ – http://emjclub.com http://emjclub.com

29. Critical Appraisal Worksheets http://www.cebm.net/index.aspx?o=1157

32. CAT Maker and Critically Appraised Topics

33. Flipped Classroom : Teaching and Learning More Efficiently Need to be information managers, not encyclopedias of knowledge. Need to be life long learners… to ask questions and go out and find answers for themselves. Need to search and filter vast quantities of information on the internet - “Free Open Access MedEd”(FOAMed) by way of blogs, podcasts, and videos, apps and mobile websites

34. Flipped Classroom Example One hour of ‘individualized learning’ at home Provide a clinical scenario and 3-5 questions for residents to answer in advance Provide some electronic resources Interactive discussion in classroom Evaluate residents by written submission of their findings or conference participation

35. Fresno Test 2 clinical scenarios with open ended questions Must complete 4 key steps of EBM practice 7 short answer questions, 2 mathematical calculations, and three fill-in-the-blank questions The only validated, standardized, and objective measure of EBM competence currently available 30 minute test 13 page rubric

36. Fresno Test

37. Knowledge Translation Shift Residents and an EBM faculty leader spend one shift per month in the ED answering EBM queries for all of the residents/faculty seeing patients.

38. Sub-competency 20: Patient Follow-Up MethodsPercent Log of patient follow up that resident does through their own review of medical record 77.5 M&M or continuous quality improvement (review of specific CASES) 65.7 Case conference presentations (not related to M&M or continuous quality improvement) 47.1 Calling back patients23.5 Continuous quality review (review specific patient care areas) 12.7 Chart review with dedicated faculty9.8

39. Sub-competency 20: Self Assessment 62% do a self-assessment tool or questionnaire 61% discuss with residency leadership 55% discuss with their advisor or mentor 42% talk about it during clinical shifts Most review semi-annually

40. Sub-competency 20: Process Improvement (PI) 55% have PI project in the ED 32% do a presentation at weekly conference 28% have resident participate in a committee 24% do not assess this

41. Sub-competency 21: Patient Safety (System Based Based Practice 1 or SBP1)

42. Sub-competency 21 (SBP1): Participates in performance improvement to optimize patient SAFETY. Level 1Level 2Level 3Level 4 Adheres to standards for maintenance of a safe working environment Describes medical errors and adverse events Routinely uses basic patient safety practices, such as time- outs and ‘calls for help’ Describes patient safety concepts Employs processes (e.g., checklists, SBAR), personnel, and technologies that optimize patient safety (SBAR= Situation – Background – Assessment – Recommendation) Appropriately uses system resources to improve both patient care and medical knowledge Participates in an institutional process improvement plan to optimize ED practice and patient safety Leads team reflection such as code debriefings, root cause analysis, or M&M to improve ED performance Identifies situations when the breakdown in teamwork or communication may contribute to medical error

43. Sub-competency 21: What type of educational formats do you use to teach your residents about "Patient Safety“? Percent M&M or CQI conferences97.0 Simulation58.6 Hospital based online modules41.4 Dedicated patient safety conferences37.4 Mock oral boards36.4 Small group discussions30.3 Institute of Healthcare Improvement (IHI) Modules4.0 Other4.0

44. Sub-competency 21- Tools to assess the Patient Safety Sub-competency

45. Sub-competency 21: Do you use any checklists or tools to assist with "Transitions of Care" such as the SBAR or Team Huddle?

46. Sub-competency 21: Tools iPASS (EPIC) TEAM STEPPS Bedside Rounding Sign-Out Tab in EPIC Standardized HMED Transfer of Care Note Evaluation of Handoff Form Developed Own Transition Sheet/Template Printed tracker board and physically rounding in ED at transition time Check list at sign out

50. After implementation of multiple medical team training programs: Improved observed team behaviors. Enhanced staff attitudes toward teamwork. Reduced observed clinical errors. Morey, JC, Simon, R, Jay GD, et al. Error reduction and performance improvement in the emergency department through formal teamwork training: Evaluation results of the MedTeams project. Health Serv Res. 37:1553-1581, 2002

52. Sub-competency 22: Systems-based Management (SBP2)

53. Sub-competency 22 (SBP2): Participates in strategies to improve healthcare delivery and flow. Demonstrates an awareness of and responsiveness to the larger context and system of health care. Level 1Level 2Level 3Level 4 Describes members of ED team (e.g., nurses, technicians, and security) Mobilizes institutional resources to assist in patient care Participates in patient satisfaction initiatives Practices cost-effective care Demonstrates the ability to call effectively on other resources in the system to provide optimal health care Participates in processes and logistics to improve patient flow and decrease turnaround times (e.g., rapid triage, bedside registration, Fast Tracks, bedside testing, rapid treatment units, standard protocols, and observation units) Recommends strategies by which patients’ access to care can be improved. Coordinates system resources to optimize a patient’s care for complicated medical situations

54. Sub-competency 22: What type of educational formats do you use to teach your residents about "Systems-based Management" to improve healthcare delivery and flow? Percent Didactics69.7 Morbidity and Mortality Conference68.7 Small group discussions39.4 Simulation38.4 Multi-disciplinary teaching by other hospital staff36.4 Mock oral boards28.3 We do not have any formal teaching on "Systems-based Management" for our residents 7.1 Online modules5.1 Other1.0

55. Sub-competency 22: Tools for Systems- based Management

56. www.improvediagnosis.org

58. Sub-competency 23: Technology (SBP3)

59. Sub-competency 23 (SBP3): Uses technology to accomplish and document safe healthcare delivery. Level 1Level 2Level 3Level 4 Uses the Electronic Health Record (EHR) to order tests, medications and document notes, and respond to alerts Reviews medications for patients Ensures that medical records are complete, with attention to preventing confusion and error Effectively and ethically uses technology for patient care, medical communication and learning Recognizes the risk of computer shortcuts and reliance upon computer information on accurate patient care and documentation Uses decision support systems in EHR (as applicable in institution)

60. Sub-competency 23: Methods used to assess the use of technology for safe health care delivery Percent Evaluations by supervising faculty in the ED71.6 Chart review with the resident28.4 Conference presentation that demonstrates ability to identify and use educational resources readily available while working in the ED (e.g. PBL) 22.5 Review of billing records17.6 No mechanism exists to assess this milestone16.7 Identification of clinical dashboards8.8

61. EM Milestones Wiki

63. emmilestones.pbworks.com OR Google “EM Milestones Wiki” For assistance or to add to the wiki site: romron1@jhmi.edu romron1@jhmi.edu

64. Summary EBM Safety Technology Systems Based Management Wiki site – How to navigate – How to access resources – How to contribute

65. Thank you!

66. Questions sakhtar@chpnet.org romron1@jhmi.edu lynn.roppolo@utsouthwestern.edu

67. References Tews MC, Liu JM, Treat R. Situation-Background-Assessment-Recommendation (SBAR) and Emergency Medicine Residents' Learning of Case Presentation Skills. J Grad Med Educ. Sep 2012; 4(3): 370–373. Morey, JC, Simon, R, Jay GD, et al. Error reduction and performance improvement in the emergency department through formal teamwork training: Evaluation results of the MedTeams project. Health Serv Res. 37:1553-1581, 2002 http://teamstepps.ahrq.gov www.improvediagnosis.org