1. Nadim Lalani MD

3. Vanilla Sky Tom Cruise 2001 Existential Mind warp Deals with cryogenics and the possibility of living a virtual life after death Blending of the technologic and biologic worlds plugged in ?Virtual [technologic] world to supplement [real world] EM medical education

4. Objectives Definition Background Literature Review Med Ed Resident Ed Professional Development What it might look like Future Directions

5. What is Online learning? Online Learning [e-learning] = is digital Evolved from CD/computer labs Everyone does it! Performance Support [ for software e.g] Web page [e.g. Uptodate] Self-paced Web-based [CME] Leader-led [ Distance Learning] Blended [or hybrid] learning combines conventional with digital learning

6. Advantages of e-learning Rich environment: Media-filled [esp in EM] transfer of difficult concepts Links to sources Convenient, efficient & flexible Asynchronous Can be accessed from a distance Adult learning principles: Self-paced and self-directed Flexible/ home access efficiency

7. Background: Why bother? U of C Medical School current enrollment = 130 students goal 150 Mandatory EM rotation / increasing competencies Resident numbers also increasing Result more learners in the ED Relative shortage of preceptors, relevant clinical encounters and curricular time Will be worse when our program expands usurp learning opportunities

8. Why bother Deficiency in learning encounters = a performance gap future physicians do not have the adequate exposure to emergent problems. imperative we equip students, clerks & residents with the skills and training.

9. why bother Increased digitalisation is a key strategic goal of the U of C Learners are unique with mulitdimensional learning styles. Adult learning principles Attract the best candidates Provide a method of training students and clerks at two different campuses Provide consistency in teaching

10. Why bother? Provide efficient means of knowledge transfer to residents Increasing number of competencies [CANMEDs] Better use of academic half-day. Provide more effective professional development: Asynchronous dont have to be there Interactive discussion board Consistent, evidence-based standard of practice Increased self-efficacy

11. E-learning not a panacea there is more to training and education than e-learning Certain skills do not lend themselves to e- learning The key will be selecting the best delivery method. Cannot simply upload old material. Learner – focused no one solution [blended may work for residents].

12. Process: Can it be done? Fail to prepare … prepare to fail Need to address several key questions: 1. Purpose? Added value? 2. What support and expertise exist? 3. Ongoing upkeep? 4. Stakeholders? 5. Team? 6. Instructional design/Pedagogy

13. Literature Review Same Search terms in PUBMED Bibliographies of relevant articles scanned Missing 1 Med Ed & 1 CME [both foreign language]

14. Literature general Comments More Literature exists for Med Ed Pre 1990 Limited by lack of internal validity Few Randomised Controlled Studies Emerging Lit wrt Resident experience Despite lots of experience with online CME Little Literature … mostly Descriptive

15. Literature General Comments Terminology inconsistent Interventions vary. ? prototypes of todays technology? Dont address some of the uniqueness [internet] Comparing apples to oranges

16. Can E-learning be used to replace/augment Traditional Methods? E Learning & Med Ed

17. Study Dartmouth Med School 328 Students randomised to: Interactive Case-based study guide on Computer* Case-Based Printed study guide Anemia and Cardiology Courses Outcomes: Performance on higher order MCQ tests, exams Self-reported Efficiency * media-rich images, blood smears and EKGs

18. Results No Difference in Test Scores No difference on board exams The vehicle is an acceptable means of delivery Time Spent Cardiology Computer 4.4 ± 0.6h Cardiology Workbook 9.4 ± 0.5hP = 0.0001 Anemia Computer5.5 ± 0.5h Anemia Workbook8.0 ± 0.5hP = 0.001

19. Limitations Self reporting of efficiency! Confounders [other text books/practice exams/time-spent cramming] Doesnt really tell us about dynamic problem- solving/ clinical judgment Board ExamsAnemiaCV Class + CAL81.5 ± 6.580.0 ± 6.7NS CAL82.7 ± 6.082.4 ± 5.6NS

20. 179 Paeds Clerks in [2 sites Chicago] Randomised to Lecture via: Multimedia Text Book* Lecture Printed Text No intervention Paeds airway diseases Outomes: MCQ Test Score [at end of rotation & at 1 y later] * Only different in audio/video

21. Results Score / 26MMTBPrinted Book LectureControl N = 8921192326 Initial Test16.6 [2.5]16.5 [3.6]15.9 [2.5]14.2 [2.9] Final Test15.3 [2.8]15.3 [2.9]14.6 [2.3]14.5 [3.4]

22. Limitations 51% Attrition rate! Clerks at one site had mail-in repeat exam Confounders One hour instruction embedded in a 6 week clerkship

23. 75 Med students [Brisbane Australia] Randomised after pretest to: Computer Tutorial Focus on knowledge Computer tutorial create ideal patient for dx + feedback [every 10 cases] Computer Tutorial both knowledge & decision + three different types of feedback [after every 10 cases] Looking at diagnosing abdo pain [ 30 cases]

24. Outcomes and Results Outcomes: Attained knowledge Diagnostic accuracy Decision-making confidence [self reported] Results: Students focusing on facts did not improve decision-making All feedback groups improved diagnostic accuracy Type of Feedback not important. Self reported confidence improved

25. Limitations Small study Very convoluted method ?reliability

26. Can E-learning be used to Teach Procedural Skills? E- Learning & Med Ed

27. 82 Medical Students [Toronto & Augusta] Randomised to: Computer Tutorial + knot board Lecture with Feedback + knot board Two-Handed Knot tying Tested right after [filmed] Outcomes: Proportion Square/ Time to tie Knot Performance score [blinded surgeons] Student Questionnaire

28. Results NO difference in Cognitive portion Lower performance score in CAL group 89% Students would have preferred Lecture Session Lack of feedback cited as negative

29. Limitations Apples and Oranges! ? Not controlled for hands-on feedback Maybe CAL better if it described pitfalls / showed video of good and bad knots? Reliability of performance score [not included]

30. 42 Clerks U of T Randomised to: Computer Tutorial [rich text, animations, interactive –Q&A, no audio/video] small group seminar [also interactive] Epistaxis Management Outcomes: Short Answer written Test Practical Test [16 point performance scale]

31. Results: Poor Prior knowledge No difference in written scores Slightly better practical skills with CBL

32. Limitations: Small numbers Examiners NOT blinded ?reliability of performance score [not included] Practical was on dummy ? transferability

33. 69 Medical Students [Wisconsin] After pre-test Randomised to: Didactic Session/Q&A ¥ Video-Tape Session* Computer Tutorial* Post Intervention: MCQ test, Practical Skills test [2 blinded obs] Repeat testing at one month ¥ no feedback. * Instructor present

34. Outcomes: MCQ Test Scores Timed observation of skills Critical Skills evaluated via checklist Performance Quotient calculated

35. Results Higher initial mean % correct / % complete in CBT group [p<0.01] Significantly better PQ in CBT group at 1 month [p < 0.01]

36. Results Didactic group better on immediate MCQ [63% vs 49% for video/CBT p < 0.01] Difference in MCQ still there at 1 month

37. Results Bigger change in PQ with CBT at 1 month [ P< 0.01]

38. Limitations Small study Video vs CBT essentially the same intervention ? Why CBT would do better than Video ? Reliability of checklist and PQ?

39. Can E-Learning be used for Emergency Medicine Rotations? What About the ED Experience?

40. 100 Clerks [Mt Sinai] Randomised by blocks EM rotation with access to EM Website Modules [ACLS, Tox], Xrays, Pix, Paeds Cases EM Rotation without access Outomes: Exam Scores Student Satisfaction

41. Results ONLY 28% intervention group used it. 72% Cited lack of time NON sig difference in exam score [72.8 vs 68.2 p = 0.058] Non sig difference in satisfaction [ 77.5% vs 66% p = 0.23] Baseline only 26% > 1h /wk online [cf 96% next class] Baseline 65% wanted online component

42. Limitations < 30% in intervention group didnt reach power. WAS ITT so results would have been +ve with more participants Problems with randomising by block rotations given away on lottery [ CARMs] Unmotivated learners? ?generalisable to clerks in 2008

43. 23 Clerks [U of T Sick Kids] Volunteered for study, Randomised to Access to Web-based Modules No Access to Web-based modules ED Procedures [lac, LP, splint] Outcome: Performance on MCQ Test

44. Results Statistically higher competence [ p = 0.0001] Cohens d Effect size r = 0.79

45. Limitations Small sample size Volunteers [EM /techno gung-ho] Methodology: Unclear when test was administered in relation to rotation ?randomised to learning vs no learning? Validity of MCQ vs Observed skills Transfer of knowledge? MCQ vs Observed skills

46. 350 Urology Clerks [4 med schools US] Randomised [two-group crossover] to: Web-based Tutorials [BPH,ED,PC,PSA] No Access to Web Tutorials Served as the controls for the modules they didnt have access to online Outcomes: Performance on test [pre/post] [Cr =.79] Durability of learning/ Learning efficiency in SubG

47. Results

48. Results:

49. Results Learning Efficiency 0.10 vs 0.03 [p<0.001] Test scores still improved without WBT [12% BPH, 6% ED, 24% PC, 20% PSA] Web-based alone had Cohens r = 24.9!

50. Limitations: Volunteers with unequal participation b/w sites [93% HMS vs 52% BUSM] High Drop out rate 210 /350 completed ? Generalisability of repeated measures ? Generalisability to EM

51. Summary: E-Learning & Med Ed Content can be delivered Appears to be transfer of learning of Cognitive skills … Perhaps also Psychomotor skills Still need for experienced clinician feedback Increased student satisfaction Attempt to make instructional design identical [validity] undermines uniqueness of EL Always will be apples and oranges

52. Summary E-Learning & EM: Controlled Interventional Literature conflicts Learners: Identify time constraints as a barrier in ER Want more visual aids [Ekg, XR, photo & WEB] Effective strategies for ER teaching include: Using Resources Going beyond patient care Improving the learning environment Adult principles/ learners may be driving force despite dearth of evidence

53. How Can we Augment Med Ed? Currently Clerks Use LMS Osler Download PDFs of Content/Cases Core Content Reviewed with Preceptor

54. We can do more … Procedures/ Anatomy Pictures Xray / Ekg interactive ppt

55. E Learning for Residents

56. 109 IM Residents [US] during clinics year Randomised [Crossover] to: Access to Web-based Practice Guidelines On WebCT [rich format with links to sources] Printed Practice Guidelines Outcomes: Format Preference Performance on Final test

57. Results Strong preference for Web Based material Men > Women NO difference on test between groups Non-Significant reduction in time spent ++ problems with WEbsite

58. Limitations Volunteer Significant dropout 145 eligible 109 enrolled 51 completed all aspects ? generalisability

59. 162 FM & IM Residents Randomised to: Web-based tutorial [hyperlinks, graphics] Printed material ACC/AHA Guidelines Management post AMI Outomes: Test Score Efficiency Satisfaction

60. Results:

61. Limitations Only enrolled 30% [550 eligible] Participants not blinded to hypothesis ? Generalisable Voluntary + monetary honorarium

62. 22 EM residents & Staff [U of T] Noninferiority for U/S vascular access course Randomised [after pretest] to: Web Tutorial + practice* 1h Lecture + practice* Outcomes: [2 weeks later..] MCQ test 4 OSCE stations [blinded obs used checklist] *Non-precepted

63. Results Same pass rate All web users logged on from home

64. Limitations Small study [did have power though] No controlling of practice session Stronger coaching the weaker? No interrater reliability for OSCE

65. Summary: E-Learning + Residents Learning can be delivered this way Weak evidence of non-inferiority for learning guidelines in other specialties Modest evidence for use prepping for a hands-on session relating to EM. Weak evidence for Psychomotor skill acquisition Consistent satisfaction with the method

66. How to Augment Resident Ed Website / Subscriptions / RemergS HPS / AHD

67. Better Use of AHD … Longitudinal course on LMS? Already Licensed by U of C Used by Anesthesia

68. Interaction is key Learner-Sources, Learner-learner, Learner- Clinician

69. E Learning and CME

70. 52 Physicians [US] randomised to: Web-Based CME Normal Instruction Office Dermatologic procedures Outcomes: Performance on test Satisfaction Self-reported performance change

71. Results: More improvement in test score with WBL [13.2 vs 9.6] General satisfaction with WBL Increased self-reported competency Fair amount of interaction in asynchronous forums

72. Limitations Small study Self-reporting ? Reliability of MCQ test ? Direct observation of procedures

73. 99 Physicians [US] Randomised to: Web based CME for domestic violence Regular Instruction [only did two surveys] Outomes: Self-Efficacy Externally validated Survey instrument [CR 0.7] Change in screening

74. Results: Increase in self efficacy + 18% intervention vs - 0.6% control [p =0.01] Positive other endpoints in survey NO difference in DV screening

75. Limitations Honoraria to WEB participants Methods ?No learning to controls? generalisability

76. 103 physicians [US] randomised to: Web-based CME F-2-F small group CME session Cholesterol management Outcomes: Performance on tests Chart audit [20 docs from each group 25 charts] Satisfaction survey

77. Results: More improvement with Online NO difference in Satifaction between groups Online spent 3.8 h / 3 sessions Chart review statistically signif [small difference] in guideline adherence [but no difference in cholesterol screening]

78. Limitations 170 eligible only 103 included ? Different groups? [online better at baseline] Live CME event happened before Online [?cross contamination] Hard to reconcile change in behaviour when no change in screening ? generalisability

79. 87 Physicians self-selected into: Online CME [3 sessions 8h] F-2-F CME [one 8 h session] Treatment of opioid dependence Outcomes: Post intervention test Satisfaction

80. Results NO difference in improvement No difference in satisfaction scores

81. Limitations 30% attrition [87 entered data for 62] NON-randomised ?reliability of MCQ test ?Generalisability

82. Summary: E-Learning and CME Certainly can be used to disseminate info In keeping with adult learning principles Transfer of cognitive component ? Psychomotor NO change in practice patterns Subjective increased self-efficacy

83. Enhancing CME Combine LMS with narrated slides Provide facilitation for your topic

84. Narrated Options for Quizzes

85. Future Directions International Emergency Medicine Disseminate info before experts hit the ground Collaboration with other programs Online Electives? EBM Toxicology Radiology

86. Thoughts?

87. Appendix: How you get there

88. Discerning the Context Learning Context Power Dynamics PGME Regionalisation People dynamics Resistance change Relatively few champions of Med Ed

89. Supporting Cast Stakeholders: University UME /PGME RMES Planning Committee: Content experts Educational specialists/Instructional design IT [graphic design/ media] End-users

90. Identifying Program Ideas Core content Literature Current material Needs analysis Narrow focus Not redundant

91. Sorting and Prioritising Prioritise content Important? Feasible?

92. Developing Objectives Needs to be competency based Reflect goals of UME/ Colleges

93. Instructional Plan How are you going to deliver this? Need interaction: Content, Instructors & peers Dedicated ED computer WEBSITE LMS

94. Instructional Plan Instructors: few innovators Clinicians Well suited many hats Need to train the trainers Learners: Self-directed & Computer savvy Clinical/ procedural Skills Blended approach

95. Instructional Plan Variety & content: Anatomy and radiology PowerPoint, Flash files, video streaming. e-case modules LMS online discussion Logisitics: LMS powerful tool Free ones available STARS uses Moodle U of C has licenses & tech Support Regionalisation