Presentation on theme: "Self-Assessment within a Learner-Prescribed CME/CE Curriculum Destry Sulkes M.D., Vice President, Educational Services Medscape LLC."— Presentation transcript:
Self-Assessment within a Learner-Prescribed CME/CE Curriculum Destry Sulkes M.D., Vice President, Educational Services Medscape LLC
Acknowledgements From We Move: Judy Blazer From Outcomes, Inc.: Mazi Abdolrasulnia, PhD Linda Casebeer, PhD Hamid Doroodchi, MD Nancy Roepke, MBA From Medscape LLC: Martin Irvine, PhD We wish to acknowledge the following for their contributions and hard work:
Disclosure This activity was made possible by an unrestricted educational grant from Boehringer-Ingelheim Pharmaceuticals, Inc.
Sometimes things aren’t quite what they seem…
Starting Point CME has been shown to improve knowledge, skills (cognitive) and attitude 1 Self-assessments have been shown to be inaccurate when not externally informed 2 Feedback from externally informed self-assessments, provided to both clinicians and Faculty, can be used to increase effectiveness of CME 3 1.Evid Rep Technol Assess (Full Rep) Jan;(149):1-69. Effectiveness of continuing medical education. Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, Miller RG, Thomas PA, Prokopowicz GP, Qayyum R, Bass EB.Marinopoulos SSDorman TRatanawongsa NWilson LMAshar BHMagaziner JLMiller RGThomas PAProkopowicz GPQayyum RBass EB 2.JAMA Sep 6;296(9): Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L.Davis DAMazmanian PEFordis M Van Harrison RThorpe KEPerrier L 3.Med Teach Mar;30(2): The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Colthart I, Bagnall G, Evans A, Allbutt H, Haig A, Illing J, McKinstry B. NHS Education for Scotland. Colthart IBagnall GEvans AAllbutt HHaig AIlling JMcKinstry B
Current Model of Online Medical Education Medical education programs for healthcare professionals are designed using broad needs assessments across target audiences Individual activities within courses are not typically developed for specific segments of learners The courses clinicians select to maintain their professional licensure may or may not be appropriate to address gaps in specific individuals’ clinical knowledge, skills and attitude (K/S/A)
Hypothesis for an Externally Informed Self Assessment Learners who participate in an externally informed self- assessment and engage in the corresponding prescribed educational activities are more likely to: Engage in the educational activities Enhance their K/S/A relative to those who find the activities in an undirected fashion
Methodology To address K/S/A gaps identified through a needs assessment of Medscape members, an online CME/CE curriculum of activities was developed in collaboration with We Move and Outcomes Inc One part of the curriculum is a CME/CE-certified self-assessment activity that measures K/S/A related to recognition, diagnosis, and non-pharmacologic and pharmacologic management of RLS The online self-assessment provides automated activity recommendations based on the learners’ responses to K/S/A questions Post-assessments follow each activity to assess the relative efficacy of this new educational approach vs the standard, undirected approach, to improving clinical management of RLS
Course Formats/Titles/Focus FormatsTitlesFocus Virtual Lecture Restless Legs Syndrome: Understanding the Patient Experience to Improve Outcomes Attitude (A1)* Expert Interview The Genetics and Pathogenesis of restless Legs Syndrome Knowledge (K1)* Mini Review Article 1 Restless Legs Syndrome in Special Populations Skills (S1)* Mini Review Article 2 Treatment of RLS in Patients with Psychiatric Comorbidities Knowledge Expert Interview What’s New in the RLS Literature? Skills Mini Clinical Case 1 Key Considerations in the Diagnosis and Management of RLS in the Family Practice Setting Skills Mini Clinical Case 2 Choosing the Appropriate Treatment Regimen for a Patient with RLS and Comorbid Conditions Skills Test and Teach Case 1 Is It Restless Legs Syndrome? A Case Study in Diagnosis Skills Test and Teach Case 2 The Importance of Accurately Diagnosing RLS in the Primary Care Setting Knowledge Review Article Maximizing Outcomes in Patients with RLS: Focus on Relevance of the Disorder, Diagnosis, and Management Attitude, Knowledge, & Skills * prescribed courses
Technical Platform Supporting Individualized Feedback Feedback What: Each learner receives an listing the recommended activities based on the question responses provided in the self-assessment. How: Each night, a script that queries the database for any new self- assessment participants. The script evaluates each participant’s answer choices and maps the choices to the recommended CME activities. The data is fed to an application where it is formatted and sent to each participant with individualized activity recommendations, plus a link to the curriculum page.
Sample immediate feedback / message Medscape CME Self- Assessment Dear Dr. Dana Ricci, Based on your answers to the Restless Legs Syndrome Self- Assessment CME activity, Medscape would like to recommend the following readings: CME/CE The Genetics and Pathogenesis of Restless Legs Syndrome: Implications for the Clinician In this expert interview, Dr. David Rye provides an overview of what is currently known about the genetics and pathogenesis of RLS and how to apply that knowledge to clinical practice. CME/CE Restless Legs Syndrome in Special Populations Dr John Winkelman provides a comprehensive overview of the similarities and differences in the diagnosis and management of primary RLS and RLS that is secondary to other conditions. The Genetics and Pathogenesis of Restless Legs Syndrome: Implications for the Clinician Restless Legs Syndrome in Special Populations
Process Schematic Prescribed Learning: Learner sent recommendation from Externally Informed Self Assessment Undirected Learning: Learner finds curriculum and/or individual activities by standard search criteria Attitude Course 1 Knowledge Course 1 Skill Course 1 Post Test A1: Includes mapped Attitude questions from Self Assessment Automated response directing learner to specific K, S, or A course Post Test S1: Includes mapped Skills questions from Self Assessment Post Test K1: Includes mapped Knowledge questions from Self Assessment
Datasets and Minimums Required for Analysis Non-Medscape Membership Baseline Survey: N=400 (Identical to E-I self- assessment) Externally Informed Self- Assessment on Medscape: N=400 (Identical to baseline survey) Prescribed Post Assessment A1 N=~100 Undirected Post Assessment A1 N=~100 Undirected Post Assessment: A N= ~1400 over 7 activities Prescribed Post Assessment K1 N=~100 Prescribed Post Assessment S1 N=~100 Undirected Post Assessment S1 N=~100 Undirected Post Assessment K1 N=~100 Undirected Post Assessment: K N= ~1400 over 7 activities Undirected Post Assessment: S N= ~1400 over 7 activities Outcomes 1 – Measures the courses prescribed vs. the courses actually completed in a three month time period, click-thru rate of A1/K1/S1 prescriptions that resulted in accessing the activities Outcomes 2 – n=~100; Measures the Prescribed Post Assessment to the E. I. Self Assessment by Knowledge, Skills and Attitude Outcomes 3 – n=~100; Measures the Prescribed Post Assessment to the Baseline Survey by Knowledge, Skills and Attitude Outcomes 4 – n=~100; Measures the Prescribed Post Assessment to the Undirected Post Assessment by Knowledge, Skills and Attitude Outcomes 5 – n=~100; Measures the Undirected Post Assessment (A1, K1, S1) to the Baseline survey by Knowledge, Skills and Attitude Outcomes 6 – n=200; Measures the Undirected Post Assessment (4-10) to the Baseline Survey by Knowledge, Skills, Attitude
Learnings to date… K/S/A model helpful and could benefit from further refinement incorporating established competencies feedback is somewhat effective, but Medscape learners access activities primarily from their specialty homepage 65%: specialty homepage 25%: outbound with editorial focus (note: different than the prescriptive s in this initiative) 10%: external search Recognition that current CME/CE activity “tagging” for site-location and rotation purposes can be enhanced to include competencies, thus allowing all activities (>3000 in 2007) to be targeted as needed in the future, perhaps on learners’ specialty homepages, and perhaps without explicitly communicating the targeting