Continuing Education/Knowledge Translation University of Toronto Needs Assessment WHY needs assessments? The supply side vs demand side story TOOLS of the needs assessment trade APPLICATIONS: Using the tools

Continuing Education/Knowledge Translation University of Toronto Needs Assessment: some definitions NEEDS ASSESSMENT: The process of determining the size and nature of the gap between ideal and real performance or competence NOTE two concepts: PERCEIVED (SUBJECTIVE) NEEDS: those needs identified by the learner UNPERCEIVED NEEDS (OBJECTIVELY DETERMINED): not identified by the learner

Continuing Education/Knowledge Translation University of Toronto WHY bother with needs assessments? (1) The demand side of the story: the Sibley Study (2) The supply side: the world-famous GI Refresher Day (3) The evidence “side”: reviews of the literature in CME

Continuing Education/Knowledge Translation University of Toronto (1) Stories from the supply side The Change Interviews: Fox The Sibley Study, Ont. fp/gps - mailed CME packages - hi-choice items, no change; lo-choice, + change

Continuing Education/Knowledge Translation University of Toronto (2) A story from the demand side The World-famous (okay, well almost) GI Refresher Day, early 1980’s McMaster

Continuing Education/Knowledge Translation University of Toronto BEFORE…………… hallway conversation with Dr C, chief GI Guy +++ drug bucks visit of his buddy, Smedley-Heathfield course aimed at gp/fp’s 3-colour brochure (big deal back then)

Continuing Education/Knowledge Translation University of Toronto THE COURSE 8-9 registration, breakfast 9-9:59 Smedley-Heathcroft: a lecture on drug bioavailability studies related to the new H2 antagonists 9:59-10:00 questions after coffee: Dr. C’s fellow on the newest endoscope, more lectures…you get the picture great lunch though

Continuing Education/Knowledge Translation University of Toronto - Evaluation of the program: - 65 attenders, mostly family docs, many stayed till lunch, vamoosed - post lunch faculty/participant (Davis) ratio > evaluation forms turned in: 21 in the lousy- awful range, 3 in the excellent range - 3 in excellent range turned out to be Barnes’ residents, fellows, maybe buddies, who knows

Continuing Education/Knowledge Translation University of Toronto FEATURES OF SUPPLY-SIDE CME: - who does the supplying? industry faculty providers - why do we let it happen? habit, ease (both content and format) - what’s the evidence for its effectiveness?

Continuing Education/Knowledge Translation University of Toronto DEMAND-SIDE CME: Some principles of adult learning - age/stage of development - personal experience - motivation - influences of colleagues, society - reinforcement for learning - learning style

Continuing Education/Knowledge Translation University of Toronto SUPPLY-SIDE VS DEMAND- SIDE CME: Summary IssueSUPPLY-SIDEDEMAND-SIDE originsfacultylearner industrypatient traditional UG ednew curricula needs ass’tnone+++ formatlectureinteractive, problem-based targetteacherphysician/patient outcome^competence^performance, ?health care outcomes

Continuing Education/Knowledge Translation University of Toronto (3) Some evidence about needs assessment Changing Physician Performance – a systematic review of the effect of CME strategies JAMA 1995;274:

Continuing Education/Knowledge Translation University of Toronto 1995 Review  >1,500 met broad criteria; 99 met all  70% showed change in performance  48% showed a change in HC outcomes  interventions: -educational materials -formal CME activities -outreach visits -opinion leaders -patient-mediated -audit/feedback -reminders -multiple activities

Continuing Education/Knowledge Translation University of Toronto NEEDS/GAP ANALYSIS Cumulative Scoring Method 0  No mention of clinical needs 1  Clinical literature Referenced 2  National guidelines employed in study 3  Local guidelines developed +/or national guidelines adapted in local consensus process 4  Intervention targeted at identified gaps in performance; +/or barriers to change addressed

Continuing Education/Knowledge Translation University of Toronto EFFECTIVENESS OF INTERVENTIONS by Needs/Gap analysis

Continuing Education/Knowledge Translation University of Toronto  Tools of Needs Assessments  1) Perceptions of the Health Professional  2) Competency Assessment Measures  3) Performance Measures  4) Health Care/Patient Outcomes

Continuing Education/Knowledge Translation University of Toronto  1) Perception  questionnaires – problem-based vs topic- based  interviews: phone, in-person, focus groups  idea-stimulating methods: case, problems, tables of content  in-program methods

Continuing Education/Knowledge Translation University of Toronto  2) Competency Assessment  multiple choice exams,  simulated problems: OSCE’s (objective structured clinical exams); paper problems, std pts - note the OSCME  chart-stimulated recall  Likert Rating Scales …why go to the trouble of Type3/4 measures?: note the Sibley Study

Continuing Education/Knowledge Translation University of Toronto  3) Performance Measures (workplace measures)  chart/case review (individual or group)  audit and feedback  utilization data review  health services literature, clinical practice guidelines

Continuing Education/Knowledge Translation University of Toronto  4) Patient/Health care outcomes  hospital data: morbidity/mortality review - eg., surgical infections, length of stay  provincial data: eg., ICES, burden of illness  local/practice data following chart review, eg., BP control in the elderly