1. Extending Use of Assessment Data to Support Long-term Curricular Improvement
Leslie Wimsatt Ph.D., James Cooke M.D., Eric Skye M.D.
Department of Family Medicine
University of Michigan

2. Disclosures
No disclosures or conflicts of interest

3. Background/Context Department of Family Medicine:
Online module curriculum grew from 19 to 31 modules in 3 years
Pre-/post-tests showed statistical improvement
Positive resident evaluations (value, satisfaction, relevance)
Best fit between module content educational goals and day-to-day patient care needs remained unclear
Study goal:
To pilot an assessment protocol and examine its utility for use in ongoing curricular review
Online modules offer a way to overcome barriers to educational delivery
Advantages...
Accessibility
Economies of scale
Flexible scheduling
Ease in updating
Individualized instruction
Incorporation of novel instructional methods
Successful management requires…
Commitment to measuring results based on meaningful criteria
Using data to inform curriculum evaluation
Initiating change to improve learning results
Rosenberg M. E-learning: strategies for delivering knowledge in the digital age. NY: McGraw-Hill; 2001.
Cook D. Where are we with Web-based learning in medical education? Med Teach 2006;28(7):

4. Methods
Study population (N= 30 residents) Department of Family Medicine, University of Michigan
Data Collection
Two focus group discussions (4 questions w/probes)
Modules completed to date
Application of module learning to clinical practice
Potential for effectively using other online tools
Module testing exercises (6 modules)
Content-specific post-post testing (7-17 items per module)
FSE/IUPC
Hip Exam
Knee Exam
Joint Injection
Skin Surgery
Vasectomy

5. Methods (cont.) Data Analysis Qualitative data Quantitative data
Thematic analysis of participant feedback
Single author review
Quantitative data
Frequency distributions
Descriptive statistics
Paired t-tests

6. Results Response rate Content analysis – pre-/post-testing
73% (22/30)
Content analysis – pre-/post-testing
Significant improvement in pre-/post knowledge test scores for all 6 modules previously tested (p<.05) *
Post-post testing exercise
Significant improvement in pre-/post-post knowledge scores for 4 of 6 modules
Joint injection (p=.003), (p=.Knee (p=.049), Skin Surgery (p=.035), Vasectomy (p=.028)
Knowledge scores fell significantly across most modules (post-/post-post)
*Skye EP, Wimsatt LA, Master-Hunter TA, Locke AB. Developing Online Learning Modules in a Family Medicine Residency. Family Medicine. 2011;43(3):185-92

7. Among residents who took all three tests – pre, post, and post-post…
Pre
Post-Post p=
Post
FSE-IUPC (N=7)
66.8
(20.43)
85.6
(4.73)
.054
86.15
(8.38)
.916
Joint Injection (N=8)
71.9
(19.39)
81.8
(18.07)
.250
97.4
(2.17)
.041
Knee (N=10)
65.5
(14.31)
81.1
(18.20)
.044
100.0
(0.00)
(18.2)
.009
Hip (N=7)
One very low score, HOI
55.2
(19.55)
63.1
(17.24)
.449
99.4
(1.59)
.002
Skin Surgery (N=11)
71.8
(17.57)
82.8
(10.02)
.068
93.4
(10.39)
.060
Vasectomy (N=8)
2 lower scores HO2, HO3
50.6
(16.22)
77.1
(20.34)
.033
97.5
(4.63)
.040
Among residents who took all three tests – pre, post, and post-post…
Column 2-4: Pre- and Post-Post Test Results
Column 6-8: Post- and Post-Post Test Results

8. Results (cont.) Experience with online module curriculum
All resident participants - 22/22
How/what information from online modules is applied in clinic
Procedural skill modules = most beneficial/transferrable knowledge
Knowledge/content-heavy modules = least transferrable information
Video provides highly transferrable information
Access to modules in clinic = an essential element
How to improve access to module content in clinic
Centralized web portal = essential
Module learning most frequently applied to patient care
All procedural modules
Specifically mentioned all musculoskeletal modules, skin surgery, IUD insertion

9. Results (cont.) Online modules vs. lecture material …
“If I’m going back to a module I know exactly what I want to access and I know where it is because I’ve already clicked through it… as opposed to a lecture where it’s just kind of there and I’m not sure of the order of the slides… it just seems like if I go back to reference a module, it’s faster. ”

10. Results (cont.) Ability to sort, control, manipulate material…
“I think pacing is the biggest thing. Like in a lecture, some lectures are too slow for you. Some lectures are too fast. The control is the biggest thing. Really, if you have a part that you are weaker at, you can spend a lot of time and maybe look up an additional resource on the same side or on the same screen. I do that all the time for things that I’m not so good at, whereas things that I’m better at, I can sort of power through a bit more.”
Ability to sort, control, manipulate material…

11. Limitations
Small sample size limited subgroup comparison of knowledge gains
Focus groups were semi-structured, thereby confining resident comments to targeted areas of questioning
Single program experience limits generalizability

12. Results (cont.) Active use during clinic…
“I use it when I [have] somebody on the schedule … with knee pain or something, I’ll go look at the video real quick … or remind myself of the maneuvers and the names.”
“There’s a document on those that tells you how to document the exam, so you can read through it while you are dictating. I’ve done it that way too.”

13. Results (cont.)
“I think that one of the benefits of the modules is that [they are] mixed media so you can use to prepare for something ahead of time, you can use it in the clinic at that moment, even if it is something that you kind of know but you want to document. It is kind of a resource on multiple levels.”

14. Discussion
Post-Post testing protocol provided a method to assess resident knowledge
Resident approaches to module use vary
Residents value the organizational framework that some modules provide
Ease of access and use emerged as essential elements

15. Future Directions
Protocol will be incorporated into future curriculum assessment efforts
Results will be used to inform module content revisions, enhance integration of module use throughout the department, contribute to improvements in faculty precepting, and expand application of module learning to patient care
Additional research needed to fully identify long-term differences in learning retention

16. Open Access Modules Musculoskeletal Medicine Women’s Health
Women’s Health

17. Open Access Modules Pain Management Integrated Medicine Modules
Integrated Medicine Modules

18. Extending Use of Assessment Data to Support Long-term Curricular Improvement
For additional information, contact: Leslie Wimsatt, Ph.D.