1. THE OSTE AS A FACULTY DEVELOPMENT TOOL
38th STFM Annual Spring Meeting, May 4, 2005
New Orleans, Louisiana
University of California, Irvine School of Medicine
Désirée Lie, MD, MSED, Penny Murata, MD, John Boker, PhD, Emily Dow, MD, Elizabeth Morrison, MD, MSED

2. OSTE AND FACULTY DEVELOPMENT
Session Objectives
By the end of the session, participants will be able to:
Describe the OSTE and its uses.
Discuss strategies for faculty development of community preceptors.
Examine the role of the OSTE in community faculty development.
Identify ways in which the OSTE may be useful in their own programs.

3. OSTE AND FACULTY DEVELOPMENT
Session schedule
Intro and background (10 mins.)
Sample OSTE stations for comparison and discussion (20 mins.)
Results of UCI OSTE (5 mins.)
Audience feedback and suggestions (10 mins.)

4. USES OF OSTE Reported uses have been formative rather than summative
Validated as assessment for residents only
References
FM clerkship faculty (Prislin et al. Acad. Med. 1998, 73: )
Residents-as-teachers (Morrison et al Oct 2002, Acad. Med. 77:10;S29-32)

5. COMMUNITY FACULTY DEVELOPMENT
Purposes:
Orient preceptors to goals and students
Increase networking between faculty and community teachers
Improve teaching skills
Retain and recruit
Other purposes?

6. COMMUNITY FACULTY DEVELOPMENT
Strategies (discuss pros and cons of each)
CME-based teaching skills workshops
Appreciation dinners with keynote speakers
One-on one teaching consult at site visits
Mailed materials (CD-ROMs, syllabi)
Web-based programs

7. FACULTY DEVELOPMENT: OUTCOME MEASURES
How do we measure program ‘success’?
Student evaluations?
Preceptor self-ratings?
Retention and recruitment rate?
Attendance?
How often is teacher evaluation needed?
When there is a problem/complaint?
Regular interval assessment?

8. UCI COMMUNITY FACULTY PROGRAM
ECE for MS2
FM, IM and Peds community faculty
Heterogenous history of teaching
Multiple roles and learners
Need to retain and recruit
Limited rewards

9. UCI COMMUNITY FACULTY PROGRAM
Strategies (multiple)
Noon series on teaching skills (6 hours CME)
Evening and weekend EBM workshops (3 hours CME)
OSTE workshop (3 hours CME)
Site visits (teaching consults)

10. UCI OSTE Formative, used as teaching tool
Adapted cases developed for residents using validated checklists
CME-based (3 hours)
Participants were also observers
Cross-checked performance with observed real clinical teaching

11. UCI OSTE: CASES
Based on most frequent teaching encounters and baseline needs assessment of faculty:
Orienting the learner
Bedside teaching
Giving feedback
Outpatient precepting
EBM station
Other possible stations: teaching charting, the difficult learner, teaching a procedure, inpatient teaching

12. UCI OSTE Show video of two teachers
Bedside (PE) teaching case (Case C)
Show video of two teachers
Compare rating of teaching skills using checklist (audience)
Match to other observations of teaching

13. UCI OSTE RESULTS Two OSTEs offered to 17 Preceptors 2004-5
Evaluations of OSTE itself
Performance of preceptors
Correlation with other measures of teaching
Challenges
Cost effectiveness

14. Evaluation by attendee preceptors (N=17)
UCI OSTE RESULTS
Evaluation by attendee preceptors (N=17)
Question (Excellent, very good, good, fair and poor)
Excellent to Very Good responses
Overall evaluation of the activity
88%
Quality of the educational content
100%
Did this activity meet your need to upgrade your teaching skills?
Was this activity relevant to your teaching?
Did this activity increase your knowledge and/or skills in teaching patient care?
Were the objectives met? (out of 3)
88-100%

15. UCI OSTE RESULTS Comments from preceptor participants:
Question: What did you like most about the format of the program?
Ability to take different roles (Teacher vs Observer)
Workshop was primarily interacting with students (very hands on)
Question: How will you change your teaching practices as a result of this activity?
Be more observant and don’t assume what the student know
Ask more questions and have student take more responsibility for learning
Listen to medical student more and offer more positive feedback

16. UCI OSTE RESULTS Linking Preceptors and Students through Education
Summary of Case Encounters
Inter-Rater Agreement (IRA) of Ratings of Preceptor Case Performance1
Case A: Orienting the Student
Preceptor ID#
# Rater
IRA 89 3
.337 48
.447 87 2
.614 74
.781 57
.779 70
(-)2
Mean
.592
Case B: Outpatient Precepting
.504
Case C: Bedside Teaching
.439
Case D: SOAP Note Feedback
.403
Mean IRA across Cases and Preceptors3
.485
1Computed as intraclass correlation coefficient (ICC), using the consistency definition and a two-way random effects model where both measures effects and raters effects are random (SPSS 12.0 Reliability Analysis).
2Negative value resulting from negative average covariance among items.
3Computed only for obtained ICC values that were positive.

17. Inter-rater agreement on station
UCI OSTE RESULTS
Inter-rater agreement on station

18. Correlation with other measures
UCI OSTE RESULTS
Correlation with other measures
Student evaluations of preceptors
Preceptor self-ratings of teaching skills
Site visit assessments of actual teaching by UCI faculty (DL, ED and PM)

19. UCI OSTE RESULTS: CORRELATIONSID
Self-Rating (Global)
OSTE Score (Mean Item Rating)
Global Teaching Score (Site Visit) 89 4
3.44
106 3
3.81 5 73 -
3.64 2 48
4.22 87
3.90 74
3.87 57
4.81 70
4.33 83
3.37 36
3.79 29
3.78 62
4.58
102
3.39 71
3.10 67
4.11
N = 15 13 15 9

20. Motivation (CME? Networking? Fun?) Accuracy of observed performance:
UCI OSTE: CHALLENGES
Attendance:
Timing, punctuality
Motivation (CME? Networking? Fun?)
Accuracy of observed performance:
Participants also observe stations using checklists
Stations not ‘blinded’
Optimal number of stations for performance vs practice?

21. UCI OSTE: COST EFFECTIVENESS
Costs include:
Use of training center and staff
Fee for ‘standardized students’ ($75 each)
CME fees, food, parking
$$ spent:
$1500 for OSTE (10 to 16 preceptors)
$400 for food (attendees, students, faculty, staff)
Faculty time

22. UCI OSTE: UNANTICIPATED OUTCOMES
No shortage of SSs
Students enjoyed the event enormously
Preceptors pleasantly surprised by lack of didactics
New preceptors asked for preparation
Networking between students and preceptors

23. OSTE AND FAC. DEV: AUDIENCE FEEDBACK
Questions:
Would you use OSTE?
If so, in what setting?
How would you modify for your setting?
Comments?

24. OSTE AND FAC. DEV: LESSONS LEARNT
PERFORM NEEDS ASSESSMENT
AGGRESSIVE MARKETING
LOW STAKES FOR PRECEPTORS (FORMATIVE)
INCLUDE UNIVERSITY FACULTY (MIX)
NEED OTHER OUTCOME MEASURES
MAKE IT FUN!

25. OSTE AND FACULTY DEVELOPMENT
For more information:
Désirée Lie, MD, MSEd, Family Medicine
Penny Murata, MD, Pediatrics
Supported by a primary care faculty development HRSA grant from the Bureau of Health Professions, #D55HP ,