1. AGS Annual Meeting May 2005 Converting the Geriatric Functional Assessment Standardized Patient Instructor …into an OSCE Karen E. Hall, M.D., Ph.D. Clinical Assistant Professor Division of Geriatric Medicine University of Michigan, Ann Arbor VAMC

2. AGS Annual Meeting May 2005 Objective Develop a CCA module that tests geriatric functional assessment Based on the Geriatric Functional Assessment Standardized Patient Instructor (GFA SPI) funded by the D.W.Reynolds Faculty Development Program Award Function-based, not disease-based Appropriate for M3 students at end of M3 year – High Stakes exam: “Must Pass”

3. AGS Annual Meeting May 2005 GFA SPI: Characteristics “Essential Skill Set” for all physicians “Essential Skill Set” for all physicians Emphasis on function, not disease Emphasis on function, not disease Validated, rapid screening tools Validated, rapid screening tools Feedback Feedback Education Education Multiple types and level of learner Multiple types and level of learner

4. AGS Annual Meeting May 2005 GFA SPI OSCE should provide: Accurate assessment of function, cognition, and affect in a geriatric patient (tasks) Accurate assessment of function, cognition, and affect in a geriatric patient (tasks) Documentation of effective communication Documentation of effective communication specialized – hearing/vision general – respect, comprehension general – respect, comprehension Objective data of performance and pass/fail standards Objective data of performance and pass/fail standards

5. AGS Annual Meeting May 2005 Steps to convert the SPI into an OSCE 1. Review scenario for reliability and validity (high stakes exam requires rigor) (high stakes exam requires rigor) Is task is appropriate for level of learner and time allotted? Err on side of “too little content” rather than “too much” Err on side of “too little content” rather than “too much” Faculty review to ensure content is clinically relevant (validity) Review checklist items with SPs to ensure performance and scoring is accurate. Faculty should observe a percentage of OSCEs (at least 5- 10%) and score using same checklist as SPs (reliability)

6. AGS Annual Meeting May 2005 Case Scenario 85-year-old male or female 85-year-old male or female Inpatient or Emergency Room Inpatient or Emergency Room “Ready for Discharge” BUT... “Ready for Discharge” BUT... Functional impairment Functional impairment Social issue: caregiver stress Social issue: caregiver stress Focus on functional assessment Focus on functional assessment not diagnosis not diagnosis

7. AGS Annual Meeting May 2005 Administration of SPI in Learning Resource Center (LRC) Total time: 1 hour Minutes Introduction 5 SPI20 Learner Survey 5 SPI feedback10 Faculty debrief15 Learner evaluation of SPI 5

8. AGS Annual Meeting May 2005 Administration of GFA OSCE in Learning Resource Center (LRC) Total time: Minutes Review case on door 3 SP OSCE10 Learner self assessment 2 Entire CCA is 4 hours with 14 stations Acute abdomen, Health beliefs, Chest pain, Chest pain note, Geriatric, Back pain, Pediatric, XRays/Imaging, EKG, Psychiatry, Evidence Based Medicine + 2 breaks

9. AGS Annual Meeting May 2005 Type of Data available from OSCE By Student: raw scores (not done=0,needs improvement=1, done=2) by item mean +/- SD of raw scores (“1.7 +/- 0.4”) percentage score (tasks; communication) – identifies pass/fail

10. AGS Annual Meeting May 2005 Type of Data available from OSCE Whole class: Histogram of scores by item (e.g. “done vs. “not done”) - Identifies deficiencies in class performance ? Curricular deficiency? Percentage for all stations “overall score” – distribution of the class: “high”; “average”; “low”

11. AGS Annual Meeting May 2005 Educational GFA SPI:M1 Evaluative GFA OSCE:M3

12. AGS Annual Meeting May 2005 OSCE: IADL Assessment Students remembered to ask about meals and grocery shopping more frequently than medications. Students often neglected to ask in enough detail to score “done”.

13. AGS Annual Meeting May 2005 OSCE: Falls, Observe Gait, Timed Up and Go Most students remember to ask about falls and observe gait, but few do objective test (TUG).

14. AGS Annual Meeting May 2005 Special Communication Educational SPI Evaluative OSCE

15. AGS Annual Meeting May 2005 Communication (9 items) Best performance (100%): “addressed by name”; “didn’t interrupt” Worst performance (60-62%): “asked open-ended questions”; ”summarized discussion”

16. AGS Annual Meeting May 2005 GFA OSCE Data from Pilot 1. Numerical percentage score on GFA 17 task items (max score =32) Converted to percentage of 100% Mean +/- SD for tasks 43 +/- 15% Communication score 84 +/- 5 %

17. AGS Annual Meeting May 2005 GFA OSCE Data from Pilot 2. Learner self-evaluation score (% of 100) on GFA: “Estimate your average score for this station” Mean +/- SD 74 +/- 12% Clearly there is a discrepancy between the student’s assessment and their performance score of 43%!

18. AGS Annual Meeting May 2005 Overall Performance (excluding GFA) Overall score on CCA 76 +/- 6% Passing average set at…64% Out of 162 students….. 62 students passed all stations 97 failed 1-3 stations – retake failed stations 3 students failed entire CCA Significant correlation between failed 2 or more stations and very low GFA scores (less than 30) – overall lack of knowledge?

19. AGS Annual Meeting May 2005 Hofstee Method to define “pass/fail” for 2005 CCA “What is the lowest % students you would accept to fail station” - usually 0% “What is the highest % students you would accept to fail station” - high for vital material, low for unimportant “What is the lowest % cut-off you would accept to fail station” – low if hard material, high if easy “What is the highest % cut-off you would accept to fail station” – high if vital

20. AGS Annual Meeting May 2005 Hofstee Method to define cut points for 2005 GFA OSCE As of March 2005 (AGS handouts due)…. Don’t know (still being analyzed – will know before May 2005) – Still don’t know! Given poor performance and curriculum revision during 2004-2006: may need generous cutoff criteria 0%-40% failure rate, 45%-60% pass cut off May be more like 10% failure, 50% cut off

21. AGS Annual Meeting May 2005 Take home points for converting an SPI to an OSCE Keep it simple Be prepared to justify your choice of material – to faculty and students (particularly those that fail!) Make sure tasks are taught in curriculum! Be prepared to review a lot of performances Keep checking the reliability of the SPs Fight off all attempts to change the OSCE checklist in mid-assessment (data will not be analyzable!)

22. AGS Annual Meeting May 2005 Reference sites Talk will be on my website: http://sitemaker.umich.edu/khallinfo GFA Student manual and pocket card on POGOe (registration required): http://www.pogoe.org/px/login.cfm I can email other material if needed: kehall@umich.edu