1. Slide 1 Authentic Performance Assessment Ellen J. Lehning, Ph.D. Department of Anesthesiology

2. Slide 2 Objectives Define clinical competence and authentic performance Define clinical competence and authentic performance Design an authentic performance assessment Design an authentic performance assessment Implement an authentic performance assessment Implement an authentic performance assessment

3. Slide 3 Teach to and Test Desired Endpoints: What is the First Principle of Education? Educational Outcomes

4. Slide 4 Pass Oral Boards Pass Written Boards Achieve Clinical Competence Desired Outcome: Assess:

5. Slide 5 What is Clinical Competence? “Integration of knowledge, skills and attitudes.” Rice et al., Competency-based objectives for clinical training. Can. J. Med. Technol. 57:136, 1995

6. Slide 6 What is Clinical Competence? “The habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.” Epstein and Hundert, Designing and assessing professional competence. JAMA 287:226, 2002

7. Slide 7 What is Clinical Competence? Miller, The assessment of clinical skills/competencies/performance. Acad. Med. 65:563-567, 1990. Knows Knows how Shows how Does Miller’s Pyramid of Competence

8. Slide 8 What is Clinical Competence? “Competence... means being able to function in context.” Chambers and Gerrow, Manual for Developing and Formatting Competency Statements. J. Dental Educ. 58:361, 1994.

9. Slide 9 How is Clinical Competence Assessed? “Competence is concerned with what people can do..., that is, potential.” While, Competence versus performance: which is more important? J. Adv. Nurs. 20:525, 1994. “Performance is concerned with what people do..., that is, what is actually done in the real-life context.” Competence is inferred from performance!

10. Slide 10 What is Authentic Performance Assessment? Performance can be simulated or authentic Performance can be simulated or authentic Simulated performance assessment uses an artificial context: Simulated performance assessment uses an artificial context: SPs, OSCEs, Long Case - Standardized or Unstandardized Authentic performance assessment uses a real context: Authentic performance assessment uses a real context: Direct observation of a resident caring for a real patient with a real problem in a real setting – Unstandardized!

11. Slide 11 Why Use Authentic Performance? “Testing should be as close as possible to the situation in which one attacks the problem.” “Ill-structured problems are not found in simulated and/or standardized tests.” “The variation inherent in professional practice will always elude capture by a set of rules.” Wiggins, Assessing Student Performance: Exploring the Purpose and Limits of Testing, Jossey-Bass, Inc. 1993

12. Slide 12 How is it Done? “There are few validated strategies to assess actual clinical practice.” “Assessment at the apex of Miller’s pyramid, the does, is the international challenge of the century for all involved in clinical competence testing.” - Ronald Epstein - Val Wass

13. Slide 13 Design Step 1 – Select Competencies Breakdown ACGME’s six global competencies into specific competencies Breakdown ACGME’s six global competencies into specific competencies Technical skills, case management skills, clinical decision-making skills, etc. Technical skills, case management skills, clinical decision-making skills, etc. Avoid too broad Avoid too broad Avoid too specific Avoid too specific Appropriate for the resident’s training level Appropriate for the resident’s training level Sequenced appropriately over the training period Sequenced appropriately over the training period

14. Slide 14 Design Step 1 – Select Competencies Write a competency statement: Write a competency statement: Chambers and Gerrow, Manual for developing and formatting competency statements. J. Dental Educ. 58:361, 1994. Verb Direct Object Qualifying conditions

15. Slide 15 Example Competency Statement Induce anesthesia for an ASA I-II patient by the end of the three month period

16. Slide 16 Example Competency Statements The Global Communication Skills Competency The Global Communication Skills Competency Communicate accurately, efficiently and supportively with the patient, patient’s family and the health care team Specific Communication Skill Competency Specific Communication Skill Competency Build a Relationship (Rapport Building) Component Skills used to Build a Relationship Establish initial rapport Support Empathy Partnership Reflection Legitimation Respect Apology

17. Slide 17 The Performance Rating Process Specific performances are judged by expert rater(s) who synthesize multiple impressions in comparison to criteria given in a rating task and filtered through the experience of the rater. Chambers, A primer on competency-based evaluation. J. Dent. Educ. 61:651, 1997. Specific performance(s) JudgmentExpert Criteria (Criterion referencing)

18. Slide 18 Design Step 2 – Select Sampling Which performances or tasks will be observed? How many observations? When?

19. Slide 19 Design Step 3 – Select Raters Raters are qualified individuals who have an opportunity to observe and draw conclusions about residents – they judge the presence and the quality of the competency to be assessed How many raters Attendings Other professional staff PatientsSelf

20. Slide 20 Design Step 4 – Select Rating Criteria Rating criteria can either be checklists or rubrics Rating criteria can either be checklists or rubrics Checklists Checklists Presence or absence of a competency or of the components of a competency Presence or absence of a competency or of the components of a competency Do not judge quality Do not judge quality May be appropriate for technical procedures or beginning level competencies May be appropriate for technical procedures or beginning level competencies May not capture sophistication of complex performances: May not capture sophistication of complex performances: Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., and Hanson, M. 1999. OSCE checklists do not capture increasing levels of expertise. Acad. Med. 74:1129-1134.

21. Slide 21 Example Checklist YesNo Competency: Built a relationship?  Specific Skills: Established initial rapport?  Employed Legitimation?  Provided support ?  Demonstrated empathy?  Demonstrated legitimation?  Demonstrated apology?  Showed respect? 

22. Slide 22 Design Step 4 – Select Rating Criteria Rubrics Rubrics Guidelines, rules, or principles by which performances are judged Guidelines, rules, or principles by which performances are judged Reflect best thinking as to what constitutes a good performance Reflect best thinking as to what constitutes a good performance Contain multiple quality levels Contain multiple quality levels All rating levels must be written-down, defined and described All rating levels must be written-down, defined and described Best rubrics contain anchor(s) to illustrate the different points on the quality scale Best rubrics contain anchor(s) to illustrate the different points on the quality scale Arter & McTighe, Scoring Rubrics in the Classroom, Corwin Press, Inc. 2001

23. Slide 23 Example Rubric Excellent – Smooth and efficient. Able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure. Competent – Lacks smoothness and efficiency but is able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure. Beginner – Lacks smoothness and efficiency. Able to manage the case but exhibits limited use of personal judgment and responsiveness to the specifics of the patient and operative procedure. Requires some limited coaching or attending intervention. Novice – Can only manage the case with extensive coaching and attending intervention.

24. Slide 24 What are the Desirable Characteristics of Performance Assessment? Face validity Low Cost Feasible Non-reactive Generalizable Valid Reliable

25. Slide 25 Psychometric Issues Write good rubrics Write good rubrics Train the raters Train the raters –Only need 1 rater/performance Sample over a broad array of cases/patients Sample over a broad array of cases/patients Our intercase agreements range from 0.2 – 0.9 Our intercase agreements range from 0.2 – 0.9 Wass et al., Generalizability in range of the OSCE Wass et al., Generalizability in range of the OSCE ACGME preliminary data shows adequate generalizability, validity and reliability ACGME preliminary data shows adequate generalizability, validity and reliability

26. Slide 26 Authentic Performances vs. Authentic Products Authentic Performances Patient Care Professionalism Communication Skills Authentic Products Patient Care Ethics PBLI Systems-Based Assessing an: Authentic Performance versus the Product of an Authentic Performance

27. Slide 27 Why Focus on Authentic Performance Assessment? ACGME outcomes mandate ACGME outcomes mandate Current assessment is rudimentary Current assessment is rudimentary Accountability Accountability Murray et al., The accountability of clinical education: its definition and assessment. Med. Educ. 34:871, 2000 Bordage et al., Education in ambulatory settings: Developing valid measures of educational outcomes, and other research priorities. Acad. Med. 73:743, 1998