Competency and Performance: What Should We Measure and How Zubair Amin Assoc Professor; Dept of Pediatrics Senior Consultant; Dept of Neonatology National University of Singapore National University Hospital paeza@nus.edu.sg

Agendas of the Talk Is there a difference between competency and performance? Why do we need to assess competency and performance? How should we assess performance in our own context? What are the models and methods available for the performance measurement?

Areas of Confusion To perform: “To portray a role or demonstrate a skill before an audience” (Nelson’s Canadian Dictionary. Toronto: Nelson 1997; cited in Hodges B; Medical Education. 2003) “Performance-based assessment” Clinical examinations: Short cases, long cases OSCE

Areas of Confusion Miller G. The assessment of clinical skills/competence/performance. Acad Med. 1990.

Areas of Confusion Miller G. The assessment of clinical skills/competence/performance. Academic Medicine. 1990.

Competency versus Performance What doctors “can do” in controlled representations of professional practice What a doctor is capable of doing Performance: What doctors “do” in actual professional practice Assessment of day-to-day practices undertaken in the working environment Rethan et al. Medical Education. 2002

Competency Performance What List of attributes required for good practice Those attributes observed in practice as behavior Why To prove that professionals have attributes for practice To prove that professionals apply those attributes in their professional practice How Measures what can be shown in controlled representation of professional practice Measuring that professionals are applying competency in their practice Tools Range of competency measures Range of competency measures PLUS a sample of measures in practice Adopted from Lambert Schuwirth

Challenges for Performance Assessment Isolating outcomes that are directly attributable to a doctor In complex healthcare delivery, outcomes of the patients are not solely attributable to an individual Variability in the complexity of the patients Complexity of the patient varies, more competent doctors tend to see more complex patients with multiple co-morbidities “Lake Wobegon Effect” “Where all the women are strong, all the men are good looking, and all the children are above average.” Conventional global ratings of supervisors are hopelessly unreliable (Streiner 1995; Gray 1996) No universally accepted norms for standard of care Numbers are approximation and guess works

Why do we need to assess competency and performance? Knowing that performance assessment is more challenging, does it add any additional value to assessment of competency?

Need for Competency and Performance Measurement Differences exist between what doctors can do in high stake controlled environments and what they actually practice in real-life working environments. Correlation between competency (testing conditions) and performance (real practice) varies from being negative to moderate to high. (Rethans J-J et al. Med Ed. 2002) Performance assessment can have a unique developmental role.

Relationship between Competency and Performance Set-up: Family Medicine Practice Four representative cases: tension headache, acute diarrhea, shoulder pain, and NIDDM Standard domains: history, physical examination, guidance and advice, treatment, and return visit Obligatory, intermediate, and superfluous Rethans J-J. BMJ. 1991. 303. 1377-80.

Rethans J-J. BMJ. 1991. 303. 1377-80.

Obligatory Score Intermediate Score Superfluous Score Total Score Time (min) Aggregate Performance Competency Max 68 37.1 49.1 Max 29 8.5 12.4 9.1 20.3 54.7 (10.1) 81.8 (11) 38.7 55.6 Headache Max 13 9.0 10.6 Max 12 4.1 5.6 2.7 5.2 15.8 (2.8) 21.5 (4.2) 11.6 17.1 Diarrhea Max 15 8.7 12.1 Max 5 2.5 2.9 1.9 3.2 13.1 (4.0) 18.2 (2.4) 6.8 9.7 Shoulder Pain Max 19 12.3 14.2 Max 6 1.8 5.1 16.6 (4.3) 22.0 (3.4) 8.0 Diabetes Max 21 7.1 0.2 1.2 2.0 9.3 (4.0) 20.1 (7.9) 16.7 Competency scores are always higher than performance scores. Time taken for performance-based assessment is lower.

We are more efficient in real practice We are more through and express more compassion under testing conditions We play by the rule! We spend less time with patients in actual practice than what we do in tests We take short cuts! We are more efficient in real practice We use experience to guide clinical decision making!

Assessment of Clinical Competency Using OSCE Participants: Students, Resident, and Physicians Naturalistic instruction: “Do what you normally do. This is not an examination. We are trying to understand how you function under normal condition.” OSCE instruction: “This is an OSCE station I which your questions and interactions are recorded in detail on checklist.” Hodges et al. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999

Assessment of Clinical Competency in OSCE with Checklist Hodges et al. Acad Med. 1999

Assessment of Clinical Competency in OSCE with Global Scores

Students are more through during their examination compared to expert clinicians Many of us would fail in the clinical examinations that we create for our students! In ‘subjective’ global rating expert clinicians would do better than the students Checklists do not capture many attributes that we associate with clinical expertise!

Developmental Aspects of Performance Assessment Typical examination set-ups rarely provide opportunities for feedback and improvement Practice-based performance assessments provide un-paralleled opportunity for direct observation and feedback

Competency and Performance Assessment Methods Well-defined Evolving Set-ups Controlled Naturalistic Context Limited Highly variable Players Examinee, Examiners, Patients Multiple; includes peers, patients, and other HCW Scale Quantitative Qualitative/Descriptive Focus Judgmental Developmental/Monitoring

How should we assess performance in our own context? What is the practical way of assessing performance?

Screening Program (all) with Moderate Rigor Good Performers (majority) with Minimal Rigor Poor Performers (minority) with Maximum Rigor Rethans et al. Med Ed. 2002

Screening Program (all) Comprehensive, periodic review; most feasible performance and competency assessments Good Performers (majority) Choice, Reflection, continuous quality improvement Poor Performers (minority) Diagnostic tests, remediation, rehabilitation Rethans et al. Med Ed. 2002

General Principles in Performance Assessment Have a higher tolerance for more subjective, experts’ judgments Always take into the account unique contextual variables of the practice Focus on the holistic profiling rather than individual instruments Emphasize feedback and developmental aspects of performance assessment