1. Competency and Performance: What Should We Measure and How
Zubair Amin
Assoc Professor; Dept of Pediatrics
Senior Consultant; Dept of Neonatology
National University of Singapore
National University Hospital

2. Agendas of the Talk
Is there a difference between competency and performance?
Why do we need to assess competency and performance?
How should we assess performance in our own context? What are the models and methods available for the performance measurement?

3. Areas of Confusion
To perform: “To portray a role or demonstrate a skill before an audience”
(Nelson’s Canadian Dictionary. Toronto: Nelson 1997; cited in Hodges B; Medical Education. 2003)
“Performance-based assessment”
Clinical examinations: Short cases, long cases
OSCE

4. Areas of Confusion
Miller G. The assessment of clinical skills/competence/performance. Acad Med

5. Areas of Confusion
Miller G. The assessment of clinical skills/competence/performance. Academic Medicine

6. Competency versus Performance
What doctors “can do” in controlled representations of professional practice
What a doctor is capable of doing
Performance:
What doctors “do” in actual professional practice
Assessment of day-to-day practices undertaken in the working environment
Rethan et al. Medical Education. 2002

7. Competency Performance What
List of attributes required for good practice
Those attributes observed in practice as behavior
Why
To prove that professionals have attributes for practice
To prove that professionals apply those attributes in their professional practice
How
Measures what can be shown in controlled representation of professional practice
Measuring that professionals are applying competency in their practice
Tools
Range of competency measures
Range of competency measures PLUS a sample of measures in practice
Adopted from Lambert Schuwirth

8. Challenges for Performance Assessment
Isolating outcomes that are directly attributable to a doctor
In complex healthcare delivery, outcomes of the patients are not solely attributable to an individual
Variability in the complexity of the patients
Complexity of the patient varies, more competent doctors tend to see more complex patients with multiple co-morbidities
“Lake Wobegon Effect”
“Where all the women are strong, all the men are good looking, and all the children are above average.”
Conventional global ratings of supervisors are hopelessly unreliable (Streiner 1995; Gray 1996)
No universally accepted norms for standard of care
Numbers are approximation and guess works

9. Why do we need to assess competency and performance?
Knowing that performance assessment is more challenging, does it add any additional value to assessment of competency?

10. Need for Competency and Performance Measurement
Differences exist between what doctors can do in high stake controlled environments and what they actually practice in real-life working environments.
Correlation between competency (testing conditions) and performance (real practice) varies from being negative to moderate to high.
(Rethans J-J et al. Med Ed. 2002)
Performance assessment can have a unique developmental role.

11. Relationship between Competency and Performance
Set-up: Family Medicine Practice
Four representative cases: tension headache, acute diarrhea, shoulder pain, and NIDDM
Standard domains: history, physical examination, guidance and advice, treatment, and return visit
Obligatory, intermediate, and superfluous
Rethans J-J. BMJ

12. Rethans J-J. BMJ

13. Obligatory Score Intermediate Score Superfluous Score Total Score
Time (min)
Aggregate
Performance
Competency
Max 68
37.1
49.1
Max 29
8.5
12.4
9.1
20.3
54.7 (10.1)
81.8 (11)
38.7
55.6
Headache
Max 13
9.0
10.6
Max 12
4.1
5.6
2.7
5.2
15.8 (2.8)
21.5 (4.2)
11.6
17.1
Diarrhea
Max 15
8.7
12.1
Max 5
2.5
2.9
1.9
3.2
13.1 (4.0)
18.2 (2.4)
6.8
9.7
Shoulder Pain
Max 19
12.3
14.2
Max 6
1.8
5.1
16.6 (4.3)
22.0 (3.4)
8.0
Diabetes
Max 21
7.1
0.2
1.2
2.0
9.3 (4.0)
20.1 (7.9)
16.7
Competency scores are always higher than performance scores. Time taken for performance-based assessment is lower.

14. We are more efficient in real practice
We are more through and express more compassion under testing conditions
We play by the rule!
We spend less time with patients in actual practice than what we do in tests
We take short cuts!
We are more efficient in real practice
We use experience to guide clinical decision making!

15. Assessment of Clinical Competency Using OSCE
Participants: Students, Resident, and Physicians
Naturalistic instruction: “Do what you normally do. This is not an examination. We are trying to understand how you function under normal condition.”
OSCE instruction: “This is an OSCE station I which your questions and interactions are recorded in detail on checklist.”
Hodges et al. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999

16. Assessment of Clinical Competency in OSCE with Checklist
Hodges et al. Acad Med. 1999

17. Assessment of Clinical Competency in OSCE with Global Scores

18. Students are more through during their examination compared to expert clinicians
Many of us would fail in the clinical examinations that we create for our students!
In ‘subjective’ global rating expert clinicians would do better than the students
Checklists do not capture many attributes that we associate with clinical expertise!

19. Developmental Aspects of Performance Assessment
Typical examination set-ups rarely provide opportunities for feedback and improvement
Practice-based performance assessments provide un-paralleled opportunity for direct observation and feedback

20. Competency and Performance Assessment
Methods
Well-defined
Evolving
Set-ups
Controlled
Naturalistic
Context
Limited
Highly variable
Players
Examinee, Examiners, Patients
Multiple; includes peers, patients, and other HCW
Scale
Quantitative
Qualitative/Descriptive
Focus
Judgmental
Developmental/Monitoring

21. How should we assess performance in our own context?
What is the practical way of assessing performance?

22. Screening Program (all) with Moderate Rigor
Good Performers (majority)
with
Minimal Rigor
Poor Performers (minority)
with
Maximum Rigor
Rethans et al. Med Ed. 2002

23. Screening Program (all) Comprehensive, periodic review;
most feasible performance and
competency assessments
Good Performers (majority)
Choice, Reflection,
continuous quality improvement
Poor Performers (minority)
Diagnostic tests, remediation, rehabilitation
Rethans et al. Med Ed. 2002

24. General Principles in Performance Assessment
Have a higher tolerance for more subjective, experts’ judgments
Always take into the account unique contextual variables of the practice
Focus on the holistic profiling rather than individual instruments
Emphasize feedback and developmental aspects of performance assessment