Assessment of Professionals M. Schürch

How do you assess performance? How do you currently assess the performance of your residents? What standards do you apply? What evidence can you provide? Is the evidence robust enough? Could the resident challenge your assessment?

Outline Principles of assessment Methods of assessment What standards do apply Assessment in the new curriculum Conclusion

Why assess? Provide direction and motivation for future learning: assessment drives learning … Screening out those who are incompetent Meet public expectations Choose applicants for advanced training

What to assess? Knowledge Skills –technical –non-technical Attitude

How to assess? Here is the problem Depends on why and what!

Formative vs summative assessment Formative assessment –guides future learning –provides reassurance –promotes reflection –shapes values Summative assessment –overall jugement of competence –fitness to practice –qualification for advancement can: reinforce intrinsic motivation inspire to set higher standards

Formative vs summative assessment Formative assessment –guides future learning –provides reassurance –promotes reflection –shapes values Summative assessment –overall jugement of competence –fitness to practice –qualification for advancement can: reinforce intrinsic motivation inspire to set higher standards

Methods of assessment Simulation –standardized patients –computer based simulation computer programms, high fidelty simulators Work-based assessment –case based discussion –mini-clinical evaluation exercise (mCEX) –direct observation of … (DOPS, DOCE) –multisource (360°) assessments

Hierarchy of professionalism Competent Knowledge Technical Skills Non-Technical Skills Pro- fessional Behaviour MCQ VIVA OSCE VIVA MCQ: multiple choice questions OSCE: objective structured clinical examination VIVA: oral exam

Hierarchy of professionalism Competent Knowledge Technical Skills Non-Technical Skills Pro- fessional Behaviour mini- CEX CBD DOCE MSF (RET?) mini-CEX: clinical evaluation exercise CBD: case based discussion DOCE: direct observation of clinical encounter MSF: multi source feedback

Criteria to choose methods Reliability accurate, reproducible Validity measures what it claims to Impact on future learning and practice Acceptability to learners and faculty fair, feasible Costs trainee, institution, society

What standards do apply? Limen referenced –Written exam, MCQ –Standardized patients, simulation Relying on assessors judgement of competency against expectation at a particular level of training –Workplace based assessment Patient difficulty Faculty stringency Equivalence of assessments?

Challanges of work based methods No absolute standards –most trainees will be satisfactory Alternative assessments for those who are unsatisfactory Importance of selection of assessors Equivalence –not useful for ranking Trainees  Multiple assessments by multiple assessors

Who should assess? „ … one of the capabilities of a competent individual is the skill of recognising competence in others.“ Ronald Burnett: The Limits of Competence 1991

Assessment in the new Curriculum

defines competencies and standards

Assessments in the curriculum Formative Summative

Formative assessments

DOCE

RET (Resident Evaluation Tool) the Resident Evaluation Tool RET (Anästhesie Basel, Abbott) can be used to give Multisource-Feedback RET combines items of mini-CEX and ANTS (non technical Skills)

Example of a rating, that needs an action plan

Example of an excellent rating

Conclusion Assessment is essential for learning Good assessment is a challenge

„ … one of the capabilities of a competent individual is the skill of recognising competence in others.“ Ronald Burnett, 1991