1. WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery, Ulster Hospital Dundonald, Northern Ireland 1. School of Surgery, Northern Ireland Medical & Dental Training Agency 2, Royal College of Surgeons of England 3. The Royal College of Surgeons of England Regional Representatives Meeting Wednesday 10 th November 2010

2. Introduction of a Curriculum Web based curricula Workplace Based Assessments (WBAs) Valid and reliable Formative and summative Clear indicators of professional progress GMC April 2010

3. Aims To determine if WBAs: Differentiate between trainees of different abilities (summative) Demonstrate progression towards competence (summative) Are completed at regular intervals through training (formative)

4. Methods Core Surgical trainees appointed Aug 2008 Trainees not completing 24 months excluded Trainees were ranked according to interview score (Q1, Q2, Q3 and Q4) 4 x 6 month placements (P1, P2, P3 and P4) Median (range): ANOVA

5. Methods CBDs, DOPS, Mini-CEXs – score 1-6 1-2 - Below expectations 3 -Borderline 4 -Meets expectations 5-6 -Above expectations PBAs – score 0-4 Level 0 – Insufficient evidence to support judgement Level 1 – Unable to perform procedure Level 2 – Able to perform procedure, or part under supervision Level 3 – Minimum supervision Level 4 – Unsupervised, able to deal with complications

6. Results 39 trainees 12 excluded n=27 trainees 1,256 assessments 381 CBDs 408 DOPS 346 Mini-CEXs 121 PBAs

7. p=0.39

8. Can WBAs differentiate between trainees of different abilities?

9. Trainees from Q1 would out perform trainees from Q4 number of assessments performed scores obtained

11. Can WBAs demonstrate progression towards competence?

12. Scores should show improvement over 4 placements

13. CBD p=0.47, DOPS p=0.19, Mini-CEX p=0.96

15. Are WBAs being used in formative assessment?

16. WBAs should be completed at regular intervals through training placements

18. Summary No inter-quartile differences with regards to the numbers of WBAs performed or scores obtained CBD, DOPS, Mini-CEX scores do not improve with time PBAs scores improve with time WBAs are clustered immediately prior to appraisal

19. Conclusions WBAs do not fulfil GMC requirements Need modification Assessors require to be more robust in their evaluations

22. Is one assessor type more robust than the other?

23. ANOVA p=0.42