1. Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM. MD 1, Nouh T. MD 1, Boutros M. MD 1, Gagnon R. PhD 3, Meterissian S. MD 1, 2 Department of Surgery, McGill University 1 Center for Medical Education, McGill University 2 Faculty of Medicine, University of Montreal 3 Association for Surgical Education Paper Session 1 March 22, 2012

2. Disclosures The authors have no disclosures to declare

5. Current Assessment Tools MCQ Rich context MCQ Orals Short answer questions Long answer questions The perfect assessment tool?

6. Graduating general surgery residents Multiple choice examination Oral examination +

7. Institute of Applied Research in Health Sciences Education http://www.script.md/portal/en/script_en.html What is Reasoning?

8. SCT: from memorizing to reasoning New tool intended to assess mental processes in medical practice Can be used to evaluate a trainee’s approach to ill- defined problems including those encountered by expert surgeons

9. Example -2 Contra-indicated or almost totally -1 Not useful or even detrimental 0 Nor less nor more useful +1 Useful +2 Necessary or absolutely necessary

10. The Scoring Grid (Modal Experts’ Choice) Nouh, T., et al. (2012). The script concordance test as a measure of clinical reasoning: A national validation study. The American Journal of Surgery, In press

11. What have we done in the past?

12. Preliminary Study: McGill University n= 36 Meterissian, S. et al. (2007). Is the script concordance test a valid instrument for assessment of intraoperative decision-making skills? The American Journal of Surgery, 193, 248-251.

13. Pan Canadian Study n= 202 Nouh, T., et al. (2012). The script concordance test as a measure of clinical reasoning: A national validation study. The American Journal of Surgery, In press

15. Purpose To determine whether using a specialty- specific scoring key would result in a progression of scores from R1 to R5, with the R5 residents having the highest scores and therefore improving the validity of the scoring key

16. Methods

17. Participants n= 25 McGill n= 202 (Nouh, T., et al, 2012) n tot = 227

18. Specialty Specific Scoring Key Colorectal Hepatobiliary Endocrine Surgical Oncology Thoracic Trauma/ ACS EXPERTS

19. Colorectal11111 Hepatobiliary11111 Endocrine 11111 Surgical Oncology11111 Thoracic11111 Trauma/ ACS 11111 EXPERT 2EXPERT 3EXPERT 4EXPERT 5EXPERT 1 Creation of amalgamated expert exams

20. Reliability: Cronbach alpha coefficient Analysis of item-to-total item correlation used to select the best items for the final analysis Construct validity tested with a one-way ANOVA with post-hoc comparisons test and planned contrasts All p values at alpha<5% were considered significant Statistical Analysis

21. RESULTS

22. Final exam 100-question test Cronbach α = 0.81 153-question test Cronbach α = 0.67 Elimination of items with a negative item-to-total item correlation

23. Results ? Scores rising with increasing level of residency: p< 0.001

24. Number of expert panel members matters! Gagnon et al. Medical Education, 2005: The mean of residents’ scores increased with increasing panel size. “…the number of points earned on an SCT is influenced by the number of members used in the panel”.

25. Mean Score Graph Scores rising with increasing level of residency: p< 0.001

26. Juniors vs Seniors TotalN R1/ R245.1 (7.6)109 R3/ R4/ R550.4 (8.0)118 The average score of junior residents proved to be significantly lower than the average score of senior residents: p< 0.001

27. Discussion The results of this study suggest that the scoring key should be developed through the use of specialty- specific experts This may be due to the increasing subspecialization of General Surgery This has important implications in the application of the SCT on a wider level

28. Limitations We used only 5 amalgamated general surgery experts for each surgical discipline instead of the recommended >10 Gagnon et al. Medical Education 2005 The resident pool consisted mostly of the same residents from our last study with the addition of 25 new residents from our institution

29. Future implications… National wide study using the specialty- specific expert scoring key. The SCT might be ready for a national in- training formative examination.

30. Acknowledgements Dr. Sarkis Meterissian Dr. Robert Gagnon Dr. Thamer Nouh Dr. Marylise Boutros

31. Thank you