1. Standard setting Determining the pass mark - OSCEs

2. The Old Way (1) …..I think that the pass mark in this is exam is probably about here….. there is a natural break I can see dividing the bottom from the rest

3. The Old Way (2) …..The pass mark is 60%.

4. Assessments

5. Match Outcome Objectives

6. Assessments Match Outcome Objectives Integrated

7. Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout

8. Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing

9. Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing House style

10. Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing House style From approved list

11. Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing House style From approved list Student information

12. Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing House style From approved list Student information External Examiners

13. Standard setting KnowledgeSkills

14. Angoff Hofstee Borderline method KnowledgeSkills Standard setting

15. Borderline procedure Used for clinical (OSCE) examinations

17. Borderline procedure Used for clinical (OSCE) examinations Examiners score the students’s performance at the station e.g 17/20 Examiners judge the overall performance clear pass / borderline / clear fail Mark sheets rated borderline identified and the scores of borderline students averaged Process repeated for each station Calculate the median borderline score for all stations

18. Borderline Method

19. A 1 A 2 A 3 Pass/Fail Station 15 (static) Oral Lesions 0.001.002.003.004.005.006.007.008.009.0010.00 Score 0 10 20 30 40 50 Count AAAA A A A AAAAAAAAAA A A A A A

20. Borderline group pass fail Median Borderline score 58.41

21. Borderline group pass fail Median Borderline score 58.41 How wide should this band be?

22. Borderline group pass fail Median Borderline score 58.41 How wide should this band be? +/- 1 - standard deviation - standard error - or what?

23. The Standard Error of Measurement depends on the reliability of the test (R) depends on the standard deviation of the test (SD) SEM = SD  1 - R acts as a confidence interval in high stakes situations

24. The Standard Error of Measurement depends on the reliability of the test (R) depends on the standard deviation of the test (SD) SEM = SD  1 - R acts as a confidence interval in high stakes situations R = Variance Variance x Error

25. Borderline group Pass Fail Median Borderline score 58.41 Pass Score 60.93 } - 1 SEM Fail score 55.88 } + 1 SEM

26. Number of students Mean Score (%) Overall Borderline Score (%) Standard deviation Standard Error of Measurement Pass mark (%) Reliability 0419573.2558.285.502.4060.680.81 0321574.458.415.772.5260.930.81 0221473. 059. 814.952.9862.790.64 0119571.8558.905.863.2162.110.70

27. Internal Reliability of the Exam Cronbach’s Alpha Cronbach’s alpha Shows whether randomly split halves of the exam by item vary together Split half Correlations (SH) Item / Stations  Population (students) 

28. Borderline Grading System Borderline Fail Pass Good Pass Top Mark Borderline Mark Distinction Pass Mark Pass Fail

29. Reducing measurement error 1. Increase reliability of exam 2. Compromise with feasibility/cost * Content specificity * More items in the exam * Blueprinting * Quality assurance of item writing * Training examiners/standardised patients * Feedback from exam performance