Standard setting Determining the pass mark - OSCEs

The Old Way (1) …..I think that the pass mark in this is exam is probably about here….. there is a natural break I can see dividing the bottom from the rest

The Old Way (2) …..The pass mark is 60%.

Assessments

Match Outcome Objectives

Assessments Match Outcome Objectives Integrated

Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout

Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing

Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing House style

Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing House style From approved list

Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing House style From approved list Student information

Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout Progressive testing House style From approved list Student information External Examiners

Standard setting KnowledgeSkills

Angoff Hofstee Borderline method KnowledgeSkills Standard setting

Borderline procedure Used for clinical (OSCE) examinations

Borderline procedure Used for clinical (OSCE) examinations Examiners score the students’s performance at the station e.g 17/20 Examiners judge the overall performance clear pass / borderline / clear fail Mark sheets rated borderline identified and the scores of borderline students averaged Process repeated for each station Calculate the median borderline score for all stations

Borderline Method

A 1 A 2 A 3 Pass/Fail Station 15 (static) Oral Lesions Score Count AAAA A A A AAAAAAAAAA A A A A A

Borderline group pass fail Median Borderline score 58.41

Borderline group pass fail Median Borderline score How wide should this band be?

Borderline group pass fail Median Borderline score How wide should this band be? +/- 1 - standard deviation - standard error - or what?

The Standard Error of Measurement depends on the reliability of the test (R) depends on the standard deviation of the test (SD) SEM = SD  1 - R acts as a confidence interval in high stakes situations

The Standard Error of Measurement depends on the reliability of the test (R) depends on the standard deviation of the test (SD) SEM = SD  1 - R acts as a confidence interval in high stakes situations R = Variance Variance x Error

Borderline group Pass Fail Median Borderline score Pass Score } - 1 SEM Fail score } + 1 SEM

Number of students Mean Score (%) Overall Borderline Score (%) Standard deviation Standard Error of Measurement Pass mark (%) Reliability

Internal Reliability of the Exam Cronbach’s Alpha Cronbach’s alpha Shows whether randomly split halves of the exam by item vary together Split half Correlations (SH) Item / Stations  Population (students) 

Borderline Grading System Borderline Fail Pass Good Pass Top Mark Borderline Mark Distinction Pass Mark Pass Fail

Reducing measurement error 1. Increase reliability of exam 2. Compromise with feasibility/cost * Content specificity * More items in the exam * Blueprinting * Quality assurance of item writing * Training examiners/standardised patients * Feedback from exam performance