1. SETTING & MAINTAINING EXAM STANDARDS Raja C. Bandaranayake

2. DEFINITIONS Standard setting is a process of determining how much is good enough. The standard or criterion level of performance is a point on the scale of measurement at which separation of competence and incompetence occurs. Cut-score, cut-off score or passing score represents this standard on a given test for making decisions pertaining to the purpose for which the test was conducted, e.g., to certify competence.

3. ERROR IN MEASUREMENT True score is a conceptual measure indicating true extent of competence in a given subject, e.g., Anatomy. Observed score is the score assigned as a result of taking a test, say in Anatomy. The difference between true and observed scores is indicative of the amount of error in the measurement. The reliability of a test and the associated standard error of measurement are estimates of the amount of error in the measurement.

4. DECISION ERRORS False positive: passing an incompetent examinee False negative: failing a competent examinee

5. NORM- & CRITERION-REFERENCED STANDARDS NORM-REFERENCED Relative Based on peer- performance Varies with each group Cut-off point not related to competence CRITERION- REFERENCED Absolute Not related to peer performance Standard set prior to exam Referenced to a defined level of performance

6. METHODS OF STANDARD SETTING 1.Test-centred methods Standards derived from hypothetical decisions based on test content before the test is answered. 2.Examinee-centred methods Standards derived from reviewing examinees’ performance before deciding cut-off score. 3.Compromise methods Provide flexibility for adjusting the standard based on the examinees’ performance on the test.

7. NEDELSKY (1954) METHOD: Example Consider N judges and n MCQ items of 1 in 5 type Judge A identifies 2 options in item 1 as those which a minimally competent examinee should eliminate as incorrect. MPL for that item for Judge A [MPL A1 ] = 1/(5-2) = 1/3 Similarly, in item 2 he identifies 3 options, giving an MPL A2 = 1/(5-3) = 1/2 He repeats this process for each item. The exam MPL for Judge A [MPL A ] = MPL A1 +MPL A2 + MPL A3 + ………….MPL An Similarly, Judge B’s MPL [MPL B ] is determined The MPL for the exam (= cut-off score) is: (MPL A + MPL B + MPL C +….MPL N ) / N

8. ANGOFF (1971) METHOD Example N judges consider 100 minimally competent examinees taking an MCQ exam of n items. Judge A estimates that, of these examinees, 50 should answer item 1 correctly, 20 item 2 correctly, 70 item 3 correctly, and so on to item n. The MPL for Judge A [MPL A ] = (0.5 + 0.2 + 0.7 +. x n ) / n X 100 = (say) A%. Similarly, for Judges B, C, D, E, …..N, the MPLs would be B%, C%, D%, E% ……N%, respectively. The MPL (cut-off score) for the exam is: (A% + B% + C% + D% + E% +....N%) / N

9. EBEL (1972) METHOD Example Assume that Judge A assigns items in a 200-item MCQ test to the cells of a “relevance-by-difficulty” matrix, as follows. He then estimates the percentage of items in each cell of the matrix that a minimally competent examinee should be able to answer correctly (as indicated within the cell). Each cell also includes the products of these two values. EASY MEDIUM HARD ESSENTIAL 15 x 100% = 1500 25 x 80% =2000 10 x 60% = 600 IMPORTANT 20 x 80% = 1600 40 x 60% =2400 20 x 50% =1000 ACCEPTABLE 10 x 50% = 500 25 x 40% = 1000 5 x 10% = 50 QUESTIONABLE 10 x 30% = 300 15 x 20% = 300 5 x 0% = 0

10. EBEL (1972) METHOD - contd. Example The MPL for Judge A [MPL A ] is then: (1500 + 1600 + 500 + 300 + 2000 + 1000 + 300 + 600 + 1000 + 50 + 0) / 200 = 56.25 % Similarly, the MPL for Judges B [MPL B ], C [MPL c ], D [MPL D ] …..N [MPL N ] are determined. The MPL for the exam (cut-off score) is: (MPL A + MPL B + MPL c + MPL D + …..MPL N ) / N

11. PROPOSED EBEL MODIFICATION PROPOSED EBEL MODIFICATION EASY MEDIUM HARD ESSENT. 6x 100% = 600 12 x 80% = 960 7 x 50% = 350 IMPORT. 12 x 80% = 960 24 x 60% = 1440 19 x 40% = 760 ACCEPT. 5 x 60% = 300 12 x 50% = 600 3 x 10% = 30 MPL: =600 + 960 + 350 + 960 + 1440 + 760 + 300 + 600 + 30 =6000/100 = 60

12. Failure Rate % Cut-off score(%) 10 15 20 35404550 f min f ma x c min c max A B HOFSTEE METHOD

13. Example A plot of cut-off scores for a given exam against resulting failure rates is given cmin = 40% cmax = 45% fmin = 10% fmax = 20% A = point representing cmin,fmax B = point representing cmax,fmin Line AB intersects the curve at a cut-off point of 42.5% Thus, operational cut-off score = 42.5%

15. CUT-OFF SCORE FOR 1 IN 5 MCQ [FRACS PART 1] Probability of guessing (=1 in 5) = 20% ‘Total ignorance’ score= 20% Maximum possible score=100% Effective range of scores= 20% to 100% Mid-point of this range= 60% Additional factor (as PG exam)= 5% Nominal cut-off score (60%+5%) = 65%

16. CUT-OFF SCORES: “MARKER QUESTIONS” 1. Comparison of exam scores Mean score in this exam: 56.7% Average exam mean score over last 4 years: 59.4% Thus mean score in this exam is: 2.7% lower Assuming this candidate group is of same standard as in last 4 yrs, this exam is: 2.7% harder

17. CUT-OFF SCORES: “MARKER QUESTIONS” - contd. 2. Comparison of “marker” scores Mean score in this exam on previously used questions (N=162): 62.5% Mean score on same questions when they were each last used: 60.5% Thus, when compared with previous candidates, this group of candidates, on these items, scored (62.5-60.5)% = 2.0% higher Thus this group of candidates is: 2.0% better than previous groups

18. CUT-OFF SCORES: “MARKER QUESTIONS” – contd. 3. Estimating examination difficulty Thus it is expected that their mean score in this exam would be:2.0%higher But their mean score in this exam is: 2.7%lower Thus this exam is really: 4.7%harder

19. CUT-OFF SCORES: “MARKER QUESTIONS” –contd. 4. Determining cut-off score The cut-off level for an average exam is:65.0% Thus the cut-off level for this exam should be (65 – 4.7)% = 60.3% Cut-off score = 60.3%

20. Failure Rate % Cut-off score(%) 10 15 20 55606570 HOFSTEE CURVE