1. SETTING & MAINTAINING EXAM STANDARDS
Raja C. Bandaranayake

2. NORM- & CRITERION-REFERENCED STANDARDS
Absolute
Not related to peer performance
Standard set prior to exam
Referenced to a defined level of performance
NORM-REFERENCED
Relative
Based on peer-performance
Varies with each group
Cut-off point not related to competence

3. NEDELSKY (1954) METHOD: Example
Consider N judges and n MCQ items of 1 in 5 type
Judge A identifies 2 options in item 1 as those which a minimally competent examinee should eliminate as incorrect.
MPL for that item for Judge A [MPLA1] = 1/(5-2) = 1/3
Similarly, in item 2 he identifies 3 options, giving an MPLA2 = 1/(5-3) = 1/2
He repeats this process for each item.
The exam MPL for Judge A [MPLA] = MPLA1 +MPLA2 + MPLA3 + ………….MPLAn
Similarly, Judge B’s MPL [MPLB] is determined
The MPL for the exam (= cut-off score) is: (MPLA + MPLB + MPLC +….MPLN) / N

4. ANGOFF (1971) METHOD Example
N judges consider 100 minimally competent examinees taking an MCQ exam of n items.
Judge A estimates that, of these examinees, 50 should answer item 1 correctly, 20 item 2 correctly, 70 item 3 correctly, and so on to item n.
The MPL for Judge A [MPLA] = ( xn) / n X 100 = (say) A%.
Similarly, for Judges B, C, D, E, …..N, the MPLs would be B%, C%, D%, E% ……N%, respectively.
The MPL (cut-off score) for the exam is: (A% + B% + C% + D% + E% +....N%) / N

5. EBEL (1972) METHOD Example
Assume that Judge A assigns items in a 200-item MCQ test to the cells of a “relevance-by-difficulty” matrix, as follows.
He then estimates the percentage of items in each cell of the matrix that a minimally competent examinee should be able to answer correctly (as indicated within the cell).
Each cell also includes the products of these two values.
EASY MEDIUM HARD
ESSENTIAL x 100% = x 80% = x 60% = 600
IMPORTANT x 80% = x 60% = x 50% =1000
ACCEPTABLE x 50% = x 40% = x 10% = 50
QUESTIONABLE 10 x 30% = x 20% = x 0% = 0

6. EBEL (1972) METHOD - contd. Example
The MPL for Judge A [MPLA] is then:
( ) / 200 = %
Similarly, the MPL for Judges B [MPLB], C [MPLc], D [MPLD] …..N [MPLN] are determined.
The MPL for the exam (cut-off score) is:
(MPLA+ MPLB+ MPLc+ MPLD + …..MPLN) / N

7. PROPOSED EBEL MODIFICATION. EASY. MEDIUM. HARD ESSENT. 6x 100% = 600
PROPOSED EBEL MODIFICATION EASY MEDIUM HARD ESSENT. 6x 100% = x 80% = x 50% = IMPORT x 80% = x 60% = x 40% = ACCEPT x 60% = x 50% = x 10% = MPL: = =6000/100 = 60

8. HOFSTEE METHOD fmin fmax cmin cmax A B Failure Rate% Cut-off score(%)10 15 20 35 40 45 50
fmin
fmax
cmin
cmax A B

9. HOFSTEE METHOD
Example
A plot of cut-off scores for a given exam against resulting failure rates is given
cmin = 40%
cmax = 45%
fmin = 10%
fmax = 20%
A = point representing cmin,fmax
B = point representing cmax,fmin
Line AB intersects the curve at a cut-off point of 42.5%
Thus, operational cut-off score = 42.5%

10. CUT-OFF SCORE FOR 1 IN 5 MCQ [FRACS PART 1]
Probability of guessing (=1 in 5) = 20%
‘Total ignorance’ score = 20%
Maximum possible score =100%
Effective range of scores = 20% to 100%
Mid-point of this range = 60%
Additional factor (as PG exam) = 5%
Nominal cut-off score (60%+5%) = 65%

11. CUT-OFF SCORES: “MARKER QUESTIONS”
1. Comparison of exam scores
Mean score in this exam: %
Average exam mean score over last 4 years: 59.4%
Thus mean score in this exam is: % lower
Assuming this candidate group is of same
standard as in last 4 yrs, this exam is: % harder

12. CUT-OFF SCORES: “MARKER QUESTIONS” - contd.
2. Comparison of “marker” scores
Mean score in this exam on previously used
questions (N=162): %
Mean score on same questions when they
were each last used: %
Thus, when compared with previous
candidates, this group of candidates, on
these items, scored ( )% = 2.0% higher
Thus this group of candidates is: % better than previous groups

13. CUT-OFF SCORES: “MARKER QUESTIONS” – contd.
3. Estimating examination difficulty
Thus it is expected that their mean score
in this exam would be: 2.0% higher
But their mean score in this exam is: 2.7% lower
Thus this exam is really: % harder

14. CUT-OFF SCORES: “MARKER QUESTIONS” –contd.
4. Determining cut-off score
The cut-off level for an average exam is: 65.0%
Thus the cut-off level for this exam
should be (65 – 4.7)% = %
Cut-off score = 60.3%

15. HOFSTEE CURVE
Failure
Rate%
Cut-off score(%) 10 15 20 55 60 65 70