Item Analysis What makes a question good??? Answer options?

Multiple Choice Questions Elements of a good multiple choice question????

Item Analysis How can we determine if it is a good question????? n Distractor power n Item difficulty n Item discrimination

Distractor power Strength of each distractor (should be equal) # answering incorrect # of distractors

E.G. n There are 4 possible answers n Option a is correct n if 9 people choose option b (9/3 = 3) n and 12 people choose option c (12/3 = 4) n and 15 people choose option d (15/3 = 5) n then option d is the strongest distractor

Item Difficulty = p (percent of people passing) p = number answering correctly total number answering n If 100 people take the test and 34 get question 1 correct, what is the difficulty of that question? p 1 = 34/100 =.34 n For 65 correct p would be 65/100 =.65 SO… the higher the p, the easier the question (score must be between 0 and 1)

Item Difficulty is... n A behavioral measure n Based on the current group n Allows internal comparison n Also allows external comparison

In construct based tests n Item difficulty is similar to item endorsement n Number agreeing with an item

Item endorsement also... n A behavioral measure n Based on the current group n Allows internal comparison n Also allows external comparison

Item discrimination = d Does the question discriminate between those who do well overall and those who do poorly? (or on a psychological test, those scoring high or low on the construct)

Distribution To determine level upper (U) = top 25 to 33 % of scores lower (L) = bottom 25 to 33 % of scores

Determining item discrimination d = U-L d = U-L n where d = item discrimination U = number in the upper group who get question correct L = number in the lower group who get question correct n = number in one group (U or L)

Examples of d n With 33 in U and L n if all in U and all in L get it correct / 33 = 0 n if all in U get it correct, and none in L / 33 = 1 n if none in U get it correct, and all in L / 33 = -1 n if 30 in U and 10 in L get it correct / 33 =.61

Value of d n Range for d = -1 to 1 n higher d indicates more discriminating (1 = all in upper group got it correct and none in lower group) n negative d indicates a really bad question (-1 indicates all in lower group got it correct and none in upper group)