1. Lesson Seven Item Analysis

2. Contents Item Analysis Item Analysis Item difficulty (item facility) Item difficulty (item facility) Item difficulty Item difficulty Item discrimination Item discrimination Item discrimination Item discrimination Distrctor (efficiency) analysis Distrctor (efficiency) analysis Distrctor (efficiency) analysis Distrctor (efficiency) analysis

3. Item Difficulty (1) Also called item facility (Bailey 132-33) Also called item facility (Bailey 132-33) Difficulty index: P or IF Difficulty index: P or IF In terms of how many people got this item correct (i.e., the percentage of Ss who correctly answer a given item). In terms of how many people got this item correct (i.e., the percentage of Ss who correctly answer a given item). P= The number of correct responses divided by the number of total test takers. P= The number of correct responses divided by the number of total test takers. P= N correct / N total = (%U+ %L) / 2 P= N correct / N total = (%U+ %L) / 2

4. Item Difficulty (2) 0 ≤ P ≤ 1 (0= difficult item, 1= easy item) 0 ≤ P ≤ 1 (0= difficult item, 1= easy item) Achievement Test (CRT): P=0 or P= 1  possible Achievement Test (CRT): P=0 or P= 1  possible NRT: P=0 or P=1  means the designer has to go back and check the question NRT: P=0 or P=1  means the designer has to go back and check the question How high should the P be? How high should the P be? Depending on purpose of giving the test Depending on purpose of giving the test If for mastery of the essential  P>=.8 If for mastery of the essential  P>=.8 If for spreading people out  P should be lower If for spreading people out  P should be lower

5. Item Discrimination (1) Discrmination index: D or ID Discrmination index: D or ID Indicate the degree to which the task differentiates correctly among test takers; the difference of the percentage passing between upper group and lower group Indicate the degree to which the task differentiates correctly among test takers; the difference of the percentage passing between upper group and lower group D= % of upper group - % of lower group OR D = D= % of upper group - % of lower group OR D = Correct Answer in Upper Group – Correct Answer in Lower Group No. of people in each group (one group)

6. Item Discrimination (2) How to interpret the discrimination index? -1<= D<= 1 How to interpret the discrimination index? -1<= D<= 1 When D= 0 (i.e., %U = %L) When D= 0 (i.e., %U = %L) ok/great  if testing for mastery ok/great  if testing for mastery terrible  if testing for spreading people out terrible  if testing for spreading people out D>=.25 or.35 (Bailey 135) D>=.25 or.35 (Bailey 135) D =.7 (realistic D): probably the best D that we could get D =.7 (realistic D): probably the best D that we could get

7. Item Discrimination (3) Definition of upper/lower group: Definition of upper/lower group: Sharper differentiation when using more extreme ends (e.g., 10% top & bottom) Sharper differentiation when using more extreme ends (e.g., 10% top & bottom) But, reduce reliability of the results (because of small # of cases utilized) But, reduce reliability of the results (because of small # of cases utilized) Q: What’s the optimum percentage (used to define upper & lower group) so that the characteristics of differentiation & reliability can be maximized? Q: What’s the optimum percentage (used to define upper & lower group) so that the characteristics of differentiation & reliability can be maximized?

8. Item Discrimination (4) The optimum percentage = 27% (Anastasi 1988) The optimum percentage = 27% (Anastasi 1988) Any convenient number between 25% ~ 33% will do. (Yuan 1998) Any convenient number between 25% ~ 33% will do. (Yuan 1998) The JCEE uses 33% The JCEE uses 33% Now, do exercises on Bailey 146. Now, do exercises on Bailey 146.

9. Distractor Analysis (1) Also called distractor efficiency analysis Also called distractor efficiency analysis To investigate the degree to which the distractors are functioning efficiently (i.e., attracting Ss who don’t know the correct answer). To investigate the degree to which the distractors are functioning efficiently (i.e., attracting Ss who don’t know the correct answer). Specifically related to the multiple- choice format. Specifically related to the multiple- choice format.

10. Distractor Analysis (2) Choices A B C D* Choices A B C D* U% 20 0 0 80 U% 20 0 0 80 L% 10 40 0 50 L% 10 40 0 50 P = ? D = ? P = ? D = ? P = (.8 +.5)/2 = 1.3/2 =.65 P = (.8 +.5)/2 = 1.3/2 =.65 D =.8 -.5 =.3 D =.8 -.5 =.3 Now, look at each of the distractors. Now, look at each of the distractors. A  negatively discriminative A  negatively discriminative C  no appeal at all C  no appeal at all

11. Distractor Analysis (3) Choices A B C* D Choices A B C* D high.40.0.60.0 P=.6 high.40.0.60.0 P=.6 low.13.07.60.20 D= 0 low.13.07.60.20 D= 0 Examine the original item carefully (in terms of its format and content) Examine the original item carefully (in terms of its format and content) Which choices should be revised? Which choices should be revised?

12. Distractor Analysis (4) Choices A* B C D Choices A* B C D high.00.80.08.12 P=.5 high.00.80.08.12 P=.5 low 1.00.00.00.00 D= - 1.00 low 1.00.00.00.00 D= - 1.00 What does the result of distractor analysis tell us? What does the result of distractor analysis tell us? The item might be miskeyed? Should B be the correct answer? The item might be miskeyed? Should B be the correct answer?