1. Using Data to Drive Decision Making:
Scantron Tests

2. Agenda Assessment terminology Response rate data
Overall Multiple Choice (MC) test item statistics
Correct Responses as a Percentage
Item discrimination
Distractor analysis
Review test items

3. Test Results: Other aspects I want to know…

4. Terminology Multiple Choice Question Mechanics
Item
Multiple Choice Question Mechanics
A 30-yr old man presented with a 4-month history of dyspnea, low-grade fever, cough, and fatigue.
Given the following chest radiograph, what is the most likely diagnosis?
pulmonary edema
streptococcal pneumonia
adult respiratory distress syndrome
pulmonary alveolar proteinosis
Stem
Keyed response
Options
Distractors

5. Using Data Different forms of assessment yield varyied types of data.
The most common output generated by assessment tools are:
response rates
achievement mastery levels

6. Response Rate Data Scantron results from multiple choice assessments
Overall test statistics
Correct response rates as percentages
Item discrimination
Distractor analysis

7. Overall MC Test Statistics
Mean, median, and mode
Skewness
Kurtosis
Reliability
Instructor: Wyle E. Coyote
Class: Intro. to Epidemiology
Time/day: 3/20/10
Test: Exam 1
Form: A
Total Possible Points:
Students in this Group:
Mean score:
Median Score:
Mode Score:
Variance:
Standard Deviation:
Measure of Skewness:
Degree of Kurtosis:
Highest Score:
Lowest Score:
Reliability Coefficient (KR20):
Standard Error of Measurement:

8. ParScore Reliability Range
.90 or higher - High reliability Suitable for making a decision about an examinee based on a single test score.
.80 to Good reliability Suitable for use in evaluating individual examinees if averaged with a small number of other scores of similar reliability.
.60 to Low to moderate reliability Suitable for evaluating individuals only if averaged with several other scores of similar reliability.
.40 to Doubtful reliability Should be used only with caution in the evaluation of individual examinees. May be satisfactory for determination of average score differences between groups.

9. Correct Responses as a Percentage
Question difficulty is the percentage of the total group of students who
answered the item
correctly.
Item#
Total (responses in) Group
Upper and Lower 27% of Group
Correct Responses as a Percentage of
Item#
Total Group
Upper 27% of Group
Lower 27% of Group 1
100 2 20 25 3 40 50

10. Correct Responses as a Percentage of
Item Discrimination
Discrimination measures the effectiveness of an item.
Discriminates between those who have mastered the material and those who have not.
Indices:
< 0.00 (Negative) Unacceptable – check item for error
0.00 – Room for improvement
0.25 – Good item
Excellent item
Correct Responses as a Percentage of
Discrimination
Item#
Total Group
Upper 27% of Group
Lower 27% of Group 1
100
0.00 2 20 25
0.33 3 40 50
-0.02
Review this Item!

11. Distractor Analysis – Pt. I
Alt. = correct answer
% correct = percentage of the class who selected a particular distractor
Biserial (Biser): correlation between correct answer among all answers for that item
Pt. Biserial (Biser): correlation between score of item and total score on test

12. Distractor Analysis – Pt. II
Point Biserial (Pt-Biser.): correlation between the score of an item and the total score on a test.
details how well an item predicts student performance on the entire exam by comparing how well students did answering one item, relative to how well they did answering all the items.
Scale Range
Indication
.30 or above
excellent distractor
.20 to .29
reasonably good distractor
.09 to .19
needs improvement
below .09
poor distractor

13. Point Biserial
If it is a low positive or negative it can be used to identify problematic areas such as:
A questionable correct answer
> 1 correct answer
No real correct answer
An ambiguous or confusing question stem

14. Correct Responses as a Percentage of
Discrimination
Distractor Analysis
Item#
Total Group
Upper 27% of Group
Lower 27% of Group
Alt.
% Correct
Biser.
Pt-Biser. 1
100
0.00 A 0% *B
100.00 C D E 2 20 25
0.33
60%
-0.10
-0.08
20%
0.48
13%
-0.65
-0.41 7%
0.34
0.18 3 40 50
-0.02
0.57
0.36
Review this Item! B
47%
-0.28
-0.22 *D
40%
-0.03

15. Reviewing Test Items - handout
What is problematic about the item?
How does the item need to be revised?

16. Timing Review results prior to student release
Assess item effectiveness
Consider item discard or revision
Highlight problematic items and seek input
Colleagues
Students
CLD Assessment Specialist

17. More Information… Using Data
Christine Savi, PhD
CLD & QEP: Assessment
If you have any questions pertaining to Using Data, please contact the Center for Learning and Development.