1. 46-320-01 Tests and Measurements
Intersession 2006

2. Writing Items DeVellis (1991) Cultural/ethnic sensitivity Define
Item Pool
Avoid long items
Appropriate level of reading
Avoid double-barreled items
Mix positively and negatively worded items
Cultural/ethnic sensitivity

3. Item Format Dichotomous format Two alternatives
Pros: Ease of construction and scoring, absolute judgment
Cons: memorization, chance of being correct

4. Item Format Polytomous format More than two alternatives
Pros: less chance guessing, fast time, distractors
Corrected scores:
Guessing?

5. Item Format Likert format Category format Visual Analogue scale
Degree of agreement
Five alternatives vs. six
Reverse scoring
Category format
10-point scale – why 10?
Remember context
Visual Analogue scale
100 cm line

6. Item Format Checklist Q-Sort Usually adjectives Increases options (9)
Form normal distribution

7. Item Analysis
Purpose: shorten a test and increase reliability and validity
Item difficulty
Proportion who get the item correct
Probability of chance
Optimum level
Variable difficulty (0.3 to 0.7)
Internal criteria = test score

8. Discriminability Extreme group method Point Biserial method
Discrimination index
Negative discriminator
Point Biserial method
Small test n
Higher correlation, better the item

9. Discrimination Item U (20) M L Difficulty (U+M+L) Discrimination (U-L)1 15 9 7 31 8 2 20 16 56 4 3 19 18 46 10 11 37 -6 5 13 35 6 14 39

10. Table Explained Class n = 60 Discrimination: rough index = U – L
Item Difficulty: U + M + L
Items:
2 = too easy
7 = too difficult
4 & 5 = negative discriminative value

11. Further Item Analysis Response Options Item Group 1 2 3 4 5 Upper
Lower 20 16 10 9 11 7 8

12. Discrimination Index: Percentages
Percent Passing
Index of Discrimination (D)
Item
Upper
Lower 1 75 35 40 2
100 80 20 3 95 45 50 4
-30 5 55 6 7 25

13. Item Characteristic Curve
X axis: total test score (trait estimate)
Y axis: proportion of test-takers with the item correct
Often use class intervals

14. Discriminability
Best scenario

15. Item Response Theory Each item has an item characteristic curve
Specific range of difficulty can be identified with a test characteristic curve
Difficulty and discriminability
Sample items
Peaked conventional vs. rectangular conventional vs. adaptive

16. Criterion-Referenced Tests
Specify objectives – aids learning
Give test to two groups
Exposed vs. not
Antimode – cutting score
Any problems with this?

17. Test Manuals Proprietary - qualifications Nonproprietary
Standards for Educational and Psychological Testing
*reflects changes in federal law and measurement trends affecting validity
testing individuals with disabilities or different linguistic backgrounds
new types of tests as well as new uses of existing tests
* Taken from apa.org

18. Test Manuals Should include: Be critical!
How to administer (standard conditions)
How to score
How to interpret
Information on reliability, validity, norms
Be critical!

19. Base Rates and Hit Rates
What does this test contribute beyond what is already know?
Cutting score not necessarily correct decision
Hit rate vs. base rate comparison
False negatives and false positives

20. Taylor-Russell Tables
What does the test contribute beyond base?
Need
Definition of success
Base rate
Selection ratio
Test validity coefficient
Determines likelihood someone selected on basis of test will succeed

21. Taylor-Russell Tables
Source: Fisher, Schoenfeldt, & Shaw (2003), Table 7.2

22. Taylor-Russell Tables
Best: validity high, selection rate low
Bad: validity low, selection rate high
Useless: no validity
Selecting low scorers?

23. Incremental Validity Unique information from using a test
Predicting future behavior and self-ratings
Prediction should consider:
Simpler method?
Less expensive method?
Less subject strain?

24. Mental Measurements Yearbook
Test reviews