1. The ABC’s of Pattern Scoring Dr. Cornelia Orr

2. Slide 2 Vocabulary Measurement – Psychometrics is a type of measurement Classical test theory Item Response Theory – IRT (AKA logistic trait theory) 1, 2, & 3-parameter IRT models Pattern Scoring

3. Slide 3 General & Specialized Measurement Assign numbers to objects or events Ex. – hurricanes, earthquakes, time, stock market, height, weight Psychometrics Assigning numbers to psychological characteristics Ex. – achievement personality, IQ, opinion, interests

4. Slide 4 Different Theories of Psychometrics Classical Test Theory Item discrimination values Item difficulty values (p-values) Guessing (penalty) Number correct scoring Item Response Theory a)Item discrimination values b)Item difficulty values c)Guessing (pseudo- guessing) values Pattern scoring Similar constructs – Different derivations

5. Slide 5 Different Methods of Scoring Number-Correct Scoring Simple Mathematics Raw scores (# of points) –Mean, SD, SEM, % correct Number right scale Score conversions –Scale scores, percentile ranks, etc. Pattern Scoring Complex Mathematics Maximum likelihood estimates –Item statistics, student’s answer pattern, SEM Theta scale (mean=0, standard dev=1) Score conversions –Scale scores, percentile ranks, etc.

6. Slide 6 Comparison: Number Correct and Pattern Scoring Similarities The relationship of derived scores is the same, e.g., –High correlation, (0.95) of number right scores and scale scores –Scale score has the same percentile rank for both methods Differences Methods of deriving scores The number of scale scores possible –Number right = limited to the number of items –IRT = unlimited or is limited by the scale (ex. 100-500)

7. Slide 7 Choosing the Scoring Method Which model? Simple vs. Complex? Best estimates? Advantages/Disadvantages? Ex. – Why do the same number correct get different scale scores? Ex. – Flat screen TV – how do they do that?

8. Slide 8 Advantages of IRT and Pattern Scoring Better estimates of an examinee’s ability –the score that is most likely, given the student’s responses to the questions on the test (maximum likelihood scoring) More information about students and items are used More reliability than number right scoring Less measurement error (SEM)

9. Slide 9 Disadvantages of IRT and Pattern Scoring Technical - Complex Mathematics – –Difficult to understand –Difficult to explain Not common – Not like my experience. Perceived as “Hocus Pocus”

10. Slide 10 Item Characteristic Curve (ICC)

11. Slide 11 Examples Effect Of Item Difficulty No Type a b c 1 1 MC 0.0150 250.000 0.1 2 1 MC 0.0150 275.000 0.1 3 1 MC 0.0150 300.000 0.1 4 1 MC 0.0150 325.000 0.1 5 1 MC 0.0150 350.000 0.1 Response Patterns (1=correct) Pattern SEM SS 12345 11100 43 300 01110 43 305 10101 43 305 10011 43 310 Answering more difficult items (b-parameter) can result in higher scores.

12. Slide 12 Examples 5 Items (Effects of Item Discrimination) No Type a b c 1 MC 0.0050 300.000 0.2 2 MC 0.0100 300.000 0.2 3 MC 0.0150 300.000 0.2 4 MC 0.0200 300.000 0.2 5 MC 0.0250 300.000 0.2 Response patterns (1=correct) Pattern SEM SS 12345 11001 94 280 1110061 270 01110 46 300 00111 39 305 Answering more discriminating items (a-parameter) can result in higher scores.