September 23, 1999COMET1 How to convince your friends NOT to misuse raw scores Benjamin D. Wright Institute for Objective Measurement & MESA Psychometric Laboratory

September 23, 1999COMET2 THE TROUBLE WITH RAW SCORES The BROKEN BUCKET of Missing Data The CRUMBLING CATEGORIES of Lumpy Ratings The DIRTY DATA of Unpredictable Responses The PERVERSE PRECISION of Extreme Scores The RUBBER RULER of Irregular Intervals and Squashed Extremes

September 23, 1999COMET3 The BROKEN BUCKET of Missing Data Compare Two Patients on an 8-item, 7-category Functional Independence Measure: –Patient A: m m m m 4 = 12 –Patient B: = 16 Which Patient is more Able? Patient B has the higher score 16 > 12 on all 8 items, BUT On the 4 Items A and B have in common –Patient A: m m m m 4 = 12 –Patient B: m m m m 3 = 8 Patient A has the higher score of 12 > 8 Are you sure you want to misuse missing-data- leaking raw scores for missing-data-impervious measures?

September 23, 1999COMET4 The CRUMBLING CATEGORIES of Lumpy Ratings Rating Forms Offer Equally Spaced Categories – But Raters Reply with Unequally Spaced Responses – The Measure Distance of One More Point from Category 1 to 2 can be FOUR times BIGGER than The Measure Distance of One More Point from Category 3 to 4 !! Are you sure you want to mistake Lumpy Ratings for Equal Interval Measures?

September 23, 1999COMET5 The DIRTY DATA of Unpredictable Responses When Item Responses are Arranged from Easy Items to Hard Items you Expect Response Patterns like: – = 46 and = 19 BUT suppose you get: – {1}5 5 4 = 41 ? or {7}1 1 1 = 24 ? What then? Raw Scores are Blind to Unpredictable Responses. Only Quality Control of Well-Constructed Measures Tells you about Response Surprises Are you sure you want to suffer raw-score-dirty-data blindness instead of enjoying data-vigilant measures?

September 23, 1999COMET6 The PERVERSE PRECISION of Extreme Scores The Statistical Precision of a Raw Score is MAXIMUM at exactly the place where the Information a Raw Score Provides is MINIMUM When a person gets the lowest possible score, their raw score precision is perfect. We know exactly the score their low ability implies. But we have no idea how far Below that Score their ability might be! It is the same with the highest possible score. We know exactly the score their high ability implies. But we have no idea how far Above that Score their ability might be! They are Off-Our-Scale and Our Precision for their Unknown Measure is ZERO! Are you sure you want to mistake imprecise raw scores for precise measures?

September 23, 1999COMET7 The RUBBER RULER of Irregular Intervals and Squashed Extremes When our items bunch in clumps of equally difficult items then a count of one more right answer within a clump implies only a little increase in our ability. But when we leap ahead and our next right answer is in a distinctly harder clump, then we see that this one more right implies a large increase in our ability. As for the ends of the test where one more right is from 0 to 1 or from all but one to all. Then the implied change in our ability is infinite.

September 23, 1999COMET8

September 23, 1999COMET9 WHAT ARE VARIABLES? Length and weight may be real variables. But we construct their units of measure. Inches and ounces are our creations - Our own imaginative constructions. A variable is an amount of something which we can always picture as a distance >From Less > To More We can arrange to experience evidence of this "something". But its measurement line and its units of measurement are up to us to construct.

September 23, 1999COMET10 EVIDENCE OF A VARIABLE? The variable and its evidence could be: –length benchmarks exceeded –health symptoms absent –ability problems solved –skill tasks completed –attitude assertions condoned We can arrange to provoke occurrences of evidence and count how many pieces occur. But these counts are not measures.

September 23, 1999COMET11 REQUIREMENTS FOR MEASUREMENT Pieces of evidence must be concrete to be observed. This necessary reality keeps them uneven in size. To measure we need an even abstraction, a line marked out in abstractly equal units. Pieces of evidence are unstable. They appear and disappear by accident. They are only probable signs of the variable which they are designed to manifest. To measure, we must find a way to connect the pieces of evidence we can arrange to observe to the probabilities of the measures we want.

September 23, 1999COMET12 WHAT IS MEASUREMENT? DISTANCE (Length) was our First Variable COUNTING Steps and Fingers was our First Measuring Operation The Trouble with Counting is its UNEQUAL UNITS How many apples fill a basket? How many oranges squeeze a glass? You may not believe it. But can we mix apples and oranges? We do it all of the time, by WEIGHING them!

September 23, 1999COMET13 CONSTRUCTING MEASURES But weighing is a constructed abstraction.There are no tangible equal units. We have to invent them. Equal feet are abstracted from real feet. Equal pounds are abstract real weights We construct our instrumentation of the variables: length and weight to approximate units equal enough to serve our practical purposes We measure so that we can use the past to plan and navigate the future. But the future is by definition UNCERTAIN

September 23, 1999COMET14 HANDLING UNCERTAINTY Imagine two batters: Smith bats 400 and Jones bats 200 So which one will hit at their next batter-up? No way to know ahead of time. Even Smith has only a 4 out of 10 record. We can’t wait to find out. So which one shall we send to the plate? Smith's odds for a hit are 2/3; Jones' odds for a hit are only 1/4 Smith odds for a hit are 8/3 times better than Jones'. Even though we know nothing for sure, Does any doubt remain as to who to send to bat? That's how we handle uncertainty. We use past experience to estimate PROBABILITIES and use these probabilities to forsee the future.

September 23, 1999COMET15 COUNTING ABSTRACT UNITS To finish this job we have to construct a reproducible transition from counting concrete events, like right answers, observed or reported symptoms, relative agreements, frequency or importance categories to counting abstract units of equal size and wide generality. How can we do this?

September 23, 1999COMET16 INVERSE PROBABILITY To deal with the uncertainty we ask Bernoulli, Bayes and Laplace and interpret our observation X as evidence of its occurence probability Px To construct unit equality we ask Campbell, Thurstone, Rasch and Luce & Tukey and define Px to satisfy the equation: log[Pnix/(1-Pnix)] = Bn - Di –Pnix is the probability of a successful response Xni being produced by person n to item i –Bn is the ability of person n –Di is the difficulty of item I The construction of equal size and hence additive units is called CONJOINT ADDITIVITY

September 23, 1999COMET17 For situations where Xni occurs in incremental steps such as Xni = 0,1,2,3,,,M This simple solution generalizes to –log[Pnix/Pnix-1] = Bn - Di - Fix For situations where Xnijk = 0,M occurs as the result of –a Rater j rating the performance of –a Person n on –a Task k This MEASUREMENT MODEL becomes –log[Pnijkx/Pnijkx-1] = Bn - Di - Cj - Ak - Fix CONJOINT ADDITIVITY

September 23, 1999COMET18 CLOSING THE DEAL Raw scores cause problems with: –missing data –lumpy ratings –unpredictable responses –extreme scores –irregular intervals and squashed extremes Rasch measurement provides a solution to these problems by: –abstracting units of measurement –using probabilities to predict futures –constructing equal-sized intervals