Download presentation

Presentation is loading. Please wait.

Published byJaheim Hammond Modified about 1 year ago

1

2
The measurement model: what does it mean and what you can do with it? Presented by Michael Nering, Ph. D.

3
Goals of this session Present commonly used measurement models –IRT Show how these models form the backbone of any large scale assessment program –Equating –Scaling To discuss the meaning of the measurement models –Ability estimation –Item characteristics

4
Now, really …. This session is an introduction to the world of psychometrics My goal is for you to understand that psychometrics: –Is not a black box –Is really just a set of procedures

5
A little about me Yes, I’m a psychometrican B.A. in psychology at Kent State Ph. D. in psychology at Univ of Minn Working at Measured Progress since 1999 Research areas of interest: IRT, equating, scaling, person fit, adaptive testing

6

7
Why Psychology? Psychometricians typically come from: –Educational measurement programs –Psychometric programs –I/O programs Ultimately, we are all after the pursuit of understanding people by way of “quantification”

8
Psychometrics Defined Psychological Measurement Psycho metrics The business of measuring psychological “things”

9
What are psychological things? Any “latent” trait Any “characteristic” that is not directly “observable” Examples: –Depression, bi-polar, personality disorder –Math, reading, writing, science abilities We don’t care – let’s use “ ”

10
Counterparts to Psychometrics Econometrics –Measurement of economic things Sociometrics –Measurement of social things

11
All “metrics” are ultimately a blend of things Psycho- metrics PsychStatsMath

12
Quantification in Psychology Deep roots that came originally from philosophy Philosophy in the 1500s branched into several disciplines because of the need to quantify certain things to better understand human beings

13
Philosophy’s Many Branches This desire to better understand humans lead to two primary areas of study –Physiology 1543 Belgian physiologists practices the dissection of cadavers –Psychology 1524 Marco Marulik publishes The Psychology of Human Thought

14
Yes, I did just use the word “cadaver” … but trust me it’s okay

15
The last 100 years of psychometrics Classical test theory and Spearman’s 1904 contribution –True score theory –Reliability theory, p-values, point biserial coefficients Item response theory

16
Let’s talk about IRT When I say “measurement model” I really do mean some sort of IRT model Lots of historical developments Lord & Novick text of 1968 Many advantages over CTT

17
So, what is IRT? A family of mathematical models that describe the interaction between examinees and test items Examinee performance can be predicted in terms of the underlying trait Provides a means for estimating scores for people and characteristics of items Common framework for describing people and items

18
The ogive Natural occurring form that describes something about people Used throughout science, engineering, and the social sciences Also, used in architecture, carpentry, engineering, photograph, art, and so forth

19
The ogive

20

21
A little jargon The item characteristic curve (ICC) Also called: –Item response function –Trace line –Etc. Stochastic: 1) involving a random variable, or 2) involving chance or probability

22
The ICC Does this one little function really do everything? –Scale items & people onto a common metric? –Help in standard setting? –Foundation of equating? –Some meaning in terms of student ability?

23
Does this one little function really do everything? Let’s talk more about the ICC

24
The ICC Any line in a Cartesian system can be defined by a formula The simplest formula for the ogive is the logistic function:

25
The ICC Where is the item parameter, and is the person parameter The function represents the probability of responding correctly to item i given the ability of person j.

26
is the inflection point Item i i =0.125

27
We can now use the item parameter to calculate p Let’s assume we have a student with =1.0, and we have our = Then we can simply plug in the numbers into our formula

28
Using the item parameters to calculate p p = i =1.00

29
Wait a minute What do you mean a student with an ability of 1.0?? Does an ability of 0.0 mean that a student has NO ability? What if my student has a reading ability estimate of -1.2? What in the world does that mean????

30
The ability scale Ability is on an arbitrary scale that just so happens to be centered around 0.0 We use arbitrary all the time: –Fahrenheit –Celsius –Decibels –DJIA

31
Scaled Scores Although ability estimates are centered around zero – reported scores are not However, scaled scores are typically a linear transformation of ability estimates Example of a linear transformation: –(Ability x Slope) + Intercept

32
The need for scaled scores ½ the kids will have negative ability estimates

33
Scaled Scores

34
Use of scaled scores Student/parent level report School/district report Cross year comparisons Performance level categorization

35
There’s a lot here Scaled scores are surface level information Behind the scenes: –we use fancy formulas to depict interaction between students and test items –there’s a “probabilistic” relationship between students and test items

36
Unfortunately, life can get a lot worse Items vary from one another in a variety of ways: –Difficulty –Discrimination –Guessing –Item type (MC vs. CR)

37
Items can vary in terms of difficulty Ability of a student Easier item Harder item

38
Items can vary in terms of discrimination Discrimination is reflected by the “pitch” in the ICC Thus, we allow the ICCs to vary in terms of their slope

39
Good item discrimination 2 close ability levels Noticeable difference in p

40
Poor item discrimination smaller difference Same 2 ability levels

41
Guessing This item is asymptotically approaching 0.25

42
Polytomous Items

43
I’m sure by now you might be having a couple of thoughts How can I get up, open the door, and walk out without anybody noticing? I’m stuck in a “psycho”metric prison … help me!

44
But, trust me … I’m really trying to make a simple point

45
Items and people Interact in a variety of ways We can use IRT to show that there exists a nice little s-shaped curve that shows this interaction As ability increases – the probability of a correct response increases

46
Advantages of IRT Because of the stochastic nature of IRT there are many statistical principles we can take advantage of A test is a sum of its parts

47
The test characteristic curve A test is made up of many items The TCC can be used to summarize across all of our items The TCC is simply the summation of ICCs along our ability continuum For any ability level we can use the TCC to estimate the overall test score for an examinee

48
A bunch of ICCs are on a test

49
The test characteristic curve

50
From an observed test score (i.e., a student’s total test score) we can estimate ability The TCC is used in standard setting to establish performance levels The TCC can also be used to equate tests from one year to the next The test characteristic curve

51
Estimating Ability Total score = 3 Ability≈0.175

52
Standard Setting Advanced Prof. Basic Below Failing Basic

53
Equating Year 1 TCC

54
Equating Year 2 TCC & Scale

55
Equating Our 2 nd scale goes away and our TCC are closer together

56
Equating Remaining differences due to non-common items

57
Equating The adjustment to the TCC can be done a variety of different ways Let’s take a look a one commonly used method of equating, namely the Mean Shift method

58
Mean shift method of equating

59
These items are common between the two years

60
Mean shift method of equating

61
Mean shift method of equating

62
Mean shift method of equating The difference between 1 and 2 is our “scaling constant” This is used to make an adjustment to all the items administered in Year 2, so that they are then on the same scale as Year 1

63
Example = 0.20 2 = We need to add 0.30 ( 1- 2:.2+.1=.3) in order for the equating items to have the same mean This 0.30 difference is due to an arbitrary scaling difference and NOT due to any differences in ability

64
Mean shift method of equating 0.30 is then added to all the item difficulty values

65
Mean shift method of equating By shifting all our item difficulties to last years scale we are ultimately putting this year’s TCC onto last year’s scale

66
Equating The example we just saw was merely one example of an equating methods There are several methods (Kolen & Brennan) that are available

67
What have we learned? IRT: used to model interaction between items and people Item characteristics: item vary in terms of difficulty, discrimination, guessing, etc. Equating: used to relate test from one year to the next Scaling: used to represent student ability

68
The Assessment Cycle Administration ICCs & TCCs Equating Ability estimates & scaling Reporting

69
So, how is this all done?

70
Psychometricians often play the role of the magical wizard of assessments

71
But, really This session has served as your training in psychometric methods For career opportunities please send along a copy of your vita to: Measured Progress Attn: psychometric Department 171 Watson Road Dover, Nh 03820

Similar presentations

© 2016 SlidePlayer.com Inc.

All rights reserved.

Ads by Google