Download presentation

Presentation is loading. Please wait.

Published byShayna Spark Modified over 2 years ago

1
Measure for Measure(ment) James Shanteau University Distinguished Professor of Psychology Emeritus Kansas State University Bayes Conference – 2/14

2
Goals and Caveats Three goals: (1)Summarize why measurement is essential (2)Review some prior studies using measurement (3)Consider what more needs to be done

3
Goals and Caveats Three goals: (1)Summarize why measurement is essential (2)Review some prior studies using measurement (3)Consider what more needs to be done Caveats: (1)Concern with psychological measurement of stim- ulus values, sometimes called psychological scaling (2)Focus on graphical analyses, which are based on underlying numerical estimates of psychological measures

4
Quotes About Measurement “Thou shall not have in thine house diverse measures. But a perfect & just measure shalt thou have” (Deuteronomy) “If you cannot measure it then it is not science” (Lord Kelvin) “The first steps in the path of discovery … the first approximate measures, are those which add most to the existing knowledge of mankind” (Charles Babbage) “To understand God’s thoughts one must study statistics … the measure of his purpose” (Florence Nightingale) “Whenever you can, count” (Sir Francis Galton) “Research is 4 things: brains with which to think, eyes with which to see, machines with which to measure, & 4 th money” m (Albert Szent-Gyorgyi)

5
Quote by Psychologists on Measurement “In order to formulate quantitative laws (in psychology), the relevant properties must be expressible by numbers. The process by which scientists (do this) is called measure-ment.” (Coombs, Dawes, & Tversky, 1970)

6
Where Is Discussion of Measurement? To find additional comments on Measurement in Psychology, I looked at recent books with “Measurement” in title, eg: Edwards, W., (Ed.) (1992). “Utility theories: Measurements and applications.” Boston: Kluwer Academic Press. Weiss, D. J. (2006) “Analysis of variance and functional measurement.” Oxford: Oxford University Press. Lockhart, P. (2012). “Measurement.” Cambridge, MA: Belknap Press of Harvard University Press. Only 1 of 3 included any content on psych measurement Guess which one?

7
Where Is Discussion of Measurement? To find additional comments on Measurement in Psychology, I looked at recent books with “Measurement” in title, eg: Edwards, W., (Ed.) (1992). “Utility theories: Measurements and applications.” Boston: Kluwer Academic Press. Weiss, D. J. (2006) “Analysis of variance and functional measurement.” Oxford: Oxford University Press. Lockhart, P. (2012). “Measurement.” Cambridge, MA: Belknap Press of Harvard University Press. Only 1 of 3 included any content on psych measurement Guess which one? Only Weiss book includes discussion of actual measurements The Dilemma: Why is “Measurement” widely talked about, but so rarely done?

8
Some Preliminary Comments (1) Measures = parameters of models, stated or unstated => Validity of measures depends on validity of models (2) Estimating Scaling values ≠ Weighting values => Different estimation methods required (3) Typically try to get interval-scale estimates => Need to establish 2 points, eg, slope & intercept (4) Computer programs simplify getting actual measures => “Its too hard to do measurement” is no longer an excuse

9
Example of Weight Measurement My 1 st study as grad student involved estimation of serial position weights in dynamic decision making tasks, eg, probability revision These curves show weights of serial positions for different lengths of stimuli, Resp 4 to Resp 15 See generalized recency Note: weight estimates are not normalized

10
Example of Weight Measurement These are 2 plots of ser- ial position weights from Shanteau (probability revision) and from Farkas (attitude change) Both show recency, but are they similar? Hard to tell, since dif- ferent methods used Neither normalized to sum to one

11
How to Compare Weights? In order to make compar- isons, need to normalize Where ∑weights = 1 When normalized, can see clear patterns (1)Generalized recency in both (2) Larger recency effect in attitude change (3) Less recency in pro- bability revision. Why?

12
When Are Weight Measures Important? In study of expertise by Ettenson, Krogstad, & Shanteau, 3 groups of auditors judged common set of cases described by 8 cues Expert (Partners) & intermediate (Seniors) auditors relied primarily on 1 cue, whereas graduate (Stu- dents) made use of many more cues Has implications for teaching students to focus on what is vital in a given context

13
Example of Scale Value Measurement Anderson’s assessment of subjective values of personality adjectives Parallelism in plot shows consistency of values when combined with 3 other adjectives Averages provide estimate of psychological scale values

14
Why Care About Measurement of Scale Values? In an analysis of willingness to donate various organs while living, Skowronski & Shanteau had 4 groups evaluate various donations while living Found all groups (even Anti-Donors) were willing to donate to relatives This finding has important implications for increasing rates of organ donation Everyone willing to donate … under some situations No one is Anti-Donation But most are Anti-Stranger

15
Example of Joint Weight & Scale Value Measurements My contribution to measurements involved using bilinear models to estimate both weight & scale values; bilinearity => diverging fan of straight lines On left, objective values => irregular pattern unclear On right, subjective values => bilinear pattern evident Shows linkage between model testing & measurement See support for SEU model: SEU = Subj Prob x Utility SEU = Subj Prob x Utility

16
What About Non-Numeric, Verbal Stimuli? Traditionally measurement approaches transform numeric stimuli into psych values, eg, psychophysics But measurement not constrained to numeric stimuli Challenge is getting interval scale values This can be solved by using anchor stimuli, eg, “No Chance” & “Sure Thing” “$.50” & “$60”

17
Other Examples of Measurement Here are some of my favorite examples of measurement Shows versatility of having measures to clarify findings

18
More Quotes About Measurement “If measurement matters at all, it is because it must have some conceivable effect on decisions and behaviour” (Douglas W. Hubbard) “ ’By measurement to knowledge’ I should write as a motto above the entrance to every labora-tory” (Heike Kamerlingh Onnes) “An experiment is a question which science poses to nature and a measurement is the recording of nature’s answer” (Max Planck)

19
What’s Holding Back Measurement? Given benefits of measurement, why isn’t everyone estimating subjective values? Three possible reasons: (1) Confusions over producing interval scales => Need more examples to illustrate “how to do” (2) Seemingly arbitrary assumption required => That is why measurement & model testing linked (3) “Forrest Young Problem” => He said MDS too sophisticated for most users => Need to make measurement easy

20
Some Caveats About Measurement “Those who think ‘Science is Measurement’ should search Darwin’s works for numbers and equations” (David Hubbel) “Our clocks do not measure time … time is defined to be what our clocks measure” (Anonymous, NIST) “Not everything that counts can be counted and not everything that can be counted counts” (Albert Einstein)

21
Comments and Questions? Your turn …

22
Norman Anderson Norman Anderson was my major professor at UCSD Emphasized link between model testing and measurement Persists in emphasizing role of measurement in psychology

23
Measure for Measure(ment) James Shanteau University Distinguished Professor of Psychology Emeritus Kansas State University Bayes Conference – 2/14

24
Issue 3: What About Individual Differences? Group analyses often conceal sizeable individual differences One advantage of FM is ability to analyze both levels See differences in both model shape & values

25
Norman Anderson on Measurement “The logic of (FM) consists in using the postulated behavior- al laws to induce a scaling on the dependent variable” (1962) “A guiding principle of FM is that measurement scales are derivative from substantive theory” (1970) “Measurement is fundamental” (1996) “FM reverses the traditional approach & makes measurement an organic component of substantive investigation” (1981) “All that is necessary is to have a valid integration function. That function provides base & frame for measurement” (1981) “Measurement… is the link between the world of behavior & the world of science … (it) is a vital feature of science … meas- urement is thus an integral part of substantive theory” (2001) “Much of progress consists in improvements in measurement, both in its empirical base & its conceptual base” (2001)

26
Norman Anderson (NHA) Story 50 years ago, NHA was debating with advocates of Conjoint Measurement (CM). One of his major arguments was that CM wasn’t being used to actually measure anything, but FM was. However, there were almost no examples of actual measures by FM at that time. NHA quickly put together a paper with actual measures of adjectives => it was published in Psych Reports (?) => I have been unable to find either the paper or the citation However, my memory of this story was the motivation for the present talk, ie, we talk-the-talk about measurement, but how often do we walk-the-walk? If measurement is such a vital part of FM, why don’t we see measured values reported in most every study But, such reports of numerical measures is rare. Why?

27
Purpose & Organization Purpose: To revisit role that measurement plays in Functional Measurement (FM) Organization of Talk: (1) Describe role of measurement in FM logic (2) Present examples of measurement in FM studies (3) Consider some possible barriers to use of FM (4) Take comments & answer questions (if I can)

28
Example of Joint Weight & Scale Value Measurements Can see application of bilinear analysis in reanalysis of Tversky (left panel) See similar patterns in Slovic & Lichenstein data Note: both original analyses involved additivity (par- allelism) assumptions based on objective values

29
Issue 3: What About Individual Differences? Group analyses often conceal sizeable individual differences One advantage of FM is ability to analyze both levels

30
Example of FM Using RTs Nearly all studies using FM have used rating data One exception: Weiss’ work using ordinal data Another exception: FM analysis of RTs in question- answering task introduced by Anderson Results provided new insights into priming effects Key is bilinear analysis of cumulative Response Time Allows bilinear scaling ∴ Can “measure” times => just like rating data Priming found in both mem- ory & decision times Decision priming over 2x effect of memory priming

31
More Quotes on Measurement “Measurement is the 1 st step to control & eventually to improvement. If you can’t measure something, you can’t understand it” (H. James Herrington) If measurement matters at all, it is because it must have some conceivable effect on decisions & behaviour” (Douglas W. Hubbard) “’By measurement to knowledge’ I should write as a motto above entrance to every laboratory” (Heike Kamerlingh Onnes) “An experiment is a question which science poses to nature, & a measurement is the recording of nature’s answer” (Max Planck) “If you cannot measure it you cannot control it” (J Grebe) “Man is the measure of all things” (Protagoras)

32
Quote by Psychologists on Measurement “Although measurement & scaling have sometimes been used interchangeably in the literature, we have chosen to distin- guish between them. Measurement is concerned with the con- ditions under which various types of scales can be construct- ed. The actual process of assigning numbers … is called scal- ing” (Coombs, 1964) “In order to formulate quantitative laws (in psychology), the relevant properties must be expressible by numbers. The process by which scientists (do this) is called measure-ment.” (Coombs, Dawes, & Tversky, 1970)

33
Some Preliminary Comments (1) Measures = parameters of models, stated or unstated => Validity of measures depends on validity of models => There is direct link between model fit & measurement (2) Estimating Scaling values ≠ Weighting values => Different estimation methods required => Lead to different interpretation and usage (3) Typically try to get interval-scale estimates => Need to establish 2 points, eg, slope & intercept (4) Computer programs simplify getting actual measures => “Its hard to do measurement” is no longer an excuse

34
Some Caveats on Measurement “Not everything that is observable & measurable is predictable, no matter how complete our past observations may have been” (Sir William McCrea) “Those who think ‘Science is Measurement’ should search Darwin’s works for numbers & equations” (David Hubel) “Investigators seem to have settled for what is measur- able instead of measuring what they would really like to know” (E D Pellegrino) => Note: Drunk & Lamppost problem “Our clocks do not measure time … Time is defined to be what our clocks measure” (Anonymous, NIST) “Not everything that counts can be counted & not every- thing that can be counted counts” (Albert Einstein)

35
Are Measured Values Constant Across Tasks? Within many tasks, eg, impression formation, mea- sured values appear to be more-or-less constant But across tasks, see sizable differences, eg, in sub- jective probability values for gambles vs dating

Similar presentations

OK

Chapter 7 Measuring of data Reliability of measuring instruments The reliability* of instrument is the consistency with which it measures the target attribute.

Chapter 7 Measuring of data Reliability of measuring instruments The reliability* of instrument is the consistency with which it measures the target attribute.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google