# 1 Class 3 Classical Methods of Scale Construction October 13, 2005 Anita L. Stewart Institute for Health & Aging University of California, San Francisco.

## Presentation on theme: "1 Class 3 Classical Methods of Scale Construction October 13, 2005 Anita L. Stewart Institute for Health & Aging University of California, San Francisco."— Presentation transcript:

1 Class 3 Classical Methods of Scale Construction October 13, 2005 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2 Readings and Homework u Homework as stated in syllabus is for the following week u Readings are relevant to the current week

3 Overview of class u Types of measurement scales u Rationale for multi-item measures u Scale construction methods u Error concepts

4 Types of Measurement Scales u Categorical (nominal) –Classification –Numbers are labels for categories u Continuous (along a continuum) –Ordinal –Interval –Ratio

5 Classification vs. Continuous Scores u CES-D continuous score –20 items summed using Likert scaling methods –Range of sum is 0-60, used as continuous score in correlational studies u CES-D classification score: –Those scoring 16 or higher are “classified” as having likely depression »Referred for further screening

6 Categorical (Nominal) Scales/Measures u Primary language 1 Spanish 2 English 3 Other u Can you walk without help? 1 Yes 2 No Numbers have no inherent meaning

7 Ordinal Scales: Numbers Reflect Increasing Level u Change in health: 1 Better 2 No change 3 Worse u Income: 1 < \$10,000 2 \$10,000 - <\$20,000 3 \$20,000 - <\$30,000 4 >\$30,000 Numbers have no inherent meaning other than “more” or “less.”

8 Another Example of Ordinal Scale How much pain did you have this past week? 1 None 2 Very mild 3 Mild 4 Moderate 5 Severe 6 Very severe

9 Feature of Ordinal Scales u Distances between numbers are unknown and probably vary –some closer together in meaning than others u When ordinal responses are determining extent of agreement (agree, disagree) –referred to as a Likert scale u Likert scale has since come to have other meanings in health measurement

10 Interval Scales u Numbers have equal intervals u A unit change is constant across the scale u Example - temperature –can add and subtract scores –a 2 unit change is the same at lower temperatures as higher temperatures

11 Ratio Scale u Has a meaningful zero point u Change scores have specific meaning u and can multiply –e.g., one score can be 2 or 3 times another u Examples –Weight in pounds – Income in dollars – Number of visits

12 Types of Measurement Scales and Their Properties Property of Numbers Type of scaleRank order Equal interval Absolute zero NominalNo OrdinalYesNo IntervalYes No RatioYes

13 Overview of class u Types of measurement scales u Rationale for multi-item measures u Scale construction methods u Error concepts

14 Single- and Multi-Item Measures u Advantages of single items –Response choices are interpretable u Disadvantages –Numbers are not easily interpretable –Limited variability »Easy to get skewed distributions –Reliability is usually low –Difficult to assess a complex concept with one item

15 Interpretability of “Numbers” in Single Item Ordinal Scale How much pain did you have this past week? 1 - none 2 – very mild 3 - mild 4 - moderate 5 - severe 6 – very severe

16 Interpretability of “Numbers” in Single Item Ordinal Scale How much pain did you have this past week? 1 - none 2 – very mild 3 - mild 4 - moderate 5 - severe 6 – very severe Is “very severe” twice as painful as “mild”?

17 Estimated Distance Between Levels in Ordinal Scale (N=2,928) (0-100 scale) How much pain did you have this past week? 0-100 transform M pain scale 1 - none0 3.30 2 – very mild2012.19 3 - mild4021.89 4 - moderate6038.76 5 - severe8059.43 6 – very severe10075.38

18 Distance Between Levels in an Ordinal Scale (N=2,928) How much pain did you have this past week? Mean: pain scale 1 - none 3.30 2 – very mild12.19 3 - mild21.89 4 - moderate38.76 5 - severe59.43 6 – very severe75.38 9 10 17 20 16

19 Distance Between Levels: “In general, how would you rate your health?” Mean: current health scale Screening N=~11,000 Baseline N=3,054 1 – poor 0 10.8 2 – fair 25 30.030.6 3 – good 50 57.655.9 4 – very good 75 75.575.4 5 – excellent 1 00 87.986.9

20 Distance Between Levels: “In general, how would you rate your health?” Mean: current health scale Screening N=~11,000 Baseline N=3,054 1 – poor10.8 2 – fair30.030.6 3 – good57.655.9 4 – very good75.575.4 5 – excellent87.986.9 20 26 18 11

21 Multi-Item Measures or Scales u Multi-item measures are created by combining two or more items into an overall measure or scale score

22 Advantages of Multi-item measures u More scale values (enhances sensitivity) u Improves score distribution (more normal) u Reduces number of variables needed to measure one concept u Improves reliability (reduces random error) u Can estimate a score if some items are missing u Enriches the concept being measured (more valid)

23 Overview of class u Types of measurement scales u Rationale for multi-item measures u Scale construction methods u Error concepts

24 Types of Scale Construction u Summated ratings scales –Likert scaling u Utility weighting or preference-based measures (econometric scales) u Guttman scaling u Thurstone scales u Many others

25 Example of a 2-item Summated Ratings Scale How much of the time.... tired? 1 - All of the time 2 - Most of the time 3 - Some of the time 4 - A little of the time 5 - None of the time How much of the time …. full of energy? 1 - All of the time 2 - Most of the time 3 - Some of the time 4 - A little of the time 5 - None of the time

26 Step 1: Reverse One Item So They Are All in the Same Direction How much of the time.... tired? 1 - All of the time 2 - Most of the time 3 - Some of the time 4 - A little of the time 5 - None of the time How much of the time …. full of energy? 1=5 All of the time 2=4 Most of the time 3=3 Some of the time 4=2 A little of the time 5=1 None of the time Reverse “energy” item so high score = more energy

27 Step 2: Sum the Two Items How much of the time.... tired? 1 - All of the time 2 - Most of the time 3 - Some of the time 4 - A little of the time 5 - None of the time How much of the time …. full of energy? 5 All of the time 4 Most of the time 3 Some of the time 2 A little of the time 1 None of the time Highest = 10 (tired none of the time, full of energy all of the time) Lowest = 2 (tired all of the time, full of energy none of the time)

28 Step 2: Average the Two Items How much of the time.... tired? 1 - All of the time 2 - Most of the time 3 - Some of the time 4 - A little of the time 5 - None of the time How much of the time …. full of energy? 5 All of the time 4 Most of the time 3 Some of the time 2 A little of the time 1 None of the time Highest = 5.0 (tired none of the time, full of energy all of the time) Lowest = 1.0 (tired all of the time, full of energy none of the time)

29 Summed or Averaged: Increase Number of Levels from 5 to 9 SummedAveraged 21.0 31.5 42.0 52.5 63.0 73.5 84.0 94.5 105.0

30 Summated Scales: Scaling Analyses u To create a summated scale, one needs to first test whether a set of items that appear to measure the same concept can be combined –Need to test hypothesis that the items do indeed belong together to form a single concept u Five criteria need to be met to combine items into a summated scale

31 Five Criteria to Meet to Qualify as a Summated Scale u Item convergence u Item discrimination u No unhypothesized dimensions u Items contribute similar proportion of information to score u Items have equal variances

32 First Criterion: Item Convergence u Each item correlates substantially with the total score of all items –with the item taken out or “corrected for overlap” u Typical criterion is >=.30 –for well-developed scales, often set at >=.40

33 Example: Analyzing Convergent Validity for Adaptive Coping Scale Item-scale correlations Adaptive coping (alpha =.70) 5 Get emotional support from others.49 11 See it in a different light.62 18 Accept the reality of it.25 20 Find comfort in religion.58 13 Get comfort from someone.45 21 Learn to live with it.21 23 Pray or meditate.39 Moody-Ayers SY et al. Prevalence and correlates of perceived societal racism in older African American adults with type 2 diabetes mellitus. J Amer Geriatr Soc, 2005, in press.

34 Example: Analyzing Convergent Validity for Adaptive Coping Scale Item-scale correlations Adaptive coping (alpha =.70) 5 Get emotional support from others.49 11 See it in a different light.62 18 Accept the reality of it.25 <.30 20 Find comfort in religion.58 13 Get comfort from someone.45 21 Learn to live with it.21 <.30 23 Pray or meditate.39

35 Example: Analyzing Convergent Validity for Adaptive Coping Scale Item-scale correlations Adaptive coping (alpha =.76) 5 Get emotional support from others.45 11 See it in a different light.59 20 Find comfort in religion.73 13 Get comfort from someone.45 23 Pray or meditate.51 Acceptance (alpha =.67) 21 Learn to live with it.50 18 Accept the reality of it.50

36 SAS/SPSS Make Item Convergence Analysis Easy u Reliability programs provide this –Item-scale correlations corrected for overlap –Internal consistency reliability (coefficient alpha) –Reliability with each item removed »To see effect of removing a bad item

37 Second Criterion: Item Discrimination u Each item correlates significantly higher with the construct it is hypothesized to measure than with other constructs –Item discrimination u Statistical significance is determined by standard error of the correlation –Determined by sample size

38 Multitrait Scaling - An Approach to Constructing Multi-item Scales u Confirms whether hypothesized item groupings can be summed into a scale score u Examines extent to which all five criteria are met u Examines resulting scales

39 Example: Two Subscales Being Developed u Depression and Anxiety subscales of MOS Psychological Distress measure

40 Example of Multitrait Scaling Matrix: Hypothesized Scales ANXIETY DEPRESSION ANXIETY Nervous person.80.65 Tense, high strung.83.70 Anxious, worried.78.78 Restless, fidgety.76.68 DEPRESSION Low spirits.75.89 Downhearted.74.88 Depressed.76.90 Moody.77.82

41 Example of Multitrait Scaling Matrix: Item Convergence ANXIETY DEPRESSION ANXIETY Nervous person.80*.65 Tense, high strung.83*.70 Anxious, worried.78*.78 Restless, fidgety.76*.68 DEPRESSION Low spirits.75.89* Downhearted.74.88* Depressed.76.90* Moody.77.82*

42 Example of Multitrait Scaling Matrix: Item Convergence ANXIETY DEPRESSION ANXIETY Nervous person.80*.65 Tense, high strung.83*.70 Anxious, worried.78*.78 Restless, fidgety.76*.68 DEPRESSION Low spirits.75.89* Downhearted.74.88* Depressed.76.90* Moody.77.82*

43 Example of Multitrait Scaling Matrix: Item Discrimination ANXIETY DEPRESSION ANXIETY Nervous person.80*.65 Tense, high strung.83*.70 Anxious, worried.78*.78 Restless, fidgety.76*.68 DEPRESSION Low spirits.75.89* Downhearted.74.88* Depressed.76.90* Moody.77.82*

44 Preference Based or Utility Measures u Utilities are numeric measurements that reflect the desirability people associate with a health state or condition –Value of that health state –Preference for that health state (rather than another)

45 Methods for Assigning Values? u Four steps: –Identify the population of judges who will assign “preferences” –Sample and describe health states to be assigned utilities –Select a preference measurement method –Collect preference judgments, analyze the data, and assign weights to the health states

46 Preference Based or Utility Measures (cont.) u Advantages –Combine complex health states into a single number u Score reflects the value or preference for the overall health state u Need two absolute reference points –0 represents death –1 represents perfect health u Methods for obtaining value weights –Time tradeoff, standard gamble, rating scales

47 Readings on Utility Measurement u A huge literature u Some readings available on request

48 Overview u Types of measurement scales u Rationale for multi-item measures u Scale construction methods u Error concepts

49 Concepts of Error u How to depict error u Distinction between random error and systematic error

50 Components of an Individual’s Observed Item Score (NOTE: Simplistic view) Observed true item score score =+ error

51 Components of Variability in Item Scores of a Group of Individuals Observed true score score variance variance Total variance (Variation is the sum of all observed item scores) =+ error variance

52 Combining Items into Multi-Item Scales u When items are combined into a scale score, error cancels out to some extent –Error variance is reduced as more items are combined –As you reduce random error, amount of “true score” increases –Multi-item scale is thus more reliable than any single item

53 Sources of Error u Subjects u Observers or interviewers u Measure or instrument

54 Measuring Weight in Pounds of Children: Weight without shoes u Observed scores is a linear combination of many sources of variation for an individual

55 Measuring Weight in Pounds of Children: Weight without shoes Scale is miscalibrated True weight Amount of water past 30 min Weight of clothes Observed weight Person weighing children is not very precise = + + ++

56 Measuring Weight in Pounds of Children: Weight without shoes Scale is miscalibrated +1 lb True weight 80 lbs Amount of water past 30 min +.25 lb Weight of clothes +.75 lb Observed weight 83 lbs Person weighing children is not very precise +1 lb = + + ++ 83 = 80 +.25 +.75 +1 +1

57 Sources of Error u Weight of clothes –Subject source of error u Person weighing child is not precise –Observer source of error u Scale is miscalibrated –Instrument source of error

58 Measuring Depressive Symptoms in Asian and Latino Men Unwillingness to tell interviewer “True” depression Low awareness of negative affect Observed depression score Depression measure not culturally sensitive = + ++

59 Measuring Depressive Symptoms in Asian and Latino Men Unwillingness to tell interviewer -3 “True” depression 16 Hard to choose one number on the 1-6 response choices +2 Observed depression score 13 Measure not culturally Sensitive -2 = + ++ 13 = 16 +2 -3 -2

60 Return to Components of an Individual’s Observed Item Score Observed true item score score =+ error

61 Components of an Individual’s Observed Item Score Observed true item score score =+ error random systematic

62 Sources of Error in Measuring Weight u Weight of clothes –Subject source of random error u Scale is miscalibrated –Instrument source of systematic error u Person weighing child is not precise –Observer source of random error

63 Sources of Error in Measuring Depression u Hard to choose one number on 1-6 response scale –Subject source of random error u Unwillingness to tell interviewer –Subject source of systematic error (underreporting true depression) u Instrument is not culturally sensitive (missing some components) –Instrument source of systematic error

64 Next Week – Week 4 u Variability u Reliability u Interpretability

65 Homework for Week 4 u Complete rows 1-12 on the matrix for each measure you want to review –Handout –On the web site for this class u Download matrix and fill it in

Download ppt "1 Class 3 Classical Methods of Scale Construction October 13, 2005 Anita L. Stewart Institute for Health & Aging University of California, San Francisco."

Similar presentations