1. Lee Sechrest, Katherine McKnight, & Mei-kuang Chen

2. 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA How NOT to Construct a measure

3. Focus The concerns we are going to talk about should not be avoided Think hard(er) about measurement Measures should be deliberately and carefully conceptualized, designed, and constructed Types of measures and related assumptions, construction, interpretation, reliability and validity issues 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

4. Why measures matter: “Garbage in, garbage out”! Gang involvement: single item in the instrument Treatment dosage (Treatment Received Scale): direct, family and external type of substance abuse treatment services in the past 90 days Family treatment: Work with you at your home? Meet with family members of yours more than one time? Work with members of your family on communication? Hook your family up with services? 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

5. What are Composite Variables? Two or more measures are combined to form a single value of a measure Individual indicators are compiled to a single index based on an underlying model Ideally measure multi-dimensional concepts that cannot be captured by a single indicator alone 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

6. Simple example Count data: The U of A football record in the PAC-10 is now 4 wins and 1 loss The U of A football record is now (4wins/5 games) =.80  a composite measure 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

7. Examples from Program Evaluation Socioeconomic Status Earned income Education Zip code Quality of Life Health status (physical & mental) Employment status Social support Quality of Law School LSAT entrance scores Job placement Starting salary

8. 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA Some things do NOT make sensible composites!

9. Before constructing a measure Think more about your construct Theory informs how we construct our measures Indicators of the construct Measurement model (e.g., latent vs. emergent) In the end your measure should map onto your understanding of the construct 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

10. There is nothing more practical than a good theory

11. Types of composites Measured (observed) variable Common factor: Latent variable measures Quasi-latent variable measures: constructed variables with (implicit) assumed model Emergent (constructed) variables with defined weights 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

12. Measured (observed) variables Defined by the specific measure: e.g., income = paycheck; height = inches; speed = distance/time Has no other meaning Assumed to be measured without error Is this an accurate assumption? 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

13. Common factor/latent variable Assume an underlying factor that causes its indicators, i.e. all indicators have a common cause Therefore, all indicators are positively correlated The value (quantity) of the unobserved (latent) variable can be inferred from the values of its indicators 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

14. Short-term memory Problem- solving Pattern perception Intelligence Vocabulary Factual knowledge Latent Variable Effects Indicators

15. Common factor/latent variable (cont’d) Because all indicators have variance in common, they are to that degree equivalent Therefore, missing indicators can be dealt with by using means of values for observed indicators The weights given to indicators are determined by their statistical inter-relationships 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

16. Quasi-Latent & Emergent Variables Constructed variables with (implicit) assumed model: Variables are constructed from conceptual models that should exhaust their intended meaning Weights for indicators cannot be determined statistically because they are not necessarily correlated They must be specified: Expert judgment Analytic approach Criterion Arbitrarily 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

17. Mental Health Financial Status Job Satisfaction Quality of life Physical Health Relationship With Spouse Quasi-latent or Emergent Variable Causal Indicators

18. Quasi-Latent & Emergent Variables (Cont’d) Indicators need not be correlated- correlation of indicators not verified Missing values cannot be imputed Any indicator omitted, any indicator added changes the definition of the variable Assumption of but no correction of error 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

19. Example: Gang involvement – What kind of measure? Is it a measured variable? “Are you part of a gang?” YES/NO Is it a latent variable? What does “gang” mean? Entitativity (Campbell)- similarity, proximity, high coefficient of common fate What does involvement mean? Loyalty Frequency of contact Is it a quasi-latent or emergent variable? Frequency of defined gang activities (e.g., “tagging,” “jumping” members into the group) + s elf-identification as a gang member + reported loyalty 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

20. Defining the construct Think carefully about what it is you want to measure Then think carefully about what it is you want to measure and what you want that measure to be used for The measure and what it is to be used for ought to be fairly closely related 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

21. Defining the construct Theoretical framework- weighting and aggregation Take method variance into consideration To the extent that we rely on a single method of measurement, then the variance associated with that method is integral to the definition of the variable 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

22. Defining the construct There is not a method more common, more used and abused, than self-report Are you interested in: Self-reported gang involvement Self-reported depression Self-reported income Self-reported job satisfaction Self-reported social support Self-reported intelligence Self-reported substance use problem 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

23. Defining the construct Best way to deal with unwanted method variance: Multiple measures that are not subject to the same sources of error Look at other behavior- not just the behavior of filling out self-report measure “I like a look of anguish, because I know it’s true” Emily Dickinson 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

24. Behavioral measures They are often difficult and expensive to acquire They may be useful even if they are available on only a portion of the sample 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

25. Creating composite variables Use our knowledge & conceptions about a given construct to : Identify appropriate indicators Determine the best method for combining the indicators -Linear vs. nonlinear -Weighting schemes -Scaling 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

26. Identifying Key Indicators Errors of Omission & Co-mission Latent/Common Factors: universe of correlated indicators, random subset serves equally Quasi-Latent & Emergent Variables: may be uncorrelated, defined by their indicators - Emergent Variables are the result of the indicators

27. Examples Latent/Common Factors: Intelligence - E.g., Short-term memory (recall a string of #s), vocabulary, spatial ability… Quasi-Latent: Good Teacher - Personality, content knowledge, organizational skills, behavioral management… Emergent: Livable Cities - K-12 education, housing price, clean air, climate, traffic…

28. Constructing the Algorithm for Combining Indicators Should be dictated by how we conceive the construct: Adiposity Ponderal Index = height/weight 1/3 Weight/Height Ratio = weight/height Body Mass Index = weight/height 2

29. A measure of adiposity should have the following characteristics (Billewicz et al, 1962): 1) Allow us to rank individuals in order of their true relative adiposity 2) A given value, for each sex, should imply on average the same degree of relative adiposity at all heights 3) Easy to compute and invariant with respect the units of measurement

30. Source: Billewicz et al. (1962)

31. These examples show that an index which contains an implicit assumption about the relationship of adiposity to height cannot be safely used in statistical analyses unless height is introduced as an independent variable. This conclusion is of particular importance in relation to analyses of body build in diseases which have a marked social class gradient. There is no doubt that average stature increases with rise of social status, so that, for example, a disease which occurs more commonly in the upper social strata will tend also to occur more commonly in tall individuals. Incautious application of the ponderal index or the height/weight ratio to groups of sufferers from such a disease could readily give misleading results. -- (Billewicz, Kemsley & Thomson, 1962, p. 186)

32. Constructing the Algorithm for Combining Indicators Should be dictated by how we conceive of the construct, and what we know about the phenomenon of interest: In humans, WEIGHT is NINE TIMES more variable than HEIGHT & therefore measures that include both vary disproportionately. Billewicz et al (1962) recommend a more useful index: (observed wt)/(standard wt.)

33. Common Algorithms: Linear Combination Sum the indicators - Stressful Life Events - Child Behavior Checklists Assumptions: - Indicators are additive - Their contributions are unique/independent - Their importance (contribution to the sum) is equal to their variance

34. Example: Risk for Heart Disease *Smoking *High Cholesterol *High blood pressure *Body fat *Diet *Genetics Do we sum these? Do we weight them?

35. Example: Delinquency *Auto theft *Stealing < $20 *Vandalism *Disrespecting authority *Drug use *Skipping school Do we sum these? Do we weight them?

36. Weighting Schemes Implicit vs. Explicit - Weight by variance vs. determine purposefully Unit weighting (z-scores, 1/variance) Expert opinion ( e.g. Sellin & Wolfgang (1966); US News & World Report; meta-analyses ) - Stealing item valued at $5 = 1 - Assault resulting in death = 26 Analytically ( e.g., unstandardized regression weights) Criterion (e.g., weighting bball hits based on their consequence to the game)

37. Other Methods for Combining Indicators Nonlinear (non-additive) - Offensive efficiency = (on-base %) x (slugging avg) - BMI = weight/height 2 Cumulative scaling – indicators ordered via probability of endorsement - Attitudes toward immigration* Allow into my country Live in my community Live on my block Live next door My child is allowed to date My child is allowed to marry “Hurdles” scaling – must “clear” one indicator to move onto the next, e.g., DSM diagnoses *Source: http://trochim.human.cornell.edu/kb/scalgutt.htm

38. Creating Composite Variables Our examples are not exhaustive but illustrative Algorithms are limited only by our imagination for combining indicators meaningfully so they accurately reflect our knowledge of the construct we are measuring

39. Psychometric properties of measures Reliability (repeatability, consistency) and validity (are we measuring what we claim?) Not inherent properties of a measure Dependent on measurement model Dependent on use of measure (context)

40. Psychometric consideration Reliability Validity- Construct validity Criterion validity 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

41. Reliability Index of consistency items (internal consistency, split-half) Index of repeatability time (test-retest) Classical test theory reliability = true score / (true score + error) observed score

42. True score Observed score True error reliability

43. Validity Extent to which a measure corresponds to the construct it is intended to measure Face: looks like it measures the construct Content: adequate sampling of full range of content to be measured Concurrent: does it measure concept as it appears “now” Predictive: does it measure concept as it will be in some future point in time

44. Criterion validity of a measure Can be estimated with precision and specified by some statistical process (e.g., a correlation) A measure can be considered to have criterion validity for any criterion with which it is correlated Criterion validity does not require any understanding of the process underlying the correlation- practical prediction 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

45. Construct validity of measures is not about statistics or even psychometrics It is about careful thinking, systematic work, careful data collection, and intensive data analysis Construct validity cannot be quantified with any precision Ultimately, it is a matter of consensus among those who use the measures 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

46. Measurement Models: Latent vs. Quasi-latent or Emergent variables Effects vs. causal indicators (Bollen & Lennox, 1991) Effects indicators: ‘latent’ variables, common factors Indicators are the result of the underlying construct Measured indirectly by the indicators (i.e. latent) Causal indicators: ‘quasi-latent or emergent’ variables, formative indicators The construct is the result of the indicators Measured directly (i.e. measured)

47. Validity issues Common factor measures: Construct validity is the essential issue Quasi-latent or emergent variable: Utility is the essential issue 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA

48. Reliability & Latent Variables Measurement model: Indicators correlated due to common cause unidimensional Internal consistency relevant Indicators imperfect reflection of underlying cause true score ≠ observed score Latent Variable V1V2V3V4

49. Reliability & Quasi-latent or Emergent Variables Measurement Model:  Indicators create the construct Multidimensional Internal consistency NOT relevant  Construct is defined by indicators Accuracy is relevant true score = observed score Quasi-latent or emergent Variable V1V2V3V4

50. Reliability of Quasi-latent or Emergent Variables Reliability analyses usually limited to Cronbach’s α and retest coefficients (Socan, 2000) Cronbach’s α underestimates reliability of multidimensional composites (Lord & Novick, 1974) Test-retest of alternate forms (for each indicator), if available (Nunnally & Berstein, 1994), e.g., Quality of Life form 1Quality of Life form 2 Health status, form1Health status, form 2 Financial status, form1Financial status, form2 Job satisfaction, form1Job satisfaction, form2

51. Reliability of multidimensional linear composites (Nunnally & Bernstein, 1994) where = variance of individual indicators in the linear combination = variance of linear combination = estimates of each indicator’s reliability

52. Reliability vs. Accuracy Emergent variables are defined by their indicators — “it is what it is” Therefore, they are measured variables Therefore, no measurement error in the CTT sense True & Observed Score

53. Reliability vs. Accuracy Quality of U.S. graduate business schools (US News & World Report, 2002) Peer assessment (1 = marginal to 5 = outstanding) Recruiter assessment (1 = marginal to 5 = outstanding) Average undergraduate GPA Average GMAT score Acceptance rate Average starting salary & bonus % employed at graduation % employed 3 months after graduation Out-of-state tuition & fees Total full-time enrollment

54. Validity issues What is being measured? To what extent does the composite measure what it is designed to measure? Is more than one trait being measured? How well does the composite variable correlate with external criteria?

55. Validity & Latent variables Measurement model: Indicators imperfect reflection of underlying cause “Entitativity” (Campbell, 1958) Correspondence ≠ 100% Latent Variable V1V2V3V4

56. Validity & Quasi-latent or Emergent variables Measurement Model: Indicators create the construct Construct is defined by indicators – “it is what it is” indexes Correspondence = 100% Quasi-latent or Emergent Variable V1V2V3V4

57. Utility of Quasi-latent or Emergent Variables Extent to which it measures what it is intended to is not a relevant issue, but… How well does this measure do what it is supposed to do? What does this measure tell us?

58. Measuring Quality of Business Schools (U.S. News & World Report Rankings) Peer assessment (1 = marginal to 5 = outstanding) Recruiter assessment (1 = marginal to 5 = outstanding) Average undergraduate GPA Average GMAT score Acceptance rate Average starting salary & bonus % employed at graduation % employed 3 months after graduation Out-of-state tuition & fees Total full-time enrollment

59. Is this index useful? What can this index do for us? Help applicants select schools for quality education Index should predict knowledge gained Help employers select graduates for quality employees Index should predict quality of work produced

60. Conclusion We need more attention on our measurement models before we focus on content Appropriate measures of psychometric properties should be determined by the measurement model Emergent variable models reflect different assumptions about reliability & validity as traditionally construed Accuracy & utility are more applicable

61. Useful Resource E. Giovannini et al. (2005), " Handbook on Constructing Composite Indicators: Methodology and User Guide “, OECD Statistics Working Papers, 2005/3, OECD Publishing. doi:10.1787/533411815016 11/11/2009Design and Construction of Measures, EGAD, Orlando, AEA