Download presentation

Presentation is loading. Please wait.

Published byKaley Blakemore Modified over 2 years ago

1
Agenda Levels of measurement Measurement reliability Measurement validity Some examples Need for Cognition Horn-honking

2
Levels of measurement Nominal Ordinal Interval Ratio

3
Linking concepts to data Conceptual definition: Theoretical variables Units of analysis Operational definition: Procedures for measuring variables Subject units

4
X TV exposure Z Reading time Y Reading skills X Self-reported TV watching Z Self-reported reading Y Scores on reading test ?? ? Operation- alization Theory of Measurement

5
Two key qualities Measurement Reliability The extent to which repeated measurements produce same results Inversely related to the amount of random error Measurement Validity The extent to which a measure does what it is intended to do

6
Random error and reliability Measures have at least two components: Measure = True Value + Random Error Variation comes from both sources: Total Variation = True Variation + Random Variation The reliability of a measure is: True Variation /Total Variation

7
Estimating reliability Need at least two measures of same concept Each measure has random error Variation shared is not due to random error True Value X1X1 Error 1 X2X2 Error 2 Correlation reflects reliability

8
Reliability coefficients Some coefficients estimate reliability of individual measures (items) Test/Retest correlation Same item repeated on (unchanging) true value Inter-item correlation Different items measure same true value Inter-coder correlation (agreement) Different coders measure same true value

9
Increasing reliability How to counteract noisy measurements? Careful conceptualization Employ precise quantitative measures Combine multiple measures of the same theoretical concept

10
Multi-item scales Example 10 vocabulary test items Each is subject to some random error Combining (e.g., adding) items will compound what is common to the measures the true vocabulary scores Combining items will not compound what is unique the random errors So combining increases proportion of true variation to total variation

11
Reliability coefficients Some coefficients estimate reliability of multiple-item scales Split-half method Total set of items randomly divided in half Each half summed to form a scale Scores on the two halves correlated Example: Spearman-Brown reliability coefficient Internal consistency method Calculate all inter-item correlations Average them, and adjust for the number of items Example: Cronbachs alpha reliability coefficient

12
Examining scales Which items produce the most reliable scale? Item-total correlations Correlate each item with the total (of other items) Weak correlations suggest item doesnt share much variance with the overall scale Comparative scale reliability Calculate scale reliability (e.g., Spearman-Brown or Cronbachs alpha) with and without particular item If a scales reliability doesnt increase with additional item, we suspect it is weak

13
What reliability insures High proportion of variance is systematic, not random However … Systematic variance may stem from shared bias Acquiescence response bias, social desirability Systematic variance may stem from the wrong concept Confusing intelligence with socialized learning Valid measures must be reliable, but reliability does not guarantee validity

14
Measurement validity One validates, not a test, but an interpretation of data arising from a test (Lee Cronbach) How should a measure be interpreted? What empirical data can help insure that a given interpretation is valid?

15
Face validity Simple examination of measure Does it manifestly address the right concept? Weak form of validation Largely matter of interpretation

16
Content validity Focuses on extent to which a measure reflects a specific domain of conceptual content Addresses coverage of a measure Largely matter of interpretation Requires conceptual definition of domain

17
Criterion-related validity Involves correlating a measure with some external phenomenon Concurrent: distinguishing some co-existing difference Predictive: forecasting future difference Depends upon validity of criterion May not always be applicable

18
Construct validity Extent to which a measure relates to other measures consistent with theoretically derived hypotheses Sometimes termed nomological validity E.g., age abstract reasoning ability Focuses on pattern of relationships among various concepts and measures

19
Construct validity Convergent validity Similar data result from measurements of similar concepts using different operational techniques Discriminant validity Dissimilar data result from measurements of different concepts (particularly those which might be easily confused operationally)

20
Trait Leadership Cooperation Questionnaire Method Observer ratings Same trait/ different methods should agree Different traits/ same methods should not agree Multi-Trait/Multi-Method Matrix Convergent Discriminant

21
An example Cacioppo & Petty (1982) The Need for Cognition What is the concept? How measured? What evidence of reliability? Item-total coefficients (Study 1) Spearman- Brown coefficient (Study 1) Factor analysis confirms single underlying factor (Study 1 & 2)

22
Factor Need for Cognition x1x1 NFC Item Factor Other Concept x2x2 NFC Item x3x3 x4x4 x5x5 Factor Loading correlation Random Error Random Error Random Error Random Error Random Error

23
Example, cont. What evidence of validity? Concurrent validity Distinguishes faculty from assembly-line workers (Study 1) Discriminant validity Only small relationship with cognitive style (Study 2) No relationship with test anxiety (Study 2) Significant and modest relationship with ACT scores (Study 3) Weak relationship with social desirability (Study 3 & 4) Weak negative correlation with dogmatism (Study 3 & 4) Predictive validity Predicts enjoyment of a cognitive task (Study 4)

24
Another example Gross & Doob (1982) Status of Frustrator as an Inhibitor of Horn-Honking Responses What is the theory? How are the concepts measured? What evidence of reliability? What evidence of validity? Construct validity: Status and gender differences consistent with prior research Convergent validity? Authors cast doubt on the validity of questionnaire measures

25
Reliability and validity Conceptualization and measurement are primary concerns in all research Always look for evidence of measurement reliability and validity Of the two, validity is probably more important Unreliable measures increases the odds that we wont find anything Invalid measures increase the odds that well find the wrong thing

26
For Thursday Question-asking Sudman & Bradburn, Ch. 1-5 Begin working on fourth individual assignment (scaling exercise)

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google