Discussion Overview: Measurement I) Reliability of Measures I) Reliability of Measures II) Construct Validity II) Construct Validity III) Measurement scales III) Measurement scales

I) Reliability of Measures Reliability Reliability –The consistency or stability of a measure Assessing a restaurant’s food Assessing a restaurant’s food Three important variables Three important variables –How many testers? (Observers)  Interrater reliability –How many different entrees? (Observations)  Internal consistency –How many times? (Occasions)  Test-retest

Interrater Reliability The degree to which independent raters agree on an observation The degree to which independent raters agree on an observation Have two (or more) judges rate the same people Have two (or more) judges rate the same people Trained and independent raters, using a coding scheme Trained and independent raters, using a coding scheme

Observer 1Observer 2 Complain about injection-23 First negative comment01 Second negative comment -22 Rip up questionnaire-23 Interrater Reliability

Observer 1Observer 2 Complain about injection22 First negative comment00 Second negative comment-2-2 Rip up questionnaire23 Interrater Reliability

Internal Consistency Internal consistency – the degree to which all specific items of a measure behave the same way Internal consistency – the degree to which all specific items of a measure behave the same way Measure the same people with multiple items Measure the same people with multiple items –Different questions in a survey –Different behaviors in observation

Extraversion Not at all true Very true 1.I am outgoing.____ 2.I am friendly. ____ 3.I am talkative.____ 4.I am gregarious.____

Internal consistency Split-half reliability – correlation of scores on one half of the test with scores on the other half Split-half reliability – correlation of scores on one half of the test with scores on the other half Cronbach’s alpha – the average of all possible correlations between items Cronbach’s alpha – the average of all possible correlations between items

‘ One of these things just doesn’t belong’ One of these things is not like the others, One of these things just doesn't belong One of these things is not like the others, One of these things just doesn't belong Student 1Student 2Student 3 Ques 1 (Chpt 12) 1029 Ques 2 (Chpt 12) 938 Ques 3 (Chpt 3) 261 Ques 4 (Chpt 12) 1029

Test-Retest Reliability The degree to which a measure correlates positively with itself over time The degree to which a measure correlates positively with itself over time –Consistency of the measure over time Measure the same people at two (or more) points in time Measure the same people at two (or more) points in time Desirable for stable traits, but not for transient states Desirable for stable traits, but not for transient states

The “More is Better Rule” Reliability is likely to increase as we increase the number of… Reliability is likely to increase as we increase the number of… –Observers (or raters) –Observations (or items) –Occasions Measurement error will average out Measurement error will average out

II) Construct Validity How well an operational definition represents the construct of interest How well an operational definition represents the construct of interest The degree to which the construct can be inferred from the operational definition of that construct The degree to which the construct can be inferred from the operational definition of that construct

Indicators of Construct Validity Face validity Face validity Criterion validity Criterion validity –Predictive validity –Concurrent validity –Convergent validity –Discriminant validity

Face Validity Face validity – Does the measure appear to measure the construct of interest? Face validity – Does the measure appear to measure the construct of interest? –Does the measure “on the face of it” look like what it’s supposed to measure? Not necessary or sufficient for a good measure Not necessary or sufficient for a good measure

Predictive Validity Predictive validity – Is the measure associated with variables it should theoretically predict? Predictive validity – Is the measure associated with variables it should theoretically predict? LSAT – Law school performance LSAT – Law school performance Self-esteem – Depression Self-esteem – Depression Shyness – Social anxiety Shyness – Social anxiety

Concurrent Validity Concurrent validity – Does the measure differ between groups it ought to differ between? Concurrent validity – Does the measure differ between groups it ought to differ between? –Also called “known groups validity” E.g., clinically depressed versus non- depressed groups E.g., clinically depressed versus non- depressed groups

Convergent Validity Convergent validity – Is the measure associated with other established measures of the same construct? Convergent validity – Is the measure associated with other established measures of the same construct? Self-report - Observations Self-report - Observations Physiological measure - Self-report Physiological measure - Self-report Self-report 1 – Self-report 2 Self-report 1 – Self-report 2

Discriminant Validity Discriminant validity – Is the measure NOT associated with measures of other constructs? Discriminant validity – Is the measure NOT associated with measures of other constructs? Self-esteem scores not associated with locus of control scores Self-esteem scores not associated with locus of control scores Problem solving knowledge not associated with factual knowledge Problem solving knowledge not associated with factual knowledge

Measurement Reliability & Validity Reliability: Is the measure consistent? Reliability: Is the measure consistent? Validity: Does the measure adequately reflect the construct of interest? Validity: Does the measure adequately reflect the construct of interest? Reliable and ValidReliable, not ValidNot Reliable, not Valid

Relationship between Reliability and Validity Can be reliable but not valid Can be reliable but not valid To be valid it must be reliable – –But reliability is not sole condition for validity Both reliability and validity are necessary for accurate measurement in a research study.

Measurement Scales Nominal scales Nominal scales Ordinal scales Ordinal scales Interval scales Interval scales Ratio scales Ratio scales

Nominal Scales AKA Categorical scales AKA Categorical scales No numerical/quantitative properties. Categories or group simply differ from one another No numerical/quantitative properties. Categories or group simply differ from one another Examples: Examples: –Men or women –Right or left handed –Catholic, Protestant, Jewish, Hindu, Buddhist… –Numbers on basketball jerseys –Zip codes

Ordinal Scales Allow us to rank order the levels of the variables being studied Allow us to rank order the levels of the variables being studied Examples Examples –Social class  lower class, working class, middle class, and upper class –College football standings –Letterman’s Top Ten

Top Ten Questions to ask Yourself Before Eating Spinach? 10. Was my spinach properly sprayed with Lysol? 10. Was my spinach properly sprayed with Lysol? 9. Isn't it still safer than eating a New York City hot dog? 9. Isn't it still safer than eating a New York City hot dog? 8. So all those years my mom made me eat spinach, she was trying to kill me? 8. So all those years my mom made me eat spinach, she was trying to kill me? 7. Is this the right side dish for my Mad Cow burger? 7. Is this the right side dish for my Mad Cow burger? 6. Are my papers in order? 6. Are my papers in order? 5. If I get sick, will my wife TiVo Ventriloquist Week on the Late Show? 5. If I get sick, will my wife TiVo Ventriloquist Week on the Late Show? 4. Should I also avoid kale? 4. Should I also avoid kale? 3. If I'm going to eat something deadly, shouldn't it be delicious Pop-Tarts? 3. If I'm going to eat something deadly, shouldn't it be delicious Pop-Tarts? 2. What would Popeye do? 2. What would Popeye do? 1. Do I really want my obituary to read: "Man Dies A La Florentine?" 1. Do I really want my obituary to read: "Man Dies A La Florentine?"

Interval Scales The difference between the numbers on the scale is meaningful The difference between the numbers on the scale is meaningful Scores separated by equal intervals Scores separated by equal intervals Examples Examples –Temperature (Fahrenheit or Celsius) –Scores on personality measure

Ratio Scales Scores separated by equal intervals and there is an absolute zero Scores separated by equal intervals and there is an absolute zero Examples Examples –Length –Weight –Time –Number of responses

Level Qualitative Info Has inherent order ‘more to less’ Equal Intervals Has zero point NominalX OrdinalXX IntervalXXX RatioXXXX Scales of Measurement

Concept Check Which scale of measurement best describes the following: Which scale of measurement best describes the following: –Telephone numbers –Distances from Budapest to cities in the US –Scores on an extraversion personality assessment –Ranking of basketball teams in the Big Ten