Measurements and Validity Julia Braverman, PhD Division on Addictions

Types of measures Michael John ANXIETY??

Types of measurement 1. Objective/Physiological measures Bodily activity, nervous system. Response time 2. Observational measures Direct observing participants. 3. Self-report Participants provide information about themselves.

Converging operations Using several measurement approaches to measure a particular variable

Basics of psychometrics: How to build a trait/state assessment measure? Concept Affect E.g. I feel sad Behavior E.g. I cannot sleep, I cry a lot Cognition E.g. I think about suicide. Question format (Likert scale, yes/no, reverse scale)

Measure quality 1. Reliability 2. Validity

Reliability The degree of consistency between observations made by the same measurement tool.

Measurement Error. No measure is perfect. Observed score = True score + Measurement error. True score – is the score that the participant would have obtained if our measure were perfect.

Sources of measurement errors 1. Transient states Mood, health, anxiety 2. Stable attributes Suspicious participant may distort their answers 3. Situational factors Weather outside, baseball game. 4. Characteristics of the measure E.g. instruction ambiguity 5. Actual mistakes

Theoretical concept of reliability. Systematic variance Reliability = Total variance 0 < Reliability < 1

Assessing reliability 1. Test-retest reliability Measuring the same thing twice. Reliability = correlation ( r) between results of the first and the second measurements. High reliability >.70

Assessing reliability 1. Test-retest reliability Problems Memory Experience

Assessing reliability Interitem Reliability - Measure of consistency among the items on a scale. 1. Item-total correlation  For each item how it is correlated with the sum of other items. > Split-item reliability  Divide the items on the scale into 2 sets and test the correlation (instead of test-retest). 3. Cronbach’s alpha coefficient  Average of all possible split-half reliabilities.

Benevolent sexism scale: 1 (disagree) – 7 (agree) 1. Women should be cherished and protected by men. 2. Women, compared to men, tend to have a superior moral sensibility. 3. Men should be willing to sacrifice their own well-being in order to provide financially for the women in their lives. 4. Many women have a quality of purity that few men possess. 5. A good woman should be set on a pedestal by her man. 6. Men are complete without women.

Made-up table of item-total correlations Item #r

Made-up table of item-total correlations Chronbach α =.85 Item #r Chronbach α (without the item)

Assessing reliability Interrater reliability – consistency between two or more raters or judges who observe the same behavior. High reliability >.70

Increasing the Reliability Measures 1. Standardize administration of the measure Same test conditions 2. Clarify instructions and questions. To reduce ambiguity and misinterpretations. Pretest questionnaires if possible. 3. Train observers. To increase interrater reliability. 4. Minimize error in coding data.

Validity If the measurement actually measures what it is supposed to measure Different from reliability Same measure maybe valid for one purpose and invalid for another one.

Assessing validity 1. Face validity – if a measure appears to be valid. Does not mean actual validity. E.g. SAT reading comprehension test Does it measure reading comprehension or common sense? (Katz et al., 1990) Affect motivation to participate?

Assessing validity 2. Construct validity Relation to other measures.  Convergent validity  High correlation with conceptually relevant measures. Discriminate validity Low correlation with conceptually unrelated constructs

Assessing validity 3. Criterion-Related validity – the correlation between the measure and some current behavior. E.g. IQ and GPA Doctor’s productivity Peer evaluation Patient evaluation

Assessing validity 3. Predictive validity – the ability of a measure to predict a certain behavior/situation in a future. E.g. SAT and GPA or GPA and after-college salary. Doctor’s productivity ?

Test bias Test is biased if it is not equally valid for everyone who takes the test. Groups with the same ability obtain different scores on the test.

Reliability and Validity If reliable May be valid or not. If not reliable Not valid

Threats to measurement validity Using non-validated measures Solution Validate the measure Use pre-validated measures

Threats to measurement validity Loose connection between theory and method. Disagreement between conceptional and operational definitions. E.g. putting more pepper as a measurement of aggression? Solution Validate your measure with previous measurements

Threats to measurement validity Social desirability (evaluation apprehension) – Desire to look “normal” or to be judged favorably by another person (including the experimenter). Solutions Anonymity Ask indirect questions “How many drinks an average college student have during a party?”

Threats to measurement validity Yes-bias Extreme-score bias Solution Reverse score. Z-transformation within an individual.

Threats to measurement validity Testing effects Most participants perform better on a test of personality/behavior/IQ measure the second time they take it. Reasons Learning (e.g. IQ test) Practice (e.g. physical skills) Learn the test goal (e.g. personality test) Attitude polarization Thinking about their attitudes

Threats to measurement validity Testing effects Solutions Control group No pretest Long waiting period

Validity of experiment Internal validity Extent to which a study provides evidence of a cause-effect relationship between the variables. External validity The ability to generalize results of the experiment.

Internal validity 3 conditions to determine causality Covariation Temporal sequence No confounds Low internal validity – the conclusion that A affects B is wrong.

Threats to internal validity Role demands – participants’ expectations to what an experiment requires them to do Good-subject tendency E.g. hypnosis and antisocial acts Participants reactance E.g. What is the weather today?

Threats to internal validity Role demands Solution Cover story E.g. Independent studies Add non-relevant tasks, items (For measurements)

Threats to internal validity Experimenter bias E.g. Gratitude study Solution Double-blind

Threats to internal validity Hawthorne effect – Increases in productivity that occur when participants know they are being studied. Workers responded to any change in working conditions by working harder than usual. Solution Control group

Common Threats to Internal Validity of Quasi-experiments History Something occurred between the pretest and posttest. Maturation Normal time changes Regression to the mean If extreme scored Ss. were selected. Pretest sensitization Pretest affects the posttest results Selection bias Comparison groups differed from the beginning Local history Contemporary history Attrition/mortality Only most motivated participants stay Only participants who experience less adverse effects of treatment stay

External validity How well the findings of an experiment generalize to other situations or populations.

Threats to external validity Other subjects Sampling/selection bias Other times Other settings

Threats to external validity Sampling bias Motivated volunteers Those available (at home, have phone)

Threats to external validity Other setting Artificial experimental environment

External validity External validity - the ability to generalize results of the experiment. Tight control - highly specific and artificial situation -> less external validity. Internal validity External validity

You are a researcher. In your experiment, you assign the first 20 people in your study to the experimental condition and the second 20 people to your control condition. This could pose a threat to: Internal validity Reliability External validity Construct validity

Saying that some measure is ________ definitely means it is also __________. valid, reliable reliable, valid nominal, numerical observational, self-report none of the above

An experimenter wants to examine if a new behavioral intervention program increases compliance among hypertension patients. For this purpose she recruits hypertension patients with low medication compliance and tests their compliance before and after the intervention. What are the potential threats to internal validity: Regression to the mean Maturation History Pretest sensitization All of the above

Find a threat/threats to internal validity The Alzheimers Center wants to evaluate the effectiveness of their support groups for caregivers of individuals with Alzheimers Disease. The caregivers are given the choice when they first come to the center as to whether they want to join these support groups. The center gives a stress measure to the caregivers that attend these weekly meetings, once they have attended meetings for three months. They also administer the same stress measure to the caregivers who have not attended the support groups, as a control group. Both groups of caregivers are married to the person with Alzheimers disease and both groups have been involved with the center for the same length of time

Any questions?