Validity Threats to Validity Threats to validity – definition Theory vs. measurement Types of validity – Conclusion validity – Internal validity – Construct validity – External validity Particular threats to validity

Validity Threats to validity - definition Researchers do not simply report facts. The researcher’s task is like describing an object covered by a thick blanket. The description is not like a photograph – it is not “veridical.” It can be wrong in a variety of ways. Problems that cause our descriptions to be wrong are problems of validity.

Validity Theory vs. measurement Our interest is always in the validity of a conclusion or inference That validity can be threatened either by poor measurement or by poor theory Theory Measurement

Validity Theory vs. measurement Our theory predicts that we will observe a relationship between measurable variables (that is, variables we have operationalized). We won’t see the expected relationship if – the theory is incorrect, or – the operationalizations are flawed.

Validity Cause Independent Variable Effect Dependent Variable

Validity Theory vs. measurement Suppose you believe that absence makes the heart grow fonder. Your theory has two constructs: – Absence – Fondness. Both must be ‘operationalized.’

Validity AbsenceFondness # days apart# phone calls We might not see the expected relationship because the theory is wrong # days apart# phone callsAbsenceFondness We might not see the expected relationship because our operationalizations are poor.

Validity Types of Validity Does our study have: – conclusion validity? – Is the relationship between variables seen in the data the one that our theory claims should be there?

Validity Types of Validity Does our study have: – conclusion validity? – internal validity? – Is the relationship between variables causal? (A  B)

Validity Types of Validity Does our study have: – conclusion validity? – internal validity? – construct validity? Does the I.V. capture our theoretical cause? Does the D.V. capture our theoretical effect?

Validity Types of Validity Does our study have: – conclusion validity? – internal validity? – construct validity? – internal validity? – Will the findings in our study be replicated in other settings?

Validity Conclusion Validity I.V.D.V.

Validity # days apart# phone calls money left Internal Validity If there is some confounding variable – one that varies with our I.V. – then our study lacks internal validity

Validity Construct Validity CauseEffect I.V.D.V.

Validity Absence Distance Fondness # thoughts about loved one A different operationalization might produce different conclusions – e.g, if distance varies with the purpose of the trip, which in turn determines how busy the traveler is through the day.

Validity External Validity CauseEffect

Validity Particular threats to validity On previous slides, we’ve discussed several types of validity. The next slides list some threats to each type. You should be able to discuss and give examples of each of these threats.

Validity Threats to Internal Validity History changes happening to only one of two or more groups that are compared

Validity Threats to Internal Validity History Maturation changes in performance associated with development

Validity Threats to Internal Validity History Maturation Test practice changes in performance due to experience with the test format

Validity Threats to Internal Validity History Maturation Test practice Regression to the mean extreme values of measurements are unlikely to be repeated

Validity Threats to Internal Validity History Maturation Test practice Regression to the mean Selection between-group differences that are present at the start

Validity Threats to Internal Validity History Maturation Test practice Regression to the mean Selection Mortality differences produced by different rates of drop- out between groups

Validity Threats to Conclusion validity Conclusion validity may be low because of: – poor reliability increase number of items used in the task or use more precise measuring instruments

Validity Threats to Conclusion validity Conclusion validity may be low because of: – poor reliability – weak relationship increase strength by increasing “dose” decrease “noise” by increasing reliability or decreasing distractions in the environment

Validity Threats to Conclusion validity Conclusion validity may be low because of: – poor reliability – weak relationship – lack of statistical power Increase number of observations, by – increasing number of subjects – increasing number of trials per subject

Validity Threats to Construct Validity Loose connection between theory and method Ambiguous effect of independent variables – Hawthorne effect – Good subject tendency – Evaluation apprehension (social desirability, white coat syndrome)

Validity Threats to Construct Validity Interaction of different treatments Interaction of testing and treatment.

Validity Threats to External Validity Any dimension on which studies can differ might produce a failure to replicate. – Other people Our subjects might be special in some way

Validity Threats to External Validity Any dimension on which people can differ might produce a failure to replicate. – Other people – Other places our environment might be special in some way especially if we work in a lab.

Validity Threats to External Validity Any dimension on which people can differ might produce a failure to replicate. – Other people – Other places – Other times our time might be special in some way e.g., right after a government report on smoking is released, you survey smokers

Validity Challenge For each of the following maxims, suggest some operationalization of the relevant constructs: Too many cooks spoil the broth Many hands make light work You snooze, you lose A rolling stone gathers no moss