External Validity Do the research findings generalize beyond the specific context/situation in which the data were collected? Generally, do your results generalize –To other populations –Across situations –Over time
Threats to External Validity Shadish, Cook, & Campbell (2002) identify four threats to external validity, –Interaction of the causal relationship with units –Interaction of the causal relationship over treatment variations –Interaction of the causal relationship with outcomes –Interaction of the causal relationship with settings
Units? “Interaction of the causal relationship with units” refers to the possibility that one’s findings may be dependent on the people sampled Thus, are the results based on a sample of students biased by the characteristics of those students?
Examples Are the outcomes of a study on racism equally valid, if the data are collected only in a predominantly white northern city? a racially-mixed southern city? Are the results of a study on endoscopic surgery valid if the data are collected with non-surgeons?
Treatment Variations? “Interaction of the causal relationship over treatment variations” refers to the possibility that one’s findings may be dependent on how the Independent Variable was manipulated Thus, are the results biased by a peculiar feature of how the Independent Variable was implemented?
Examples Are the outcomes of a study on depth perception dependent on the instructions given to the participants? Are the results of a study on helping behavior dependent on the fact that confederate carried a great deal of material?
Outcomes? “Interaction of the causal relationship with outcomes” refers to the possibility that one’s findings may be dependent on how the Dependent Variable was measured Thus, are the results biased by a peculiar feature of how the Dependent Variable was measured?
Examples Are the conclusions of a study on distance education different if you use multiple-choice or essay tests? Are the conclusions of a study on mental workload different if you measure it by looking at performance, or how well they do on a secondary task?
Settings? “Interaction of the causal relationship with settings” refers to the possibility that one’s findings may be dependent on the experiment’s setting, i.e., where/when it was conducted Thus, are the results biased by a peculiar feature of when/where the study was conducted?
Examples Are the results of a depth perception study biased because it was conducted indoors? Are the results of a study on conformity biased because it was conducted when conformity was valued?
Impact Over the last 20 years or so, the topic of external validity has become a major concern in the minds of scientists and lay-people alike This causes folks to dismiss some research (possibly rightly so), because it does not meet some minimal level of external validity
Mook’s (1983) Retort Mook points out that the ability to generalize to other populations, settings, variables, is not always intended or even desirable Furthermore, he argues that external invalidity can be quite appropriate –It all just depends on what you are trying to do with your experiment
Harlow (1958) The studies were designed to test the drive-reduction theories of attachment that were paramount in the 1940’s and 50’s
Harlow (1958) How would Harlow’s study stand up to the threats to external validity? –Interaction of the causal relationship with units –Interaction of the causal relationship over treatment variations –Interaction of the causal relationship with outcomes –Interaction of the causal relationship with settings
Other Research Goals Asking whether or not something can happen, rather than does it happen For example, can the presence/absence of eyeglasses on a person affect someone’s impression of that person’s intelligence?
Other Research Goals Asking whether or not something should happen in the lab For example, if the tension-reduction hypothesis is correct, then shouldn’t subjects threatened with electric shock drink more than subjects who are not so threatened?
Other Research Goals Showing the power of a phenomenon because it shows up under unnatural circumstances For example, aren’t the results of the famous obedience studies more amazing, because the subjects really had nothing stopping them from non-compliance?
Other Research Goals Using the lab to examine processes that should occur in real-life as well For example, a lot is known about the process of dark adaptation –That knowledge was acquired by asking subjects to report whether or not they saw a light, while sitting quietly in a dark room
Mook’s (1983) Point Mook was not dismissing external validity He was asking each of us to think about whether or not external validity is desirable in our studies So, he asks that each of us think through, case by case, –What conclusion do I want to make about my study’s outcome? –Do the sample, setting, and/or procedures prevent us from drawing that conclusion?
Our studies
The goal of our studies is to determine whether or not Verbal Estimate-based training can affect subsequent verbal or active behaviors –Criticisms of our methodology based on concerns about external validity are unfounded