Internal Validity EAF 410 July 25, 2001

Is the observed relationship due to something other than what I studied?

Many alternative hypotheses. They are “threats to internal validity.”

Validity -- 3 types b Instrument (measurement) b External (generalization) b Internal (ambiguity)

Internal Validity b Observed relationship among two or more variables unambiguous. b Relationship not due to “something else.”

Threats to Internal Validity

Subject Characteristics b Individuals/groups differ in unintended ways agegenderagegender ethnicityintelligenceethnicityintelligence attitudevocabularyattitudevocabulary

Mortality b Loss of subjects drop outdrop out don’t complete testsdon’t complete tests don’t complete questionnairesdon’t complete questionnaires

b Limits generalizability b Introduces bias different responses from those who dropped outdifferent responses from those who dropped out b What can be done about this threat?

Location b Influence of where research is conducted resourcesresources environmentenvironment powerpower b Hold location constant

Instrumentation b Instrument Decay nature of instrument changednature of instrument changed –scoring changes –item changes

Instrumentation b Data Collector Characteristics characteristics of researcher affects datacharacteristics of researcher affects data –gender –age –ethnicity –language

Instrumentation b Data Collector Bias unconsciously distort dataunconsciously distort data –treat groups differently –leading questions –judgement/interpretation Standardize proceduresStandardize procedures Planned ignorancePlanned ignorance

Testing b Improvement due to pretest subjects “alerted” to topic by pretestsubjects “alerted” to topic by pretest practice on pretest helpspractice on pretest helps

History b Unanticipated, unplanned for events occur during study b Individual/group experiential differences

Maturation b Change due to passing of time b Serious threat to studies using pre-post data for intervention group b Control with well-selected comparison group

Attitude of Subjects b How subjects view the study b Hawthorne Effect effects because of increased attention to subjectseffects because of increased attention to subjects

Regression b Group selected because of low or high scores will, on average, score closer to the average on subsequent tests b Control with equivalent comparison group

Implementation b Experimental group treated in unintended ways different individuals implement different treatmentsdifferent individuals implement different treatments personal bias for one treatmentpersonal bias for one treatment

Finding a Relationship b Threats to internal validity can reduce likelihood of finding a relationship

Minimize the Threats b Standardize study conditions b Obtain subject information b Obtain study details where, when, extraneous eventswhere, when, extraneous events b Choose appropriate design

b Threats reduced by planning. b Planning requires collection of information before the study begins. b Often too late to control them once the data have been collected.