Experiments: Validity, Reliability and Other Design Considerations Lecture 6

Controls At least three different meanings: Controlled Studies Control in an Experiment Controls used in a study or analysis

Experiments and Observational Studies Assigning people to groups vs. observing people who ‘assign’ themselves Example of pitfalls in experimental assignment: Portacaval Shunt Example Examples of key pitfalls of observational studies: Cervical cancer and circumcision study Alcohol consumption and lung cancer

Measurement and Design Validity Measurement Concerns Construct Validity Design Concerns Internal Validity External Validity Ecological Validity Construct validity: measuring our intended concept? Internal validity: External: Ecological:

Construct Validity How do we know that our independent variable is reflecting the intended causal construct and nothing else? “Face” validity deals with subjective judgement of appropriate operationalization “Content” validity is a more direct check against relevant content domain for the given construct. Examples from operationalization exercise?

Internal Validity Internal Validity deals with questions about whether changes in the dependent variable were caused by the treatment.

? ? Cause Effect ? ? What about “confounds”

Threats to Internal Validity History additional I.V. that occurs between pre-test and post-test Maturation Subjects simply get older and change during experiment Testing Subjects “get used” to being tested Regression to the Mean Issue with studies of extremes on some variable

Contamination and Internal Validity Demand Characteristics Anything in the experiment that could guide subjects to expected outcome Experimenter Expectancy Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy)

General Demand Characteristics Evaluation Apprehension “Hawthorne Effect” Temporary improvement based on observation Solutions Double-blind experiments Experiments in natural setting (i.e., subjects do not know they are in an experiment) Cover stories Hidden measurements Subjects know that they are being evaluated and this changes their behavior

Reducing the role of the experimenter: solving expectancy effects Naïve experimenter Those conducting study are not aware of theory or hypotheses in the experiment Blind Researcher is unaware of the experiment condition that he/she is administering Standardization Experimenter follows a script, and only the script “Canned” Experimenter Audio/Video/Print material gives instructions

And More! Selection Bias Mortality Diffusion, Sharing of Treatments Issue with non-random selection of subjects Mortality Departure of subjects in the experiment Diffusion, Sharing of Treatments Control group unexpectedly obtains treatment Other ‘social’ threats? Compensatory rivalry, resentful demoralization, etc.

Three threats to external validity (generalizability) in experiments External Validity– How far does the given experiment generalize to similar groups, individuals, etc? Setting Population History Setting:Physical and social context of the experiment Population:Is there something specific about the sample that interacts with the treatment? Histor: Is there something about the time that interacts with the treatment?

Ecological Validity Approximation of ‘real-life’ situations

The Validity Tradeoff: Truth and Myth Internal Validity External & Ecological Validity Why is internal validity considered so much more important than external validity? Are there situations where we can more easily give up internal validity for the sake of ecological or external validity? Balance is important between the types of validity, but internal validity is usually (if not always) the more important factor.

True Experiments in the Field Some experiments can be conducted in a real-world setting while maintaining random assignment and manipulation of treatments Is this a random sample? Random assignment? Milliman (1986) Study of music tempo and restaurant customer behavior Cheshire and Antin (2008) Study of Incentives and Contributions of Information in an Online Setting

Natural Experiments 1998 Total Solar Eclipse: testing temperature of sun’s corona

Pro’s and Con’s of Experiments Gives researcher tight control over independent factors Allows researcher to test key relationships with as few confounding factors as possible Allows for direct causal testing Con’s Often very small N; enough for statistical purposes but not ideal for generalizability Sometimes give up large amounts of external validity in favor of construct validity and direct causal analysis Require a large amount of planning, training, and time– sometimes to test relationship between only 2 factors!

Additional considerations before using experiments Cost and Effort Is the effort worth it to test the concepts you are interested in? Manipulation and Control Will you actually be able to manipulate the key concept(s)? Importance of Generalizability Are you testing theory, or trying to establish a real-world test?