Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is internal validity? Evaluating the Experiment from the Inside An experiment is internally valid when the effects on the dependent variable are due to the independent variable. An internally valid experiment is free of confounding.

What is a manipulation check? Evaluating the Experiment from the Inside A manipulation check evaluates how well the experimenter manipulated the experimental situation. A manipulation check determines whether subjects followed directions and were appropriately affected by our treatments.

What did Orne (1969) mean by a pact of ignorance? Evaluating the Experiment from the Inside Subjects expect their data to be discarded if they guess the experimental hypothesis, and don’t volunteer this information to the experimenter. Experimenters don’t want to test additional subjects and may take subject reports at face value.

How can researchers overcome this problem? Evaluating the Experiment from the Inside Debrief subjects after the experiment and convey that you want to know if they guessed the hypothesis. Provide incentives for guessing the hypothesis.

Which mistakes could produce this threat to internal validity? Evaluating the Experiment from the Inside Selecting the wrong statistical test Using a t-test to analyze ordinal data. Improperly using a statistical test Calculating multiple t-tests. Drawing the wrong conclusions from the test Reporting p =.07 as a trend.

Explain external validity. The Problem of External Validity An experiment is externally valid when its findings can be extended to other situations and populations.

What two requirements must an externally valid study satisfy? The Problem of External Validity 1. The experiment must be internally valid. 2. The experimental findings can be replicated.

What does it mean to generalize across subjects? Why is this important? The Problem of External Validity The findings can be extended to a larger group than our sample. Generalizing across subjects is critical to the external validity and usefulness of experimental findings.

Which problems prevent us from generalizing across subjects? The Problem of External Validity The samples used in psychological research are often biased and may not represent the larger population. The samples may not always represent even college sophomores since we heavily depend on volunteers.

Explain generalizing from procedures to concepts. The Problem of External Validity Experimental variables like anger may have multiple operational definitions. When we generalize from our experimental results, we move from discussing our specific operational definition of anger to discussing the concept of anger itself.

Why is this a problem in research? The Problem of External Validity It is dangerous to generalize from a single experiment’s operational definition of anger. We cannot be sure of the reliability or validity of our procedures.

What is research significance? The Problem of External Validity A study achieves research significance when its findings clarify or extend knowledge gained from previous studies and raise implications for broader theoretical issues.

When should we question novel findings? The Problem of External Validity We should question novel findings when they contradict prior findings that have been successfully replicated. The burden of proof is on the experimenter who claims novel findings to explain this discrepancy.

Why do we want to generalize beyond the laboratory? The Problem of External Validity We want to generalize beyond the laboratory to increase the external validity of our findings.

What is the problem in extending laboratory findings? The Problem of External Validity Since extraneous variables are uncontrolled in real world setting and operate in complex combinations, they can modify the influence of our individual variables.

What is the trade-off between laboratory and field experiments? The Problem of External Validity The trade-off is between the laboratory’s more precise control of extraneous variables and the field experiment’s greater realism and external validity.

What did Hanson (1980) find about the agreement between laboratory and field studies? The Problem of External Validity Hanson (1980) found that more laboratory than field studies reported a positive correlation between reported attitudes and behavior.

When can we extend results from an experiment to everyday life? The Problem of External Validity We can’t confirm external validity until additional studies are completed in field settings. Researchers can increase and verify the external validity of laboratory findings using aggregation, multivariate designs, nonreactive measurements, field experiments, and naturalistic observation.

What is aggregation? What is its logic? The Problem of External Validity Aggregation is the grouping together and averaging of data to increase external validity. Combining the results of experiments with different subjects and methodologies increases the generality and external validity of our findings.

How is this logic shared by meta-analysis? The Problem of External Validity Meta-analysis uses statistical analysis to combine and quantify data from many comparable experiments to calculate an average effect size.

How does aggregation establish external validity? The Problem of External Validity Aggregation establishes external validity by combining the results of experiments performed using different subjects, stimuli and/or situations, trials or occasions, and measures.

What is a multivariate design? The Problem of External Validity A multivariate design studies multiple DVs. For example, a study of repetitive strain places a computer keyboard at different distances from the subject IV) and measures the effect on three different muscle groups (3 DVs).

What is the advantage of multivariate designs? The Problem of External Validity Multivariate designs allow us to study the effect of an independent variable on combinations of dependent variables. These designs better simulate the complexity of the real world than univariate designs and provide more detailed information.

How are multivariate experiments analyzed? The Problem of External Validity We analyze multivariate experiments with a multivariate analysis of variance (MANOVA).

How should researchers handle a nonsignificant outcome? Handling a Nonsignificant Outcome Accept the outcome, don’t reframe your result as “almost significant.” Examine the experimental procedures for design flaws. If the design appears sound, decide whether the hypothesis was reasonable.

How should we handle the possibility of faulty procedures? Handling a Nonsignificant Outcome Check for possible causes of a nonsignificant outcome like: 1.confounding 2.extraneous variables that increase within- subjects variability 3.weak manipulation of the IV

How should we handle the possibility of faulty procedures? Handling a Nonsignificant Outcome 4.inconsistent or flawed procedures 5.ceiling and floor effects 6.insufficient power

How should we handle the possibility of a faulty hypothesis? Handling a Nonsignificant Outcome If previous studies supported the hypothesis and ours did not, look for differences in experimental design or sample. If there was no previous support and our design and execution were good, we may have to revise or discard our hypothesis.