1. Experimental and Quasi-Experimental Designs
Chapter 5
Experimental and Quasi-Experimental Designs

2. Introduction
Experimentation is an approach to research best suited for explanation and evaluation
An experiment is “a process of observation, to be carried out in a situation expressly brought about for that purpose”
Experiments involve:
Taking action
Observing the consequences of that action
Especially suited for hypothesis testing

3. The Classical Experiment
Variables, time order, measures, and groups are the central features of the classical experiment
Involves three major pairs of components:
Independent and dependent variables
Pretesting and posttesting
Experimental and control groups

4. PRE-EXPERIMENTAL RESEARCH DESIGN
One-shot case study - single group of subjects is measured on a variable following experimental stimulus
One-group pretest-posttest design - adds a pre-test for the group, but lacks a control group
Static-group comparison - includes experimental and control group, but no pre-test

5. One-Shot Case Study
A man who exercises is observed to be in trim shape

6. One-Group Pretest-Posttest Design
An overweight man who exercises is later observed to be in trim shape

7. Static-Group Comparison
A man who exercises is observed to be in trim shape while one who doesn’t is observed to be overweight

8. Independent Variables
The Independent Variable takes the form of a dichotomous stimulus that is either present or absent
It varies (i.e., is independent) in our experimental process
“The Cause”

9. Dependent Variables The outcome, the effect we expect to see
Depends on the Independent Variable
Might be physical conditions, social behavior, attitudes, feelings, or beliefs
“The Effect”

10. Pretesting and Posttesting
Subjects are initially measured in terms of the Dependent Variable prior to association with the Independent Variable (pretested)
Then, they are exposed to the Independent Variable
Then, they are re-measured in terms of the Dependent Variable (posttested)
Differences noted between the measurements on the Dependent Variable are attributed to influence of the Independent Variable

11. Experimental and Control Groups
Experimental group – Exposed to whatever treatment, policy, initiative we are testing
Control group – Very similar to experimental group, except that they are NOT exposed
If we see a difference, we want to make sure it is due to the Independent Variable, and not to a difference between the two groups

12. DIAGRAM OF BASIC EXPERIMENTAL DESIGN

13. Double-Blind Experiment
Experimenters may be more likely to “observe” improvements among those who received drug
In a Double-Blind experiment, neither the subjects nor the experimenters know which is the experimental group and which is the control group
Broward County Florida and Portland, Oregon domestic violence policing units study: “keeping safe” strategies

14. Selecting Subjects
First, must decide on target population – the group to which the results of your experiment will apply
Second, must decide how to select particular members from that group for your experiment
Cardinal rule – ensure that Experimental and Control groups are as similar as possible
Randomization purposes towards this

15. SELECTING SUBJECTS: OPEN MATRIX ILLUSTRATION
Three Requirements of Selecting Subjects:
Probability sampling
Randomization
Matching

16. Random assignment “Randomization”
Central feature of the classical experiment
Produces experimental and control groups that are statistically equivalent
Farrington and associates:
“Randomization insures that the average unit in the treatment group is approx. equivalent to the average unit in another group before the treatment is applied”
“All Other Things are Equal”

17. EVALUATION OF EXPERIMENTAL METHOD
Strengths:
Isolation of the experimental variable over time
Experiments can be replicated several times using different groups of subjects
Weaknesses:
Artificiality of laboratory setting
Social processes that occur in a lab might not occur in a more natural social setting

18. Experiments and Causal Inference
Experiments potentially control for many threats to the validity of causal inference
Experimental design ensures:
Cause precedes effect via taking posttest
Empirical correlation exists via comparing pretest to posttest
No spurious 3rd variable influencing correlation via posttest comparison between experimental and control groups, and via randomization

19. Threats to Internal Validity
Conclusions drawn from experimental results may not reflect what went on in experiment
History: External events may occur during the course of the experiment
Maturation: People constantly are growing
Testing: The process of testing and retesting

20. Threats to Internal Validity
4. Instrumentation: Changes in the measurement process
5. Statistical regression: Extreme scores regress to the mean
6. Selection biases: The way in which subjects are chosen (use random assignment)
7. Experimental mortality: Subjects may drop out prior to completion of experiment

21. Generalizability and threats to validity
Potential threats to internal validity are only some of the complications faced by experimenters; they also have the problem of generalizing from experimental findings to the real world
Two dimensions of generalizability:
Construct Validity
External Validity

22. Threats to Construct Validity
Concerned with generalizing from experiment to actual causal processes in the real world
Link construct and measures to theory
Clearly indicate what constructs are represented by what measures
Decide how much treatment is required to produce change in Dependent Variable

23. Threats to External Validity
Significant for experiments conducted under carefully controlled conditions rather than more natural conditions
Reduces internal validity threats
John Eck (2002): "diabolical dilemma."
Suggestion:
explanatory studies  internal validity
applied studies  external validity

24. Threats to Statistical Conclusion Validity
Becomes an issue when findings are based on small samples
More cases allows you to reliably detect small differences; less cases result in detection of only large differences
Finding cause-and-effect relationships through experiments depends on two related factors:
Number of Subjects
Magnitude of posttest differences between the experimental and control groups

25. Variations in the classical experimental design
Four basic building blocks present in experimental designs:
The number of experimental & control groups
The number & variation of experimental stimuli
The number of pretest & posttest measurements
The procedures used to select subjects and assign them to groups
Variations on the classical experiment can be produced by manipulating the building blocks of experiments

26. Quasi-Experimental Designs
When randomization isn’t possible for legal or ethical reasons
Renders them subject to Internal Validity threats
Quasi = “to a certain degree”
Two categories:
Nonequivalent-groups designs
Time series designs

27. Nonequivalent-Groups Designs
When we cannot randomize, we cannot assume equivalency; hence the name
We take steps to make groups as comparable as possible
Match subjects in Experimental and Control groups using important variables likely related to Dependent Variable under study
Aggregate matching – comparable average characteristics

28. Cohort Designs
Cohort – Group of subjects who enter or leave an institution at the same time
Ex: A class of police officers who graduate from a training academy at the same time, All persons who were sentenced to probation in May
Necessary to ensure that two cohorts being examined against one another are actually comparable

29. Time-Series Designs Longitudinal Studies
Examine a series of observations over time
Interrupted – Observations compared before and after some intervention
Used in cause-and-effect studies
Instrumentation threat to internal validity is likely because changes in measurements may occur over a long period of time
Often use measures produced by CJ organizations

30. Variable-Oriented Research and scientific realism
A large number of variables are studied for a small number of cases or subjects
Case-oriented research: Many cases are examined to understand a small number of variables (Boston Gun Project)
Variable-oriented research: A large number of variables are studied for a small number of cases or subjects
Case Study Design: Centered on an in-depth examination of one or a few cases on many dimensions