Research Designs for Explanation Experimental, Quasi-experimental, Non-experimental, Observational

Causation Three types of evidence necessary to demonstrate that one variable ‘causes’ another A statistical association of the two variables (they co-vary) The cause (independent variable) occurs ‘before’ the effect (dependent variable) Variables other than the independent variable are eliminated as causes of the dependent variable

Experimental Designs Provide best means of obtaining evidence necessary to infer causality Researcher can: Assign subjects to different groups (random assignment) Manipulate the independent variable (expose particular groups to treatment) Control most environmental influences (changes are due to the independent variable) Two types – classical & randomized posttest only – if understand, can understand many other types

Experimental Designs: Classical Experiment Characteristics Subject randomly assigned to experimental or control groups (no systematic differences) Pretest administered measuring I.V. Experimental group receives treatment (all else held equal) Posttest administered to both groups Difference between the two groups is attributed to I.V. Quality of evidence – Very High All 3 types of evidence provided Random assignment to Exp and Control rules out most alternative explanations (threats to internal validity) Internal Validity still susceptible to selection-mortality and design contamination Construct Validity susceptible to Interaction of Testing & Treatment Example 3.1 R O 1 X O 2 R O 1 O 2

Experimental Designs: Classical Experiment Creating Equivalent Groups Preferred method: randomly assign half of a pool of subjects to the experimental group and half to the control group If enough subjects are used, differences between groups begin to disappear if each subject has the same probability of being in either group Alternative method: match each individual in the experimental group with an individual in the control group Researcher identifies characteristics that may effect dependent variable (confounding variables) Researcher pairs individuals based on similar characteristics Un-pair-able individuals are eliminated (from the study) Researcher randomly assigns one member to each group Can reduce the number of subjects needed – reduces variation Can be very hard to identify all contributing characteristics R O 1 X O 2 R O 1 O 2

Experimental Designs: Randomized Posttest Only No pretest Advantages over classic design Eliminates Interaction of Testing and Treatment Assumption is pretest is unnecessary because random sampling was used Most valid for ‘large’ population studies Recidivism example – how do you pretest? Example 3.3 R X O 2 R O 2

Experimental Designs: Limitations for Administrative Research Mostly see in program evaluations Used more in times of scarce resources Loss of research control is biggest issue Especially for studies of large programs Decisions to evaluate programs are usually after the fact and made by a different set of decision makers Data wasn’t collected in an adequate fashion Have to reconstruct with what you have

Quasi-experimental Designs If you cannot: Manipulate at least one independent variable Randomly assign groups Randomly assign treatments to groups Control the exposure of the experimental group to the treatment in isolation from other factors. Need to use a quasi-experimental design (the best available approximation of an experimental design) Three most common types: comparison group pretest/posttest; interrupted time-series; multiple-group interrupted time-series

Quasi-experimental Designs: Comparison Group Pretest-Posttest Characteristics Subjects not randomly assigned Consider police compressed work week to increase work satisfaction example Consider the effects of day-care example Internal and external validity considerations Selection threats become a hazard Selection-history, selection-maturation, and selection- instrumentation can be controlled by design Need to carefully document each Statistical regression is of particular concern – what if experimental police barracks had very high turnover rate? Harder to generalize for external validity – but can still replicate Example 3.4 (N) O 1 X O 2 (N) O 1 O 2

Quasi-experimental Designs: Interrupted Time Series Design Characteristics Incorporates independent variable other than time into the time- series design Several observations before and after change independent variable Variations in independent variable result in: Abrupt permanent change Abrupt temporary change (possibly example 3.2) Gradual permanent change (finding a new level) Need to differentiate from long-term trends, cyclical variations, seasonal trends, and random fluctuations (figure 3.1) Internal and external validity considerations Most internal threats are controlled for by design or easily controlled for (is statistical regression controlled for?) History is not controlled for Interaction between selection and treatment (are the people being measured the same throughout? Consider the year-round schooling example – how might you control for this? Figures 3.1 and 3.2 (N) O 1 O 2 O 3 O 4 X O 5 O 6 O 7 O 8

Quasi-experimental Designs: Multiple Group Interrupted Time Series Characteristics Addition of a comparison time series More easily control for selection threats to internal validity Example 3.5 (N) O 1 O 2 O 3 O 4 X O 5 O 6 O 7 O 8 (N) O 1 O 2 O 3 O 4 O 5 O 6 O 7 O 8

Non-experimental Designs No control for threats of internal validity But, can be quick, cheap and very useful to decision makers – Chinese newspaper transit advertisement example Can also have serious limitations – South Carolina welfare-to-work example Three types: single group posttest; single group pretest/posttest; Nonequivalent groups posttest only

Non-experimental Designs: Single Group Posttest Why use? Quick method to determine if intervention met expectations – Chinese newspaper advertisement example (N) X O 2

Non-experimental Designs: Single Group Pretest-Posttest Characteristics Indicates in ‘something’ changed between pretest and posttest It ‘might’ be the intervention – might not Consider tax preparer example p.86 (N) O 1 X O 2

Non-experimental Designs: Non-equivalent Groups Posttest Could this be called comparison group posttest? Again, no internal validity, but useful for quick, cheap identification of major aspects of what is being studies Best used when the decision to be made will cost little and can be easily changed again Example 3.6 (N) X O 1 (N) O 1

In-Class Probllk;’ems In 5 groups of 3 (2 if need be), each group answer one of the following: Questions for Review #7 Problems for Homework #s 1, 3, 4, 8 Each group will present their answers