Quasi-Experiments

The Basic Nonequivalent Groups Design (NEGD) l Key Feature: Nonequivalent assignment NOXONOONOXONOO

What Does Nonequivalent Mean? l Assignment is nonrandom. l Researcher didn’t control assignment. l Groups may be different. l Group differences may affect outcomes.

Equivalence l “Equivalent” groups are not necessarily identical on any pre-test measure. l Merely implies that if the random assignment procedure was repeated, the groups would tend toward equivalence.

Non-Equivalence l Non-equivalent groups do not necessarily differ on any pre-test measure. l Merely implies that If the same non- random assignment procedure was repeated, the groups would tend to toward non-equivalence. l If assignment to groups was based partly on income, then groups would tend to have different expected mean levels of income – but any two groups you picked might well be similar in income levels.

The Point l Equivalence or non-equivalence is defined by the selection procedure. l Even if the difference in pre-test means across groups is “small,” this does not imply that the groups are equivalent. –Small differences can introduce big threats.

Quasi- vs. Natural vs. Experiment l In a true experiment, the researcher performs the random assignment –Can be in a lab or the field l In a natural experiment, someone else assigns through a “random” process. l In a quasi-experiment, assignment is not random, introducing selection threats. –Much stronger if the selection is not done by the cases themselves (exogenous sorting).

What is a Natural Experiment l Strict Definition: –Some truly natural process, such as rainfall or weather patterns, assigns IV. l Definition we all use in our own work: –Some exogenous process, rather than our cases, ourselves, or a causal process relevant to our theory, assigns IV.

Genres of Natural Experiments The natural border or natural disaster –Jared Diamond’s islands –Dan Posner’s rivers –Caroline Hoxby’s streams –Settler mortality (Acemoglu, Johnson, and Robinson) –Hurricane Katrina –Strength is that nature doesn’t care about your cases or IV The Rule Change –House seniority system (Crooks and Hibbing) –GAVEL amendment in Colorado –Connecticut speeding law –New Zealand electoral reform –Propositions –Relatively easy to spot, hard to defend

Genres of Natural Experiments The Court Decision l Roe V. Wade for Levitt and Donohue l Iowa item veto decision l Strength is that court is not a blatant political actor responding to societal shifts or societal pressures The Lottery l James Fowler’s use of Canadian bill introduction privilege l US House Clerk conducts a randomization of the order in which members choose office l Strength is true randomness in first step, but human action in 2nd

Genres of Natural Experiments Staged Implementation l Two-step reapportionment revolution in the United States l Lots of program evaluations in development l Helps to rule out history and maturation threats The Threshold l Mail ballot assignment in precincts with <250 voters l Need to make the threshold unrelated to DV, or else use Trochim- style regression discontinuity

What Makes a Convincing Natural Experiment? l You can show that the process of selection was not related to characteristics of the cases that are relevant to your DV l In a cross-sectional experiment, demonstrate that the two groups are quite similar l In a time-series experiment, demonstrate that little else changed when the treatment took place. l In a word, show equivalence

Any purported causal test of needs to take into consideration all of the two-group threats to validity. RXORORXORO NXONONXONO Can be a valid causal test. Fully exposed to threats.

NEGD Design has Multiple Groups AND Multiple Measures N O XO N OO This helps rule out (or at least recognize) threats.

Pre-Tests v. Covariates N O XO N OO N O 1 XO 2 N O 1 O 2 Proxy Pre-Test Design: First observations are covariates on which you collect data. Pre- Post-Test Design: Observations are tests you administer.

Problems of Internal Validity in NEGDs

Internal Validity NOXONOXONOONOONOXONOXONOONOO Selection-history Selection-maturation Selection-testing Selection-instrumentation Selection-regression Selection-mortality All designs suffer from threats to validity. In addition to all the single group threats, quasi-experiments are particularly likely to suffer from multi-group threats.

The Bivariate Distribution

Program Group has a 5-point pretest advantage.

The Bivariate Distribution Program group has a 5-point pretest advantage, Program group scores 15-points higher on Posttest.

Graph of Means pretestposttestpretestposttest MEANMEANSTD DEVSTD DEV Comp Prog ALL

Possible Outcome #1 l Possible: local event l Possible: PG initially higher l Unlikely: no change in CG l Possible: scale effects l Unlikely: expect change in CG l Possible: PG loses low scorers Selection-history Selection-maturation Selection-testing Selection-instrumentation Selection-regression Selection-mortality

Possible Outcome #2 l Likely: PG initially higher l Possible l Unlikely: expect change in CG l Possible: both lose low scorers Selection-history Selection-maturation Selection-testing Selection-instrumentation Selection-regression Selection-mortality

Possible Outcome #3 l Possible: local event l Unlikely: no change in CG l Possible: scale effects l Likely l Possible: PG loses high scorers Selection-history Selection-maturation Selection-testing Selection-instrumentation Selection-regression Selection-mortality

Possible Outcome #4 l Possible: local event l Unlikely: no change in CG l Possible: scale effects l Very Likely l Possible: PG loses low scorers Selection-history Selection-maturation Selection-testing Selection-instrumentation Selection-regression Selection-mortality

Possible Outcome #5 “And you should be so lucky…” Selection-history Selection-maturation Selection-testing Selection-instrumentation Selection-regression Selection-mortality

Analysis Requirements l Pre-post (or covariates) l Two-group l Treatment-control (dummy = 0, 1) NOXONOONOXONOO

Analysis of Covariance (ANCOVA) y i = outcome score for the i th unit  0 =coefficient for the intercept  1 =pretest coefficient  2 =mean difference for treatment X i =covariate Z i =dummy variable for treatment(0 = control, 1= treatment) e i =residual for the i th unit y i =  0 +  1 X i +  2 Z i + e i where:

The Bivariate Distribution Program group has a 5-point pretest Advantage. Program group scores 15-points higher on Posttest.

The Bivariate Distribution Slope is B 1 Vertical Distance is Mean Treatment Effect, or B 2

Why Add Covariates to Analysis? l ANCOVA can include more than one pretest or “control” variable. l Additional pretests further adjust for initial group differences. l Ideally, in the absence of any treatment effect, the covariates would perfectly predict the posttest. l Additional covariates will often improve the accuracy of the estimate of the treatment effect.

Irrelevant Covariates l Adding pretests that are completely unrelated to the posttest, however, actually decreases precision. l “Irrelevant covariates” contribute nothing to the analysis, but subtract a degree of freedom from the error term. l This reduces the efficiency of the estimate.

Omitted Covariates l Covariates that are related to the posttest but not to the treatment can be ignored without biasing the estimate of the treatment effect. l Covariates that are related to the posttest and the treatment but that are omitted will bias the estimate of the treatment effect. l We can safely omit control variables even if they are highly correlated with the posttest as long as they do not correlate with the treatment.

Omitted Variables Bias l Omitted (relevant) covariates that are positively correlated with the treatment will lead us to overestimate the treatment effect. l Omitted (relevant) covariates that are negatively correlated with the treatment will lead us to underestimate the treatment effect.

Bottom Line l We should always try to include omitted relevant covariates, except l When the omitted covariate is itself a consequence of the treatment. l If cannot include a relevant covariate, we can at least predict the direction if not magnitude of the likely bias.

But…What about measurement error? l With multiple covariates, measurement error does not always lead to a pseudo- effect. l As measurement error in any single variable increases, it becomes “as if” the variable is not included in the ANCOVA. l This then mimics an omitted variables problem, and the direction of bias depends upon the relationship between the “noisy” covariate and the treatment.

Other Quasi-Experimental Designs

Separate Pre-Post Samples l Groups with the same subscript come from the same context. l Here, N 1 might be people who were in the program at Agency 1 last year, with those in N 2 at Agency 2 last year. l This is like having a proxy pretest on a different group. N1ON1XON2ON2ON1ON1XON2ON2O

Separate Pre-Post Samples l Take random samples at two times of people at two nonequivalent agencies. l Useful when you routinely measure with surveys. l You can assume that the pre and post samples are equivalent, but the two agencies may not be. R1OR1XOR2OR2OR1OR1XOR2OR2O N N

Double-Pretest Design l Strong in internal validity l Helps address selection-maturation NOOXONOOONOOXONOOO

Switching Replications l Strong design for both internal and external validity l Strong against social threats to internal validity l Strong ethically NOXOONOOXONOXOONOOXO

Nonequivalent Dependent Variables Design (NEDV) l The variables have to be similar enough that they are affected the same way by all threats. l The program has to target one variable and not the other. l In simple form, weak internal validity. NO1XO1O2O2NO1XO1O2O2

NEDV Example l Only works if we can assume that geometry scores show what would have happened to algebra if untreated. l The variable is the control. l Note that there is no control group here.

NEDV Pattern Matching l Have many outcome variables. l Have theory that tells how affected (from most to least) each variable will be by the program. l Match observed gains with predicted ones. l With pattern, NEDV can be extremely powerful.

NEDV Pattern Matching l A “ladder” graph. r =.997

NEDV: Lake and O’Mahony 2006 Hypothesis: As territory declines in value in 20 th century (measured by average state size), wars fought over territory should decline in frequency. There should be no pattern in other Issues.