1. EVAL 6970: Experimental and Quasi-Experimental Designs
Dr. Chris L. S. Coryn
Dr. Anne Cullen
Spring 2012

2. Agenda Basic design elements and notation
Quasi-experimental designs that either lack a control group or lack pretest observations on the outcome
Midterm examination
Case study

3. Questions to Consider
What are the limitations of designs lacking either control groups and/or pretest observations?
What simple strategies can be used to improve these types of designs?
Why are such designs sometimes the only ones that can be used?

4. Basic Design Elements and Notation

5. Assignment Random assignment Cutoff-based assignment
Other nonrandom assignment
Matching and stratifying
Masking

6. Measurement Posttest observations Pretest observations
Single posttests
Nonequivalent dependent variables
Multiple substantive posttests
Pretest observations
Single pretest
Retrospective pretest
Proxy pretest
Repeated pretests over time
Pretests on independent samples
Moderator variable with predicted interaction
Measuring threats to validity

7. Comparison Groups Single nonequivalent groups
Multiple nonequivalent groups
Cohorts
Internal versus external controls
Constructed contrasts
Regression extrapolation contrasts
Normed contrasts
Secondary data contrasts

8. Treatments Switching replications Reversed treatments
Removed treatments
Repeated treatments

9. Notation
X = treatment O = observation R = random assignment NR = nonrandom assignment X = removed treatment X+ = treatment expected to produce an effect in one direction X- = conceptually opposite treatment expected to reverse an effect C = cutting score = non-randomly formed groups … = cohort

10. Logic of Quasi-Experimentation

11. Rationale
Quasi-experiments are often a necessity given practical and logistical constraints
Greater emphasis on construct or external validity rather than cause-effect associations (least common)
Funding, ethics, administration (somewhat common)
The intervention has already occurred (most common)
Sometimes they are the best alternative, even if causal inferences are weaker than is possible with other designs
Even so, great care must be taken when planning such studies as numerous threats that cannot be controlled are often operating

12. Central Principles
Identification and study of plausible threats to internal validity
Careful scrutiny of plausible alternative explanations for treatment-outcome covariation
Primacy of control by design
Use carefully planned and implemented design elements rather than statistical controls for anticipated confounds
Coherent pattern matching
Complex (a priori) causal hypotheses that reduce the plausibility of alternative explanations

13. Designs without Control Groups

14. One-Group Posttest Only Design
Absence of pretest makes it difficult to know if change has occurred and absence of a control group makes it difficult to know what would have happened without treatment X O1

15. One-Group Pretest-Posttest Design
Adding a pretest provides weak information concerning what might have happened to participants had the treatment not occurred O1 X O2

16. One-Group Pretest-Posttest Design with Double Pretest
Adding multiple pretests reduces the plausibility of maturation and regression effects
Additional pretests can confirm maturational trends O1 O2 X O3

17. One-Group Pretest-Posttest Design Using a Nonequivalent Variable
Measure A is expected to change because of treatment, B is not
Both A and B are expected to respond to the same validity threats in the same way
{O1A , O1B} X
{O2A , O2B}

18. A = sale of lottery tickets
B = sale of alcohol
C = sale of tobacco A C C B B A
Lottery ticket sales in convenience stores after introduction of signs in store windows reading “did you buy your ticket?”

19. Removed-Treatment Design
Demonstrates that outcomes rise and fall with the presence or absence of treatment O1 X O2 O3 O4

20. Generally interpretable outcome pattern
Better
Outcome
Uninterpretable outcome
Worse
Interpretable outcome O1 X O2 O3 X O4
Generally interpretable outcome pattern

21. Repeated-Treatment Design
Few threats could explain a close relationship between treatment introductions and removals and parallel outcome changes O1 X O2 O3 O4

22. Mean narcotics use over multiple Methadone maintenance on/off conditions

23. A-B Designs
Multiple-baseline design (a class of single-subject designs), or collection of A-B designs, to assess the effects of an intervention across separate baselines
A = baseline
B = treatment
The intervention is introduced in a staggered manner and the baseline provides a predicted level of the dependent variable in absence of the treatment
A-B-A designs are sometimes called removal designs (i.e., the treatment is removed)

24. Site 1 Number of Accidents Site 2 Site 3 Weeks
Baseline
Treatment
Effect
Number of Accidents
Site 2
Baseline
Treatment
Effect
Site 3
Baseline
Treatment
Effect
Weeks

25. Designs that use a Control Group but no Pretest

26. Posttest-Only Design with Nonequivalent Control Group
Unknown pretest group differences make it extremely difficult to separate treatment effects from selection effects NR X O1

27. Posttest-Only Design using an Independent Sample Pretest
Assumes overlapping group membership
Useful when pretest measurements may be reactive, cannot follow same groups over time, or when interested in studying intact communities whose members change over time NR O1 X O2

28. Posttest-Only Design using Proxy Pretest
Proxy measures should be conceptually related to and correlated with outcome
Can be used for a variety of purposes including indexing selection bias and/or attrition NR
OA1 X
OB2

29. Case Control Studies
Predominant method for many forms of epidemiological research
Used to identify factors that may contribute to a condition by comparing subjects who have that condition (i.e., 'cases') with those who do not have the condition but are otherwise similar (i.e., 'controls')
Famously, the association between smoking and lung cancer
Similar in many respects to Scriven’s GEM and MOM

30. Midterm Examination

31. Midterm Examination
The examination will consist of multiple-choice items, scored as 0 or 1
You will have 2½ hours to complete the examination
You may use one page of notes (front and back) on 8½” X 11’’ paper
You will be asked questions about statistical power, but will not be required to calculate power

32. Case Study

33. Case Study Activity
An aid agency implemented a project in Bangladesh with the objective of improving the nutritional and health status of men and women
The intervention consisted of a package of services including: nutrition education, primary health care, and other activities
To determine whether the intervention might be effective, the project was field-tested in a small rural community prior to large-scale implementation throughout the country
A small monetary incentive was provided and slightly more than half of the community’s men and women participated in the study
All men and women in the community were weighed prior to the intervention and then were measured for body mass index (BMI) six months after the intervention
Those who did not participate were used as a control group and the evaluators found significant improvements in nutritional and health indicators for the treatment group contrasted with the control

34. Questions What is the design of the study?
What internal validity threats are most plausible?
How might the design feasibly be improved?