EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013

Agenda Quasi-experimental designs that use both control groups and pretests Interrupted time-series designs Design and power problems

Designs that Use Both Control Groups and Pretests

Untreated Control Group Design with Dependent Pretest and Posttest Samples A selection bias is always present, but the pretest observation allows for determining the magnitude and direction of bias NRO1O1 XO2O2 O1O1 O2O2

Treatment Control This pattern is consistent with treatment effects and can sometimes be causally interpreted, but it is subject to numerous threats, especially selection-maturation Both groups grow apart at different average rates in the same direction Outcome Pattern 1

Not a lot of reliance can be placed on this pattern as the reasons why spontaneous growth only occurred in the treatment group must be explained (e.g., selection-maturation) Spontaneous growth only occurs in the treatment group Treatment Control Outcome Pattern 2

Same internal validity threats as outcome patterns #1 and #2 except that selection- maturation threats are less plausible Initial pretest differences favoring the treatment group diminish over time Treatment Control Outcome Pattern 3

Subject to numerous validity threats (e.g., selection-instrumentation, selection-history), but generally can be causally interpreted Initial pretest differences favoring the control group diminish over time Treatment Control Outcome Pattern 4

Most amenable to causal interpretation and most threats cannot plausibly explain this pattern Outcomes that crossover in the direction of relationships Treatment Control Outcome Pattern 5

Modeling Selection Bias Simple matching and stratifying – Overt biases with respect to measured variables/characteristics Instrumental variable analysis – Statistical modeling of covariates believed to explain selection biases Hidden bias analysis – Difference with respect to unmeasured variables/characteristics – Sensitivity analysis (how much hidden bias would need to be present to explain observed differences) Propensity score analysis – Predicted probabilities of group membership – Propensities then used for matching or as covariate

Large Small Program Onset Program Termination Response Time Large Small Program Onset Program Termination Response Time Large Small Program Onset Program Termination Response Time Large Small Program Onset Program Termination Response Time Immediate Effect, No Decay Delayed Effect Immediate Effect, Rapid Decay Early Effect, Slow Decay Effect-Decay Functions

Permits assessment of selection-maturation on the assumption that the rates between O 1 and O 2 will continue between O 2 and O 3 Testable only on the control group NRO1O1 O2O2 XO3O3 O1O1 O2O2 O3O3 Untreated Control Group Design with Dependent Pretest and Posttest Samples Using a Double Pretest

A strong design and only a pattern of historical changes that mimics the time sequence of the treatment introductions can serve as an alternate explanation The addition of treatment removal (X) can strengthen cause-effect claims NRO1O1 XO2O2 O3O3 O1O1 O2O2 XO3O3 Untreated Control Group Design with Dependent Pretest and Posttest Samples Using Switching Replications

Interpretation of this design depends on producing two effects with opposite signs Adding a control is useful Ethically, often difficult to use a reversed treatment NRO1O1 X+X+ O2O2 O1O1 X-X- O2O2 Untreated Control Group Design with Dependent Pretest and Posttest Samples Using Reversed Treatment Control Group

Interrupted Time-Series Designs

Interuppted Time-Series A large series of observations made on the same variable consecutively over time – Observations can be made on the same units (e.g., people) or on constantly changing units (e.g., populations) Must know the exact point at which a treatment or intervention occurred (i.e., the interruption) Interrupted time-series designs are powerful cause-probing designs when experimental designs cannot be used and when a time series is feasible

Types of Effects Form of the effect (slope or intercept) Permanence of the effect (continuous or discontinuous) Immediacy of the effect (immediate or delayed)

Analytic Considerations Independence of observations – (Most) statistical analyses assume observations are independent (one observation is independent of another) – In interrupted time-series, observations are autocorrelated (related to prior observations or lags) – Requires a large number of observations to estimate autocorrelation Seasonality – Observations that coincide with seasonal patterns – Seasonality effects must be modeled and removed from a time-series before assessing treatment impact

Simple Interrupted Time- Series Design The basic interrupted time-series design requires one treatment group with many observations before and after a treatment O1O1 O2O2 O3O3 O4O4 O5O5 XO6O6 O7O7 O8O8 O9O9 O 10

Change in Intercept Intervention Change in intercept

Change in Slope Intervention Change in slope

Weak and Delayed Effects Intervention Impact begins

Validity Threats With most interrupted time-series designs, the major validity threat is history – Events that occur at the same time as the treatment was introduced Instrumentation is also often a threat – Over long time periods, methods of data collection may change, how variables are defined and/or measured may change Selection is sometimes a threat – If group membership changes abruptly

Additional Designs (1) nonequivalent control group, (2) nonequivalent dependent variable, and (3) removed treatment O1O1 O2O2 O3O3 O4O4 O5O5 XO6O6 O7O7 O8O8 O9O9 O 10 O1O1 O2O2 O3O3 O4O4 O5O5 O6O6 O7O7 O8O8 O9O9 OA1OA1 OA2OA2 OA3OA3 OA4OA4 OA5OA5 XOA6OA6 OA7OA7 OA8OA8 OA9OA9 O A10 OB1OB1 OB2OB2 OB3OB3 OB4OB4 OB5OB5 XOB6OB6 OB7OB7 OB8OB8 OB9OB9 O B10 O1O1 O2O2 O3O3 O4O4 XO5O5 O6O6 O7O7 O8O8 XO9O9 O 10 O 11 O 12

Nonequivalent Control Group Intervention Control group Treatment group

Nonequivalent Dependent Variable Intervention Dependent variable Nonequivalent dependent variable

Removed Treatment Introduction Treatment period Removal

Design and Power Problems

Problem #1 A school administrator wants to know whether students in his district are scoring better or worse than the national norm of 500 on the SAT He decides that a difference of points or more from this normative value would be important to detect He anticipates that the standard deviation of scores in his district is about 80 points – Determine the number of students necessary for power at 95% to detect a difference of 20 and 25 points – Graph both – Diagram the design of the study

Problem #2 Patients suffering from allergies are nonrandomly assigned to a treatment and placebo condition and asked to rate their comfort level on a scale of 0 to 100 The expected standard deviation is 20 and a difference of is expected (treatment = and placebo = 40) – Determine the number of patients necessary for power at 95% to detect a difference of 10 and 20 points – Graph both – Diagram the design of the study

Problem #3 The cure rate for two current treatments are 10% and 60%, respectively The alternative treatments are expected to increase the cure rate by 10% – Determine the number of patients necessary for power at 95% to detect a difference of 10% for both scenarios – Graph both – Diagram the design of the studies