Presentation is loading. Please wait.

Presentation is loading. Please wait.

Quasi Experimental Designs Chapters 4 & 5 Som Nwegbu and Binh Le H615: Advanced Evaluation and Research Design October 18, 2013.

Similar presentations


Presentation on theme: "Quasi Experimental Designs Chapters 4 & 5 Som Nwegbu and Binh Le H615: Advanced Evaluation and Research Design October 18, 2013."— Presentation transcript:

1 Quasi Experimental Designs Chapters 4 & 5 Som Nwegbu and Binh Le H615: Advanced Evaluation and Research Design October 18, 2013

2 A Few Helpful Definitions Quasi – “seemingly; apparently but not really.” Synonyms: supposedly, seemingly, apparently, allegedly, ostensibly, on the face of it, on the surface, to all intents and purposes, outwardly, superficially, purportedly, nominally” Experiment – “A test under controlled conditions that is made to demonstrate a known truth, examine the validity of a hypothesis, or determine the efficacy of something previously untried.” Quasi Experiment – Why is it ‘quasi’ ? It’s “…an experiment in which units are not assigned to conditions randomly.” (SCC)

3 Quasi-Experimental Designs Lack a Control Group or Lack Pretest Observations on Outcome Why use designs? Devote more resources to construct validity and external validity Necessities imposed by funding, ethics, administrators, or logistical constraints Sometimes the best design for the study, even if causal inference might be weaker

4

5 Logic of Quasi Experiments Causal inference must meet requirements: That cause precede effect, that cause covary with effect, and alternative explanations unlikely. Randomized and quasi-experiments manipulate treatment to force it to occur before the effect Randomized and quasi-experiments manipulate treatment to force it to occur before the effect Covariation between cause and effect accomplished during statistical analysis Covariation between cause and effect accomplished during statistical analysis Alternative explanations implausible by ensuring random distribution Alternative explanations implausible by ensuring random distribution Identification and study of plausible threats to internal validity Primary of control by design Coherent pattern matching

6 Quasi Experimental Designs w/o Control Groups One-Group Posttest-Only XO 1 Examples…Weaknesses… One-Group Posttest-Only with Multiple Posttests X 1 {O 1A O 1B …O 1N } Examples…Weaknesses…

7 Quasi Experimental Designs w/o Control Groups One-Group Pretest-Posttest O 1 XO 2 Examples…Weaknesses… One-Group Pretest-Posttest Using Double Pretest O 1 O 2 X O 3 Examples…Weaknesses… One-Group Pretest-Posttest Using Nonequivalent Dependent Variable {O 1A, O 1B } X {O 2A, O 2B }

8 Quasi Experimental Designs w/o Control Groups Removed-Treatment O 1 X O 2 O 3 χO 4 Examples…Weaknesses…Repeated-Treatment O 1 X O 2 χ O 3 X O 4 Examples…Weaknesses…

9 Quasi-Experimental Designs w/Control Groups but no Pretest Posttest-Only Design with Nonequivalent Groups NRXO 1 NRO 2 Posttest-Only Design Independent Pretest Sample NRO 1 | X O 2 NR O 1 | O 2 Posttest-Only Design Proxy Pretests NRO A1 X O B2 NR O A1 O B2

10 Improving the Posttest-Only Design Using Matching or Stratifying Internal Controls Multiple Control Groups Predicted Interaction

11 Constructing Contrasts other than with Independent Groups Regression Extrapolation Contrasts Compares obtained posttest score of the treatment with the score predicted from other information Normed Comparison Contrasts Treatment group at pretest and posttest compared with published norms Secondary Source Contrasts Construct opportunistic contrasts from secondary sources

12 Case Control Design Also called case-referent, case-comparative, case-history, or retrospective design One group of cases with outcome of interest and another group of controls without outcome Typically dichotomous outcome Generating hypotheses about causal connections More feasible than experiments in cases, logistically easier to conduct, decrease risk of participants, and easy examination of multiple causes

13 Case Control Design Methodological problems Decision on what counts as the presence or absence of an outcome Disagreement about the decision, if they do, assessing the outcome may be unreliable or low validity Selection of control cases is difficult Randomly sampled controls are ideal but when not feasible, matching is the next option Matching controls can still differ from cases in unobserved ways

14 Threats to Validity Reading on the field(5): One-sided reference bias Positive results bias Hot stuff bias Specifying and selecting the study sample(22): Diagnostic access bias Unacceptable disease bias Membership bias Executing the experimental maneuvers (5): Contamination bias Withdrawal bias

15 Threats to Validity Measuring Exposures and Outcomes (13): Underlying cause bias Expectation bias Attention bias Analyzing the Data (5): Scale degradation bias Tidying-up bias Interpreting the Analysis (6): Magnitude bias Significance bias Correlation bias

16 Quasi-experimental Designs that Use Both Control Groups and Pretests

17 Benefits of a Pretest Addresses the issue of bias resulting from non- random selection NB: However, ‘no difference’ between intervention and control groups at pretest does not guarantee zero selection bias. Gives us a baseline to compare against (statistical analysis)

18 Limitations of a Pretest We cannot assume that any covariates unaccounted for, but present at pretest, are unrelated to outcome In a randomized experiment, this would have been controlled for by random selection.

19 Key: NR = Non-random assignment NR = Non-random assignment 01 = pre-test 01 = pre-test 02 = second pre-test (if any) 02 = second pre-test (if any) 03 = post-test 03 = post-test X = Test/intervention X = Test/interventionX+ } = Reversed treatment } = Reversed treatmentX-

20 Untreated Control Group Design with Dependent Pre-test & Post-test samples NR O1 X O2 NR O1 X O2 NR O1 O2 NR O1 O2 Most common and most basic used Others (to come) are an attempt to improve internal validity and vary depending on context and resources available to the researcher. Why do you think this is the most commonly used?

21 Limitations and/or weaknesses aka ‘Threats to internal validity’ Selection-maturation (various subtypes) Selection-maturation (various subtypes) Pretest difference b/w intervention and con groups increases when intervention leads to improvement in group that was better to begin with Selection-instrumentation Selection-instrumentation Detectable pretest difference between intervention and control groups - pretest started at different points e.g., one group starts at Q50 and another at Q1 Selection-regression??? Pg 139*** Selection-regression??? Pg 139*** Selection History Selection History Events occurring midway b/w pre and posttest affect one group more than the other, thus widening/narrowing the observed pretest difference.

22 Possible versus plausible threat Possible – might have occurred, but highly unlikely. Plausible – most probably did occur We want to be able to, as much as we can, rule out all the possibles and pursue the plausibles Question: How do we know which cases to worry about (plausibles) and which we can safely ignore (possibles)? (discuss)

23 Answer: Analyze results in context Outside of your study, what do you already know about the threats? What is the observed pattern of outcomes: Groups grow apart in the same direction No change in control group Initial pretest difference (in favor of the treatment group) but then diminishes over time Initial pretest difference (in favor of control group) but then diminishes over time Outcomes that crossover (you wish!)

24 Point to note In view of the sub-types of sub-maturation threat, one must be prepared to present and justify a study’s assumptions about maturational differences (if using the basic quasi-experimental design).

25 Ways to improve on the internal validity of inferences made using the basic design: (1) DOUBLE PRETEST NR O 1 O 2 X O 3 NR O 1 O 2 X O 3 NR O 1 O 2 O 3 NR O 1 O 2 O 3 Exposes selection maturation where present Helps reveal regression effects if present Helps statistical analysis by establishing more precise correlation between observations. Put simply, it gives us a baseline to compare against.

26 (2)SWITCHING REPLICATIONS NR O 1 X O 2 O 3 NR O 1 X O 2 O 3 NR O 1 O 2 X O 3 NR O 1 O 2 X O 3 Treatment is administered at a later time point for the group that initially served as a control. Treatment is administered at a later time point for the group that initially served as a control. May be employed where it would be unethical to withhold treatment/intervention (particularly if the treatment has been proven to be beneficial). May be employed where it would be unethical to withhold treatment/intervention (particularly if the treatment has been proven to be beneficial). Helps test both internal and external validity. (Discuss) Helps test both internal and external validity. (Discuss)

27 (3) REVERSED TREATMENT- CONTROL GROUP NR O 1 X + O 2 NR O 1 X + O 2 NR O 1 X ─ O 2 NR O 1 X ─ O 2 Advantageous, particularly in terms of potential for improved construct validity. (Discuss) Advantageous, particularly in terms of potential for improved construct validity. (Discuss) Allows for ruling out potential of Hawthorne effect. Allows for ruling out potential of Hawthorne effect. Assumptions/weakness – This design depends on the assumption that there are no historical or other extrinsic behavior modifying events occurring while the study is ongoing. Assumptions/weakness – This design depends on the assumption that there are no historical or other extrinsic behavior modifying events occurring while the study is ongoing.

28 (4) DIRECT MEASUREMENT OF THREAT Researcher tries to conceive of every possible threat to validity and then put in checks to reduce such threats. Demerits?Merits?

29 MATCHING USING COHORTS AS CONTROLS Cohort (in this context) – “…a group of subjects who have shared a particular event together during a particular time span e.g., people born in Europe between 1918 and 1939.” “…a group of subjects who have shared a particular event together during a particular time span e.g., people born in Europe between 1918 and 1939.”Benefits?CostConvenience Allows one to make good use of well kept records where available Others as outlined in SCC page 149 [paragraph 1]

30 It gets even better... (Don’t you mean more complicated?) DESIGNS THAT COMBINE MANY DESIGN ELEMENTS Depending on context and need, one may use combinations of the above examples, or further variations, to improve internal validity and enable causal inference (or something close). Depending on context and need, one may use combinations of the above examples, or further variations, to improve internal validity and enable causal inference (or something close). Be prepared to explain and defend it though. Be prepared to explain and defend it though.

31 Moving on from threats to Internal validity to: Improving Statistical conclusion validity HDFS 532 (or equivalent) highly recommended Topics such as, SEM and Selection Bias Modeling covered in detail. Further comments or contributions?

32 To wrap up We can never be 100% certain of our claims when using quasi-experimental designs and we must be prepared to own it and state our limitations upfront. Still, there are numerous ways to strengthen the validity of our claims and these can be applied at different stages of the research study: Assignment Assignment Measurement Measurement Use of comparison groups Use of comparison groups Treatment Treatment Statistical Analysis Statistical Analysis


Download ppt "Quasi Experimental Designs Chapters 4 & 5 Som Nwegbu and Binh Le H615: Advanced Evaluation and Research Design October 18, 2013."

Similar presentations


Ads by Google