Quasi-Experimental Designs Whenever it is not possible to establish cause-and-effect relations because there is not complete control over the variables in the study

Common Quasi-Experiments The two most common quasi-experimental designs include –Nonequivalent control group design –Interrupted time series designs Both of these are very common in program evaluation research

Nonequivalent Control Group Design Design –Experimental Group Pre-test --> Treatment --> Post-test –Control Group Pre-test --> Nothing --> Post-test The distinguishing aspect of this design is that the experimental and control groups are not equivalent at the beginning of the experiment

Example Testing the usefulness of a new technique for teaching research methods Design –Experimental Group (Texas Tech students) Pre-test (Final from Fall) --> Treatment (using the new technique Spring) --> Post-test (Final from Spring) –Control Group (Minnesota students) Pre-test (Final from Fall) --> Nothing (i.e., they keep teaching via their same technique) --> Post-test (Final from Spring)

Example Because of the non-equivalence of the groups at the beginning of the study, the researcher needs to be especially concerned about the various threats to internal validity.

Example History –If something significant happens during Spring ‘05 at Tech (e.g., abnormal amounts of bad weather, unrest on campus, etc.), then this could affect test scores and effect the outcome of the study All you can do is record these kinds of events and try to defend that nothing like this happened during your study

Example Selection –The experimental and control groups each contain people from different schools. If students at one school were more capable than students at the other, then this could create post-test differences To deal with this, you look not just at post-test scores, but the improvement from pre-test to post-test for both groups

Cook & Campbell (1979) When doing a non-equivalent control group design, pay special attention to the following threats to internal validity –Maturation –Instrumentation –Regression to the Mean –Selection/History

Interrupted Time Series Design Design –Experimental Group Pre-test --> Pre-test --> etc. --> Treatment --> Post-test --> Post-test --> etc. The distinguishing aspects of this design are –there is no control group –measurements are taken for an extended period of time before and after the treatment

Example Testing the usefulness of a new technique for teaching research methods Design –Experimental Group (Texas Tech students) Pre-test (Final from Fall) --> Pre-test (Final from Spring) --> Pre- test (Final from Fall) Treatment (using the new technique Spring) Post-test (Final from Spring) --> Post-test (Final from Fall) --> Post-test (Final from Spring)

Example Because of the lack of a control group, the researcher needs to be especially concerned about the various threats to internal validity

Example History/Selection –If Tech changes its entrance requirements around the time of the implementation of the new technique, then this might account for any improvement in test scores To deal with this, it would be best to examine grades in other unrelated courses (which essentially would serve like a control group)

Example Instrumentation –If the new teaching technique is associated with new tests, then the pre-test/post-test measures for the experimental group would be different. Ideally, you would use the same test before and after treatment, but that may not always be possible.

Example Attrition –If the post-treatment drop-out rate is higher than the pre-treatment drop-out rate, then (assuming that folks not doing well drop the courses) this could inflate the post-treatment grades To deal with this, you would need to record the drop- out rate and demonstrate that a change in those rates did not account for your results

Cook & Campbell (1979) When doing a interrupted time series design, pay special attention to the following threats to internal validity –History –Maturation –Instrumentation

Summary Quasi-experiments do not have the components to allow for causal statements In addition, the missing components (e.g., control groups, random assignment, etc.) increase the likelihood of encountering problems with the threats to internal validity