William M. Trochim James P. Donnelly Kanika Arora 8 Introduction to Design

Research design is the glue that holds pieces of the research project together –The sample –The measures –The treatments or programs –The method of assignment 8.1 Foundations of Design

Causal –Pertaining to a cause-effect question, hypothesis, or relationship –Something is causal if it leads to an outcome or makes an outcome happen –Don’t confuse this word with casual! 8.2 Research Design and Causality

Causal relationship –A cause-effect relationship: for example, when you evaluate whether your treatment or program causes an outcome to occur, you are examining a causal relationship Three criteria: –Temporal precedence –Covariation of the cause and effect –No plausible alternative explanations 8.2a Establishing Cause and Effect in Research Design

An unobserved variable that accounts for a correlation between two variables Correlation does not equal causation! 8.2a The Third-Variable Problem

8.2a Control Groups

8.2b Internal Validity

The three main types of threats –Single-group threats –Multi-group threats –Social-interaction threats 8.2b Internal Validity

Single Group Considered non-experimental, or, pre- experimental. Design Notation - Sketch 8.2b Internal Validity – Single Group Design

Occurs in single group post-test or pretest- post-test designs –Types History Maturation Testing Instrumentation Mortality Regression 8.2b Internal Validity – Single Group Threats

Deal with single-group threats by adding a second group! –Called a control group Once you’ve added a control group, you need to address the multi-group threats to internal validity 8.2b Internal Validity – Single Group Threats (cont’d.)

8.2b Multi-group Design Notation – Two-group Post-test Only Design Sketch

There really is only one multiple-group threat to internal validity: that the groups were not comparable before the study –See Fine et al. (2003) – Section 3.3c: including prisoners as researchers provided critical insights 8.2b Internal Validity – Multi-Group Threats

Selection-history threat Selection-maturation threat Selection-testing Threat Selection-instrumentation threat Selection-mortality threat Selection-regression threat 8.2b Internal Validity – Types of Selection Bias

When you do not use randomization to groups, you have a quasi-experimental design Quasi-experimental designs have weaker internal validity Example Sketch – Two-group Pre-Post Design 8.2b Internal Validity – Multi-Group Threats

Diffusion or imitation of treatment Compensatory rivalry Resentful demoralization Compensatory equalization of treatment 8.2b Internal Validity – Social-Interaction Threats

Other ways to rule out threats to internal validity –By argument –By measurement or observation –By analysis Main Effect Covariance Analysis –By preventative action 8.2b Internal Validity – Other Methods

Four elements to any research design: –Time –Treatments or programs (IV) –Measures or observations (DV) –Groups or individuals 8.3 Developing a Research Design

8.3 Design Notation

8.4 Types of Designs

8.4 Notational Examples

Why is internal validity so important in research design? What is the purpose of randomization in research design? Discuss and Debate

Program group Treatment group Comparison group Control group Probabilistically equivalent 9.2a Distinguishing Features of Experimental Design

9.2b Experimental Design and Threats to Internal Validity

Two-group Post-test Only Design – Sketch Design Notation Post-Test Only Examples

Two-group Post-test Only Design - Sketch Switching Replications Design – Sketch Design Notation Post-Test Only Examples

Two-group Post-test Only Design - Sketch Switching Replications Design - Sketch 2 x 2 Factorial Design – see next slides –Allows one to examine “levels” of the variables, and, most importantly, interactions between variable levels. –Let’s use the example from the book: Time in Instruction (1 hour vs. 4 hour) and Setting (In- Class vs. Pull-out) Randomized Block Design – see next slides Design Notation Post-Test Only Examples

9.4a The Basic 2 x 2 Factorial Design

9.4a The Basic 2 x 2 Factorial Design: Possible Outcome

9.4a A Main Effect of Time in Instruction in a 2 x 2 Factorial design

9.4a A Main Effect of Setting in a Factorial Design

9.4a Main Effects of Both Time and Setting in a 2 x 2 Factorial Design

9.4a An Interaction in a 2 x 2 Factorial Design

9.4a A Crossover Interaction in a 2 x 2 Factorial Design

Benefits –Enhances the signal –Efficient design –Only design that allows you to examine interactions Limitations –Complex –More participants required 9.4b Benefits and Limitations of Factorial Designs

9.4c Factorial Design Variations: 2 x 3

Helps minimize noise through the grouping of units (e.g., participants) into one or more classifications (blocks) that account for some of the variability in the outcome 9.5 Noise-Reducing Designs: Randomized Block Designs

Differential drop out (mortality threat) Ethical problems Social threats to internal validity Difficult to generalize to the real world 9.8 Limitations of Experimental Design

What is the difference between random selection and random assignment? What are some strengths and weaknesses of experimental designs? Can you think of some research topics for which a factorial design may be a good approach? –Need at least two variables with multiple levels. Discuss and Debate

“Quasi” means “sort of”, Quasi- experiments have: –A control group –A treatment (or program) group –Variables Quasi-experiments do not have: –Random assignment to groups 10.1 Foundations of Quasi-Experimental Design

One of the most frequently used quasi-experimental designs –Looks just like a pretest-posttest design –Lacks random assignment to groups –As a result, the treatment and control groups may be different at the study’s start –Raises a selection threat to internal validity 10-2 The Nonequivalent Groups Design

10.2a Plot of Pretest and Posttest Means for Possible Outcome 1

10.2a Plot of Pretest and Posttest Means for Possible Outcome 2

10.2a Plot of Pretest and Posttest Means for Possible Outcome 3

10.2a Plot of Pretest and Posttest Means for Possible Outcome 4

10.2a Plot of Pretest and Posttest Means for Possible Outcome 5

A pretest- posttest program comparison- group quasi-experimental design in which a cutoff criterion on the preprogram measure is the method of assignment to a group 10.3 The Regression-Discontinuity Design

Notation –C indicates that groups are assigned by means of a cutoff score on the premeasure –An O stands for the administration of a measure to a group. –An X depicts the implementation of a program Each group is described on a single line 10.3a The Basic RD Design

A line that describes the relationship between two or more variables 10.3a Regression Line

A post-only design in which, after the fact, a pretest measure is constructed from preexisting data –Usually done to make up for the fact that the research did not include a true pretest 10.4a The Proxy Pretest Design

A design in which the people who receive the pretest are not the same as the people who take the posttest 10.4b The Separate Pre-Post Samples Design

A design that includes two waves of measurement prior to the program –Addresses selection-maturation threats 10.4c The Double-Pretest Design

A two-group design in two phases defined by three waves of measurement –In the repetition of the treatment, the two groups switch roles: The original control group in phase 1 becomes the treatment group in phase 2, whereas the original treatment group acts as the control 10.4d The Switching-Replications Design

At first, looks like a weak design But pattern matching gives researchers a powerful tool for assessing causality –The degree of correspondence between two data items 10.4e The Nonequivalent Dependent Variables (NEDV) Design

A pre-post quasi-experimental research design where the treatment is given to only one unit in the sample, with all remaining units acting as controls –This design is particularly useful to study the effects of community level interventions 10.4f The Regression Point Displacement (RPD) Design

Why can quasi-experiments be more ethical than randomized experiments? What are the strengths and the weaknesses of quasi-experimental designs? Discuss and Debate