Experimental Design 1

Experimental Design Strongest design with respect to internal validity 2

If X then Y and If not X, then not Y or If the program is given, then the outcome occurs and If the program is not given, then the outcome does not occur 3

Dilemma 2 identical groups 2 identical contexts Same time …. 4 similarity

Course of Action Randomly assign people from a pool to the 2 groups –  probabilistically equivalent One group gets the treatment and the other does not 5

Random Selection and Assignment Random selection is how you draw the sample of people for your study from a population. Random assignment is how you assign the sample that you draw to different groups or treatments in your study.

Probabilistic Equivalence Means that we know perfectly the odds that we will find a difference between two groups. When we randomly assign to groups, we can calculate the chance that the two groups will differ just because of the random assignment. 7

External validity Experiments are difficult to carry out  artificial situation  high internal validity Limited generalization to real contexts –> limited external validity 8 ?

Two-Group Experimental Design Simplest form: two-group posttest-only randomized experiment No pretest required Test for differences: t-test or ANOVA 9

Advantages Strong against single-group threats and multi- group threats (except selection-mortality) Strong against selection testing and selection- instrumentation 10

Classifying Experimental Designs Two components: signal and noise signal-enhancing experimental design (factorial design) Noise-reducing experimental design (covariance designs or blocking designs) 11

Factorial Designs A factor is a major independent variable – Time and setting A level is a subdivision of a factor. – Time (1h/4h), setting (pull-out/in-class) 2 x 2 factorial design 12

Factorial Design X 11 = 1h and in-class X 12 = 1h and pull-out X 21 = 4h and in-class X 22 = 4h and pull-out 13

The Null Outcome The null case is a situation where both treatments have no effect. 14

The Main Effects A main effect is an outcome that is a consistent difference between levels of a factor. 15

Main Effects 16

Main Effects 17

Interaction Effects An interaction factor exists when differences on one factor depending on the level of the other factor. 18

How do you know if there is an interaction in a factorial design? Statistical analysis When it can be talked about one factor without mentioning the other factor In graphs of group means – the lines are not parallel 19

Interaction Effects 20

Interaction Effects 21

Factorial Design Variations 2 x 3 Example Factor 1: Treatment – psychotherapy – behavior modification Factor 2: Setting – inpatient – day treatment – outpatient 22

Main Effects 23

Main Effects 24

Interaction Effect 25

Interaction Effect 26

Factorial Design Variations A Three-Factor Example (2 x 2 x 3) Factor 1: Dosage – 100 mg – 300 mg Factor 2: Treatment – Psychotherapy – Behavior modification Factor 3: Setting – Inpatient – Day treatment – Outpatient 27

2 x 2 x 3 Design 28

Incomplete Factorial Design Common use is to allow for a control or placebo group that receives no treatment 29

Randomized Block Design Stratified random sampling To reduce noise or variance in the data Division into homogeneous subgroups Treatment implemented to each subgroup Variability within each block is less than the variability of the entire sample or each block is more homogenous than the entire group 30

Randomized Block Design Stundents are a homogenous group with exception of semester 31 freshman sophomore junior senior

How blocking reduces noise? 32

Covariance Designs (ANCOVA) Pretest-posttest randomized design Pre-program measure = covariate Covary it with the outcome variable Covariates are the variables you adjust for – Effect is going to be removed 33

How does a Covariate reduce Noise? 34

How does a Covariate reduce Noise? 35

How does a Covariate reduce Noise? 36

How does a Covariate reduce Noise? 37

How does a Covariate reduce Noise? 38

Hybrid Experimental Designs Are new strains that are formed by combining features of more established designs. 39

The Solomon Four-Group Design Is designed to deal with a certain testing threat 2 groups are pre-tested, 2 are not 2 groups get a treatment, 2 do not 40

The Solomon Four-Group Design T = Treatment Group, C = Control Group 41

The Solomon Four-Group Design T = Treatment Group, C = Control Group 42

Switching Replication Design The implementation of the treatment is repeated or replicated. In the repetition, the two groups switch roles Finally, all participants have received the treatment Reduces social threats 43

Switching Replication Design Period 1 – group 1 gets the treatment Period 2 – group 2 gets the treatment 44

Switching Replication Design Longterm treatment effect  group 1 improves even though no further treatment was given 45