1. Experiment Basics: Designs
Psych 231: Research Methods in Psychology

2. So far we’ve covered a lot of the about details experiments generally
Now let’s consider some specific experimental designs.
Some bad (but common) designs
Some good designs
1 Factor, two levels
1 Factor, multi-levels
Factorial (more than 1 factor)
Between & within factors
Experimental designs

3. 1 factor - 2 levels How does anxiety level affect test performance?
participants
Low
Moderate
Test
Random
Assignment
Anxiety
Dependent Variable
low
moderate
test performance
anxiety
1 factor - 2 levels

4. 1 factor - 3 levels participants Low Moderate Test Random Assignment
Anxiety
Dependent Variable
High
low
mod
test performance
anxiety
high
1 factor - 3 levels

5. Factorial experiments
Two or more factors
Some vocabulary
Factors - independent variables
Levels - the levels of your independent variables
2 x 4 design means two independent variables, one with 2 levels and one with 4 levels
“Conditions” or “groups” is calculated by multiplying the levels, so a 2x4 design has 8 different conditions A1 A2 B1 B2 B3 B4
Factorial experiments

6. Factorial experiments
Two or more factors (cont.)
Main effects - the effects of your independent variables ignoring (collapsed across) the other independent variables
Interaction effects - how your independent variables affect each other
Example: 2x2 design, factors A and B
Interaction:
At A1, B1 is bigger than B2
At A2, B1 and B2 don’t differ
Everyday interaction = “it depends on …”
Factorial experiments

7. Rate how much you would want to see a new movie (1 no interest, 5 high interest)
Ask men and women – looking for an effect of gender
Not much of a difference
Interaction effects

8. Maybe the gender effect depends on whether you know who is in the movie. So you add another factor:
Suppose that George Clooney might star. You rate the preference if he were to star and if he were not to star.
Effect of gender depends on whether George stars in the movie or not
This is an interaction
Interaction effects

9. Results of a 2x2 factorial design
The complexity & number of outcomes increases:
A = main effect of factor A
B = main effect of factor B
AB = interaction of A and B
With 2 factors there are 8 basic possible patterns of results:
1) No effects at all
2) A only
3) B only
4) AB only
5) A & B
6) A & AB
7) B & AB
8) A & B & AB
Results of a 2x2 factorial design

10. 2 x 2 factorial design Interaction of AB A1 A2 B2 B1 Marginal means
What’s the effect of A at B1?
What’s the effect of A at B2?
Condition
mean
A1B1
Condition
mean
A2B1
Marginal means
B1 mean
B2 mean
A1 mean
A2 mean
Main effect of B
Condition
mean
A1B2
Condition
mean
A2B2
Main effect of A
2 x 2 factorial design

11. Examples of outcomes Main effect of A ✓ Main effect of B
Dependent Variable B1 B2 30 60 45 60 45 30 30 60
Main Effect
of A
Main effect of A ✓
Main effect of B X
Interaction of A x B X
Examples of outcomes

12. Examples of outcomes Main effect of A Main effect of B ✓
Dependent Variable B1 B2 60 60 60 30 30 30 45 45
Main Effect
of A
Main effect of A X
Main effect of B ✓
Interaction of A x B X
Examples of outcomes

13. Examples of outcomes Main effect of A Main effect of B
Dependent Variable B1 B2 60 30 45 60 45 30 45 45
Main Effect
of A
Main effect of A X
Main effect of B X
Interaction of A x B ✓
Examples of outcomes

14. Examples of outcomes Main effect of A ✓ Main effect of B ✓
Dependent Variable B1 B2 30 60 45 30 30 30 30 45
Main Effect
of A
Main effect of A ✓
Main effect of B ✓
Interaction of A x B ✓
Examples of outcomes

15. Anxiety and Test Performance
Let’s add another variable: test difficulty.
anxiety
low
mod
high 80 35 50 70 80
main effect
of difficulty
test performance
high
low
mod
anxiety
easy
easy
medium
hard
Test difficulty 80 80 80
medium 65 80
hard 65 80 60
main effect
of anxiety
Yes: effect of anxiety depends on level of test difficulty
Interaction ?
Anxiety and Test Performance

16. Factorial Designs Advantages Interaction effects
Always consider the interaction effects before trying to interpret the main effects
Adding factors decreases the variability
Because you’re controlling more of the variables that influence the dependent variable
This increases the statistical Power of the statistical tests
Increases generalizability of the results
Because you have a situation closer to the real world (where all sorts of variables are interacting)
Factorial Designs

17. Factorial Designs Disadvantages
Experiments become very large, and unwieldy
The statistical analyses get much more complex
Interpretation of the results can get hard
In particular for higher-order interactions
Higher-order interactions (when you have more than two interactions, e.g., ABC).
Factorial Designs

18. Factorial designs Consider the results of our class experiment
Main effect of word type
Main effect of depth of processing
Interaction between word type and depth of processing
Dr. Kahn's reporting stats page
Factorial designs

19. So far we’ve covered a lot of the about details experiments generally
Now let’s consider some specific experimental designs.
Some bad (but common) designs
Some good designs
1 Factor, two levels
1 Factor, multi-levels
Factorial (more than 1 factor)
Between & within factors
Experimental designs

20. What is the effect of presenting words in color on memory for those words?
Clock
Chair
Cab
So you present lists of words for recall either in color or in black-and-white.
Two different designs to examine this question
Example

21. Between-Groups Factor
2-levels
Each of the participants is in only one level of the IV
levels
Clock
Chair
Cab
Colored
words
participants
Test
Clock
Chair
Cab BW
words

22. Within-Groups Factor levels participants Colored words BW Test Clock
Sometimes called “repeated measures” design
2-levels, All of the participants are in both levels of the IV
levels
participants
Colored
words BW
Test
Clock
Chair
Cab
Clock
Chair
Cab

23. Between vs. Within Subjects Designs
All participants participate in all of the conditions of the experiment.
Between-subjects designs
Each participant participates in one and only one condition of the experiment.
participants
Colored
words BW
Test
participants
Colored
words BW
Test
Between vs. Within Subjects Designs

24. Between vs. Within Subjects Designs
All participants participate in all of the conditions of the experiment.
Between-subjects designs
Each participant participates in one and only one condition of the experiment.
participants
Colored
words BW
Test
participants
Colored
words BW
Test
Between vs. Within Subjects Designs

25. Between subjects designs
Advantages:
Independence of groups (levels of the IV)
Harder to guess what the experiment is about without experiencing the other levels of IV
Exposure to different levels of the independent variable(s) cannot “contaminate” the dependent variable
Sometimes this is a ‘must,’ because you can’t reverse the effects of prior exposure to other levels of the IV
No order effects to worry about
Counterbalancing is not required
participants
Colored
words BW
Test
Clock
Chair
Cab
Between subjects designs

26. Between subjects designs
participants
Colored
words BW
Test
Clock
Chair
Cab
Disadvantages
Individual differences between the people in the groups
Excessive variability
Non-Equivalent groups
Between subjects designs

27. Individual differences
The groups are composed of different individuals
participants
Colored
words BW
Test
Individual differences

28. Individual differences
The groups are composed of different individuals
participants
Colored
words BW
Test
Excessive variability due to individual differences
Harder to detect the effect of the IV if there is one R NR
Individual differences

29. Individual differences
The groups are composed of different individuals
participants
Colored
words BW
Test
Non-Equivalent groups (possible confound)
The groups may differ not only because of the IV, but also because the groups are composed of different individuals
Individual differences

30. Dealing with Individual Differences
Strive for Equivalent groups
Created equally - use the same process to create both groups
Treated equally - keep the experience as similar as possible for the two groups
Composed of equivalent individuals
Random assignment to groups - eliminate bias
Matching groups - match each individuals in one group to an individual in the other group on relevant characteristics
Dealing with Individual Differences

31. Matching groups Group A Group B Matched groups
Trying to create equivalent groups
Also trying to reduce some of the overall variability
Eliminating variability from the variables that you matched people on
Red
Short
21yrs
matched
Red
Short
21yrs
matched
Blue
tall
23yrs
Blue
tall
23yrs
matched
Green
average
22yrs
Green
average
22yrs
Color
Height
Age
matched
Brown
tall
22yrs
Brown
tall
22yrs
Matching groups

32. Between vs. Within Subjects Designs
Between-subjects designs
Each participant participates in one and only one condition of the experiment.
Within-subjects designs
All participants participate in all of the conditions of the experiment.
participants
Colored
words BW
Test
participants
Colored
words BW
Test
Between vs. Within Subjects Designs

33. Within subjects designs
Advantages:
Don’t have to worry about individual differences
Same people in all the conditions
Variability between conditions is smaller (statistical advantage)
Fewer participants are required
Within subjects designs

34. Within subjects designs
Disadvantages
Range effects
Order effects:
Carry-over effects
Progressive error
Counterbalancing is probably necessary to address these order effects
Within subjects designs

35. Within subjects designs
Range effects – (context effects) can cause a problem
The range of values for your levels may impact performance (typically best performance in middle of range).
Since all the participants get the full range of possible values, they may “adapt” their performance (the DV) to this range.
Within subjects designs

36. Order effects Carry-over effects
Transfer between conditions is possible
Effects may persist from one condition into another
e.g. Alcohol vs no alcohol experiment on the effects on hand-eye coordination. Hard to know how long the effects of alcohol may persist.
test
Condition 2
Condition 1
How long do we wait for the effects to wear off?
Order effects

37. Order effects Progressive error
Practice effects – improvement due to repeated practice
Fatigue effects – performance deteriorates as participants get bored, tired, distracted
Order effects

38. Dealing with order effects
Counterbalancing is probably necessary
This is used to control for “order effects”
Ideally, use every possible order
(n!, e.g., AB = 2! = 2 orders; ABC = 3! = 6 orders, ABCD = 4! = 24 orders, etc).
All counterbalancing assumes Symmetrical Transfer
The assumption that AB and BA have reverse effects and thus cancel out in a counterbalanced design
Dealing with order effects

39. Counterbalancing Simple case Two conditions A & B
Two counterbalanced orders: AB BA
participants
Colored
words BW
Test
Counterbalancing

40. Often it is not practical to use every possible ordering
Partial counterbalancing
Latin square designs – a form of partial counterbalancing, so that each group of trials occur in each position an equal number of times
Counterbalancing

41. Partial counterbalancing
Example: consider four conditions
Recall: ABCD = 4! = 24 possible orders
1) Unbalanced Latin square: each condition appears in each position (4 orders) D C B A
Order 1
Order 2
Order 3
Order 4 A D C B B A D C C B A D
Partial counterbalancing

42. Partial counterbalancing
Example: consider four conditions
Recall: ABCD = 4! = 24 possible orders
2) Balanced Latin square: each condition appears before and after all others (8 orders) A B C D A B D C
Partial counterbalancing

43. Mixed factorial designs
Treat some factors as within-subjects (participants get all levels of that factor) and others as between-subjects (each level of this factor gets a different group of participants).
This only works with factorial (multi-factor) designs
Mixed factorial designs