1. Treating Stimuli as a Random Factor in Social Psychology: A New and Comprehensive Solution to a Pervasive but Largely Ignored Problem Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny University of Colorado BoulderUniversity of Connecticut

2. What to do about replicability? Mandatory reporting of all DVs, studies, etc.? Journals or journal sections devoted to straight replication attempts? Pre-registration of studies? Many of the proposed solutions involve large-scale institutional changes, restructuring incentives, etc. These are good ideas worthy of discussing, but surely not quick or easy to implement

3. One way to increase replicability: Treat stimuli as random Failure to account for uncertainty associated with stimulus sampling (i.e., treating stimuli as fixed rather than random) leads to biased, overconfident estimates of effects (Clark, 1973; Coleman, 1964) The pervasive failure to model stimulus as a random factor is probably responsible for many failures to replicate when future studies use different stimulus samples

4. Doing the correct analysis is easy! Recently developed statistical methods solve the statistical problem of stimulus sampling These mixed models with crossed random effects are easy to apply and are already widely available in major statistical packages (R, SAS, SPSS, Stata, etc.)

5. Outline of rest of talk 1.The problem – Illustrative design and typical RM-ANOVA analyses – Estimated type 1 error rates 2.The solution – Introducing mixed models with crossed random effects for participants and stimuli – Applications of mixed model analyses to actual datasets

6. Illustrative Design Participants crossed with Stimuli – Each Participant responds to each Stimulus Stimuli nested under Condition – Each Stimulus always in either Condition A or Condition B Participants crossed with Condition – Participants make responses under both Conditions Sample of hypothetical dataset: 546738879565 447846967456 536745758345

7. Typical repeated measures analyses (RM-ANOVA) M Black M White Difference 5.56.671.17 5.56.170.67 5.05.330.33 546738879565 447846967456 536745758345 How variable are the stimulus ratings around each of the participant means? The variance is lost due to the aggregation “By-participant analysis”

8. Typical repeated measures analyses (RM-ANOVA) 546738879565 447846967456 536745758345 4.00 3.67 6.33 7.33 3.67 6.33 8.00 6.00 8.00 4.00 5.00 5.33 Sample 1 v.s. Sample 2 “By-stimulus analysis”

9. Simulation of type 1 error rates for typical RM-ANOVA analyses Design is the same as previously discussed Draw random samples of participants and stimuli – Variance components = 4, Error variance = 16 Number of participants ∈ {10, 30, 50, 70, 90} Number of stimuli ∈ {10, 30, 50, 70, 90} Conducted both by-participant and by-stimulus analysis on each simulated dataset True Condition effect = 0

11. Type 1 error rate simulation results The exact simulated error rates depend on the variance components, which although realistic, were ultimately arbitrary The main points to take away here are: 1.The standard analyses will virtually always show some degree of positive bias 2.In some (entirely realistic) cases, this bias can be extreme 3.The degree of bias depends in a predictable way on the design of the experiment (e.g., the sample sizes)

12. The old solution: Quasi-F statistics Although quasi-Fs successfully address the statistical problem, they suffer from a variety of limitations – Require complete orthogonal design (balanced factors) – No missing data – No continuous covariates – A different quasi-F must be derived (often laboriously) for each new experimental design – Not widely implemented in major statistical packages

13. The new solution: Mixed models Known variously as: – Mixed-effects models, multilevel models, random effect models, hierarchical linear models, etc. Most social psychologists familiar with mixed models for hierarchical random factors – E.g., students nested in classrooms Less well known is that mixed models can also easily accommodate designs with crossed random factors – E.g., participants crossed with stimuli

15. Grand mean = 100

17. Mean A = -5 Mean B = 5

19. Participant Intercepts 5.86 7.09 -1.09 -4.53

21. Stim. Intercepts: -2.84 -9.19 -1.16 18.17

23. Participant Slopes 3.02 -9.09 3.15 -1.38

25. Everything else = residual error

27. The linear mixed-effects model with crossed random effects Fixed effects Random effects

28. The linear mixed-effects model with crossed random effects Intercept Slope 6 parameters

29. Fitting mixed models is easy: Sample syntax library(lme4) model <- lmer(y ~ c + (1 | j) + (c | i)) proc mixed covtest; class i j; model y=c/solution; random intercept c/sub=i type=un; random intercept/sub=j; run; MIXED y WITH c /FIXED=c /PRINT=SOLUTION TESTCOV /RANDOM=INTERCEPT c | SUBJECT(i) COVTYPE(UN) /RANDOM=INTERCEPT | SUBJECT(j). R SAS SPSS

30. Mixed models successfully maintain the nominal type 1 error rate (α =.05)

31. Applications to existing datasets 1.Representative simulated dataset (for comparison) 2.Afrocentric features data (Blair et al., 2002, 2004, 2005) 3.Shooter data (Correll et al., 2002, 2007) 4.Psi / Retroactive priming data (Bem) – Forward-priming condition (classic evaluative priming effect) – Reverse-priming condition (psi condition)

32. Comparison of effects between RM-ANOVA and mixed model analyses DatasetRM-ANOVA (by-participant)Mixed modelStimulus ICC F ratioD.F.pF ratioD.F.p Simulated example 30.48(1, 29)<.0019.11(1, 38.52).005r = 0.191 Shooter data57.89(1, 35)<.0013.39(1, 48.1).072r = 0.317 Afrocentric features data 6.40(1, 46).0154.33(1, 51.1).043r = 0.113 Bem (2011) Forward- priming condition 22.18(1, 98)<.00114.59(1, 46.91).029Targets: r = 0.349 Primes: r = 0.035 Bem (2011) Reverse- priming condition 6.60(1, 98).0122.34(1, 27.58).136Targets: r = 0.292 Primes: r = 0.0

33. Comparison of effects between RM-ANOVA and mixed model analyses DatasetRM-ANOVA (by-participant)Mixed modelStimulus ICC F ratioD.F.pF ratioD.F.p Simulated example 30.48(1, 29)<.0019.11(1, 38.52).005r = 0.191 Shooter data57.89(1, 35)<.0013.39(1, 48.1).072r = 0.317 Afrocentric features data 6.40(1, 46).0154.33(1, 51.1).043r = 0.113 Bem (2011) Forward- priming condition 22.18(1, 98)<.00114.59(1, 46.91).029Targets: r = 0.349 Primes: r = 0.035 Bem (2011) Reverse- priming condition 6.60(1, 98).0122.34(1, 27.58).136Targets: r = 0.292 Primes: r = 0.0

34. Comparison of effects between RM-ANOVA and mixed model analyses DatasetRM-ANOVA (by-participant)Mixed modelStimulus ICC F ratioD.F.pF ratioD.F.p Simulated example 30.48(1, 29)<.0019.11(1, 38.52).005r = 0.191 Shooter data57.89(1, 35)<.0013.39(1, 48.1).072r = 0.317 Afrocentric features data 6.40(1, 46).0154.33(1, 51.1).043r = 0.113 Bem (2011) Forward- priming condition 22.18(1, 98)<.00114.59(1, 46.91).029Targets: r = 0.349 Primes: r = 0.035 Bem (2011) Reverse- priming condition 6.60(1, 98).0122.34(1, 27.58).136Targets: r = 0.292 Primes: r = 0.0

35. Conclusion Many failures of replication are probably due to sampling stimuli and the failure to take that into account Mixed models with crossed random effects allow for generalization to future studies with different samples of both stimuli and participants

36. The end Further reading: Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of personality and social psychology, 103(1), 54-69.