1. Could you repeat the question please? Uri Simonsohn SESP 2014SESP 2014 – Friday October 3 rd 1

2. Question Statistics Data needs 2

3. Question New Statistics Data needs 3

4. New Question New Statistics Data needs 4

5. New Question New Statistics Much more Data needed 5

6. Question Statistics Data needs 6

7. Question Bayesian Statistics Data needs 7

8. Unclear Question Bayesian Statistics Data needs 8

9. Unclear Question Bayesian Statistics (Slightly more) Data needed 9

10. What question are you interested on? 10

11. Results Control: 7.12 seconds Bingo: 8.44 Bingo: 7.66 Bingo: 11.08 Bingo: 7.88 Bingo: 8.24 Bingo 8.24 seconds p =.079 p =.0001 p =.0000001 p =.049 p =.0079 11

12. Our theories are about existence of effects Our questions are about existence of effects p-value: tool that informs existence of effect 12

13. Oh yeah? Null is always false! Big N  anything** Do we really care if with N=10,000: – Control: 7.12 seconds – Bingo: 7.16 seconds p<.01? Answer 1: Maybe! Answer 2: Big N  not everything is ** 13

14. Simonsohn (2011) N=12.8 million February 2 nd birthday – Live in 2 nd avenue? p =.74 14

15. Oh yeah? Null is always false! Big N  anything** Do we really care if: – Control: 7.12 seconds – Bingo: 7.16 seconds p<.01 Answer 1: maybe! Answer 2: Big N  not everything is ** Answer 3: We don’t have big Ns! 15

16. Sample Size in Psych Science 2003-2010 Median: n=19 16

17. When is N “too big” for p-values? I work near wealthy lab – N<500 – 96 th percentile In a 2x2 design – 80% power for d=.5 (!) 17

18. Wait …If lab can barely tell you if d=0 or not. Say we did care about effect size How much would we learn? 18

19. http://datacolada.org/2014/05/01/20-we-cannot-afford-to-study-effect-size-in-the-lab/ 19

20. Lab studies are never too big for p-values – However… They are always too small for confidence intervals 20

21. If we have a p-value question And give a CI answer. What happens? Case study: Bootstrapping & mediation – If p=.049 “The confidence interval does not include 0” – If p<.0001 “The confidence interval does not include 0” Confidence intervals have reduced information 21

22. What about Bayesian? 22

23. Recall: Question Statistics Data needs 23

24. Recall: Question Bayesian Statistics Data needs 24

25. Recall: Unclear Question Bayesian Statistics Data needs 25

26. Recall: Unclear Question Bayesian Statistics (Slightly more) Data needed 26

27. Bayesian Hypothesis Testing Very nice approach – Are data more compatible with null or alternative? But – Which alternative hypothesis? – What’s the question? Psych Bayesians (so far) “default” alternative Null: d=0 Alternative: d~N(0,1) 27

28. Three Problems with Default Alternative Problem 1: Who asked that question? Problem 2: If we ask that question – Equivalent to p-value with α <.01  Slightly more data Problem 3: Answer changes (a lot) with o other alternative Null: d=0 Alternative 1: d~N(0, 1) Alternative 2: d~N(0,.5) Alternative 3: d~N(0,.25) Same data, different answer. Not clear which it is we are asking 28

29. Last Slide Question  Stats Does effect exist in the lab?  p-value That may be the wrong question – Let’s debate that Be explicit about consequences Can we study our own thing? Leave the lab? Go Within-Subject? Research other things? 29