1. “Making Science Great Again”
From replication crisis to open science, how we can improve research
Roy Salomon
Gonda Center
This presentation is inspired by presentations from Daniel Lakens, Jim Grange, and Brian Nosek. Most slides are from PD Dr. Felix Schönbrodt, Ludwig-Maximilians-Universität München, and used under a CC-BY 4.0 license.

2. “Only when certain events recur in accordance with rules or regularities, as in the case of repeatable experiments, can our observations be tested—in principle—by anyone.... Only by such repetition can we convince ourselves that we are not dealing with a mere isolated ‘coincidence.” – Karl Popper (1959, p. 45)
We have a problem!
What are the causes?

3. We have a problem!

4. 2011
Bem
2012
2013
2014
2015

5. 2011
Bem
Simmons et al.: False-positive psychology
2012
2013
2014
The combination of some typical questionable research practices (QRPs) increases Type-I error rate from 5% to > 50%.
2015

6. 2011
2012
2013
2014
2015
Bem
Simmons et al.: False-positive psychology
John et al.: Prevalence of QRPs
“Self-admission rate” for many QRPs > 50%; estimated prevalence partly > 70%.

7. I see a train wreck looming.
2011
Bem
Simmons et al.: False-positive psychology
2012
John et al.: Prevalence of QRPs
Doyen et al. (2012) ➙ “The Bargh rant” Kahneman: Open Letter
2013
Cited by 4195
2014
2015
I believe that you should collectively do something about this mess.
I see a train wreck looming.

8. I see a train wreck looming.
I believe that you should collectively do something about this mess.
I see a train wreck looming.

9. n = 20 in each condition d = 0.73 95% CI[0.05; 1.41] 577 citations

10. N > 3500 in each condition n = 20 in each condition d = 0.73
95% CI[0.05; 1.41]
577 citations
p=.76
d = -0.01
95% CI[-0.05; 0.04]
577x cited

11. 2011
Bem
Simmons et al.: False-positive psychology
2012
John et al.: Prevalence of QRPs
Doyen et al. (2012) ➙ “The Bargh rant” Kahneman: Open Letter
2013
Foundation of Center for Open Science ( Open Science Framework )
2014
2015

12. Complete scientific project management
Data management, pre-registrations, version control, private/public, private read-only links for reviewers, wikis, lists, Dropbox/Figshare/Github integration, download statistics …

13. 2011
Bem
Simmons et al.: False-positive psychology
2012
John et al.: Prevalence of QRPs
Doyen et al. (2012) ➙ “The Bargh rant”
Kahneman: Open Letter
2013
Foundation of Center for Open Science ( Open Science Framework
Simonsohn et al.: p-curve
) +
2014
2015

14. p-curve: Null effect Under H₀, p-values are uniformly distributed
Simonsohn et al.: p-curve
Under H₀, p-values are uniformly distributed
Doing a study = drawing a random p-value from this distribution 5 4
Density 3 2 1 5%
0.0
0.2
0.4
0.6
0.8
1.0
p value

15. p-curve: Effect size > 0
Simonsohn et al.: p-curve
With increasing power, the p-curve gets more positively skewed 8
10% power 6
Density 4 2
10%
0.0
0.2
0.4
0.6
0.8
1.0
p value

16. p-curve: Effect size > 0
Simonsohn et al.: p-curve
With increasing power, the p-curve gets more positively skewed
35% power (average in psychology) 12 10 8
Density 6 4 2
35%
0.0
0.2
0.4
0.6
0.8
1.0
p value

17. p-curve: Effect size > 0
Simonsohn et al.: p-curve
With increasing power, the p-curve gets more positively skewed
80% power 30 25 20
Density 15 10 5
80%
0.0
0.2
0.4
0.6
0.8
1.0
p value

18. Simonsohn et al.: p-curve Elderly priming p-values
11% of all p-values are expected to be between
.025 and .05
49% of all p-values are expected to be <.025 50 40
Density 30 20 10
k=5
k=13
0.00
0.05
0.10
0.15
0.20
60% power
p value 18

19. Simonsohn et al.: p-curve
Elderly priming
p-values (k = 18): p = .043
p = .034
p = .046
p = .033
p = .017
p = .044
p = .043
p = .048
p = .039 …

20. 2011
2012
2013
2014
2015
Bem
Simmons et al.: False-positive psychology
John et al.: Prevalence of QRPs
Doyen et al. (2012) ➙ “The Bargh rant” Kahneman: Open Letter
Foundation of Center for Open Science ( Open Science Framework
Simonsohn et al.: p-curve
ManyLabs 1 & Special Issue “Replication”
) +

21. ManyLabs 1 & Special Issue “Replication”
Social Psychology: Replication Special Issue
(Nosek & Lakens, 2014)
Bayesian reanalysis
(Marsman, Schönbrodt, Morey, Wagenmakers, in prep.)
7/59 =
12% replicable

22. 2011
2012
2013
2014
2015
Bem
Simmons et al.: False-positive psychology
John et al.: Prevalence of QRPs
Doyen et al. (2012) ➙ “The Bargh rant” Kahneman: Open Letter
Foundation of Center for Open Science ( Open Science Framework
Simonsohn et al.: p-curve
ManyLabs 1 & Special Issue “Replication” Schnall-Debate
ManyLabs 3
) +

23. ManyLabs 3
10 effects, 20 labs, n > 3400

24. power in original study (n = 152): 8%
ManyLabs 3
ES: d = .09, p = .02
n for 95% power = 6708
power in original study (n = 152): 8%
10 effects, 20 labs, n > 3400

25. 2011
Bem
Simmons et al.: False-positive psychology
2012
John et al.: Prevalence of QRPs
Doyen et al. (2012) ➙ “The Bargh rant” Kahneman: Open Letter
2013
Foundation of Center for Open Science ( Open Science Framework
Simonsohn et al.: p-curve
ManyLabs 1 & Special Issue “Replication” Schnall-Debate
ManyLabs 3
Reproducibility Project: Psychology (RP:P)
) +
2014
2015

27. Reproducibility Project: Psychology (RP:P)
97 replications
36% of all replications were significant
PS - cog: 53%
JEP:LMC: 48%
PS - soc: 29%
JPSP - soc: 23%
83% of all effect sizes are smaller than the original

28. An outlook to other disciplines.
Not my problem?
An outlook to other disciplines.

29. 31

30. 53 ‘landmark studies’, not randomly selected: fresh approaches targeted for future drug development
“scientific findings were confirmed in only 6 (11%) cases. Even knowing the limitations of preclinical research, this was a shocking result.”
Bayer Healthcare: 67 target-validation projects in oncology, women’s health, and cardiovascular medicine. Only 14 (21%) could be reproduced.
Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483, 531–533. doi: /483531a Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10, 712–712. doi: /nrd3439-c1 32

31. “Our results indicate that the average statistical power of studies in the field of neuroscience is probably no more than between ~8% and ~31%, on the basis of evidence from diverse subfields within neuro-science.

32. What are the Causes?
What are the Solutions?

33. Why is reproducibility so low??
How?
We are human
We Err. We p-hack. We HARK. We use QRDs.
We are part of a system
Publication Bias
Problematic incentive scheme

35. Unintentional mistakes
The garden of forking paths
Questionable Research Practices (QRPs)
Fraud
Publication bias

36. Unintentional mistakes
The garden of forking paths
Questionable Research Practices (QRPs)
Fraud
Publication bias

37. Reproducible analysis code and open data required at submission - “inhouse checking” in review process
54% of all submissions had results in the paper that did not match the computed results from the code
wrong signs, wrong labeling of regression coefficients, erorrs in sample sizes, wrong descriptive stats

38. Solution: Open Scripts
Solution: Open Data
Solution: Open Scripts
Unintentional mistakes
The garden of forking paths
Questionable Research Practices (QRPs)
Fraud
Publication bias 39

39. Solution: Open Scripts
Solution: Open Data
Solution: Open Scripts
Unintentional mistakes
The garden of forking paths
Questionable Research Practices (QRPs)
Fraud
Publication bias 40

40. The garden of forking paths
Data
Andrew Gelman & Eric Loken, 2013
Inspired by Neurosceptic’s blog:

41. The garden of forking p-hacks
Data
P=0.04
P=0.34
P=0.66
P=0.82
P=0.17
P=0.34
P=0.07
P=0.24
Andrew Gelman & Eric Loken, 2013
Inspired by Neurosceptic’s blog:

42. Lets do this together
Inspired by Neurosceptic’s blog:

43. Solution: Preregistration
The first principle is that you must not fool yourself and you are the easiest person to fool. -Richard P. Feynman
What should be included in a preregistration?
What is a preregistration?
Hypotheses
Predictions
Models
Dependent variables
ROIs
Confounds
Exclusion criteria
Feature definition (“functional connectivity defined as…”)
Analysis plan
Statistical techniques (algorithms)
Multiple comparison correction
Parameters
It’s the introduction and methods section of your future paper.

45. Solution: Open Scripts
Solution: Open Data
Solution: Open Scripts
Unintentional mistakes
Solution: Open Data
Solution:
Pre- registration
The garden of forking paths
Questionable Research Practices (QRPs)
Fraud
Publication bias

46. Solution: Open Scripts
Solution: Open Data
Solution: Open Data
Solution: Open Scripts
Unintentional mistakes
Solution: Open Data
Solution:
Pre- registration
The garden of forking paths
Questionable Research Practices (QRPs)
Fraud
Publication bias

47. QRP

48. Solution: Reproducible Scripts
Solution: Open Data
Unintentional mistakes
Solution: Open Data
Solution:
Pre- registration
The garden of forking paths
Solution:
Pre- registration
Questionable Research Practices (QRPs)
Fraud
Publication bias

49. 92%! 34%? 21%? Psychology/Psychiatry
Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. doi: /s

50. Reviewed Pre-Registration

51. Reviewed Pre-Registration
Advances in Methodologies and Practices in Psychological ScienceAIMS Neuroscience Animal Behavior and Cognition Attention, Perception, and Psychophysics Behavioral Neuroscience Cognition and Emotion Cognitive Research: Principles and Implications Comprehensive Results in Social Psychology Cortex Drug and Alcohol Dependence European Journal of Neuroscience Experimental Psychology Health Psychology Bulletin Human Movement Science Infancy International Journal of Psychophysiology Journal of Business and Psychology Journal of Cognitive Enhancement Journal of European Psychology Students Journal of Experimental Political Science Journal of Media Psychology Journal of Personnel Psychology Journal of Research in Personality Judgment and Decision Making Management and Organization Review Memory Nature Human Behaviour NFS Journal Nicotine & Tobacco Research Perspectives on Psychological Science Royal Society Open Science Stress and Health The Leadership Quarterly Work, Aging and Retirement

52. Solution: Reproducible Scripts
Solution: Open Data
Unintentional mistakes
Solution: Open Data
Solution:
Pre- registration
The garden of forking paths
Solution:
Pre- registration
Questionable Research Practices (QRPs)
Fraud
Solution:
Pre- registration,
Registered reports
Publication bias

53. How we can improve research?

54. Summary Personal level System Level Open data
Our current system’s incentives foster questionable research practices, which decrease the truth value of our shared knowledge. To make science great again we need to adopt new approaches:
Personal level
Open data
Open scripts
Open materials
Preregistration
Transparency
Better Statistics
Peer review openness (make others participate)
System Level
How we appraise and hire people.
Expect less papers but better ones.
What journals we support?
Open Access!