1. Brian Nosek University of Virginia -- Center for Open Science http://briannosek.com/http://briannosek.com/ -- http://cos.io/http://cos.io/ Improving Openness and Reproducibility of Scientific Research

2. Norms Communality Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Counternorms Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity

3. Anderson, Martinson, & DeVries, 2007

4. Incentives for individual success are focused on getting it published, not getting it right Nosek, Spies, & Motyl, 2012

5. Problems Flexibility in analysis Selective reporting Ignoring nulls Lack of replication Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969; Greenwald, 1975; Meehl, 1978; Rosenthal, 1979

6. Silberzahn et al., 2015 Figure credit: fivethirtyeight.com

7. http://compare-trials.org/

8. Reported Tests (122) Median p-value =.02 Median effect size (d) =.29 % p <.05 = 63% Unreported Tests (147) Median p-value =.35 Median effect size (d) =.13 % p <.05 = 23% Franco, Malhotra, & Simonovits, 2015, SPPS

9. Positive Result Rate dropped from 57% to 8% after preregistration required.

10. 97%37% xx Open Science Collaboration, 2015, Science

12. Barriers 1.Perceived norms (Anderson, Martinson, & DeVries, 2007) 2.Motivated reasoning (Kunda, 1990) 3.Minimal accountability (Lerner & Tetlock, 1999) 4.I am busy (Me & You, 2016)

13. Technology to enable change Training to enact change Incentives to embrace change

14. Signals: Making Behaviors Visible Promotes Adoption

15. % Articles reporting that data was available 40% 30% 20% 10% 0%

16. % of Articles reporting that data was available 100% 75% 50% 25% 0% Reportedly available Available Correct Data Usable Data Complete Data

17. Two Modes of Research Context of Discovery Exploration Data contingent Hypothesis generating Context of Justification Confirmation Data independent Hypothesis testing

18. Preregistration Purposes 1.Discoverability: Study exists 2.Interpretability: Distinguish exploratory and confirmatory approaches Mistaking exploratory as confirmatory increases publishability of results decreases credibility of results

19. Positive Result Rate dropped from 57% to 8% after preregistration required.

20. Are you okay with someone you love receiving a treatment based on clinical trials that were not preregistered?

21. Preregistration Challenge http://cos.io/prereg

22. Registered Reports Design Collect & Analyze Report Publish PEER REVIEW PEER REVIEW

23. Registered Reports AIMS Neuroscience Attention, Perception, & Psychophysics Cognition and Emotion Comprehensive Results in Social Psychology Cortex Drug and Alcohol Dependence eLife Euro Journal of Neuroscience Experimental Psychology Journal of Accounting Research Journal of Business and Psychology Journal of Personnel Psychology Journal of Media Psychology Nutrition and Food Science Journal Perspectives on Psych. Science Royal Society Open Science Social Psychology Stress and Health Work, Aging, and Retirement

24. Infrastructure Strategy Solve problems researchers have now, Enable easy transition to openness

25. My Problems I lose stuff I forget stuff

26. Search and discover Develop idea Design study Acquire materials Collect data Store data Analyze data Interpret findings Write report Publish report OSF Open Science Framework A scholarly commons connecting the entire research lifecycle https://osf.io/

28. OpenSesame

29. ecosystem FUNDERSSOCIETIES UNIVERSITIES PUBLISHING

30. What can you do? Researchers – Try out OSF, http://osf.io/http://osf.io/ – Prereg Challenge, http://cos.io/prereg/http://cos.io/prereg/ Editors: Badges, Registered Reports, TOP Departments: OSF-Reproducibility Workshops Library: Connect IR to OSF, or use OSF as IR All: Conferences – OSF4Meetings Email: Support@cos.io or nosek@virginia.eduSupport@cos.ionosek@virginia.edu

33. http://cos.io/ http://osf.io/ Brian Nosek nosek@virginia.edu