1. Transparency increases the credibility and relevance of research
David Mellor
Center for Open Science

2. Evidence to encourage change Incentives to embrace change
Technology to enable change
Improving scientific ecosystem

3. Infrastructure
Metascience
Community

5. Transparency and Openness Promotion (TOP) Guidelines
Eight policy statements for increasing the transparency and reproducibility of the published research.
Agnostic to discipline
Low barrier to entry
Modular

6. Disclose Require Verify
Three Tiers
Disclose Require Verify
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication

7. Disclose Require Verify
Three Tiers
Disclose Require Verify
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication

8. Disclose Require Verify
Three Tiers
Disclose Require Verify
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication

9. Example Implementations

10. Data, Analytic Methods (Code), and Research Materials Transparency
Level 1: Authors must disclose action

11. Data, Analytic Methods (Code), and Research Materials Transparency
Level 2: Authors must share (exceptions permitted)

12. Data, Analytic Methods (Code), and Research Materials Transparency
Level 3: Journal or third party will verify that the data can be used to reproduce the findings presented in a paper.

13. Design and Analysis Transparency
Society or journal defines the relevant reporting standards that are appropriate for their discipline.

14. Preregistration
Preregistration of studies is a means of making research more discoverable even if it does not get published. Preregistration of Analysis Plans clarifies the distinction between confirmatory and exploratory research.

15. Exploratory research:
Finds unexpected trends
Pushes knowledge into new areas
Results in a testable hypothesis

16. Confirmatory research:
Puts a hypothesis to the test
Does not allow data to influence the hypothesis
Results are held to the highest standard of rigor

17. Presenting exploratory results as confirmatory increases publishability at the expense of credibility.

18. Direct replications: The gold standard for scientific evidence
“Nullius in verba” ~ “Take nobody's word for it”

19. Preregistration or Replication
Level 1: Disclose preregistration, encourage replication

20. Preregistration or Replication
Level 2: Results blind peer review

21. Preregistration or Replication
Level 3: Registered Reports

22. Preregistration or Replication
Level 3: Registered Reports
Are the hypotheses well founded and worth addressing?
Are the methods and proposed analyses able to address the hypotheses?
Have the authors included sufficient positive controls to confirm that the study will provide a fair test?

23. Preregistration or Replication
Level 3: Registered Reports
Did the authors follow the approved protocol?
Did positive controls succeed?
Are the conclusions justified by the data?

24. Nearly 3,000 Journal, Society, Publisher, and Funder Signatories

25. How can publishers support this effort?
Becoming a signatory provides a common framework for a diverse group of disciplines under one roof to consider different ways to adopt TOP.
It is not necessary to mandate a blanket policy across different disciplines.

26. What incentives are needed for researchers to more openly share?
How do we enable sharing without placing more burden on researchers?

27. Mandates
Tack on extra steps to a process
Appeal to everyone’s good nature

28. Mandates
Tack on extra steps to a process
Appeal to everyone’s good nature

29. Align scientific values (open sharing, value of evidence) with scientific rewards (getting published, funded, and hired).
Make tools that make researchers’ lives easier, while making open science as easy as pressing a button.

31. Collaboration Documentation Archiving
And it does things that many content management systems do. But it always balances things researchers care about while either automatically or making it very easy to align with values. Again, scholars should focus on scholarship--and we respect that they have a workflow—this has to be frictionless.

32. So here, it’s very easy to curate a project, add contributors, share, but we also keep an activity log. To the users that’s either meaningless or an easy way to keep up-to-date, but to us, this is provenance—this is important information about how science happens—science is a process, this is great time series metadata for metascience

33. OpenSesame

34. OpenSesame

35. Signals: Making Behaviors Visible Promotes Adoption
Badges
Open Data
Open Materials
Preregistration
Psychological Science (Jan 2014)

36. Data Availability in Psychological Science
10x increase from 2013 to 2015
(Kidwell et al., 2016)

37. $1,000,000 in prizes for researchers to publish the results of their preregistered work.

38. https://cos.io/prereg

39. Data collection methods
Research questions
Data collection methods
Variables
Statistical tests
Outliers

40. What can you do: Sign the TOP Guidelines
Implement rewards for ideal science such as badges or Registered Reports
Spread awareness about the Preregistration Challenge
David Mellor, Find me online: Find this presentation:
(Take a picture!)

41. General Links https://cos.io/top https://cos.io/badges

42. Links to reported policies