Transparency increases the credibility and relevance of research David Mellor Center for Open Science https://cos.io/

Evidence to encourage change Incentives to embrace change Technology to enable change Improving scientific ecosystem

Infrastructure Metascience Community

http://cos.io/top

Transparency and Openness Promotion (TOP) Guidelines Eight policy statements for increasing the transparency and reproducibility of the published research. Agnostic to discipline Low barrier to entry Modular

Disclose Require Verify Three Tiers Disclose Require Verify Eight Standards Data citation Materials transparency Data transparency Code transparency Design transparency Study Preregistration Analysis Preregistration Replication

Disclose Require Verify Three Tiers Disclose Require Verify Eight Standards Data citation Materials transparency Data transparency Code transparency Design transparency Study Preregistration Analysis Preregistration Replication

Disclose Require Verify Three Tiers Disclose Require Verify Eight Standards Data citation Materials transparency Data transparency Code transparency Design transparency Study Preregistration Analysis Preregistration Replication

Example Implementations

Data, Analytic Methods (Code), and Research Materials Transparency Level 1: Authors must disclose action

Data, Analytic Methods (Code), and Research Materials Transparency Level 2: Authors must share (exceptions permitted)

Data, Analytic Methods (Code), and Research Materials Transparency Level 3: Journal or third party will verify that the data can be used to reproduce the findings presented in a paper.

Design and Analysis Transparency Society or journal defines the relevant reporting standards that are appropriate for their discipline. http://www.cell.com/star-methods http://resource-cms.springer.com/springer-cms/rest/v1/content/7117202/data/v2/Minimum+standards+of+reporting+checklist http://www.equator-network.org/ https://www.nature.com/authors/policies/ReportingSummary.pdf

Preregistration Preregistration of studies is a means of making research more discoverable even if it does not get published. Preregistration of Analysis Plans clarifies the distinction between confirmatory and exploratory research.

Exploratory research: Finds unexpected trends Pushes knowledge into new areas Results in a testable hypothesis

Confirmatory research: Puts a hypothesis to the test Does not allow data to influence the hypothesis Results are held to the highest standard of rigor

Presenting exploratory results as confirmatory increases publishability at the expense of credibility.

Direct replications: The gold standard for scientific evidence “Nullius in verba” ~ “Take nobody's word for it”

Preregistration or Replication Level 1: Disclose preregistration, encourage replication http://www.psychonomic.org/?page=journals

Preregistration or Replication Level 2: Results blind peer review https://jbp.uncc.edu/other-journals-involved-in-this-joint-initiative/

Preregistration or Replication Level 3: Registered Reports

Preregistration or Replication Level 3: Registered Reports Are the hypotheses well founded and worth addressing? Are the methods and proposed analyses able to address the hypotheses? Have the authors included sufficient positive controls to confirm that the study will provide a fair test?

Preregistration or Replication Level 3: Registered Reports Did the authors follow the approved protocol? Did positive controls succeed? Are the conclusions justified by the data?

Nearly 3,000 Journal, Society, Publisher, and Funder Signatories

How can publishers support this effort? Becoming a signatory provides a common framework for a diverse group of disciplines under one roof to consider different ways to adopt TOP. It is not necessary to mandate a blanket policy across different disciplines.

What incentives are needed for researchers to more openly share? How do we enable sharing without placing more burden on researchers?

Mandates Tack on extra steps to a process Appeal to everyone’s good nature

Mandates Tack on extra steps to a process Appeal to everyone’s good nature

Align scientific values (open sharing, value of evidence) with scientific rewards (getting published, funded, and hired). Make tools that make researchers’ lives easier, while making open science as easy as pressing a button.

Collaboration Documentation Archiving And it does things that many content management systems do. But it always balances things researchers care about while either automatically or making it very easy to align with values. Again, scholars should focus on scholarship--and we respect that they have a workflow—this has to be frictionless.

So here, it’s very easy to curate a project, add contributors, share, but we also keep an activity log. To the users that’s either meaningless or an easy way to keep up-to-date, but to us, this is provenance—this is important information about how science happens—science is a process, this is great time series metadata for metascience

OpenSesame

OpenSesame

Signals: Making Behaviors Visible Promotes Adoption Badges Open Data Open Materials Preregistration Psychological Science (Jan 2014)

Data Availability in Psychological Science 10x increase from 2013 to 2015 (Kidwell et al., 2016)

$1,000,000 in prizes for researchers to publish the results of their preregistered work.

https://cos.io/prereg

Data collection methods https://cos.io/prereg Research questions Data collection methods Variables Statistical tests Outliers

What can you do: Sign the TOP Guidelines Implement rewards for ideal science such as badges or Registered Reports Spread awareness about the Preregistration Challenge David Mellor, david@cos.io Find me online: https://osf.io/qthsf/, @EvoMellor Find this presentation: https://osf.io/GUID (Take a picture!)

General Links https://cos.io/top https://cos.io/badges https://cos.io/rr

Links to reported policies http://www.psychonomic.org/?page=journals http://journals.plos.org/plosone/s/data-availability http://publications.agu.org/author-resource-center/publication-policies/data-policy/data-policy-faq/ https://ajps.org/ajps-replication-policy/ http://www.cell.com/star-methods https://www.equator-network.org/ https://www.nature.com/authors/policies/ReportingSummary.pdf?viewType=Print&viewClass=Print http://resource-cms.springer.com/springer-cms/rest/v1/content/7117202/data/v2/Minimum+standards+of+reporting+checklist https://jbp.uncc.edu/other-journals-involved-in-this-joint-initiative/ http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002456