1. Thomas Steckler FORCE2015, Oxford, 12-13 Jan 2015
True or False? Transparency Validation and Reproducibility of Research in Scholarly Communication
Thomas Steckler
FORCE2015, Oxford, Jan 2015

2. The Pillars of Good Scientific Practice
SCIENCE
Reproducibility
Robustness
Relevance

3. Should Scholarly Research Aim for Reproducibility or Robustness?
Reproducibility: The ability of an entire experiment or study to be reproduced, ideally according to the same reproducible experimental description and procedure
Robustness: A characteristic describing a phenomenon / finding to be detected effectively while the variables of a test system are altered
A robust concept can be observed without failure under a variety of conditions
 Making obsolete the requirement for exact, point-by-point reproduction
A robust finding may be (biologically) more relevant than reproducibility
 Robustness of data may be key
Most replication studies may in fact test the robustness of reported findings, since it is difficult to exactly recapitulate all details and conditions under which original data were produced

4. What Makes Data Robust? Criteria Rationale Implication
Meaningful sampling procedure
determines data relevance to answer research question
high ethological or biomarker validity of test; translatability
Data reliability
determines consistency of findings across different experimental conditions
document comparison across different experimental conditions/replication
Proper baseline
uncertainty in a ratio is more sensitive to denominator than to numerator error
provide comparison with historical data
Limited noise
determines accuracy of the signal
acceptable variability, correct positive and negative controls
Limited bias
determines accuracy of a finding
document blinded and automated measurement, show all data, pre-specified endpoints, randomization
Appropriate sample size
determines statistical efficiency
provide power analysis and PPV
Data entry audit
verifies data in database
traceability of data
Complete documentation
transparency of data definitions, collection procedures and other meta-information
full methodological detail
Full data access
allows replication, reanalysis, new analysis, interpretation, or inclusion into meta-analyses, and facilitates reproducibility of research
raw data sharing
Robust statistics
not unduly affected by outliers
statistical review, sensitivity analysis
Visualization
highlights possible data defects
high quality figures