1. The Challenge

2. BD2K-SDC Sustainability

3. Sustainability - Agenda
Introduction: Allen Dearry 
Results From Metrics RFI: Izumi Hinkson 
Panel Presentations and Discussion
Susan Gregurick, Moderator 
David Giaretta: The Role of Trustworthy Digital Repositories in Sustainability
George Alter: The Sloan Stewardship Gap Project
Melissa Landrum: Archiving Interpretations of Variants in ClinVar 
Cathy Wu: Interoperability, Sustainability, and Impact: A UniProt Case Study
Posters
134, Combining Protein and Genome Annotation for Interpretation of Genomic Variants, Peter McGarvey
135, Interoperability of NURSA, PharmGKB, dkNET, and DataMed, Neil McKenna
Notes
(hotlink on program book p6)

4. RFI- Metrics to Assess the Value of Biomedical Digital Repositories
Izumi Hinkson
AAAS S&T Policy Fellow
NCI CBIIT
BD2K & SDC  Sustainability Working Group  November 30, 2016

5. Goals of RFI Solicit input from stakeholders
Foundation for long-term sustainability
Enable repository owners to prioritize repository management
Support decisions made by funding agencies
Support diverse domains of science
Communication between stakeholders

6. Distribution of Response Themes8%
n=21 9%
n=24
24%
n=65 6%
n=16
11%
n=29
26%
n=71
16%
n=42

7. Examples of metrics Use & Users Number of downloads
Size of user community
International reach
Cautions
User counts vs coverage
Bias of utilization statistics
24%
n=65

8. Examples of metrics Quality & Impact Number of publications,
citations, grants, and
patents
Data and metadata standards
Educational tools and protocols
Altmetrics
Cautions
Data vs repository value
Context for quantitative metrics
26%
n=71

9. Examples of metrics Quality of Service Expertise of staff
Up time and response time
Regularly scheduled updates and maintenance
Help desk and FAQs
Tutorials, webinars, and training
Cautions
Down time context
16%
n=42

10. Examples of metrics Governance Scientific Advisory Board
Legal, regulatory, and contractual framework
Documentation
Terms of use
Licensing
Encryption and security
Lifecycle management plan
Funding
11%
n=29

11. Examples of metrics Infrastructure Infrastructure funding
Technology architecture
Hardware and software
Maintenance
Sustainability
Office space
16%
n=42

12. Examples of metrics Surveys & Case Studies User experience surveys
Stakeholder interviews
Availability of alternatives
Counterfactuals
External audits
Cautions
Testimonials may be biased
Validity of counterfactuals 9%
n=24

13. Examples of metrics Other Considerations Terminology
Indicators vs metrics
Research cycle specific metrics
Tracking of missing, uncertain, contradictory, or retracted data
“Diversity in data complexity calls for different metrics”
Existing metrics and assessment resources (e.g., ISO 16363, DSA-WDS) 8%
n=21

14. Where to find more information?
Executive Summary, mid-December
datascience.nih.gov/bd2k
For more information,

15. Data Science at NIH Data Science at NIH 
 #BD2K, #BigData

16. Framing Questions for Sustainability Session at the 2016 BD2K AHM
(also on google doc notes page)
1. In thinking about data preservation, scientific quality and impact, what do you find are the most important elements, or indicators, that promote data quality and ensure data impact?
2. As data integration and database cross integration become commonplace, how will this affect attribution and adherence to the FAIR principles*? For example, are there best practices for using format standards or for correctly referencing data identifiers that best support cross-linking data & repositories from one source/provider to another?
3. For data repositories that deal with clinical data and information, are there particular issues or challenges with adhering to the FAIR principles and performance indicators that could impact success?
4. What is the role for certifications of biomedical data repositories and trusted digital repositories? Will this differ between different biomedical and scientific domains?
5. What role does the scientific research community play in planning for and evaluating indicators for data preservation?
*FAIR principles are Findable, Accessible, Interoperable and Reusable.