1. CISL’s Research Data Archive (RDA) : Description and Methods
Joseph L “Joey” Comeaux
Computational & Information Systems Laboratory
National Center for Atmospheric Research
CISL’s Research Data Archive (RDA) : Description and Methods

2. Outline Description of CISL RDA Metadata Sustainable Data Curation
Considerations for Archiving Model Data
Lessons learned

3. CISL Research Data Archive (RDA)
Reference datasets maintained for use by research community
Receives high level of curation and stewardship
Primarily Meteorological and Oceanographic datasets
> 200 person-years invested in RDA
RDA managed by 8 staff members
438 TB (currently)
616 datasets (~ new datasets added annually)
3.7 Million files

4. Contents of the RDA 616 datasets
Content of archive important – Model, Satellite data much > than Obs …..

5. ACCESS MODES NCAR Mass Storage System Internet - Non NCAR users
- Primary mode of access NCAR Users
- Most users from US
Internet
- Non NCAR users
- Many international users
Special Request
- Provide data on media
- Allow access to data on MSS

6. Storage Metrics
MSS
Online

7. Unique Users

8. Amount of Data Delivered

9. Long-term RDA user metrics
MSS log file information first added 1990
Online web metrics – rough estimates

10. Comeaux/Worley/Dattore - SCD/DSS
4/26/2019

11. METADATA Several Levels of Metadata Dataset search and discovery
dataset usefulness
File Level
Description of file content
Relates files to datasets

12. Dataset Level Metadata
Model or Obs, Variables, Levels, POR …
Use controlled vocabularies (GCMD, ISO, THREDDS)
Guided entry via a Web-based GUI
Saved to a mysql database (and XML files as backup)
Exportable to DIF (NASA GCMD), THREDDS (UCAR CDP); can include others as needed
Dynamically create dataset web pages
Easy to create user interfaces that search the metadata and return relevant results

13. File Content Metadata Scan a data file; inventory its contents
Command-line utilities read the data files and extract the metadata
Metadata are saved to a mysql database and a system of XML files
Works with many Model and Obs formats
Provides more detailed and up-to-date search/discovery metadata, leading to better (more relevant) results when searching for datasets
Facilitates the discovery of specific data files within an RDA dataset

14. File XREF Metadata
Provides Xref from individual data files to datasets
Command line utilities archive data and create metadata
Relies on mysql
Allow for grouping and organization of files
Tracks both MSS and Web files
Tracks usage and allows metrics

15. METADATA Advantages of a GOOD, ROBUST metadata system
Allows creation of metrics in an easy fashion :
You can track dataset usage and users
Provides information on archive size and growth
Useful when analyzing future equipment and staff needs and thus funds

16. METADATA Advantages of a GOOD, ROBUST metadata system
Quality of metadata
directly related to the usefulness of search of discovery on both the dataset level and individual file level
Improves ability and speed for subset generation and automation
Improves the Long Term viability of the Archive
Reduces the chances of losing or throwing out data which is not adequately described with metadata
Facilitates preservation activities (backups, off-site replication, etc.)

17. Sustainable Data Curation
Stable Funding
Enriched Staff
Knowledgeable
Consistent Levels
Robust Storage
Backup Plans
Data Formats
Partnerships

18. Sustainable Data Curation
Focused on Data Management
Not project specific
Allows flexibility
Necessary to keep curated collection viable
Stable Funding
Knowledgeable and educated in the specific discipline
Important for checking integrity of data
Choosing organization of data
Creating adequate meta-data
Designing access system and assisting users
Consistent Staffing Levels
Dedicated to best practices in archiving and stewardship
Great deal of knowledge held by staff, regardless of documentation
Value of human based knowledge cannot be under-estimated
We find ~10 years is good
Staff

19. Sustainable Data Curation
Capable of meeting growth needs
NCAR -> tape based Mass Storage System (MSS)
Size > 2x every 2.5 years
Currently > 6PB
Must be able to handle data migration across generations of media (oozing)
Tapes size in MSS : 20GB -> 60GB -> 200GB -> 1000GB
Oozing must not interrupt normal, day-day operations
Provide access speeds able to handle daily curation and stewardship activities
Robust Storage Facilities
Loss of data attributed to 2 general causes
Equipment, Environmental
Lack of knowledge
Resolution
Store copies of irreplaceable data at separate facilities
Backup copies of data should be stored on different drives/tapes than originals
Knowledgeable Staff
Backups

20. Sustainable Data Curation
Ensure data access for long term
Fully documented to the byte level
Non-proprietary
Practices to avoid
Formats should not be dependent on OS, hardware or applications
Latest/Greatest formats not always best for your situation
Format
No single institute can “do it all”
Most users “need/want it all”
Good way to share some costs
National and international
Partnerships

21. Reanalysis Projects Prime example of data curation and stewardship
Encompass all 6 major aspects of good data curation
Main feature of the RDA and have been a very valuable resource for a wide variety of climate and weather studies

22. Most Current Reanalysis Projects
Name
Temporal
Range
Highest Resolution
Start
End
Horizontal
Vertical
NCEP/NCAR
1948
Ongoing
6 hours
209 km
17 Plvl
NCEP-DOE
1979
ECMWF
ERA-40
1957
2002
125 km
23 Plvl
NCEP NARR
3 hours
32 km
29 Plvl
Japanese
JRA

23. Considerations for Archiving Model Output
Know Your User Base
Manner in which data will be used
How to organize the data
Which model and what fields to archive
How long data from each model needs to be kept
Backups
Partnerships
Plan storage carefully
Create necessary metadata – dataset and file level

24. Considerations for Archiving Model Output
Diverse delivery system for access – web/ftp/mss/media
Transfer method for receiving archive
Data tools and formats
Known issues of models
Who/How will questions be handled
Task often larger than expected
Reorganize to meet user needs
Fixes/changes to model output
Changes in model resolution, variables, levels
Sub-setting needed
Moving large model output around

25. Considerations for Archiving Model Output
Diverse delivery system for access – web/ftp/mss/media
Transfer method for receiving archive
Data tools and formats
Known issues of models
Who/How will questions be handled
Task often larger than expected
Reorganize to meet user needs
Fixes/changes to model output
Changes in model resolution, variables, levels
Sub-setting needed
Moving large model output around

26. LESSONS LEARNED Create necessary Metadata
Do not do just minimal amount
Use standards where possible
Store in a useful, manageable system
Tightly couple files to datasets
User dynamic web interfaces to reflect current state
Organize archive files to align with ‘most’ user demands
Offer multiple modes of access to the data
Know your users
Track metrics so resources can be applied

27. LESSONS LEARNED How much software do you support
Balance between real time access and delayed mode
Simply data access where possible
Plan backup and recovery immediately
Staff educated in particular discipline needed
Assign consultants to each dataset

28. Questions and/or comments
Thank you
Questions and/or comments