R2R ↔ NODC Steve Rutz NODC Observing Systems Team Leader May 12, 2011 Presented by L. Pikula, IODE OceanTeacher Course Data Management for Information Professionals, Oostend, Be. September 26-30, 2011

2 Acquire: environmental data from U.S. and foreign sources Archive: preserve the Nation’s environmental data assets Assess: use the data in assessments and assemble it into easy to use, long-term collections Access: provide access to environmental data for business, federal, science, and many other customers Scientific Data Stewardship So, what does NODC do?

The OAIS Reference Model Open Archival Information System (OAIS) ISO Standard (14721) for Digital Archives Applies to all organizations that need to preserve digital information for the long-term Does NOT specify any particular implementation An organization conforms to the OAIS RM by discharging a minimal set of responsibilities and supporting basic information concepts

ISO 14721: ISO 14721:2003 specifies a reference model for an open archival information system (OAIS). The purpose of this ISO 14721:2003 is to establish a system for archiving information, both digitalized and physical, with an organizational scheme composed of people who accept the responsibility to preserve information and make it available to a designated community.

ISO 4721 continued This reference model addresses a full range of archival information preservation functions including ingest, archival storage, data management, access, and dissemination. It also addresses the migration of digital information to new media and forms, the data models used to represent the information, the role of software in information preservation, and the exchange of digital information among archives. It identifies both internal and external interfaces to the archive functions, and it identifies a number of high-level services at these interfaces. It provides various illustrative examples and some "best practice" recommendations. It defines a minimal set of responsibilities for an archive to be called an OAIS, and it also defines a maximal archive to provide a broad set of useful terms and concepts.

ISO The OAIS model described in ISO 14721:2003 may be applicable to any archive. It is specifically applicable to organizations with the responsibility of making information available for the long term. This includes organizations with other responsibilities, such as processing and distribution in response to programmatic needs.

The OAIS Environment Producer provides information to be preserved Management sets overall policy Consumer seeks and acquires preserved information OAIS Archive Management Producer Consumer The OAIS Environment from 30,000 ft

Elements of NODC’s process 1. Submission Information Form 2. Submission Information Package 3. Archival Information Package 4. Distribution Information Package

Are you ready to archive? Send us an ! To: From: R2R Data Manager Subject: automated archiving of R2R data

1.Submission Information Form Producer↔Archive Document their Relationship: POCs (both Producer and Archive) ID Data Types to Archive (or which Archive) Data Transport Mechanisms File Naming Conventions File Formats File Integrity Checking Feedback Mechanisms (e.g., notification)& etc.

2.Submission Information Package Data files Preferred formats - NetCDF with CF and ACDD conventions Accepted formats - ASCII Text, Microsoft Office products, databases, and everything else May include metadata Metadata records Preferred formats - FGDC or ISO Accepted formats - everything else Documentation Data processing and quality control procedures File formats and naming conventions Freely distributable publications

12 Producer → Archive NODC “pulls” SIP from Producer (Submission Information Package) FTP, HTTP, rsync Pull enables redundancy, reduces possibility of data loss NODC confirms files received are same as files produced Use cryptographic “checksum” or “digest” For example: SHA-256, MD5 NODC ensures completeness of SIP How data files are grouped in the Archive How often data files are ingested

4.Archival Information Package Accession Tracking Database Tracks metadata relevant to data discovery Records version control for long term stewardship Producer's SIP Data set is preserved exactly as submitted to Archive Archive's supplementary files Browse graphic of geographic coordinates FGDC metadata records May generate more interoperable file formats of the data for long-term stewardship

U.S.NODC’s Ocean Archive System

4.Distribution Information Package Ocean Archive System Download AIP in its entirety or select individual files HTTP and FTP View and download individual files ftp://data.nodc.noaa.gov/ OPeNDAP Hyrax server THREDDS Data Server Web Accessible Folder of Metadata

Questions?