NASA Earth Science Data Stewardship Dawn R. Lowe, Project Manager Earth Science Data & Information Systems (ESDIS) Project NASA/Goddard Space Flight Center

Data Stewardship Objective: Ensure that our Earth science data and information content are reliable, of high quality, easily accessible, and usable for as long as they are considered to be of value. Involves communications with data producing projects throughout their lifecycles Essential to plan for long term preservation

Project Lifecycle Events

General requirements: Data Preservation General requirements: Preserve bits Ensure: Discoverability and accessibility Readability Understandability Usability Reproducibility of results

Bits Checksums while transferring between subsystems Regular media migration Raw (Level 0) data from satellites held at back-up archive physically distant from DAACs Product generation software held at the DAACs and SIPSs Guards against catastrophic loss Raw data and higher level products are backed-up at the DAACs as well – for efficiency Periodic reassessment of risk of data loss and impact on users given the back-up approach being used at the DAACs

Discoverability and accessibility Standard metadata are critical for discoverability of data Processing software automatically generates metadata at individual file level Metadata repository is constantly populated Common Metadata Repository (CMR) Unites collection level and file level metadata Provides a source of unified, high-quality and reliable Earth Science metadata across NASA’s Earth science data holdings Improvements to Accessibility Migration from the near-line robotic tape libraries into on-line disk archives during 2005-2008. Assignment of Digital Object Identifiers (in progress)

Readability Hierarchical Data Format (HDF) Primary format for most EOSDIS datasets Translations into other formats such as NetCDF, GeoTIFF and binary, upon request by users HDF is a self-documenting formatting system Flexible structure for data producer to define “profile” HDF library facilitates writing and reading Need to maintain library for future users Alternative - HDF Archive Mapper Promoting structural consistency among datasets Cross-instrument team agreement (EOS Aura – 2004) HDF Product Designer to enable design of interoperable products compliant with community conventions, and to share designs across teams

Understandability, Usability & Reproducibility Algorithm Theoretical Basis Documents (ATBDs) Product information pages, guides, answers to frequently asked questions (FAQs), forums Usability Information on fitness for purpose Accuracy assessments, validation and data quality documentation Reproducibility Source code and/or software specification documents Versions of datasets or the means of regenerating them when they result in peer-reviewed publications

Preservation Content Specification (PCS) Has been in effect since November 2011; latest version dated January 2013 Covers eight categories of content plus a checklist (see next page) Rigor of application varies among completed, on-going and future missions Completed missions – requirements had not been in place; some items may no longer be available for preservation; responsible individuals may not be accessible On-going missions - requirements had not been in place; some of the relevant data and documentation generated early in mission may not be easily available; need additional work to reach responsible individuals Future missions – requirements are in place; included as part of mission planning

Preservation Content Categories Preflight/Pre-Operations: Instrument/Sensor characteristics including pre-flight/pre-operations performance measurements; calibration method; radiometric and spectral response; noise characteristics; detector offsets Science Data Products: Raw instrument data, Level 0 through Level 4 data products and associated metadata Science Data Product Documentation: Structure and format with definitions of all parameters and metadata fields; algorithm theoretical basis; processing history and product version history; quality assessment information Mission Data Calibration: Instrument/sensor calibration method (in operation) and data; calibration software used to generate lookup tables; instrument and platform events and maneuvers Science Data Product Software: Product generation software and software documentation Science Data Product Algorithm Input: Any ancillary data or other data sets used in generation or calibration of the data or derived product; ancillary data description and documentation Science Data Product Validation: Records, publications and data sets Science Data Software Tools: product access (reader) tools. Checklist: “metadata” about the above 8 categories showing how and where items in each category are preserved

Sources of Preservation Content Calibration Team Mission logs Science Data Product Documentation Mission Data Calibration Mission Operations Product Generation Support Teams (SIPSs) Instrument Teams / PI’s Science Data Software Tools Science Data Product Software Science Data Products Level 0 Data Science Data Product Algorithm Input Ancillary data sources (e.g., NOAA) Different entities hold preservation content during the life of a project, but they need to be gathered for long-term preservation. Some items are part of regular flow during active operational phase. Others need extra effort to collect. DAACs Science Data Product Validation Preflight/ Pre-Operations Instrument Developer/ Manufacturer Data gathering project (e.g., flight project) Validation Team

Examples for Scale of Effort Thousands of items had to be reviewed for deciding what had to be preserved Category Number of Items (HIRDLS) Number of Items (GLAS/ ICESat) Preflight/Pre-Operations Calibration 168 23 Product Documentation 18 34 Mission Calibration 10 12 Science Data Product Software 26* 5 Science Data Product Algorithm Inputs 1 56 Science Data Product Validation 1** 3 Science Data Software Tools 20 Total 225 153 *Includes source code and documentation **List of published papers

Standard for Preservation Content NASA would like to see a broad international standard identifying preservation content – NASA’s PCS is a good starting point, as are ESA’s Long Term Data Preservation documents ISO Technical Committee on Geographic Information/ Geomatics (TC 211) is working on a standard (ISO 19165)

Attention to preservation is needed throughout lifecycle Summary NASA has been collecting Earth observation data from many sources for over 50 years Data and derived scientific products are a valuable asset requiring stewardship and preservation Attention to preservation is needed throughout lifecycle Waiting for closeout phase of projects is too late Preservation Content Specification helps with planning ahead We would like to see an international standard on preservation content – looking for collaboration