Preservation of NASA’s Earth Observation Data EOSDIS Science Operations, ESDIS Project Code 423 Goddard Space Flight Center, Greenbelt, MD ESIP Federation Meeting, Santa Barbara, CA. July 9, 2009

EOSDIS and NASA’s Earth Science Preservation Activities NASA’s Earth Science Data and Information System Project will maintain distributed active archives to continue servicing the science and applications communities. –responsibilities for data management include, yet go beyond, the life of EOS missions and continue with missions recommended by Decadal Survey. –keep all data important to active science research and applications in place at the EOSDIS Data Centers. 11/10/20152

3 Preservation of Earth Observation Data NASA has observational data that will need long term preservation Selected data sets will be organized for the National Archives and Records Retention (NARA) based on mission end of life and other critical data needs. Our data life cycle ends with a consolidated and organized permanent record which specifically means instrument data co-located with the product generation source code, allowing for regenerated products when necessary, documentation, submitted and accepted to NARA.

Organization A number of events need to occur with each instrument for a successful archive strategy, but in general: –ESDIS will coordinate joint long term archive activities with each EOS Instrument team through appropriate working groups. –The working group will agree on source of information, responsibility for inventory and content, standards to follow and any corresponding preparation tasks. –ESDIS will align the long term archive plan with current NASA’s records retention schedule rules. –The entire archive process will be a transparent activity to all involved, including the science community. –The working group will stay active through the entire archive process to ensure goals are met. 11/10/20154

5 Characterizing EOS Electronic Records There are two primary types of EOS electronic records and 4 product categories –Science data needed for future national climate research and climate change programs and projects – not reproducible, clear need for long term preservation Instrument data and information required for re-calibration to improve historical products Observational data (e.g., EOS Level 0 or 1) needed for long term climate records; the basis for establishing operational climate monitoring programs –Earth observations of national policy significance Products (e.g., EOS Level 2 and 3) used by researchers whose results are the basis for the formulation of national policies and continue to be used to determine effectiveness of national regulations and international treaties –E.g. ozone monitoring, ice monitoring Research products (e.g., EOS Level 2 and 3) used within operational monitoring programs for the public good –E.g., weather satellites, land use, land cover, air quality EOS Data Product categories: Level 0 - Unprocessed instrument/payload data at full resolution. Level 1 - Instrument data with radiometric and geometric calibration and geo-references Level 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data. Level 3 - Variables mapped on uniform space-time grid scales

Science Mission Drivers EP-TOMS and ICESat 1 where chosen as the first data sets to enter the long term archive process for the reasons below: EP-TOMS –Data flow discontinued in Dec Currently near the end of the project, should be forming or have identified final records. –Team members should still be available to support a joint working group (OMI is the continuity mission – different team with some overlap in membership). –A critical national historical dataset of ozone records for NASA in support of Clean Air Act Amendments of ICESat 1 –Instrument may fail soon and we may suddenly need to finalize records. –The instrument is a reasonable representation of the kind of information that needs to be organized for the archive, and the learning curve faced by most other flight and science projects concerning use of preservation standards. 11/10/20156

The value of the OAIS Ref. Model A common language that enables coordinated planning and implementation by partnering agencies and institutions for permanent records retention –NOAA, USGS, NARA A structure for linking representation information with the data –Example – MODIS LTA Handbook –Provides a logical structure, avoiding having to rethink what goes into an AIP 11/10/20157

Experience using OAIS RM Several of the data centers have used the OAIS Reference Model when evolving, updating or re-designing their archive –NASA Langley (LARC) Atmospheric Sciences Data Center data system evolution –PO.DAAC (JPL) using OAIS Reference Model, ISO as design constraint –SEDAC and Columbia University using OAIS RM in implementation of COTS packages Fedora and VITAL 11/10/20158

Atmospheric Sciences Data Center Archive, Next Generation (or ANGē) ANGē’s design is based on the Open Archival Information System (OAIS) Reference Model. ANGē development is organized by each of the primary OAIS model component. One of primary goals of ANGē’s design is to minimize the changes required for the interfaces to any current data providers. An additional translation layer will be developed to allow current interface protocols to interact with ANGē’s OAIS compliant services. However, current data providers may elect to modify existing interfaces to take advance of better services (e.g. automated notification of data receipt). The diagram depicts this approach using nomenclature from the OAIS reference model. 11/10/20159

Model for Web-Based Data Submission and Workflow Data Producer Provide Files And Descriptions Data Deposit Describe Data Set Data Description Grant Intellectual Property Rights Submission Agreement Appraise And Approve Submission Information Package Review Transform Values to XML Metadata Transformation Archival Information Package In Digital Repository Ingest Login with ID & PasswordAuthentication One or More Submission Sessions Notifications and Requests Communication Source: Downs and Chen. Creating a Trustworthy Digital Repository for a Long-Term Archive of Interdisciplinary Data: A Case Study. 21st International CODATA Conference, 5-8 October, 2008 Kyiv, Ukraine. Socioeconomic Data Center SEDAC

backup 11/10/201511

11/10/ EOS Data and Information System Earth Observing System (EOS) Data and Information System (EOSDIS) was conceived and built in the 1990’s as the spacecraft control, data acquisition, processing system for the EOS flight missions and the archive and distribution system for all NASA Earth science data EOSDIS consists of several Distributed Active Archive Centers (DAACs), and Science Investigator-led Processing Systems (SIPSs), and a Core System –DAACs provide processing, archive and distribution services for EOS data. Each DAAC holds different kinds of scientific data that includes atmospheric, oceanic, land use/cover, space geodesy, cryospheric, and biospheric. The DAACs make EOS data publically available to all users, through online query, metadata, and retrieval systems directly to the user community. –SIPSs produce standard products from EOS instrument data and send them to the DAACs for archiving –Core System consists of components used to support major data flows at several DAACs and SIPSs

11/10/ EOS electronic records life cycle Mission based Earth observing research programs –Many instruments and levels (maturity) of data –Many different activity timelines, retention is event driven Typical period for operating instrument (~3-15 yrs) –Nominally the design life plus optimal performance period –Ongoing data acquisition, product generation, archive & distribution –Ongoing science research and applications for societal benefit –Algorithm adjustments and reprocessing from beginning of mission can occur –Provides inputs to next research & operational satellite programs Mission End-of-Life period, Decommissioning (~3-5 yrs) –Final reprocessing for products from the entire mission –Final research results, inputs to next missions –Organization of mission data for permanent long term retention Permanent electronic records are created during both periods

11/10/ Earth Science Missions

EOSDIS Manages Data For all EOS Measurements 11/10/201515

11/10/ Tracking & Data Relay Satellite (TDRS) Instrument Teams and Science Investigator-led Processing System (SIPS) EOS Spacecraft Direct Broadcast/ Direct Readout Stations Direct Broadcast (DB) EOS Data and Operations System (EDOS) Data Processing White Sands Complex (WSC) EOS Polar Ground Stations EOSDIS and Related Data Centers Internet (Search, order, distribution) Research Education Value-Added Providers Interagency Data Centers International Partners Earth System Models Decision Support Systems Distribution and Data Access Data Acquisition Flight Operations, Data Capture, Initial Processing, Backup Archive Data Transport to Data Centers/ SIPSs Science Data Processing, Data Mgmt, Interoperable Data Archive, & Distribution NASA Integrated Services Network (NISN) Mission Services Mission OperationsScience Operations EOS Operations Center (EOC) Mission Control