AMSR-E SIPS Processing Status Presented by Helen Conover Information Technology and Systems Center at the University of Alabama in Huntsville AMSR-E Joint Science Team Meeting Sept 11-12, 2012 Portland, OR ghrc.nsstc.nasa.gov www.itsc.uah.edu ghrc.nsstc.nasa.gov

Outline Introduction Sea Ice and Sea Ice Drift Updates Software Updates for Reprocessing – Streamlined Processing Automation – Browse Image Updates – Provenance Collection End of Mission Plan – Reprocessing Plan – Data Preservation Requirements 2 AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

AMSR-E SIPS Personnel 3 NASA MSFC: H. Michael Goodman, PI UAHuntsville ITSC (Staff shared with GHRC DAAC): Management: Dr. Sara Graves, Director, ITSC Development:Helen Conover, Lead Bruce Beaumont Mary Nair Engineering: Kathryn Regner Sandra Harper Data Management:Lamar Hawkins Sherry Harrison Systems Administration:Will Ellett, Lead Michele Garrett AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

AMSR-E SIPS Processing and Dependencies 4 Level-2A Brightness Temperatures (with Metadata and QA files) L2B Ocean (1/2-orbit) L2B Land (1/2-orbit) L2B Rain (1/2-orbit) L3 Ocean (Daily) L3 Land (Daily) L3 Snow (Daily) L3 Sea Ice (Daily, 3 Res) L3 Snow (Pentad) L3 Ocean (Weekly) L3 Ocean (Monthly) L3 Rain (Monthly) L3 Snow (Monthly) Once per day Once every five days Once per week Once per month Product Generation at SIPS-GHRC ~ 29 times per day Level-2A Brightness Temperatures L3 Sea Ice Drift ~ 29 times per day at SIPS-RSS AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

Algorithm DAP Version Summary 5 Algorithm Short NameCurrent Version L2A TbV11 L2B LandV10 L2B OceanV07 L2B RainV11 L3 LandV07 L3 OceanV05 L3 RainV07 L3 Sea IceV14 L3 Sea Ice DriftB03 L3 SnowV10 AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

Sea Ice and Sea Ice Drift Reprocessing Sea Ice V14 from beginning of mission with new land mask. Processing Sea Ice Drift from beginning of mission with B03 algorithm, using Sea Ice V14 as input Full mission processing of all Sea Ice products should be complete Spring AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

SIPS Software Updates In place for SI/SID reprocessing: Refactored full suite of processing automation software to streamline system for reprocessing Updated browse image process – New 24-bit PNG images with embedded metadata (e.g., source data, parameter, date/time, projection) – KMZ files suitable for display on map or globe – Traditional “framed” image with title, color bar and annotations To be in place for L2A and subsequent reprocessing: Incorporating provenance capture utilities ­Provenance logging added to processing automation; ready for integration and test ­Lineage metadata and DOIs to be included in data files; design complete, development in progress AMSR-E Joint Science Team Meeting Portland, OR Sept ,

AMSR-E Approach to Provenance / Lineage Why worry about data provenance and lineage? – Important to understand how a data product was generated – Data Preservation Requirements from ESDIS Specification document from Rama and John Moses includes algorithm information, processing history, input data AMSR-E approach: Incorporating results from NASA- sponsored provenance research into production – ISO-compliant lineage information to be included in data files Incorporating lineage information into the data files as additional metadata means that information stays with the files Current push toward ISO-compliant metadata for NASA ES data – Provenance Browser will provide easy access to Browse imagery Basic algorithm information with pointers to full documentation Data lineage and processing history AMSR-E Joint Science Team Meeting Portland, OR Sept ,

AMSR-E Provenance Browser Modes of Exploration Browse list by product type, file name – Mid-winter Sea Ice data was incorrect after reprocessing. Look at January data to determine what was different about this processing run. Browse images by product type, parameter/variable – Scan series of images to look for possible problems, e.g., incomplete data for Search by file name – Question about a particular data file that doesn’t look right. Search by file name; may discover that this was a problem file later replaced, and get new file. View general processing information – For information about how files from a particular product are generated, view general processing graph and get pointers to more detailed documentation. AMSR-E Joint Science Team Meeting Portland, OR Sept ,

AMSR-E Provenance Browser AMSR-E Joint Science Team Meeting Portland, OR Sept ,

AMSR-E Provenance Browser (2) AMSR-E Joint Science Team Meeting Portland, OR Sept ,

AMSR-E Provenance Browser (3) AMSR-E Joint Science Team Meeting Portland, OR Sept ,

Outline Introduction Sea Ice and Sea Ice Drift Updates Software Updates for Reprocessing – Streamlined Processing Automation – Browse Image Updates – Provenance Collection  End of Mission Plan – Reprocessing Plan – Data Preservation Requirements 13 AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

SIPS Reprocessing Planning 14 AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

Data Preservation Requirements Data Preservation Goals – No loss (or loss within scientifically acceptable bounds) as bits move across systems as well as over time – Readability over time – Long-term understandability – Repeatability of previously obtained results Without a standard, missions may not fully plan for preservation of required content –Content to be preserved is held by various organizations during mission’s life. They must provide essential information for archiving before end of their involvement. –Diverse types of supporting information and data may be well understood by originator’s community, but gaps in a- priori knowledge are difficult or impossible to fill once the originator has moved on out-eosdis/requirements AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

Categories of Content to be Preserved 1.Preflight/Pre-Operations: Instrument/Sensor characteristics including pre- flight/pre-operations performance measurements; calibration method; radiometric and spectral response; noise characteristics; detector offsets 2.Science Data Products: Raw instrument data, Level 0 through Level 4 data products and associated metadata 3.Science Data Product Documentation: Structure and format with definitions of all parameters and metadata fields; algorithm theoretical basis; processing history and product version history; quality assessment info 4.Mission Data Calibration: Instrument/sensor calibration method (in operation) and data; calibration software used to generate lookup tables; instrument and platform events and maneuvers 5.Science Data Product Software: Product generation software and software documentation 6.Science Data Product Algorithm Input: Any ancillary data or other data sets used in generation or calibration of the data or derived product; ancillary data description and documentation 7.Science Data Product Validation: Records, publications and data sets 8.Science Data Software Tools: product access (reader) tools. WHAT ELSE IS IMPORTANT FOR LONG TERM PRESERVATION OF AMSR-E DATA???

Data Preservation Checklist 17 Preflight / Pre-Operations Calibration Instrument DescriptionHigh-level description on NSIDC web site More details available? Preflight / Pre-Ops Calibration Data Science Data Products Raw and Derived ProductsAt NSIDC DAAC: L0, L1 (JAXA), L2 & L3 (US) products MetadataAt NSIDC DAAC Mission Calibration Data* Calibration Method Calibration Data * Documentation of in-line changes to calibration or to instrument or platform operations or conditions that occur throughout the mission. AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

Data Preservation Checklist (2) 18 Science Data Product Documentation Product TeamIncluded on NSIDC product pages Product Requirements and Designs Original ATBD? Processing and Algorithm Version History NSIDC “versions” web page Provenance information embedded in files at final reprocessing Product Generation Algorithm Documentation Science software Updated ATBD (all products?) More detail needed? Product Quality Documentation NSIDC product web pages Data quality flags Product Application Published articles about use of data, user feedback AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

Data Preservation Checklist (3) 19 Other Data, Software and Documentation Science Data Product Software and Documentation Delivered Algorithm Package Science Data Product Algorithm Inputs Ancillary data included in DAP All inputs (AMSR-E and ancillary) referenced in provenance metadata Science Data Product Validation: Datasets and Documentation Validation data and metadata at NSIDC Cal/val plans Detailed history of activities List of published papers Science Data Software ToolsHDF-EOS and related tools Deliverables Checklist This is a start AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

Summary Ready to begin full mission reprocessing with – Same or new science algorithms – Updated browse imagery – ISO Lineage metadata and DOIs included in data files AMSR-E SIPS will be one of the first to implement ESDIS Data Preservation requirements as part of end of mission activities 20 AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

Backup

TLSCF Science Software I&T SIPS-GHCC PO.DAAC JAXA-EOC L1A Product Generation SIPS-RSS L2A Product Generation L2B and L3 Product Generation (science, metadata, qa, ph, subsets, browse) L2 and L3 algorithms L2A algorithm NSIDC DAAC Data Archive and Distribution Level-1 Level-2 and Level-3 Level-1 NASA EDOS GBAD (Pre Ephem) & PDS Ground Stations L2B and L3 Algorithms Level-0 Product Delivery Record Server (PDRS) Algorithm Teams OceanSea Ice Rain Land Snow Data NASA Flt Dynamics GBAD (Def Ephem) AMSR-E Data Flow 22 AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012

SIPS-GHCC Hardware Operational Configuration (Dec ‘10) Near line storage of all Level-2A and most recent Level-2B and Level-3 Restricted FTP Server Multiple processing environments across three machines results in much greater flexibility for processing and reprocessing. Sun Storage 7310 With J4400 Disk Array 10TB NAS Private Network (1 GB/sec) Forward Processing Reprocessing Dev, Integration & Test Sun Fire x4270 x64 2 x 2.93 GHz Sun Intel Xeon Processors 4 x 2GB DIMM, RHEL 5 AMSR AMSR-E Joint Science Team Meeting Portland, OR Sept , 2012