Management of AMSR-E and GLAS Data by NSIDC Melinda Marquis AMSR/AMSR-E Data Management Team Lead GLAS/ICESat Data Management Team Lead National Snow and Ice Data Center PoDAG Meeting April 25 & 26, 2001

Outline Data Flows Data Distribution Services Documentation Post-launch communications Validation Operations Testing Progress & Milestones Subsetting Products tables

AMSR-E Data Flow EDOS L0

GLAS Data Flow

EOS Data Gateway (EDG) User Data Management Subsystem Data Server Subsystem Data Archive (Powderhorn) Science Data Server Database Distribution of data by ECS Search and Order Granules Distributed

Data Distribution Services Data centers provide different levels of service for different products NSIDC distributes data at the same granularity as it is archived 1.We plan to limit distribution of level 0 AMSR-E and GLAS data 2.We intend to offer additional services, such as subsetting

Data Support Services Provided by NSIDC DIFs (submitted to the Global Change Master Directory) Guide Documents (User guides) Web Site Newsletter DAAC Yearbook Outreach

Directory Interchange Formats (DIFs) A DIF is a metadata document Submitted to the NASA’s Global Change Master Directory (GCMD) GCMD is a comprehensive directory of descriptions of data sets relevant to global change research. A DIF is a collection of fields that allow user to decide whether a particular data set would be useful to them. Compatible with U.S. federally mandated Federal Geographic Data Committee’s (FCDC) Content Standard on Digital Geospatial Metadata.

Guide Documents Is a technical document (user’s guide) for data products funded by NASA. Must follow a specific template Template at ( IDE-VALID/GUIDES/TemplateECS2.htm). Contains detailed information about the data product, allowing an uninitiated user to understand and use the data fully and accurately.

Post-Launch Communications between SIPS/SCF/Science Team and NSIDC Communication will continue to be important after launch Either document protocol in Operations Agreement (OA) between NSIDC and SIPS or in Memorandum of Understanding (MOU) between NSIDC and SCF

Post-Launch Communications between SIPS/SCF/Science Team and NSIDC NSIDC’s web site will provide information on: –Availability of products –Status of Aqua and of AMSR-E and of ICESat and of GLAS –Known errors Any changes to processing will need to be conveyed so our documentation can be updated –So we can inform users, including registered users

Post-Launch Communications between SIPS/SCF/Science Team and NSIDC USER NSIDC USO NSIDC IN-HOUSE RESOURCES ProgrammersOpsScience Staff System Engineering/ Admin. OUTSIDE RESOURCES Science Team Others Documentation Specialist Documentation Science Operations Support Staff As needed SIPS

Validation – AMSR-E Richard Armstrong, Jim Maslanik and Julienne Stroeve are actively involved in validation efforts in support of the AMSR-E science team Therefore, they will perform very high level QA of AMSR-E data products

Archive Migration Old drives and media will no longer be supported by Storage Tek. Installation of new drives and media began in April All data, including ASMR-E and GLAS will be archived on the new media, which will be the 9940 tapes from Storage Tek.

PDS (Product Distribution System) Installation completed by the second week in April The PDS system capabilities: –CD-R –8MM tape –DV Automated QC of all media (not data) Automated label printing

Mission Operations and Science Systems (MOSS) Tests – Aqua/AMSR-E Formal test exercise to demonstrate Aqua Science Operations Concurrent Aqua test load with nominal Terra operations

Mission Operations and Science Systems (MOSS) Tests Aqua science systems end-to-end test from EDOS through science data processing facilities (AMSR-E SIPS) to assess readiness for: Demonstration of system performance and concurrent operations ( ingest, archive, processing, user access, and distribution) for Terra, Landsat-7 and Aqua Verify compatibility of AMSR data products with ECS data types Ability to support early post-launch science operations with DAAC/SIPS production using simulated data Demonstrate subscriptions, user pull and distribution (ftp and tape) Demonstrate nominal operations personnel and procedures Identify critical liens, problems, or defects that must be resolved prior to Aqua launch

MOSS 1 Summary Totals Data ProviderData TypeGranules Ingested EDOSAE_PMSCI 21 PM1GBAD1 19 PM1GBAD4 2 PM1GDAB8 2 GHCCAE_L2A 47 AE_Ocean 48 AE_Land 48 AE_Rain 48 AE_RnGd AE_DyOcn AE_WkOcn AE_M0Ocn AE_DySno 2 AE_5DSno 2 AE_MoSno 2 AE_S16 2 AE_SI12 2 AE_S125 2 AE_Land3 2 Browse 46 PO.DAACAMSREL1A 53

MOSS 2 Summary Totals Data ProviderData TypeGranules Ingested EDOSAE_PMSCI 32 PM1GBAD1 33 PM1GBAD4 6 PM1GDAB8 6 GHCCAE_L2A 56 AE_Ocean 56 AE_Land 56 AE_Rain 56 PO.DAACAMSREL1A 56

Future MOSS Tests (Aqua/AMSR-E) MOSS 3 : May 14 - May 18 MOSS 4 : July 16 - July 20 MOSS 5 : September 17 - September 21

GLAS Test Schedule April 24: ICT -- went well. We received PDRs and PDSs for 17 L0 APIDs. All successfully ingested. May 14-16: MOSS 3 (part of Aqua MOSS) July 16-18: MOSS 4 (part of Aqua MOSS) July 2-31: IGST-4 July 10: IGST-6 Sept. 3- Oct. 5: IGST-7

Test Schedule: AMSR-E and GLAS DateAMSR-EGLAS Dec. 4-7, 00 MOSS-1 X Feb , 01 MOSS 2 X May MOSS 3 XX July 2-31 IGST-4 X July 10 IGST-6 X July MOSS 4 XX Sept. 17-Sept. 21 MOSS 5 X Sept. 3- Oct. 5 IGST-7 X Target launch dateNET Dec., 2001 Dec. 15, 2001

Progress – AMSR-E ECS System Upgrades needed for AMSR-E –ECS 5B and 55 Integration and Test Install and check out Release 5B (this was completed by July 2000, i.e., before FY01) Install and check out Release 55 Release 55 has been split into 2 parts: –a) 5B06 (Dec. 2000) –b) IRIX upgrade/SGI (2 nd quarter FY01) 5B07 (March 12 in TS1; March 19 in OPS) oSystem Tests –Perform Regressions Tests on Release 55 (2 nd quarter FY01) –ECS 55 Site Readiness Acceptance (2 nd quarter FY01) –System Confidence Tests (SCTs) (One already during 1 st quarter; maybe one or more in future.) –MOSS (Mission Operations and Science Systems) Tests (First MOSS was in Dec., Second was in Feb., 2001, i.e., 2 nd quarter FY01) Interface Documents –OAs and ICDs are completed, except for those with NASDA.

Milestones – AMSR-E 1 st Quarter (December 31, 00) –1. Installation and Checkout 55: –a. 5B06 (Dec. 2000) –2. System Confidence Test –3. MOSS Test (Dec. 2000) 2 nd Quarter (March 31, 01) 4. IRIX upgraded 5. Second MOSS Test (Feb.) 3 rd Quarter (June 30, 01) –6. DIFs completed –7. Add additional Hardware (getting PDS, another rimage system, RAID) 4 th Quarter (September 30, 01) –8. Guide documents (summaries) completed –1st Quarter (December 31, 01) –9. Operations Readiness Review (Oct. 15) –10. Launch (NET 12/01) Level 0 ingest begins

Milestones -- GLAS 1 st Quarter (by December 31) –1. Write DIFs (drafts) 2 nd Quarter (by March 31) –2. Polish/finish (have reviewed) DIFs 3 rd Quarter (by June 31) –3. ICD b/w ECS at NSIDC DAAC and ISIPS –4. OA b/w NSIDC DAAC and ISIPS –5. OA b/w NSIDC DAAC and EDOS –6. OA or MOU b/w NSIDC DAAC and SCF ? –7. Interface Confidence Tests

Milestones – GLAS (cont.) 4 th Quarter (by September 30) –9. Guide doc – L1 Alt. (GLA01, 05 & 06) –10. Guide doc – L2 Alt. (GLA12-15) –11. Upgrades to ECS for support of ICESat –12. Installation and Checkout –13. Acceptance and Regression Tests –14. Installation and Checkout –15. Regression Tests –16. Interface Tests –17. Mission End-to-End Tests (IGSTs) Launch (Dec. 15, 01) FY02, 1 st Quarter –Guide doc – L1 &L2 Atmos. (GLA02, 7-11) –Guide doc – GLA16

Subsetting ESDIS is working to provide subsetting for data managed in the ECS –MODIS –AMSR-E –GLAS For data in HDF-EOS format, the University of Alabama’s HEW subsetter may be used GLAS data products will be in binary format, with exceptions (higher level products) –NSIDC is pursuing becoming an rSCF –NSIDC hopes to develop subsetter tool unique to GLAS

Backup Slides

AMSR-E Products

Products

GLAS Data Products and Granularity

GLAS Data Granularity and Potential User Issues User A: –GLA12 is 49MB –49MB/day * 183 days = 8,967 MB –Greenland Ice Sheet occupies 15% of GLA12 –0.15*8967MB = 1345MB User B: –GLA16 is 117MB –117MB/day * 183 days = 21,411 MB – Greenland Ice Sheet occupies 5% of GLA16 –0.05*21411MB = 1071MB

GLAS Data Granularity and Data Delivery Issues A CD-ROM holds approx. 650 MB of data, an 8mm Tape holds approx. 5 GB of data. Our typical GLA12 user would require 2 x 8mm tapes or 15 CD-ROMs. Our GLA16 user would require 5 x 8mm tapes or 36 CD-ROMs. If the data were staged to an ftp server, then the user would be required to download large amounts of data, which would require large periods of time. kb/sec, the GLA12 user would take ~44.5 hrs kb/sec, the GLA16 user would take ~106.2 hrs

GLAS Data Granularity and Data Delivery Issues However, if the data were subsetted for the Greenland Ice Sheet alone, then the GLA12 user would require –1x8mm tape, 2 CD-ROMs or ~6.6 hrs time –similarly the GLA16 user would require 1x8mm tape, 2 CD-ROMs or ~5.3 hrs of download time. This reduction in media use will reduce media distribution costs and can encourage use of ftp to deliver the data to the user. Subsetting becomes essential for efficient data distribution and is being addressed.

Spatial search via NOSE EDG ECS Science Data Server (SDSRV) Query Parameters AMSR-E Granule Inventory Spatial Indicies Granule List Spatial Query Server

Implementation of Nominal Orbital Spatial Extent (NOSE) for AMSR-E ECS Science Data Server (SDSRV) Spatial Query Server AMSR-E Granule Inventory AMSR-E NOSE Table AMSR-E DLL AMSR-E pre-defined orbit data

Insertion of AMSR-E granule into ECS ECS Science Data Server (SDSRV) AMSR-E granule metadata Spatial Query Server AMSR-E Granule Inventory Spatial Indicies AMSR-E DLL

Subscriptions Communications Subsystem (CSS) NSIDC User Services Data Server Subsystem (DSS) 2. Event occurs 1. USO enters subscription for user through GUI Ingest of data 3. DSS tells subscription Server about each event 4. Event is recognized as “belonging” to this user 5. and/or Granule push to user

QA at NSIDC Automatic QA: –If can’t ingest -> no good –We plan to exercise media: pull data off randomly to inspect quality of media – still readable (new PDS does this, too) Manual QA: –Inspect media when we distribute data on media (not via ftp) We will support QA updates if provider wishes to utilize that functionality (hope to make it automated)

AMSR-E L1A Data -- availability Proposal: One orbit of data will be available to team members only, three weeks after the AMSR- E instrument is turned on. One (1) day's worth of data will be available to team members only, four weeks after the AMSR-E instrument is turned on. All Level 1A data will be continuously available to team members after 2 months.