AMSR-E SIPS Processing Status Kathryn Regner Information Technology and Systems Center at the University of Alabama in Huntsville www.itsc.uah.edu Image provided by Matt Smith

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Outline What’s New at the SIPS Data Flow Review New System Configuration Update Processing Status –Near real time –Reprocessing –AMSR Midori Field Campaign Support Important Reminders

AMSR-E Joint Science Team Meeting Telluride, CO July 14, SIPS Highlights All processing (near real time and reprocessing) transitioned from ARIEL to AMSR cluster. –If you currently pull data from Ariel, please verify login to Reprocessing all products to ESDT Version 2 –product maturity codes are transitional or validated

AMSR-E Data Flow TLSCF Science Software I&T SIPS-GHCC PO.DAAC JAXA-EOC L1A Product Generation SIPS-RSS L2A Product Generation L2B and L3 Product Generation (science, metadata, qa, ph, subsets, browse) L2 and L3 algorithms L2A algorithm NSIDC DAAC Data Archive and Distribution Level-1 Level-2 and Level-3 Level-1 NASA EDOS RBD & PDS Ground Stations L2B and L3 Algorithms Level-0 Product Delivery Record Server (PDRS) Algorithm Teams OceanSea Ice Rain Land Snow Data NASA FDS GBAD

AMSR-E Joint Science Team Meeting Telluride, CO July 14, New System Status (1) Dell hardware configuration is complete –Storage Area Network (SAN) sized to hold all Level-2A products from beginning of mission through 2010 –evaluating procurement options for next increment –Cluster Suite and Global File System enable much greater flexibility for processing streams of data.

AMSR-E Joint Science Team Meeting Telluride, CO July 14, New System Status (2) Software configuration –Recently upgraded operating system to Red Hat Enterprise Linux 5; minor software adjustments were required –All processing automation scripts were ported to Linux and tested with science software –QA files being sent to web server on “weather” –All algorithms are ESDT Version Collection 2

AMSR-E Joint Science Team Meeting Telluride, CO July 14, SIPS-GHCC Hardware Operational Configuration 8 TB Dell EMC CX300 Storage Area Network Array Forward Processing “late” processing Reprocessing Development, Integration & Test 2 GB/sec Fibre Channel Switch Dell PowerEdge x 3.4 GHz Processors 4 GB RAM, RHEL 5 AMSR 1-4 Near line storage of all Level-2A and most recent Level-2B & Level-3 Restricted FTP Server NASA Network (100 Mb/sec) Multiple processing environments across four machines results in much greater flexibility for processing and reprocessing.

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Software Architecture (Routine Processing ) Level-2A Brightness Temperatures (with Metadata and QA files) L2B Ocean (1/2-orbit) L2B Land (1/2-orbit) L2B Rain (1/2-orbit) L3 Ocean (Daily) L3 Land (Daily) L3 Snow (Daily) L3 Sea Ice (Daily, 3 Res) L3 Snow (Pentad) L3 Ocean (Weekly) L3 Ocean (Monthly) L3 Rain (Monthly) L3 Snow (Monthly) Once per day Once every five days Once per week Once per month Product Generation Scheduling ~ 29 times per day Level-2A Brightness Temperatures Near Real Time Ingest from RSS

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Algorithm Versions and Product Maturity Codes Algorithm Short Name L2A Tb L2B Land L2B Ocean L2B Rain L3 Land L3 Ocean L3 Rain L3 Sea Ice L3 Snow Current Versions (July 2008) V09 T08 V06 V10 T05 V03 V06 T08 Product Maturity Codes T=transitional V=validated

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Forward Processing Routine forward processing is running very smoothly –automated to run 24 x 7, unattended The L2A files typically arrive at GHCC hours after observation Routinely dealing with the occasional straggler or replacement Level-2A files, which requires –regeneration of the composite products, –replacement files being sent to the DAAC for archive

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Reprocessing Status Reprocessing –standard Level-2B products began in March, using validated (V09) Level-2A files generated at RSS and ported versions of Level-2B algorithms Achieving rates of 11-12x –standard Level-3 products (land, ocean, rain and snow) began in May using validated (V09) Level-2A files generated at RSS, reprocessed Level-2B and ported versions of Level-3 algorithms Achieving rates of ~25x –Standard Level-3 Sea Ice began last week (algorithm updated to remove sea ice temperature parameter) Achieving rates >25x The reprocessed data is available to the science and validation teams on amsr1 and for public distribution at the NSIDC DAAC.

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Reprocessing Timeline 12/07 01/08 02/08 03/08 04/08 05/08 06/08 07/08 08/08 09/08 10/08 11/08 L2B products with L2A V ‘03‘02 L3 Land, Ocean, Rain, Snow products with L2A V09 and L2B ‘04 L3 Sea Ice products with L2A V09 ‘02‘04 L2B products with L2A V Available at the DAAC In progress at the SIPS

AMSR-E Joint Science Team Meeting Telluride, CO July 14, AMSR-Midori Transferred the 7 months of AMSR-Midori Level-2A products from RSS to GHCC in Jan 08 –Transferred a few new test files last month for TLSCF metadata/qa development and science software integration and test –SIPS will begin transferring full dataset to replace previous version in the near future

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Field Campaign Support Near Real Time AMSR-E data requested to assist with flight planning during three field campaigns this year –Snow depth product for ARCTAS (Arctic Research of the Composition of the Troposphere from Aircraft and Satellites) spring and summer –working with sea ice team to supply data in support of an Arctic experiment in August. “The near real-time AMSR-E data helped us immensely with our flight planning during our last field campaign in 2006.” - Don Cavalieri Real Time Mission Monitor - interactive field experiment asset management and data visualization tool

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Important Reminders Products Retention Plan Original Hardware Ariel to be decommissioned soon Teamaqua Login Information

AMSR-E Joint Science Team Meeting Telluride, CO July 14, SIPS-GHCC Products Retention Plan All latest version Level-2A files are kept on the Storage Area Network to facilitate reprocessing. Approximately six weeks of the latest Level-2B and Level-3 daily science files are kept on line –for science team QC and –to facilitate product regeneration due to late arriving L2A files The 5-day, weekly and monthly science files are kept on line for approximately 6 months

AMSR-E Joint Science Team Meeting Telluride, CO July 14, Used for:  Routine processing  Late L2A processing Used for:  Reprocessing  Special Processing  Development  Integration & test On line storage of most recent six weeks of Level-2 and Level-3 daily products for - transfer to DAAC - science team quality control 2 TB RAID file system Restricted FTP Server SGI Origin x 250Mhz Processors 2GB RAM sgi 2100 Ariel SGI Origin x 350Mhz Processors 2GB RAM sgi 2100 Melody Fibre Channel Interface Network File System Interface Original SIPS-GHCC Hardware Soon to be Decommissioned

AMSR-E Joint Science Team Meeting Telluride, CO July 14, teamaqua Login Update sftp password: same as before (call me if you need it) Directory structure is a little different: cd ops/science/data (for near real time) cd repro/science/data (for reprocessed)