AMSR-E SIPS Processing Status Kathryn Regner Information Technology and Systems Center at the University of Alabama in Huntsville kregner@itsc.uah.edu www.itsc.uah.edu 256.961.7791 Image provided by Matt Smith

Outline What’s New at the SIPS Data Flow Review New System Configuration Update Processing Status Near real time Reprocessing AMSR Midori Field Campaign Support Important Reminders AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

SIPS Highlights All processing (near real time and reprocessing) transitioned from ARIEL to AMSR cluster. If you currently pull data from Ariel, please verify login to teamaqua@amsr1 Reprocessing all products to ESDT Version 2 product maturity codes are transitional or validated AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

AMSR-E Data Flow JAXA-EOC NSIDC DAAC PO.DAAC SIPS-RSS TLSCF SIPS-GHCC NASA EDOS RBD & PDS JAXA-EOC L1A Product Generation Level-0 Ground Stations NASA FDS GBAD Level-1 NSIDC DAAC Data Archive and Distribution PO.DAAC Level-1 Level-1 SIPS-RSS TLSCF L2A algorithm Level-2 and Level-3 L2A Product Generation Science Software I&T RBD & PDS – Rate Buffered Data and Production Datasets FDS – Flight Dynamics System; generates the definitive orbit data (GBAD, ground based attitude determination) L2 and L3 algorithms Product Delivery Record Server (PDRS) SIPS-GHCC L2B and L3 Algorithms Algorithm Teams L2B and L3 Product Generation (science, metadata, qa, ph, subsets, browse) Rain Snow Data Ocean Sea Ice Land

New System Status (1) Dell hardware configuration is complete Storage Area Network (SAN) sized to hold all Level-2A products from beginning of mission through 2010 evaluating procurement options for next increment Cluster Suite and Global File System enable much greater flexibility for processing streams of data. AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

New System Status (2) Software configuration Recently upgraded operating system to Red Hat Enterprise Linux 5; minor software adjustments were required All processing automation scripts were ported to Linux and tested with science software QA files being sent to web server on “weather” All algorithms are ESDT Version Collection 2 AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

SIPS-GHCC Hardware Operational Configuration 8 TB Dell EMC CX300 Storage Area Network Array Forward Processing “late” processing Reprocessing Development, Integration & Test 2 GB/sec Fibre Channel Switch Dell PowerEdge 2850 2 x 3.4 GHz Processors 4 GB RAM, RHEL 5 AMSR 1-4 Near line storage of all Level-2A and most recent Level-2B & Level-3 Restricted FTP Server NASA Network (100 Mb/sec) Multiple processing environments across four machines results in much greater flexibility for processing and reprocessing. AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

Software Architecture (Routine Processing ) Level-2A Brightness Temperatures (with Metadata and QA files) L2B Ocean (1/2-orbit) L2B Land L2B Rain L3 Ocean (Daily) L3 Land L3 Snow L3 Sea Ice (Daily, 3 Res) (Pentad) (Weekly) (Monthly) L3 Rain Once per day Once every five days Once per week month Product Generation Scheduling ------------- ~ 29 times Near Real Time Ingest from RSS AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

Algorithm Versions and Product Maturity Codes Short Name Current Versions (July 2008) L2A Tb L2B Land L2B Ocean L2B Rain L3 Land L3 Ocean L3 Rain L3 Sea Ice L3 Snow V09 T08 V06 V10 T05 V03 Product Maturity Codes T=transitional V=validated AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

Forward Processing Routine forward processing is running very smoothly automated to run 24 x 7, unattended The L2A files typically arrive at GHCC 10-12 hours after observation Routinely dealing with the occasional straggler or replacement Level-2A files, which requires regeneration of the composite products, replacement files being sent to the DAAC for archive AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

Reprocessing Status Reprocessing standard Level-2B products began in March, using validated (V09) Level-2A files generated at RSS and ported versions of Level-2B algorithms Achieving rates of 11-12x standard Level-3 products (land, ocean, rain and snow) began in May using validated (V09) Level-2A files generated at RSS, reprocessed Level-2B and ported versions of Level-3 algorithms Achieving rates of ~25x Standard Level-3 Sea Ice began last week (algorithm updated to remove sea ice temperature parameter) Achieving rates >25x The reprocessed data is available to the science and validation teams on amsr1 and for public distribution at the NSIDC DAAC. AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

Reprocessing Timeline 12/07 01/08 02/08 03/08 04/08 05/08 06/08 07/08 08/08 09/08 10/08 11/08 L2B products with L2A V08 2002 2003 2004 L2B products with L2A V09 2002 2003 2004 2005 2006 ‘03 ‘02 L3 Land, Ocean, Rain, Snow products with L2A V09 and L2B ‘04 Available at the DAAC In progress at the SIPS L3 Sea Ice products with L2A V09 ‘02 ‘04 AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

AMSR-Midori Transferred the 7 months of AMSR-Midori Level-2A products from RSS to GHCC in Jan 08 Transferred a few new test files last month for TLSCF metadata/qa development and science software integration and test SIPS will begin transferring full dataset to replace previous version in the near future AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

Field Campaign Support Real Time Mission Monitor - interactive field experiment asset management and data visualization tool http://rtmm.nsstc.nasa.gov/ Near Real Time AMSR-E data requested to assist with flight planning during three field campaigns this year Snow depth product for ARCTAS (Arctic Research of the Composition of the Troposphere from Aircraft and Satellites) spring and summer http://www.espo.nasa.gov/arctas/ working with sea ice team to supply data in support of an Arctic experiment in August. From: Don Cavalieri [mailto:Donald.J.Cavalieri@nasa.gov] Sent: Wednesday, May 21, 2008 11:56 AM To: Regner, Kathryn Subject: New Arctic campaign Hi Kathryn, There will be a new Arctic field campaign this summer to study sea ice/atmosphere interactions. The campaign involves coordinated measurements with both the NASA DC-8 aircraft and the Swedish Icebreaker Oden. I am the scientist responsible for sea ice remote sensing. It would be extremely helpful if I could obtain near real-time AMSR-E images of our study region during the campaign (August 2008). The near real-time AMSR-E data helped us immensely with our flight planning during our last field campaign in 2006. Please let me know what I need to do to set this up. Thanks in advance for your help with this. Best regards, Don ARCTAS: Arctic Research of the Composition of the Troposphere from Aircraft and Satellites The Arctic is undergoing significant environmental changes related to global climate change. Now, NASA is extensively studying the role of air pollution in this climate-sensitive region as part of the ARCTAS field campaign, the largest airborne experiment ever to do so. The second phase of the Arctic Research of the Composition of the Troposphere from Aircraft and Satellites – or ARCTAS – mission got under way in earnest 29 June to 12 July as NASA's DC-8, P3 Orion and B200 airborne science laboratories are flying multiple coordinated atmospheric sampling mission through boreal forest fire plumes over north-central Canada (Alberta, Saskatchewan, and Manitoba). SNOW depth product “The near real-time AMSR-E data helped us immensely with our flight planning during our last field campaign in 2006.” - Don Cavalieri AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

Products Retention Plan Original Hardware Important Reminders Products Retention Plan Original Hardware Ariel to be decommissioned soon Teamaqua Login Information AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

SIPS-GHCC Products Retention Plan All latest version Level-2A files are kept on the Storage Area Network to facilitate reprocessing. Approximately six weeks of the latest Level-2B and Level-3 daily science files are kept on line for science team QC and to facilitate product regeneration due to late arriving L2A files The 5-day, weekly and monthly science files are kept on line for approximately 6 months AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

Original SIPS-GHCC Hardware Soon to be Decommissioned SGI Origin 2100 4 x 350Mhz Processors 2GB RAM SGI Origin 2100 4 x 250Mhz Processors 2GB RAM 2 TB RAID file system sgi 2100 Melody sgi 2100 Ariel Network File System Interface Fibre Channel Interface Restricted FTP Server Used for: Reprocessing Special Processing Development Integration & test Used for: Routine processing Late L2A processing On line storage of most recent six weeks of Level-2 and Level-3 daily products for - transfer to DAAC - science team quality control AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

teamaqua Login Update sftp teamaqua@amsr1.nsstc.nasa.gov password: same as before (call me if you need it) Directory structure is a little different: cd ops/science/data (for near real time) cd repro/science/data (for reprocessed) AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008