1. AMSR-E SIPS Processing Status
Kathryn Regner
Information Technology and Systems Center at the University of Alabama in Huntsville
Image provided by Matt Smith

2. Outline What’s New at the SIPS Data Flow Review
New System Configuration Update
Processing Status
Near real time
Reprocessing
AMSR Midori
Field Campaign Support
Important Reminders
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

3. SIPS Highlights
All processing (near real time and reprocessing) transitioned from ARIEL to AMSR cluster.
If you currently pull data from Ariel, please verify login to
Reprocessing all products to ESDT Version 2
product maturity codes are transitional or validated
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

4. AMSR-E Data Flow JAXA-EOC NSIDC DAAC PO.DAAC SIPS-RSS TLSCF SIPS-GHCC
NASA EDOS
RBD & PDS
JAXA-EOC
L1A Product Generation
Level-0
Ground
Stations
NASA FDS
GBAD
Level-1
NSIDC DAAC
Data Archive and Distribution
PO.DAAC
Level-1
Level-1
SIPS-RSS
TLSCF
L2A algorithm
Level-2 and Level-3
L2A Product Generation
Science
Software
I&T
RBD & PDS – Rate Buffered Data and Production Datasets
FDS – Flight Dynamics System; generates the definitive orbit data (GBAD, ground based attitude determination)
L2 and L3
algorithms
Product Delivery
Record Server (PDRS)
SIPS-GHCC
L2B and L3 Algorithms
Algorithm
Teams
L2B and L3 Product Generation
(science, metadata, qa, ph, subsets, browse)
Rain
Snow
Data
Ocean
Sea Ice
Land

5. New System Status (1) Dell hardware configuration is complete
Storage Area Network (SAN) sized to hold all Level-2A products from beginning of mission through 2010
evaluating procurement options for next increment
Cluster Suite and Global File System enable much greater flexibility for processing streams of data.
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

6. New System Status (2) Software configuration
Recently upgraded operating system to Red Hat Enterprise Linux 5; minor software adjustments were required
All processing automation scripts were ported to Linux and tested with science software
QA files being sent to web server on “weather”
All algorithms are ESDT Version Collection 2
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

7. SIPS-GHCC Hardware Operational Configuration
8 TB Dell EMC CX300
Storage Area
Network Array
Forward Processing
“late” processing
Reprocessing
Development,
Integration & Test
2 GB/sec Fibre Channel Switch
Dell PowerEdge 2850
2 x 3.4 GHz Processors
4 GB RAM, RHEL 5
AMSR 1-4
Near line storage of all Level-2A and most recent
Level-2B & Level-3
Restricted
FTP Server
NASA Network (100 Mb/sec)
Multiple processing environments across four machines results in much greater flexibility for processing and reprocessing.
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

8. Software Architecture (Routine Processing )
Level-2A Brightness Temperatures
(with Metadata and QA files)
L2B Ocean
(1/2-orbit)
L2B Land
L2B Rain
L3 Ocean
(Daily)
L3 Land
L3 Snow
L3 Sea Ice
(Daily, 3 Res)
(Pentad)
(Weekly)
(Monthly)
L3 Rain
Once
per day
Once every
five days
Once per
week
month
Product
Generation
Scheduling
~ 29 times
Near Real Time
Ingest from RSS
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

9. Algorithm Versions and Product Maturity Codes
Short Name
Current Versions
(July 2008)
L2A Tb
L2B Land
L2B Ocean
L2B Rain
L3 Land
L3 Ocean
L3 Rain
L3 Sea Ice
L3 Snow
V09
T08
V06
V10
T05
V03
Product Maturity Codes
T=transitional V=validated
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

10. Forward Processing Routine forward processing is running very smoothly
automated to run 24 x 7, unattended
The L2A files typically arrive at GHCC hours after observation
Routinely dealing with the occasional straggler or replacement Level-2A files, which requires
regeneration of the composite products,
replacement files being sent to the DAAC for archive
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

11. Reprocessing Status Reprocessing
standard Level-2B products
began in March, using validated (V09) Level-2A files generated at RSS and ported versions of Level-2B algorithms
Achieving rates of 11-12x
standard Level-3 products (land, ocean, rain and snow)
began in May using validated (V09) Level-2A files generated at RSS, reprocessed Level-2B and ported versions of Level-3 algorithms
Achieving rates of ~25x
Standard Level-3 Sea Ice began last week (algorithm updated to remove sea ice temperature parameter)
Achieving rates >25x
The reprocessed data is available to the science and validation teams on amsr1 and for public distribution at the NSIDC DAAC.
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

12. Reprocessing Timeline
12/ / / / / / / / / / / /08
L2B products with L2A V08
2002
2003
2004
L2B products with L2A V09
2002
2003
2004
2005
2006
‘03
‘02
L3 Land, Ocean, Rain, Snow products with L2A V09 and L2B
‘04
Available at the DAAC
In progress at the SIPS
L3 Sea Ice products with L2A V09
‘02
‘04
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

13. AMSR-Midori
Transferred the 7 months of AMSR-Midori Level-2A products from RSS to GHCC in Jan 08
Transferred a few new test files last month for TLSCF metadata/qa development and science software integration and test
SIPS will begin transferring full dataset to replace previous version in the near future
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

14. Field Campaign Support
Real Time Mission Monitor - interactive field experiment asset management and data visualization tool
Near Real Time AMSR-E data requested to assist with flight planning during three field campaigns this year
Snow depth product for ARCTAS (Arctic Research of the Composition of the Troposphere from Aircraft and Satellites) spring and summer
working with sea ice team to supply data in support of an Arctic experiment in August.
From: Don Cavalieri
Sent: Wednesday, May 21, :56 AM
To: Regner, Kathryn
Subject: New Arctic campaign
Hi Kathryn, There will be a new Arctic field campaign this summer to study sea ice/atmosphere interactions. The campaign involves coordinated measurements with both the NASA DC-8 aircraft and the Swedish Icebreaker Oden. I am the scientist responsible for sea ice remote sensing. It would be extremely helpful if I could obtain near real-time AMSR-E images of our study region during the campaign (August 2008). The near real-time AMSR-E data helped us immensely with our flight planning during our last field campaign in Please let me know what I need to do to set this up. Thanks in advance for your help with this.
Best regards, Don
ARCTAS: Arctic Research of the Composition of the Troposphere from Aircraft and Satellites The Arctic is undergoing significant environmental changes related to global climate change. Now, NASA is extensively studying the role of air pollution in this climate-sensitive region as part of the ARCTAS field campaign, the largest airborne experiment ever to do so.
The second phase of the Arctic Research of the Composition of the Troposphere from Aircraft and Satellites – or ARCTAS – mission got under way in earnest 29 June to 12 July as NASA's DC-8, P3 Orion and B200 airborne science laboratories are flying multiple coordinated atmospheric sampling mission through boreal forest fire plumes over north-central Canada (Alberta, Saskatchewan, and Manitoba).
SNOW depth product
“The near real-time AMSR-E data helped us immensely with our flight planning during our last field campaign in 2006.” - Don Cavalieri
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

15. Products Retention Plan Original Hardware
Important Reminders
Products Retention Plan
Original Hardware
Ariel to be decommissioned soon
Teamaqua Login Information
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

16. SIPS-GHCC Products Retention Plan
All latest version Level-2A files are kept on the Storage Area Network to facilitate reprocessing.
Approximately six weeks of the latest Level-2B and Level-3 daily science files are kept on line
for science team QC and
to facilitate product regeneration due to late arriving L2A files
The 5-day, weekly and monthly science files are kept on line for approximately 6 months
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

17. Original SIPS-GHCC Hardware Soon to be Decommissioned
SGI Origin 2100
4 x 350Mhz Processors
2GB RAM
SGI Origin 2100
4 x 250Mhz Processors
2GB RAM
2 TB RAID
file system
sgi 2100
Melody
sgi 2100
Ariel
Network
File
System
Interface
Fibre
Channel
Interface
Restricted
FTP Server
Used for:
Reprocessing
Special Processing
Development
Integration & test
Used for:
Routine processing
Late L2A processing
On line storage of most recent six weeks of Level-2 and
Level-3 daily products for
- transfer to DAAC
- science team quality control
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008

18. teamaqua Login Update sftp teamaqua@amsr1.nsstc.nasa.gov
password: same as before (call me if you need it)
Directory structure is a little different:
cd ops/science/data (for near real time)
cd repro/science/data (for reprocessed)
AMSR-E Joint Science Team Meeting Telluride, CO July 14, 2008