1. Earth Science Data and Information System (ESDIS) Project Update
EOSDIS System Manager
AMSR Science Team Meeting Sept 23-24, 2014 Huntsville, AL

2. ESDIS Project

3. EOSDIS Facilities
Data centers, collocated with centers of science discipline expertise, archive and distribute standard data products produced by Science Investigator-led Processing Systems (SIPSs)
ASF DAAC
SAR Products
Sea Ice, Polar
Processes
SEDAC
Human Interactions
in Global Change
CDDIS
Crustal Dynamics
Solid Earth
NCAR, U of Col.
HIRDLS, MOPITT, SORCE
LP DAAC
Land Processes
& Features
GSFC
GLAS, MODIS, OMI, OBPG
GES DISC
Atmos Composition &
Dynamics, Global
Modeling, Hydrology,
Radiance
Key
NSIDC DAAC
Cryosphere, Polar
Processes
OBPG
Ocean Biology &
Biogeochemistry
Data
Center
JPL
MLS, TES
ORNL DAAC
Biogeochemical
Dynamics, EOS Land
Validation
LAADS/
MODAPS
Atmosphere
SIPSs
LaRC
CERES, SAGE III
San Diego
ACRIM
ECS
Sites
PO.DAAC
Ocean Circulation
Air-Sea Interactions
GHRC
Hydrological Cycle &
Severe Weather
ASDC
Radiation Budget,
Clouds, Aerosols,
Tropo Chemistry
GHRC
AMSR-E, LIS,
AMSR2

4. EOSDIS Key Metrics EOSDIS Metrics FY2013
(Oct 1, 2012 to Sept 30, 2013)
Unique Data Products
6,861
Distinct Users of EOSDIS Data and Services
1.7 M
Average Daily Archive Growth
8.5 TB/day
Total Archive Volume
9.8 PB
End User Distribution Products
839 M
End User Average Daily Distribution Volume
22 TB/day
For the Discipline pie chart
Others:
Cryosphere
Earth Radiation
Socio-Economic
Others (not identified)
EOSDIS Science Data Volume Growth
EOSDIS Science Data Products Distribution FY2000 through FY2013

5. REVERB - Data and Service Access Client
Primary web-based client for discovering and ordering cross-discipline data from all of EOS Clearing House (ECHO) metadata holdings.
Allows users, including those without specific knowledge of the data, to search science data holdings, retrieve high-level descriptions of data sets, view browse images, and submit orders via ECHO to appropriate data providers.
Users can submit queries using spatial and temporal criteria and examine search results for relevance using built-in tools.

6. REVERB - Data and Service Access Client
Free
Text
Search
GCMD
Science
Keywords
The “Free Text Search” searches all of the metadata sent to ECHO by data providers.
The GCMD and ECHO repositories will eventually merge to form the Common Metadata Repository (CMR), a single, shared metadata repository for EOSDIS collection and granule level metadata.

7. LANCE – A Component of EOSDIS
The Land Atmosphere Near-real time Capability for EOS (LANCE) is a component of EOSDIS that generates and distributes near real-time (NRT) products from the following instruments:
Terra: MODIS
Aqua: MODIS, AIRS
Aura: OMI, MLS
AMSR-E (Aqua) – ended 2011
AMSR2 (GCOM-W1) – future
MISR (Terra) – being added
LANCE Objectives:
To leverage science processing expertise to create high quality NRT products
To provide data to applications community with a maximum latency of 3 hours from observation (The standard, science-quality products are typically available with a latency of hours)
To provide data products with high reliability using redundant systems
To provide an umbrella environment with uniform high level requirements to better serve user needs (Consistency, Collaboration, Coordination)

8. LANCE Metrics 2014

9. ESDIS Data Discovery Capabilities
Worldview (
Supports time-critical application areas such as wildfire management, air quality measurements, and weather forecasting.
Allows users to interactively browse satellite imagery in near real-time, add layers, and also provides a time series for comparison.
GIBS (Global Imagery Browse Services)
GIBS are a set of standard services to deliver global, full-resolution browse satellite imagery in a highly responsive manner
Full Resolution mosaicked layers that are chopped into imagery tiles stored at predefined zoom levels
Worldview/GIBS currently consists of:
Routine NRT production of: MODIS (C5), AIRS, MLS, OMI, Ocean Color
AMSR-E historical data in process of being added
Just added 5 Cryospheric Layers in Arctic, Antarctic and Geographic projections from
: Sea Ice Brightness Temperatures (89H GHz, 89V GHz), Sea Ice Concentration (12km,25km), Snow Depth over Ice (12km)
GIBS/Worldview upcoming plans consist of adding:
Routine NRT production of GPM from PPS
Standard SMAP products
MODIS Collection 6 (Forward + Reprocessed)

10. Continuing Browse Enhancements
GIBS and Worldview
Data granules, used to create the imagery are also searchable and can be downloaded through the Worldview interface. 
Delivery via Standard Services OGC WMTS / TWMS / WMS / KML
Clients can be built to use and display images in GIBS – WorldView is an example

11. All GIBS Product Downloads vs LANCE NRT

12. File Transfer Protocol Testing
Compared file transfer protocol performance across different network environments in ESDIS Network lab and to Alaska
FTP, FTPS, HTTP, SCP, SFTP, HTTPS, bbFTP, GridFTP
Varied file size, packet loss rate, round-trip time to emulate different network environments
Test findings:
Multiple stream file transfer protocols perform best sending large files over networks with high packet loss and long round-trip times (WAN)
bbFTP, GridFTP
Single stream file transfer protocols perform best sending small files over networks with short round-trip times (LAN)
FTP, FTPS, HTTP
Protocols that encrypt the data are consistently the slowest. For best performance, data encryption should be avoided unless it is a requirement.
SCP, SFTP, HTTPS

13. Future File Transfer Protocol Work
Continue working with DAACs and EOSDIS partners choose the best performing protocols for file transfers
Some DAACs have started incorporating multi-stream protocols, bbFTP and GridFTP, for sending large files
Test additional protocols, including Aspera and MMC ftp including tests with JAXA

14. Recent and Upcoming Missions
Launch Date
DAAC
SNPP
Oct 28, 2011
Multiple
GPM
Feb 27, 2014
GES DISC
OCO-2
Jul 2, 2014
SMAP
Dec 15, or
Jan 29, 2015
ASF / NSIDC
DSCOVR
Jan 2015
ASDC
SAGE-III on ISS
Jun 30, 2015
OCO-3 on ISS
Aug 1, 2016
CYGNSS (EV-M)
Oct 28, 2016
PO.DAAC
GRACE FO
Jul 8, 2017
ICESat-2
Jul 27, 2017
NSIDC

15. Future Suomi National Polar-orbiting Partnership Plans for NASA SDS
Current architecture
SNPP data is processed by NOAA IDPS
ESDIS manages the Science Data Segment (SDS) of SNPP and was responsible for product evaluation for VIIRS, CrIS, ATMS, OMPS Nadir as well as the production and distribution of CERES, and OMPS Limb
Archive and distribution of SNPP is at NOAA CLASS
Future Architecture
NASA will be producing and distributing it’s own products from all SNPP instruments and potentially future JPSS missions including NRT
Science Investigator-led Processing Systems (SIPS) and Science Teams have recently been selected for the following disciplines: Land, Ocean, Atmosphere, Ozone, Sounder

16. Soil Moisture Active Passive
SMAP Soil Moisture Active Profile
ORR Nov 4-5, 2014
Launch Dec 15, 2014, or January 29, 2015
Downlink to Alaska, McMurdo, Svalbard, Wallops (backup)
15 contacts per day; 135 GB/day
3 year mission timeline
SMAP data will be processed to L0 by EDOS
SMAP science processing will take place at JPL and GSFC/GMAO
Ingest, archive and distribution of SMAP data will be at ASF for Radar data (L1) and NSIDC for Radiometer data (L1-L4)

17. Preservation NASA is not a “permanent archive” agency
Must maintain “research archive” for as long as data are used for scientific research and/or transition responsibility to permanent archives
Research archive responsibilities persist well beyond lives of missions
NASA works with USGS and NARA for long-term preservation
NASA has to ensure data and other critical items are preserved and made available to permanent archival agencies
NASA has developed Preservation Content Specifications for Earth Science Data
Categories of Data to be preserved include:
Preflight/Pre-Operations
Science Data Products
Mission Data Calibration
Science Data Product Software
Science Data Product Algorithm Input
Science Data Product Validation
Science Data Software Tools

18. Acronyms AIRS Atmospheric Infrared Sounder AMSR
Advanced Microwave Scanning Radiometer
ASDC
Atmospheric Science Data Center
ASF
Alaska Satellite Facility
ATMS
Advanced Technology Microwave Sounder
bbFTP
BaBar File Transfer Protocol
CERES
Clouds and the Earth’s Radiant Energy System
CMR
Common Metadata Repository
CrIS
Cross-track Infrared Sounder
CTGNSS
Cyclone Global Navigation Satellite System
DAAC
Distributed Active Archive Center
DSCOVR
Deep Space Climate Observatory
ECHO
EOS Clearing House
EDOS
EOS Data and Operations System
EOS
Earth Observing Center
EOSDIS
Earth Observation System Data and Information System
ESDIS
Earth Science Data and Information System
ESIP
Earth Science Information Partners
FTP
File Transfer Protocol
FTPS
File Transfer Protocol Secure GB
Gigabyte
GCMD
Global Change Master Directory
GCOM
Global Change Observation Mission – 1st Water Satellite in Series
GES DISC
GSFC Earth Sciences Data and Information Services Center
GHRC
Global Hydrology Resource Center
GIBS
Global Imagery Browse Services
GMAO
Global Modeling and Assimilation Office
GPM
Global Precipitation Measurement
GRACE-FO
Gravity Recovery And Climate Experiment – Follow-on
GridFTP
Grid File Transfer Protocol
HTTP
Hypertext Transfer Protocol
ICESat
Ice, Cloud, and Land Elevation Satellite-2
IDPS
Interface Data Processing Segment
ISS
International Space Station
JAXA
Japan Aerospace Exploration Agency
JPL
Jet Propulsion Laboratory
JPSS
Joint Polar Satellite System
KML
Keyhole Markup Language
LAN
Local Area Network
LANCE
Land Atmosphere Near real-time Capability for EOS
MISR
Multi-angle Imaging SpectroRadiometer
MLS
Microwave Limb Sounder
MODIS
Moderate Resolution Imaging Spectrometer
MSFC
Marshall Space Flight Center
NARA
National Archives and Records Administration
NASA
National Aeronautics and Space Administration
NOAA
National Oceanic and Atmospheric Administration
NRT
Near Real-Time
NSIDC
National Snow and Ice Data Center Distributed Active Archive Center
OCO
Global Modeling and Assimilation Office
OGC
Open Geospatial Consortium
OMI
Ozone Monitoring Instrument
OMPS
Ozone Mapper/Profiler Suite
ORR
Operational Readiness Review
PPS
Precipitation Processing System
SAGE
Stratospheric Aerosol Gas Experiment
SCP
Secure Copy
SFTP
Secure Shell File Transfer Protocol
SIPS
Science Investigator-led Processing System
SMAP
Soil Moisture Active Passive
SNPP
Suomi National Polar-orbiting Partnership
TWMS
Tile Web Map Service
USGS
United States Geological Survey
VIIRS
Visible Infrared Imager Radiometer Suite
WAN
Wide Area Network
WMS
Web Map Service
WMTS
Web Map Tile Service