1. BEDI -Big Earth Data Initiative
WGISS -37
Cocoa Beach Florida - April 15, 2014
Andrew Mitchell
Earth Science Data and Information Systems (ESDIS)
National Aeronautics and Space Administration (NASA

2. United States Big Data Initiative
U.S. Gent Big Data Initiative
United States Big Data Initiative
President Obama’s “Big Data Research and Development Initiative” seeks to improve our ability to acquire knowledge and discover insights into large and complex collections of digital data. The initiative’s primary objectives are to:
Develop the core technologies needed to collect, archive, manage, analyze, and share large and diverse datasets.
Use these technologies to accelerate the pace of discovery and knowledge, and transform teaching and learning paradigms 2

3. Big Earth Data Initiative (BEDI)
The Big Earth Data Initiative (BEDI) invests in standardizing and optimizing the collection, management and delivery of U.S. Government’s civil Earth observation data. BEDI will improve the ability to discover, access and use Earth observations by the broader user community.
Discover: improve metadata in archives to make datasets more discoverable by popular search engines (e.g., Google or Bing)
Access: Develop better web accessible APIs by improving geospatial coordinate handling and response formats
Use: Improve compatibility with commercial GIS software (e.g., Esri) 3

4. NASA’s Role in BEDI
NASA is to advance these efforts by developing and implementing an agency-wide framework for managing and curating data that will:
maximize the availability of data and information and ensure dissemination in a timey and usable manner;
(2) facilitate the transformation of observations and data into useful information through the use of open, machine readable formats and Application Programming Interfaces (APIs);
(3) encourage the development and use of uniform tools and practices across Feral agencies for the handling of Earth system data and information to increase interoperability; and
(4) support the development of information products and tools that directly support decision-making

5. EOSDIS Approach
Enhancing our Global Imagery Browse Services (GIBS) capability to provide pre-generated full resolution browse imagery (with links to the underlying data).  Currently only a subset of EOSDIS data is available via GIBS.  The objective would be to greatly expand the number of data sets, as well as the usability of the data, by making it more compatible with commercial GIS software.
Enhancing the ability to access EOSDIS data via web accessible APIs (e.g., OPeNDAP, WEBIFICATION [W10N]
Enhancing and formalizing standards like GeoTIFF
Increasing the ability of commercial search engines to discover EOSDIS data sets
Improved support for Open Standards (via ISO and W3C, as well as OGC)

6. EOSDIS Data Trends For the Discipline pie chart Others: Cryosphere
Earth Radiation
Socio-Economic
Others (not identified)

7. Big Data Lessons Learned
Data Acquisition and Archives
Pick the best protocol for high bandwidth circuits over long distances – NASA’s recent test results indicate the most effective protocols are:
Bbftp (
GridFtp (
Monitor system health, provide system failover, database failover and replication to protect against data loss resulting from system faults
Detect data gaps and automatically reacquire missing products
Provide users with data gap reports and data archive status
Provide data consumers with subscription capabilities and a manifest file to streamline data access
User Experience
Data on spinning disk is essential for providing interactive services
Quality, consistency and flexibility of metadata services enables service orientated architectures
Open source software and standards are vital for interoperability
Machine accessible APIs
Engage users early and often

8. NASA Sentinel Mirror Leveraging Existing EOSDIS Capabilities
For Sentinel 1, 3, 5P, NASA will leverage proven mirroring and redistribution capabilities, currently used for S-NPP (Suomi National Polar-orbiting Partnership)
Single network interface relieves bandwidth load on European networks
Long term archival and end user distribution by DAACs (e.g. Sentinal-1 by ASF DAAC).
Provides metric reports back to the EC/ESA on product distribution and usage
Leverages entire suite of NASA’s EOSDIS capabilities including capturing and reporting metrics on distribution and usage of Sentinel products by U.S. scientists
The primary role of the SDS is to assess the quality of the SNPP Raw Data Records (RDRs), Sensor DRs (SDRs), Temperature DRs (TDRs), Environmental DRs (EDRs), and Intermediate Products (IPs), for accomplishing climate research and providing algorithm enhancements when possible. The SDS also provides the capability to generate, store, and disseminate climate quality data products for CERES (via Earth Radiation Budget Climate Analysis Research System (ERBCARS) and the OMPS limb (via the Ozone Product Evaluation and Analysis Tool Element (PEATE)). The SDS is composed of the SDS Data Depository and Distribution Element (SD3E), the Integration and Test System Element (I&TSE), the VIIRS Characterization Support Team (VCST), and five Product Evaluation and Analysis Tools Elements (PEATEs), one for each of the following disciplines: Atmosphere, Land, Ocean, Ozone, and Sounder.  Additionally the Earth Radiation Budget Climate and Research System (ERBCARS) is part of SDS for the production of CERES climate data records.
The SNPP instruments are: (1) the Visible Infrared Imaging Radiometer Suite (VIIRS) which provides global observations of land, ocean, and atmospheric parameters; (2) the Advanced Technology Microwave Sounder (ATMS) collects specialized data to permit, in conjunction with CrIS, the calculation of the vertical moisture, temperature and pressure profiles of the earth’s atmosphere; (3) the Cross-Track Infrared Sounder (CrIS) which will combine infrared and microwave measurements to measure atmospheric temperature, moisture, and pressure profiles; (4) the Ozone Mapper/Profile (OMPS) that monitors global ozone levels; and (5) the Clouds and the Earth’s Radiant Energy System (CERES) which measures the earth’s reflected solar and emitted thermal radiation as fluxes at the top of the earth’s atmosphere.

9. Sentinel Mirror: Leveraging experience with SNPP
Current Capabilities of NASA’s S-NPP Science Data System:
Server System dedicated to
Acquiring data from multiple locations
Storing data temporarily (~ 30 days)
Making data available to six data processing centers
Ingests 6 TB daily
Capability of distributing 2.5 times the ingest volume; routinely distributes 15TB daily
Products available to data processing centers within 30 minutes of receipt