BEDI -Big Earth Data Initiative WGISS -37 Cocoa Beach Florida - April 15, 2014 Andrew Mitchell Earth Science Data and Information Systems (ESDIS) National Aeronautics and Space Administration (NASA

United States Big Data Initiative U.S. Gent Big Data Initiative United States Big Data Initiative President Obama’s “Big Data Research and Development Initiative” seeks to improve our ability to acquire knowledge and discover insights into large and complex collections of digital data. The initiative’s primary objectives are to: Develop the core technologies needed to collect, archive, manage, analyze, and share large and diverse datasets. Use these technologies to accelerate the pace of discovery and knowledge, and transform teaching and learning paradigms 2

Big Earth Data Initiative (BEDI) The Big Earth Data Initiative (BEDI) invests in standardizing and optimizing the collection, management and delivery of U.S. Government’s civil Earth observation data. BEDI will improve the ability to discover, access and use Earth observations by the broader user community. Discover: improve metadata in archives to make datasets more discoverable by popular search engines (e.g., Google or Bing) Access: Develop better web accessible APIs by improving geospatial coordinate handling and response formats Use: Improve compatibility with commercial GIS software (e.g., Esri) 3

NASA’s Role in BEDI NASA is to advance these efforts by developing and implementing an agency-wide framework for managing and curating data that will: maximize the availability of data and information and ensure dissemination in a timey and usable manner; (2) facilitate the transformation of observations and data into useful information through the use of open, machine readable formats and Application Programming Interfaces (APIs); (3) encourage the development and use of uniform tools and practices across Feral agencies for the handling of Earth system data and information to increase interoperability; and (4) support the development of information products and tools that directly support decision-making

EOSDIS Approach Enhancing our Global Imagery Browse Services (GIBS) capability to provide pre-generated full resolution browse imagery (with links to the underlying data).  Currently only a subset of EOSDIS data is available via GIBS.  The objective would be to greatly expand the number of data sets, as well as the usability of the data, by making it more compatible with commercial GIS software. Enhancing the ability to access EOSDIS data via web accessible APIs (e.g., OPeNDAP, WEBIFICATION [W10N] Enhancing and formalizing standards like GeoTIFF Increasing the ability of commercial search engines to discover EOSDIS data sets Improved support for Open Standards (via ISO and W3C, as well as OGC)

EOSDIS Data Trends For the Discipline pie chart Others: Cryosphere Earth Radiation Socio-Economic Others (not identified)

Big Data Lessons Learned Data Acquisition and Archives Pick the best protocol for high bandwidth circuits over long distances – NASA’s recent test results indicate the most effective protocols are: Bbftp (http://doc.in2p3.fr/bbftp/) GridFtp (http://toolkit.globus.org/toolkit/data/gridftp/) Monitor system health, provide system failover, database failover and replication to protect against data loss resulting from system faults Detect data gaps and automatically reacquire missing products Provide users with data gap reports and data archive status Provide data consumers with subscription capabilities and a manifest file to streamline data access User Experience Data on spinning disk is essential for providing interactive services Quality, consistency and flexibility of metadata services enables service orientated architectures Open source software and standards are vital for interoperability Machine accessible APIs Engage users early and often

NASA Sentinel Mirror Leveraging Existing EOSDIS Capabilities For Sentinel 1, 3, 5P, NASA will leverage proven mirroring and redistribution capabilities, currently used for S-NPP (Suomi National Polar-orbiting Partnership) Single network interface relieves bandwidth load on European networks Long term archival and end user distribution by DAACs (e.g. Sentinal-1 by ASF DAAC). Provides metric reports back to the EC/ESA on product distribution and usage Leverages entire suite of NASA’s EOSDIS capabilities including capturing and reporting metrics on distribution and usage of Sentinel products by U.S. scientists The primary role of the SDS is to assess the quality of the SNPP Raw Data Records (RDRs), Sensor DRs (SDRs), Temperature DRs (TDRs), Environmental DRs (EDRs), and Intermediate Products (IPs), for accomplishing climate research and providing algorithm enhancements when possible. The SDS also provides the capability to generate, store, and disseminate climate quality data products for CERES (via Earth Radiation Budget Climate Analysis Research System (ERBCARS) and the OMPS limb (via the Ozone Product Evaluation and Analysis Tool Element (PEATE)). The SDS is composed of the SDS Data Depository and Distribution Element (SD3E), the Integration and Test System Element (I&TSE), the VIIRS Characterization Support Team (VCST), and five Product Evaluation and Analysis Tools Elements (PEATEs), one for each of the following disciplines: Atmosphere, Land, Ocean, Ozone, and Sounder.  Additionally the Earth Radiation Budget Climate and Research System (ERBCARS) is part of SDS for the production of CERES climate data records.   The SNPP instruments are: (1) the Visible Infrared Imaging Radiometer Suite (VIIRS) which provides global observations of land, ocean, and atmospheric parameters; (2) the Advanced Technology Microwave Sounder (ATMS) collects specialized data to permit, in conjunction with CrIS, the calculation of the vertical moisture, temperature and pressure profiles of the earth’s atmosphere; (3) the Cross-Track Infrared Sounder (CrIS) which will combine infrared and microwave measurements to measure atmospheric temperature, moisture, and pressure profiles; (4) the Ozone Mapper/Profile (OMPS) that monitors global ozone levels; and (5) the Clouds and the Earth’s Radiant Energy System (CERES) which measures the earth’s reflected solar and emitted thermal radiation as fluxes at the top of the earth’s atmosphere.

Sentinel Mirror: Leveraging experience with SNPP Current Capabilities of NASA’s S-NPP Science Data System: Server System dedicated to Acquiring data from multiple locations Storing data temporarily (~ 30 days) Making data available to six data processing centers Ingests 6 TB daily Capability of distributing 2.5 times the ingest volume; routinely distributes 15TB daily Products available to data processing centers within 30 minutes of receipt