Presentation is loading. Please wait.

Presentation is loading. Please wait.

BEDI -Big Earth Data Initiative Andrew Mitchell Earth Science Data and Information Systems (ESDIS) National Aeronautics and Space Administration (NASA.

Similar presentations


Presentation on theme: "BEDI -Big Earth Data Initiative Andrew Mitchell Earth Science Data and Information Systems (ESDIS) National Aeronautics and Space Administration (NASA."— Presentation transcript:

1 BEDI -Big Earth Data Initiative Andrew Mitchell Earth Science Data and Information Systems (ESDIS) National Aeronautics and Space Administration (NASA WGISS -37 Cocoa Beach Florida - April 15, 2014

2 U.S. Gent Big Data Initiative 2 United States Big Data Initiative President Obama’s “Big Data Research and Development Initiative” seeks to improve our ability to acquire knowledge and discover insights into large and complex collections of digital data. The initiative’s primary objectives are to: Develop the core technologies needed to collect, archive, manage, analyze, and share large and diverse datasets. Use these technologies to accelerate the pace of discovery and knowledge, and transform teaching and learning paradigms

3 Big Earth Data Initiative (BEDI) The Big Earth Data Initiative (BEDI) invests in standardizing and optimizing the collection, management and delivery of U.S. Government’s civil Earth observation data. BEDI will improve the ability to discover, access and use Earth observations by the broader user community. Discover: improve metadata in archives to make datasets more discoverable by popular search engines (e.g., Google or Bing) Access: Develop better web accessible APIs by improving geospatial coordinate handling and response formats Use: Improve compatibility with commercial GIS software (e.g., Esri) 3

4 NASA’s Role in BEDI NASA is to advance these efforts by developing and implementing an agency- wide framework for managing and curating data that will: (1)maximize the availability of data and information and ensure dissemination in a timey and usable manner; (2) facilitate the transformation of observations and data into useful information through the use of open, machine readable formats and Application Programming Interfaces (APIs); (3) encourage the development and use of uniform tools and practices across Feral agencies for the handling of Earth system data and information to increase interoperability; and (4) support the development of information products and tools that directly support decision-making

5 EOSDIS Approach Enhancing our Global Imagery Browse Services (GIBS) capability to provide pre-generated full resolution browse imagery (with links to the underlying data). Currently only a subset of EOSDIS data is available via GIBS. The objective would be to greatly expand the number of data sets, as well as the usability of the data, by making it more compatible with commercial GIS software. Enhancing the ability to access EOSDIS data via web accessible APIs (e.g., OPeNDAP, WEBIFICATION [W10N] Enhancing and formalizing standards like GeoTIFF Increasing the ability of commercial search engines to discover EOSDIS data sets Improved support for Open Standards (via ISO and W3C, as well as OGC)

6 EOSDIS Data Trends

7 Big Data Lessons Learned Data Acquisition and Archives – Pick the best protocol for high bandwidth circuits over long distances – NASA’s recent test results indicate the most effective protocols are: Bbftp (http://doc.in2p3.fr/bbftp/)http://doc.in2p3.fr/bbftp/ GridFtp (http://toolkit.globus.org/toolkit/data/gridftp/) – Monitor system health, provide system failover, database failover and replication to protect against data loss resulting from system faults – Detect data gaps and automatically reacquire missing products Provide users with data gap reports and data archive status – Provide data consumers with subscription capabilities and a manifest file to streamline data access User Experience – Data on spinning disk is essential for providing interactive services – Quality, consistency and flexibility of metadata services enables service orientated architectures – Open source software and standards are vital for interoperability Machine accessible APIs – Engage users early and often

8 NASA Sentinel Mirror Leveraging Existing EOSDIS Capabilities For Sentinel 1, 3, 5P, NASA will leverage proven mirroring and redistribution capabilities, currently used for S-NPP (Suomi National Polar-orbiting Partnership) – Single network interface relieves bandwidth load on European networks – Long term archival and end user distribution by DAACs (e.g. Sentinal-1 by ASF DAAC). – Provides metric reports back to the EC/ESA on product distribution and usage – Leverages entire suite of NASA’s EOSDIS capabilities including capturing and reporting metrics on distribution and usage of Sentinel products by U.S. scientists

9 Sentinel Mirror: Leveraging experience with SNPP Current Capabilities of NASA’s S-NPP Science Data System: – Server System dedicated to Acquiring data from multiple locations Storing data temporarily (~ 30 days) Making data available to six data processing centers – Ingests 6 TB daily – Capability of distributing 2.5 times the ingest volume; routinely distributes 15TB daily – Products available to data processing centers within 30 minutes of receipt


Download ppt "BEDI -Big Earth Data Initiative Andrew Mitchell Earth Science Data and Information Systems (ESDIS) National Aeronautics and Space Administration (NASA."

Similar presentations


Ads by Google