Presentation on theme: "BEDI -Big Earth Data Initiative"— Presentation transcript:
1 BEDI -Big Earth Data Initiative WGISS -37Cocoa Beach Florida - April 15, 2014Andrew MitchellEarth Science Data and Information Systems (ESDIS)National Aeronautics and Space Administration (NASA
2 United States Big Data Initiative U.S. Gent Big Data InitiativeUnited States Big Data InitiativePresident Obama’s “Big Data Research and Development Initiative” seeks to improve our ability to acquire knowledge and discover insights into large and complex collections of digital data. The initiative’s primary objectives are to:Develop the core technologies needed to collect, archive, manage, analyze, and share large and diverse datasets.Use these technologies to accelerate the pace of discovery and knowledge, and transform teaching and learning paradigms2
3 Big Earth Data Initiative (BEDI) The Big Earth Data Initiative (BEDI) invests in standardizing and optimizing the collection, management and delivery of U.S. Government’s civil Earth observation data. BEDI will improve the ability to discover, access and use Earth observations by the broader user community.Discover: improve metadata in archives to make datasets more discoverable by popular search engines (e.g., Google or Bing)Access: Develop better web accessible APIs by improving geospatial coordinate handling and response formatsUse: Improve compatibility with commercial GIS software (e.g., Esri)3
4 NASA’s Role in BEDINASA is to advance these efforts by developing and implementing an agency-wide framework for managing and curating data that will:maximize the availability of data and information and ensure dissemination in a timey and usable manner;(2) facilitate the transformation of observations and data into useful information through the use of open, machine readable formats and Application Programming Interfaces (APIs);(3) encourage the development and use of uniform tools and practices across Feral agencies for the handling of Earth system data and information to increase interoperability; and(4) support the development of information products and tools that directly support decision-making
5 EOSDIS ApproachEnhancing our Global Imagery Browse Services (GIBS) capability to provide pre-generated full resolution browse imagery (with links to the underlying data). Currently only a subset of EOSDIS data is available via GIBS. The objective would be to greatly expand the number of data sets, as well as the usability of the data, by making it more compatible with commercial GIS software.Enhancing the ability to access EOSDIS data via web accessible APIs (e.g., OPeNDAP, WEBIFICATION [W10N]Enhancing and formalizing standards like GeoTIFFIncreasing the ability of commercial search engines to discover EOSDIS data setsImproved support for Open Standards (via ISO and W3C, as well as OGC)
6 EOSDIS Data Trends For the Discipline pie chart Others: Cryosphere Earth RadiationSocio-EconomicOthers (not identified)
7 Big Data Lessons Learned Data Acquisition and ArchivesPick the best protocol for high bandwidth circuits over long distances – NASA’s recent test results indicate the most effective protocols are:Bbftp (http://doc.in2p3.fr/bbftp/)GridFtp (http://toolkit.globus.org/toolkit/data/gridftp/)Monitor system health, provide system failover, database failover and replication to protect against data loss resulting from system faultsDetect data gaps and automatically reacquire missing productsProvide users with data gap reports and data archive statusProvide data consumers with subscription capabilities and a manifest file to streamline data accessUser ExperienceData on spinning disk is essential for providing interactive servicesQuality, consistency and flexibility of metadata services enables service orientated architecturesOpen source software and standards are vital for interoperabilityMachine accessible APIsEngage users early and often
8 NASA Sentinel Mirror Leveraging Existing EOSDIS Capabilities For Sentinel 1, 3, 5P, NASA will leverage proven mirroring and redistribution capabilities, currently used for S-NPP (Suomi National Polar-orbiting Partnership)Single network interface relieves bandwidth load on European networksLong term archival and end user distribution by DAACs (e.g. Sentinal-1 by ASF DAAC).Provides metric reports back to the EC/ESA on product distribution and usageLeverages entire suite of NASA’s EOSDIS capabilities including capturing and reporting metrics on distribution and usage of Sentinel products by U.S. scientistsThe primary role of the SDS is to assess the quality of the SNPP Raw Data Records (RDRs), Sensor DRs (SDRs), Temperature DRs (TDRs), Environmental DRs (EDRs), and Intermediate Products (IPs), for accomplishing climate research and providing algorithm enhancements when possible. The SDS also provides the capability to generate, store, and disseminate climate quality data products for CERES (via Earth Radiation Budget Climate Analysis Research System (ERBCARS) and the OMPS limb (via the Ozone Product Evaluation and Analysis Tool Element (PEATE)). The SDS is composed of the SDS Data Depository and Distribution Element (SD3E), the Integration and Test System Element (I&TSE), the VIIRS Characterization Support Team (VCST), and five Product Evaluation and Analysis Tools Elements (PEATEs), one for each of the following disciplines: Atmosphere, Land, Ocean, Ozone, and Sounder. Additionally the Earth Radiation Budget Climate and Research System (ERBCARS) is part of SDS for the production of CERES climate data records.The SNPP instruments are: (1) the Visible Infrared Imaging Radiometer Suite (VIIRS) which provides global observations of land, ocean, and atmospheric parameters; (2) the Advanced Technology Microwave Sounder (ATMS) collects specialized data to permit, in conjunction with CrIS, the calculation of the vertical moisture, temperature and pressure profiles of the earth’s atmosphere; (3) the Cross-Track Infrared Sounder (CrIS) which will combine infrared and microwave measurements to measure atmospheric temperature, moisture, and pressure profiles; (4) the Ozone Mapper/Profile (OMPS) that monitors global ozone levels; and (5) the Clouds and the Earth’s Radiant Energy System (CERES) which measures the earth’s reflected solar and emitted thermal radiation as fluxes at the top of the earth’s atmosphere.
9 Sentinel Mirror: Leveraging experience with SNPP Current Capabilities of NASA’s S-NPP Science Data System:Server System dedicated toAcquiring data from multiple locationsStoring data temporarily (~ 30 days)Making data available to six data processing centersIngests 6 TB dailyCapability of distributing 2.5 times the ingest volume; routinely distributes 15TB dailyProducts available to data processing centers within 30 minutes of receipt
Your consent to our cookies if you continue to use this website.