We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byGary Bowley
Modified over 2 years ago
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann The EarthServer initiative: towards Agile Big Data Services 2nd GEOSS Science and Technology Stakeholder Workshop Bonn, Germany, 2012-aug-29 Peter Baumann Jacobs University | rasdaman GmbH Bremen, Germany firstname.lastname@example.org
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann About the Presenter Professor of CS, Jacobs University Head, Large-Scale Scientific Information Systems research group Main outcome so far: rasdaman first „Big Raster Data Analytics“ server Standardization OGC: chair of raster-relevant working groups, editor of 12+ standards & candidate standards ISO: working on Raster („Array“) SQL INSPIRE: Invited expert for coverages www.jacobs-university.de/lsis, www.rasdaman.org
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Roadmap OGC standards rasdaman EarthServer EarthServer & GEOSS Conclusions
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Feature and Coverage Data Standards Core element in OGC: geographic feature = abstraction of a real world phenomenon associated with a location relative to Earth Special kind of feature: coverage = space-time varying multi-dimensional phenomenon Typical representative: raster image...but there is more! Typically, coverages are Big Data
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann «FeatureType» Abstract Coverage MultiSolid Coverage MultiSurface Coverage MultiCurve Coverage MultiPoint Coverage Discrete Coverage Continuous Coverage as per GML 3.2.1 Rectified GridCoverage Referenceable GridCoverage Grid Coverage all n-D New subtypes possible Coverage Types
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Coverage Encoding Pure GML: complete coverage represented by GML Special Format: other suitable file format (ex: MIME type “image/tiff”) Multipart-Mixed: multipart MIME, type “multipart/mixed” GML Coverage Domain set Range type Range set App Metadata GML Coverage Domain set Range type xlink App Metadata NetCDF file NetCDF Domain set Range type Range set App Metadata GeoTIFF Range type Range set 6
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann featurecoverage data WMS imagesdata meta data WCPS WCS-T WCS FE WFS-T WFS CQL CS-T CS-W Core OGC Service Standards WMS "portrays spatial data” pictures WCS "provides data + descriptions; data with original semantics, may be interpreted, extrapolated, etc.“ [09-110r4] … … … 7
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Web Coverage Service (WCS) Core: Simple & efficient access to multi-dimensional coverages subset = trim | slice WCS Extensions for additional functionality facets “band extraction”, scaling, reprojection, interpolation, query language,... Application Profiles define domain-oriented bundling 8
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Raster Query Language: ad-hoc navigation, extraction, aggregation, analytics Time series Image processing Summary data Sensor fusion & pattern mining Web Coverage Processing Service (WCPS)
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Scalable On-Demand Processing for the Earth Sciences EU funded, 3 years, 5.85 mEUR Platform: rasdaman (Array Analytics server) Distributed query processing, integrated data/metadata search, 3D clients Strictly open standards: OGC WMS+WCS+WCPS; W3C Xquery; X3D 6 * 100+ TB databases for all Earth sciences + planetary science EarthServer: Big Earth Data Analytics
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Array DBMS for massive n-D raster data new database attribute type: array Data integration: rasters stored in standard database Extending ISO SQL with array operators: “tile streaming” architecture n-D array set of n-D tiles extensive optimization, hw/sw parallelization In operational use dozen-Terabyte objects Analytics queries in 50 ms on laptop The rasdaman Raster Analytics Server select img.green[x0:x1,y0:y1] > 130 from LandsatArchive as img www.rasdaman.org
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Value-Added Satellite Image Archive [Diedrich et al 2001]
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann WCPS peer-to-peer cloud each node accepts all requests Incoming node distributes query, semantics based Manifold optimization criteria for $a in ( A ), $b in ( B ) return encode( ( ($a.nir - $a.red) / ($a.nir + $a.red) - ($b.nir - $b.red) / ($b.nir + $b.red) ), “ HDF5 “ ) coverage A for $b in ( B ) return encode( ($b.nir - $b.red) / ($b.nir + $b.red), “ array-compressed “ ) for $a in ( A ) return encode( ($a.nir - $a.red) / ($a.nir + $a.red), “ array-compressed “ ) rasdaman: Distributed Query Processing coverage B [Owonibi 2012]
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann EarthServer Contribution to GEOSS Integrated n-D coverage data / metadata search Smooth integration with Broker [Nativi, Mazzetti 2012]
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Integrated n-D coverage data / metadata search Smooth integration with Broker EarthServer Contribution to GEOSS Including „reverse lookup“ queries: „give me metadata for data with specific properties“ Also integration with MapServer, GDAL,... Scalable n-D interfaces, based on OGC standards Working „in situ“on existing archives; no copying! Flexible ad-hoc processing & filtering Through OGC standardized query language nD visual Web clients 1D diagrams, 2D maps, 3D data cubes, 3D timeseries sets,... Dynymically composed from query results
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Conclusion Sensor, image, & statistics data = a main source of Big Data in Earth Sciences Petrol industry has „more bytes than barrels“ OGC standards offer common platform spatio-temporal coverages – a unified, cross-domain data model Web Coverage Service suite – from simple download to flexible analytics www.ogcnetwork.net/wcs EarthServer can contribute Agile Analytics to GEOSS OGC coverage standards rasdaman technology www.earthserver.eu
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann SWE O&M and SensorML (+ friends): high flexibility to accommodate virtually any data structure → upstream integration GMLCOV and WCS (+WCPS): one generic schema for all coverage types; scalable; versatile processing → downstream services coverage server O&M + SensorML GMLCOV + WCS Semantic Web Integration of OGC WCS and SWE
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann VAROS (contd.d)
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann The Integrated Geo Warehouse 1D 2D 3D Compprehensive geophysics data mgmt seismic measurement, borehole data, geophone data, geo tomograms, stratigraphy layers, geological models,... + annotations + meta data nD
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Let’s Take a Closer Look... Divergent access patterns for ingest and retrieval Alternative 1: simple access service, let client chisel result Alternative 2: Deliver to exact needs o no bandwidth waste, higher quality of service Server must mediate between access patterns (...later more) Intelligent access interfaces help t
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann interfaces: OGC or API Server: OGC interfaces as servlets: WCS 2.0, WCPS 1.0, WPS 1.0 Server engine: C++ Bindings to GDAL, MapServer, ERDAS (to be extended) Ex: VAROS project (ESA) Commercial client, ChartLink Open-source server, rasdaman petascope request translator rasdaman engine metadata standard database system WCS+WCPSWPS+WCPS System Architecture
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Just-In-Time Compilation Observation: interpreted mode slows down Approach: cluster suitable operations compile & dynamically bind Benefit: Speed up complex, repeated operations Variation: compile code for GPU Times [ms] for 512 2 * n ops [Jucovschi, Stancu-Mara 2008] select x*x*...*x from float_matrix as x
EarthServer :: GEOSS 2012, Bonn :: ©2012 Peter Baumann Query Optimization – Ex. 1
AIP Deep Dive :: Coverage Cubes :: © 2015 rasdaman - AIP-8 Results - GEO XII Plenary, Mexico City, Alexandru Mircea Dumitru Jacobs University.
INSPIRE KEN :: © 2015 AD, PB Alex Dumitru Jacobs University | rasdaman GmbH Spatio-Temporal Coverage World Through The Looking Glass.
1 SSO 1st Nov, 2011 Applying WCO Ontology to Geospatial Web Coverage Services Xia Wang and Peter Baumann Jacobs University.
The Next Generation Network Enabled Weather (NNEW) SWIM Application Asia/Pacific AMHS/SWIM Workshop Chaing Mai, Thailand March 5-7, 2012 Tom McParland,
Mapping between SOS standard specifications and INSPIRE legislation. Relationship between SOS and D2.9 Matthes Rieke, Dr. Albert Remke (m.rieke,
Big Data: Arrays :: RDA 4th Plenary :: © 2014 P. Baumann RDA 4th Plenary 2014-sep-22, Amsterdam, The Netherlands Peter Baumann Jacobs University | rasdaman.
WCS :: OGC Bonn 2005-nov-09 Some GALEON Results for WCS OGC, Bonn, 2005-nov-09 Peter Baumann International University Bremen, rasdaman GmbH
Serving society Stimulating innovation Supporting legislation Web Coverage Services (WCS) Thematic Cluster #3 Jordi Escriu Facilitator.
Page 1 CSISS Center for Spatial Information Science and Systems Access HDF-EOS data with OGC Web Coverage Service - Earth Observation Application Profile.
INTERNATIONAL INSTITUTE FOR GEO-INFORMATION SCIENCE AND EARTH OBSERVATION Towards quality-aware Infrastructures for Geographic Information Services Richard.
MapServer Support for Web Coverage Services Stephen Lime - Minnesota DNR Dr. Thomas E. Burk - University of Minnesota MUM Ottawa, Canada.
Science SQL :: EGI-GEANT :: © 2014 P. Baumann EGI-GEANT Symposium 2014-sep-25, CWI, Amsterdam, The Netherlands Peter Baumann Jacobs University | rasdaman.
Implementation of a Data Node in China's Spatial Information Grid Based on NWGISS Dengrong Zhang, Le Yu, Liping Di Institute of Spatial.
® Sponsored by Improving Access to Point Cloud Data 98th OGC Technical Committee Washington DC, USA 8 March 2016 Keith Ryden Esri Software Development.
Unidata’s TDS Workshop TDS Overview – Part II October 2012.
Crossing the Digital Divide Presented by: Fernando R. Salas David Maidment, Enrico Boldrini, Stefano Nativi, Ben Domenico OGC Technical Meeting – Met/Occean.
HMA-FO ODA, June 2011 HMA Follow-On Task 3 Online Data Access (ODA) in the frame of ESA's Heterogeneous Missions Accessibility (HMA) initiative Final Presentation.
GALEON :: OGC Huntsville, 2006-mar-08 IUB GALEON Status Update Huntsville, 2006-mar-08 Peter Baumann, Ivan Delchev International University Bremen, rasdaman.
1 OGC Web Services Kai Lin San Diego Supercomputer Center
OGC Liaison Report WGISS-20 Allan Doyle, EOGEO
SSE WebMapViewer Recent Developments Steven Smolders SSE Workshop ESA - ESRIN, Frascati, Rome.
ESIP Air Quality Jan Air Quality Cluster Air Quality Cluster Technology Track Earth Science Information Partners Partners NASA NOAA EPA (?) USGS.
Geographic Information Systems GIS Data Databases.
Ideas on Opening Up GEOSS Architecture and Extending AIP-5 Wim Hugo SAEON.
OGC Web Services with complex data Stephen Pascoe How OGC Web Services relate to GML Application Schema.
® OGC Web Services Initiative, Phase 9 (OWS-9): Innovations Thread - OPeNDAP James Gallagher and Nathan Potter, OPeNDAP © 2012 Open Geospatial Consortium.
Introduction to Apache OODT Yang Li Mar 9, What is OODT Object Oriented Data Technology Science data management Archiving Systems that span scientific.
ETICS An Environment for Distributed Software Development in Aerospace Applications SpaceTransfer09 Hannover Messe, April 2009.
® ® Geo-Processing in Cyberinfrastructure: Making the Web and easy to use geospatial computing platform George Percivall, OGC and Ingo Simonis, OGC-E Lionel.
Routemap to derive ISO models from BUFR Why do we need both ISO and BUFR models? –The BUFR data model is very large – much larger in principle than most.
WCPS :: OGC Huntsville, 2006-mar-08 Proposal for a Web Coverage Processing Service (WCPS) OGC, Huntsville, 2006-mar-08 Peter Baumann International University.
Distributed Data Analysis & Dissemination System (D-DADS ) Special Interest Group on Data Integration June 2000.
OneGeology-Europe - the first step to the European Geological SDI INSPIRE Conference 2010, Session Thematic Communities: Geology Krakow, June 24 th 2010.
ORNL DAAC SPATIAL DATA ACCESS TOOL Open Geospatial Consortium (OGC) Services Bruce E. Wilson Suresh K. Santhana Vannan Yaxing Wei Tammy W. Beaty National.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
Presentation to GEO ADC Brazil March The Challenge Information systems need to provide simple access to multi-scale, multi-discipline data sets.
Second workshop on the use of GIS/OGC standards in meteorology, Toulouse 2009 © Geomatys, 2009 Adrian Custer 25 November 2009 Three libraries, one platform,
1 The NERC DataGrid DataGrid The NERC DataGrid DataGrid AHM 2003 – 2 Sept, 2003 e-Science Centre Metadata of the NERC DataGrid Kevin O’Neill CCLRC e-Science.
Experience of application of modern GIS-technologies for environmental monitoring tasks Prof., Dr. Cheremisina Evgenia, Dr.Lyubimova Anna.
Page 1Overview of the HMA Project, 8 March 2010 Overview of the HMA Project OGC TC Opening Plenary Pier Giorgio Marchetti European Space Agency.
High level summary and recommendations from AIP-3 George Percivall Open Geospatial Consortium Task lead AR-09-01B ADC-16, May 2011.
ArcGIS for Server material assembled from the web pages at
Crossing the Digital Divide Presented by: Fernando R. Salas David Maidment, Enrico Boldrini, Ben Domenico, Stefano Nativi CUAHSI Conference on Hydrologic.
© IONIC Software 4th GML Relay.
Achieving Interoperability using the ArcGIS Platform Satish Sankaran Roberto Lucchi.
ITSC/University of Alabama in Huntsville ADaM System Architecture Rahul Ramachandran, Sara Graves and Ken Keiser Mathematical Challenges in Scientific.
1 Key Results from GALEON John Caron Ben Dominico UCAR/Unidata.
Bremen:: 15-July-2014 :: FOSS4G-Europe 2014 – INSPIREd geology1 Earth Server EU FP7-INFRA project INSPIREd access to geological data Bremen,
Proposed update of Technical Guidance for INSPIRE Download services based on SOS Matthes Rieke, Dr. Albert Remke (m.rieke, 52°North.
Web Feature Service and Web Map Service WATER FOR A HEALTHY COUNTRY FLAGSHIP SISS Workshop v2.3 Pavel Golodoniuc, Josh Vote 8 May 2013.
© 2017 SlidePlayer.com Inc. All rights reserved.