Summary previous session 1 3 D:\... 3 tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data.

Slides:



Advertisements
Similar presentations
1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.
Advertisements

Datawarehouse Workflow: ETLP Extract Transform LoadProvide Make user- friendly formats Dynamic database Charts & Maps Tools & websites Archive native formats.
Use of OpenEarthTools to make netCDF
OPeNDAP in the Cloud Optimizing the Use of Storage Systems Provided by Cloud Computing Environments OPeNDAP James Gallagher, Nathan Potter and NOAA/NODC.
Client 1 Client n Provide model output as WMS images Encode into your product ADCIRC Model outputs (surge, wind, depth, wave peak etc.) Google Maps Build.
® OGC Web Services Initiative, Phase 9 (OWS-9): Innovations Thread - OPeNDAP James Gallagher and Nathan Potter, OPeNDAP © 2012 Open Geospatial Consortium.
1 Lightning Products and Services at NOAA’s National Climatic Data Center Steve Ansari, Stephen Del Greco, Neal Lott (NOAA / NCDC)
1 86 th Annual American Meteorological Society Meeting Atlanta, Georgia January 29 – February 2, 2006 The NOAA Weather and Climate Toolkit Steve Ansari,
Dynamic Quick View, interoperability and the future Jon Blower, Keith Haines, Chunlei Liu, Alastair Gemmell Environmental Systems Science Centre University.
Internet enabled GIS GIS Topics and Applications John Reiser Rowan University.
IBIS GIS Mapping Missouri “Show and Tell”. Outline 1.What is KML 2.Why we chose KML 3.Show and Tell.
Data Standards Workflow Raw dataScriptsDatabase Store raw data in subversion to keep track of history Stored files (netcdf) accessible through the web.
The RSS Editor Programme: RSS_broker A.Annunziato, C. Best JRC Ispra
workflow. SubVersion Version control Quality control: “Something adheres to some criteria” Without a properly defined something there.
Open Earth Team, Deltares Data sharing Sharing data, tools and thoughts A brief overview of possibilities recently conducted by Deltares.
Unidata TDS Workshop THREDDS Data Server Overview October 2014.
GIS Data Science for Collaboration Across Communities: GIScience 2.0 and Beyond Dr. Brand Niemann Director and Senior Data Scientist Semantic Community.
GADS: A Web Service for accessing large environmental data sets Jon Blower, Keith Haines, Adit Santokhee Reading e-Science Centre University of Reading.
GIS 1 GIS Lecture 4 Geodatabases. GIS 2 Outline Administrative Data Example Data Tables Data Joins Common Datasets Spatial Joins ArcCatalog Geodatabases.
The HDF Group ESIP Summer Meeting HDF OPeNDAP update Kent Yang The HDF Group 1 July 8 – 11, 2014.
Bringing it All Together: NODC’s Geoportal Server as an Integration Tool for Interoperable Data Services Kenneth S. Casey, Ph.D. YuanJie Li NOAA National.
Unidata’s TDS Workshop TDS Overview – Part II October 2012.
General Assembly Meeting – SA Session | Crete Oct SA 13.1 NEMO-SN1 – WESTERN IONIAN SEA GEOPHYSICS, OCEANOGRAPHY, AND DERIVED PRODUCTS SA starting.
Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.
THREDDS Data Server Ethan Davis GEOSS Climate Workshop 23 September 2011.
U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center.
1 AJAX and Dapper: The Good, the Bad, and the Ugly Joe Sirott PMEL/NOAA.
Electronic DEM delivery (data) Dayne Broderson an all text no pictures presentation less than 9 slides.
Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.
NcBrowse A Graphical netCDF/OPeNDAP Browser Donald Denbo 1 & John Osborne 2 1 UW/JISAO-NOAA/PMEL, 2 OceanAtlas Software
1 Dapper and Argo Joe Sirott PMEL/NOAA. 2 What is Dapper? Web server that provides distributed access to in-situ data via OPeNDAP protocol Clients include.
GIS 1 GIS Lecture 4 Geodatabases Copyright – Kristen S. Kurland, Carnegie Mellon University.
NODC ↔ Data Consumers Steve Rutz NOAA/NESDIS National Oceanographic Data Center NODC Observing Systems Team Leader June 21, 2011.
Variables and ConstantstMyn1 Variables and Constants PHP stands for: ”PHP: Hypertext Preprocessor”, and it is a server-side programming language. Special.
Opendap dev - meeting, Boulder, Feb 2007 OPeNDAP infrastructure in European Operational Oceanography T Loubrieu (IFREMER) T Jolibois (CLS)
Tailored data graphics of data standard data raw data catalogue of data what where when who why how … data URLs smart phone & tablet users scientists professionals.
Unidata TDS Workshop THREDDS Data Server Overview
240-Current Research Easily Extensible Systems, Octave, Input Formats, SOA.
Sciamachy features and usage with respect to end-users The typical fate of retrieval people dealing with large datasets… C. Frankenberg, SRON team, IUP.
The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg,
THREDDS Catalogs Ethan Davis UCAR/Unidata NASA ESDSWG Standards Process Group meeting, 17 July 2007.
Unidata’s TDS Workshop TDS Overview – Part I July 2011.
Remote Data Access with OPeNDAP Dr. Dennis Heimbigner Unidata netCDF Workshop October 25, 2012.
1-2-3 February 2006 –Page 1 Mersea Integrated System How to improve Access/Downloading services ? How far do we go in terms of standardization ?
Improving Data Catalogs with Free and Open Source Software Kevin O’Brien University of Washington Joint Institute for the Study of the Atmosphere and Ocean.
The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin,
OPeNDAP Hyrax Harnessing the power of the BES OPeNDAP Hyrax Back-End Server Patrick West
Research & Development Building a science foundation for sound environmental decisions Remote Sensing Information Gateway (RSIG)
1 Adventures in Web Services for Large Geophysical Datasets Joe Sirott PMEL/NOAA.
John Porter Sheng Shan Lu M. Gastil Gastil-Buhl With special thanks to Chau-Chin Lin and Chi-Wen Hsaio.
©2010 AIR WORLDWIDE 1 Using ArcGIS Server Web Map Services in Web Map Mash-Ups Benjamin Spaulding, Ph.D. AIR Worldwide Esri Boston Dev Meet-Up 12/8/2010.
Is this going to be difficult? Identify by Radius?
Overview of the FIATOOLS web page Brief introduction to the DataMart Focus is on FIDO – Forest Inventory Data Online A tool that helps you produce summary.
ERDDAP The Next Generation of Data Servers Bob Simons DOC / NOAA / NMFS / SWFSC / ERD Monterey, CA Disclaimer: The opinions expressed.
DOC / NOAA / NMFS / SWFSC / ERD
IPCC WG II + III Requirements for AR5 Data Management GO-ESSP Meeting, Paris, Michael Lautenschlager, Hans Luthardt World Data Center Climate.
Rich Signell Roland Viger Curtis Price USGS Community for Data Integration Feb 15, 2012.
In Situ Data Access Some reasons for success or failure Nancy N. Soreide, Donald W. Denbo NOAA Pacific Marine Environmental Laboratory IIPS Session 3B.
Installing the THREDDS and Aggregation Servers ● Install and verify the Tomcat servlet engine ● Install and verify the THREDDS servlet (which also contains.
Product Designer Hub – Taking HPD to the Web
Data Are from Mars, Tools Are from Venus
Data Sharing We all need data
MERRA Data Access and Services
“Real Simple Syndication” (RSS)
TIGGE Data Archive and Access System at NCAR
Access HDF5 Datasets via OPeNDAP’s Data Access Protocol (DAP)
SAGES Scottish Alliance for Geoscience, Environment & Society
LP DAAC OPeNDAP Services
ExPLORE Complex Oceanographic Data
WPS 4 ODV: on-the-fly server-side plotting of data (in Google Earth)
Presentation transcript:

Summary previous session 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users 2a use tools > Extract Transform LoadProvide 2b read data 2c > save as netCDF

Current session: using netCDF from others 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users Extract Transform LoadProvide 2

netCDF > OPeNDAP netCDF files are already a powerfull file format to share self-describing all meta-info included lots of free inspection tools but … often netCDF files can be very big and you just want to check the meta-info (at first) or peek at a subset of the data, while keeping all meta-info instead of downloading all GB’s OPeNDAP solves this for free because netCDF is an open standard A bunch of netCDF files = a hierarchical database no difficult oploading, just copy to file server (ftp, scp) and the dynamic OPeNDAP server sees it

Just have a look at an OPeNDAP sever OpenEarth.deltares.nl > data > opendap server (THREDDS) (HYRAX) test server Data organized per copyright owner Synchonized with file simple server (still to do)

netCDF though OPeNDAP File structure can be scrolled in browser netCDF (*.nc) files offer special functionality Extract meta-info online in browser Get full netCDF: large files often used Get subset of netCDF: peeking large files Get subset as ASCII Paste url into ncBrowse matlab No need to store copy of data: one url commands in url

Example: orthogonal grid: ncBrowse

Example: orthogonal grid: Matlab

Example: timeseries grid: ncBrowse waterbase.nl NOTE: ncBrowse cannot handle really large datasets (e.g. timeseries of HvH), but Matlab can.

Example: timeseries grid: Matlab NOTE: Matlab handles also laaaarge datasets (e.g. timeseries HvH).

Example: coastlines: ncBrowse

Example: coastlines: Matlab Note: in fact, some people say GIS data should not end up in netCDF, but in shape Files. Yet shaperead.m is not standard Matlab (mapping toolbox) and netCDF is fast, and offers meta-data. oreviewing. However, OpenGIS WFS services allows do Offer this option too.

Summary: using netCDF from others via web 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users Extract Transform LoadProvide 2

Now the fun starts Easily feed models: meteo Make inventory of dataset Make easily KML files (Thijs) fore Google Earth. Add plot to Google Mass (Thijs)