Connecting HDF And ISO Metadata Ted Habermann, NASA/ESDIS Hook Hua, Barry Weiss, NASA/Jet Propulsion Lab Mike Folk, Gerd Heber, Elena Pourmal, The HDF.

Slides:

Advertisements

Similar presentations

1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.

Advertisements

EURO4M Project Kick-Off, April 2010 OGC Web Services Data visualization using OGC web services Maarten Plieger Wim Som de Cerff Royal Netherlands Meteorological.

A Draft Standard for the CF Metadata Conventions Cheryl Craig and Russ Rew UCAR.

Unidata Seminar Series - 30 January 2004 OPeNDAP and THREDDS: Access and Discovery of Distributed Scientific Data Yuan Ho Ethan Davis UCAR Unidata.

The Model Output Interoperability Experiment in the Gulf of Maine: A Success Story Made Possible By CF, NcML, NetCDF-Java and THREDDS Rich Signell (USGS,

® OGC Web Services Initiative, Phase 9 (OWS-9): Innovations Thread - OPeNDAP James Gallagher and Nathan Potter, OPeNDAP © 2012 Open Geospatial Consortium.

THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007.

Best Practices to Promote Data Interoperability Chris Lynnes Joe Glassy Technology Infusion Working Group.

FGDC, Meet the DDI Adding Geospatial Metadata to a Numeric Data Catalog Julie Linden Yale University.

Data Discovery 1.Data discovery systems 2.TDS and metadata 3.NcISO services.

Presentation Outline Metadata Coverage and Guidelines SMAP ISO Requirement Metadata Accessibility – HDF5 Group/Attribute Multiple Instantiation of the.

Steve Rutz NOAA/NESDIS National Oceanographic Data Center NODC Observing Systems Team Leader June 21, 2011.

Unidata TDS Workshop THREDDS Data Server Overview October 2014.

Documentation Developments Ted Habermann, NOAA/NESDIS/NGDC Adoption of ISO Metadata Standards - Crossing the Chasm Facilitating the Transition Documentation.

1 CF Unleashed: Introduction to Cf/Radial Joe VanAndel National Center for Atmospheric Research 2013/1/8 The National Center for Atmospheric.

1 Introduction to web mapping Dissemination of results, maps and figures ESTP course on Geographic Information Systems (GIS): Use of GIS for making statistics.

VrRBO with THREDDS data store. Paths & URLs THREDDS server THREDDS data directory.

Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned Rich Signell USGS Woods Hole, MA / NOAA Silver Spring USA Model Data.

Unidata’s TDS Workshop TDS Overview – Part II October 2012.

DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.

Interoperable Documentation Ted Habermann, NOAA/NESDIS/NGDC NCAR Earth Observing Laboratory, June 2010 Links: GEO-IDE Wiki:

XSLT transforms Mapping from Different Metadata Standards.

2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

THREDDS Data Server Ethan Davis GEOSS Climate Workshop 23 September 2011.

U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center.

Unidata’s TDS Workshop TDS Overview – Part II Unidata July 2011.

Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew

Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.

IOOS Model Data Interoperability Design ROMS POM WW3 WRF ECOM NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF)

1 International Standards for Data Interoperability GALEON Geo-interface for Air, Environment, Land, Ocean NetCDF Ben Domenico Unidata Program Center*

Metadata Implementation Ted Habermann NOAA National Geophysical Data Center Documentation: It’s not just discovery... 50% change in global average Why?

GI-cat / THREDDS notes. GI-cat/THREDDS system

Documentation from NcML to ISO Ted Habermann, NOAA NESDIS NGDC.

Sharing Metadata Recommendations Ted Habermann, John Kozimor Earth Science The HDF Group 1 John Farley Raytheon.

IOOS Modeling Testbed Cyberinfrastructure Rich Signell, USGS, Woods Hole, MA IOOS-RA-Briefing, Feb 14, 2012.

Documentation Foundation Spectrum Relational Tables XML/Relational Database (with some fields) XML Blobs (with some fields) File Systems XML Blobs in Database.

Workshop Resources.

Tailored data graphics of data standard data raw data catalogue of data what where when who why how … data URLs smart phone & tablet users scientists professionals.

Unidata TDS Workshop THREDDS Data Server Overview

Recent developments with the THREDDS Data Server (TDS) and related Tools: covering TDS, NCML, WCS, forecast aggregation and not including stuff covered.

IOOS Data Services with the THREDDS Data Server Rich Signell USGS, Woods Hole IOOS DMAC Workshop Silver Spring Sep 10, 2013 Rich Signell USGS, Woods Hole.

THREDDS Catalogs Ethan Davis UCAR/Unidata NASA ESDSWG Standards Process Group meeting, 17 July 2007.

Unidata’s TDS Workshop TDS Overview – Part I July 2011.

JPL/Caltech proprietary. Not for public release or redistribution. This document has been reviewed for export control and it does NOT contain controlled.

Documenting UAF Data Ted Habermann NOAA/NESDIS/National Geophysical Data Center.

GODAE August 2007 Contribution from F. Blanc for IGST, August 2007 Strong  Understanding of systems (homogeneous description, primary.

1 NASA CEOP Final Summary CEOS WGISS-26 Boulder, Colorado September 23, 2008 Yonsook Enloe

Creating Good Documentation NOAA National Geophysical Data Center

The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin,

The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.

Data File Formats: netCDF by Tom Whittaker University of Wisconsin-Madison SSEC/CIMSS 2009 MUG Meeting June, 2009.

Data Interoperability at the IRI: translating between data cultures Benno Blumenthal International Research Institute for Climate Prediction Columbia University.

Writing Metadata Working Towards Best Practices for SEFSC.

Standard Metadata in Scientific Data Formats September 19, 2007 Flash at:

DOC / NOAA / NMFS / SWFSC / ERD

Convergence And Trust in Earth and Space Science Data Systems Ted Habermann, NOAA National Geophysical Data Center Documentation: It’s not just discovery...

Semantic Web underpinnings of the IRI Data Library Semantic Web as a Framework for Multiple Metadata IRI Data Library: presenting Data in multiple frameworks.

New Dog You Met Conventions Ted Habermann NOAA EDMC May 2012 Data needs to be in tools Document.

Rich Signell Roland Viger Curtis Price USGS Community for Data Integration Feb 15, 2012.

1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)

NetCDF: Data Model, Programming Interfaces, Conventions and Format Adapted from Presentations by Russ Rew Unidata Program Center University Corporation.

Interoperability Day Introduction Standards-based Web Services Interfaces to Existing Atmospheric/Oceanographic Data Systems Ben Domenico Unidata Program.

Update on Unidata Technologies for Data Access Russ Rew

TSDS (HPDE DAP). Objectives (1) develop a standard API for time series-like data, (2) develop a software package, TSDS (Time Series Data Server), that.

FGDC  ISO MTX (Metadata Transformation XSLT) Bruce Westcott – Geospatial Metadata Consultant.

Product Designer Hub – Taking HPD to the Web

Data Are from Mars, Tools Are from Venus

HDF Support for NASA Data Producers

OPeNDAP/Hyrax Interfaces

Presentation transcript:

Connecting HDF And ISO Metadata Ted Habermann, NASA/ESDIS Hook Hua, Barry Weiss, NASA/Jet Propulsion Lab Mike Folk, Gerd Heber, Elena Pourmal, The HDF Group

Layers of Access HDF5 Data HDF & NetCDF Library Climate Forecast Conventions MatLab, IDL, IDV, Ferret, GMT OPeNDAP Geospatial Database SQL Community Data Models, Open GIS Specifications ArcMap, ArcIMS, WMS, WFS, WCS GML, KML, SimpleFeatures HDF/netCDFGIS

The ISO Metadata Standard (19115)

Documentation Objects

People/Organizations

Citations

ISO Lineage Model Source Step Product Processing and Algorithm Descriptions

Lineage

THREDDS Data Server Multiple Dialects: THREDDS Catalog THREDDS Catalog file1.nc file2.nc file3.nc file4.nc file1.nc file.nc file2.nc file.nc THREDDS Catalog THREDDS Catalog THREDDS Catalog THREDDS Catalog Extract Data OPeNDAP WMS WCS Extract Metadata (NcISO) NcML ISO Rubric THREDDS Metadata Server

Documentation in Three Dialects ISO ACDD NcML wiki/index.php?title=NcISO group/ncisometadata

Documentation Repository ISO 19115, , and extensions Documentation Repository ISO 19115, , and extensions THREDDS netCDF (NcML) netCDF (NcML) DIF, FGDC, Data.Gov DIF, FGDC, Data.Gov SensorML WCS, WMS, WFS, SOS Open Provenance Model, PROV Open Provenance Model, PROV KML Documentation in Multiple Dialects

Conventions Discovery Use / Mashup Understanding Climate-Forecast (CF) Conventions Standard variable names and data organizations Unidata Attribute Convention for Data Discovery ISO Conventions

Metadata Where Are Citations? XML dataset / resource sourcekeyword thesaurus & ontology evaluation procedure standard specification algorithmfeature catalog application schema alternate metadata & service standard associated resource name/metadata additional documentation constraints reference source metadata process reference & documentation software reference feature catalog format specification Documentation

Questions?

The Design Process SMAP.xml SMAP.h5 SMAPHDF.xml SMAP2.xml h5dump HDF2ISO.xsl ISO2NCML.xml ISO2NCML.xsl NCML2h5py.py NCML2h5py.xsl The content of these two files must match 1.SMAP.xml: an ISO compliant XML file that contains the metadata elements identified in the SMAP metadata model. This is the content that must traverse the system into and out of the HDF5 file. 2.ISO2NCML.xsl: an xsl file that transforms ISO metadata into a candidate NcML representation. This representation is used because it is intuative and easy to read. It also provides a connection to the netCDF/CF community. 3.ISO2NCML.xml: an NcML file that contains an extract of the SMAP content in netCDF4 compliant NcML. 4.NCML2h5py.xsl: an xsl that transforms NCML into python that is compliant with the python HDF5 library (h5py). The python that comes out of this transform instantiates the group structure from ISO2NCML.xml in HDF5 5.NCML2h5py.py: The python program that, when executed, instantiates the structure from ISO2NCML.xml into HDF5. 6.SMAP.h5: the HDF5 file created using NCML2h5py.py 7.SMAPHDF.xml: the XML representation of the content of SMAP.h5 8.HDF2ISO.xsl: an xsl that transforms the HDF/XML into ISO SMAP2.xml: the output of the process that should match the original (SMAP.xml).