HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII10/17/2008

OPeNDAP Review A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community 2HDF and HDF-EOS Workshop XII10/17/2008

Client Library (libnc-dap) DAP Proctocol (Internet) Server (apache) Handler (hdf4_handler) Visualization tools (e.g. GrADS) Data (HDF4) View Data (User) OPeNDAP Concept 3HDF and HDF-EOS Workshop XII10/17/2008 Visualization Tools (gradsdap)

Example Usage OPeNDAP Client OPeNDAP HDF4 Server HDF4 Syntactic Structure of Data Semantic Meaning of Data Actual Content of Data OPeNDAP Visualization Client 4HDF and HDF-EOS Workshop XII10/17/2008

Why is HDF5-OPeNDAP Project important? HDF5: NASA EOS / NPOESS ACCESS! NASA’s MISSION : ACCESS! Our MISSION: Build a DAP-HDF5 Bridge 5HDF and HDF-EOS Workshop XII10/17/2008

OPeNDAP HDF5 Server 1.Map HDF5 Datatypes to DAP Datatypes 2.Plus, some enhancements to allow OPeNDAP Clients to access HDF-EOS5 Files Goal: Access HDF5 data via OPeNDAP 6HDF and HDF-EOS Workshop XII10/17/2008

Mapping HDF5 Datatypes to DAP Datatypes Compound Datatype Support for Group Object/regional References 7HDF and HDF-EOS Workshop XII10/17/2008 Challenges

Some OPeNDAP Clients Need special variable names and attributes on dataset. Need shared geolocation variables outside Grid. Not all OPeNDAP clients are created equal! 8HDF and HDF-EOS Workshop XII10/17/2008

Tweaks for OPeNDAP Clients Two server configuration options --enable-short-name --enable-CF 9HDF and HDF-EOS Workshop XII10/17/2008

HDF EOS5 Grid with No Geolocation data  Clients expect Grid with Geolocation data Some Attributes stored as Extremely Long String(s) (e.g. StructMetada.0)  Clients expect structured attributes Not all HDF5 files are created for DAP! 10HDF and HDF-EOS Workshop XII10/17/2008

Tweaks for HDF-EOS Two more server configuration options --enable-eos-grid --enable-eos-meta 11HDF and HDF-EOS Workshop XII10/17/2008

Day After Server Tweaks Finally, Happy Clients! 12HDF and HDF-EOS Workshop XII10/17/2008

Goals 1st: Access HDF5 via OPeNDAP o The OPeNDAP HDF5 handler was released in March, 2008 at OPeNDAP website 2nd: Construct HDF5 data from OPeNDAP Bonus: Help clients to better access EOS Swath data 13HDF and HDF-EOS Workshop XII10/17/2008

HDF5-Friendly OPeNDAP client library 14HDF and HDF-EOS Workshop XII10/17/2008

Example: Groups in HDF5 Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5 15HDF and HDF-EOS Workshop XII10/17/2008

Example: Reference in HDF5 Important for NPOESS Object / Regional Reference Map to DAP URL 16HDF and HDF-EOS Workshop XII10/17/2008

Only ODC can display Swath properly. One more reason: Help Clients to view Swath HDF5-Friendly OPeNDAP Client Library No Latitude and Longitude Courtesy of NASA 17HDF and HDF-EOS Workshop XII10/17/2008

libnc-dap ??? liboc-dap NetCDF OPeNDAP Client Library libdap Dap server OPeNDAP NC Server gradsdap NetCDFView NetCDF HDF5 Group/Ref/Swath OPeNDAP HDF5 Server View HDF5 gradsoc View G/R/Swath HDF5-Friendly OPeNDAP Client Library 18HDF and HDF-EOS Workshop XII10/17/2008

Summary of Client Library Prototype Finished coding Tested with the GrADS visualization client Documented Working on a demo DAP to HDF5 tool by using the client library prototype Will test with NCL if time allows 10/17/2008HDF and HDF-EOS Workshop XII19

Example: dap2h5 A test application for the client library It can construct HDF5 from DAP output 20HDF and HDF-EOS Workshop XII10/17/2008

Demo: MLS swath via GrADS GrADS coupled with our client library The client library provides grid mapping from swath data GrADS displays swath through HDF-EOS5 specific client library API calls

Caution Our Client Library is ONLY PROTOTYPE!  It does NOT support all DAP data types  It does NOT support all AURA files  It does NOT support all Visualization clients 22HDF and HDF-EOS Workshop XII10/17/2008

Summary HDF5 Access via OPeNDAP Is Easy Is used by GES DISC to serve Aura files HDF5-Friendly OPeNDAP Client Library Prototype Serves HDF5 better(EOS swath) Benefits visualization clients 23HDF and HDF-EOS Workshop XII10/17/2008

Future Work HDF5 to DAP2 Mapping Document Release HDF5-friendly OPeNDAP Client Library Prototype URL: h ttp://hdfgroup.org/projects/opendap 24HDF and HDF-EOS Workshop XII10/17/2008

Credits Mike Folk (The HDF Group) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR) 25HDF and HDF-EOS Workshop XII10/17/2008

Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 26HDF and HDF-EOS Workshop XII10/17/2008