HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1 HDF and HDF-EOS Workshop XII10/17/2008

OPeNDAP Review A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community 2 HDF and HDF-EOS Workshop XII10/17/2008

Client Library (libnc-dap) DAP Protocol (via http) Server (apache) Handler (hdf4_handler) Remote Data (HDF4) View Data (User) OPeNDAP Concept 3 HDF and HDF-EOS Workshop XII10/17/2008 Visualization Tools (gradsdap) Typical Remote Access FTP/HTTP Visualization tools (e.g. grads)

DAP Protocol (via http) Server (apache) Remote HDF5View Data (User) OPeNDAP HDF5 Handler 4 HDF and HDF-EOS Workshop XII10/17/2008 HDF5 Handler

Mapping HDF5 to DAP Compound Datatype Groups Object/Regional References 5 HDF and HDF-EOS Workshop XII10/17/2008 Challenges

DAP Protocol (via http) Server (apache) View Data (User) OPeNDAP HDF5 Handler with HDF-EOS5 6 HDF and HDF-EOS Workshop XII10/17/2008 HDF5 Handler Remote HDF5 Client Library (libnc-dap) Visualization tools (grads) Remote HDF-EOS5

Challenges: HDF- EOS5 Grid with No Geolocation data  Clients expect Grid with Geolocation data Some Attributes stored as Extremely Long String(s) (e.g. StructMetada.0)  Clients expect structured attributes 7 HDF and HDF-EOS Workshop XII10/17/2008

Tweaks for HDF-EOS5 Added two default HDF5 handler configuration options: --enable-eos-grid --enable-eos-meta 8 HDF and HDF-EOS Workshop XII10/17/2008

Challenges: OPeNDAP Clients Need special attributes on dataset. Need shared geolocation variables. 9 HDF and HDF-EOS Workshop XII10/17/2008

Tweaks for OPeNDAP Clients Added two optional handler configuration options: --enable-short-name --enable-CF 10 HDF and HDF-EOS Workshop XII10/17/2008

HDF5 Handler DAP Protocol (via http) Server (apache) View Data (User) OPeNDAP HDF5 Handler with HDF-EOS5 11 HDF and HDF-EOS Workshop XII10/17/2008 HDF5 Handler w/ CF options Remote HDF5 Client Library (libnc-dap) Visualization tools (grads) Remote HDF-EOS5

Day After Server Tweaks Finally, Happy Clients! 12 HDF and HDF-EOS Workshop XII10/17/2008

Problems of Tweaking HDF5 Handler 13 HDF and HDF-EOS Workshop XII10/17/2008 Remember that we added two optional HDF5 handler configuration options: 1.--enable-short-name 2.--enable-CF Causes ambiguity among variable names (e.g. /GroupA/ozone vs. /GroupB/ozone) Drops some key attributes (e.g. StructMetdata, HDF_ROOT_GROUP)

HDF5-Friendly OPeNDAP Client Library 14 HDF and HDF-EOS Workshop XII10/17/2008 DAP Protocol (via http) Server (apache) View Data (User) HDF5 Handler Client Library (libnc-dap) Visualization tools (grads) Remote HDF5 HDF5 Handler w/ CF options HDF5-Friendly Client Library (liboc-dap) HDF5 GroupsView Groups???

Example: Groups in HDF5 Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5 15 HDF and HDF-EOS Workshop XII10/17/2008

Example: Reference in HDF5 Important for NPOESS Object / Regional Reference Map to DAP URL 16 HDF and HDF-EOS Workshop XII10/17/2008

Example: dap2h5 A test application for the client library It can construct HDF5 from DAP output 17 HDF and HDF-EOS Workshop XII10/17/2008 DAP Protocol (via http) Server (apache) View Data (User) HDF5-F. Client Library App. (dap2h5) Remote HDF5 HDF5 Handler HDF5-Friendly Client Library (liboc-dap) Group/Ref.View Group/Ref.

One more reason: Help Clients to view Swath Our Client Library(prototype)

HDF5 Handler DAP Protocol (via http) Server (apache) View Data (User) Visualizing HDF-EOS5 Grids 19 HDF and HDF-EOS Workshop XII10/17/2008 HDF5 Handler w/ CF option Remote HDF5 Client Library (libnc-dap) Visualization tools (grads) Remote HDF-EOS5 Swath Visualizing HDF-EOS5 Swath Problem ??? HDF5-Friendly Client Library (liboc-dap) Visualization Tools (gradsoc) View Swath

Demo: MLS swath via GrADS GrADS coupled with our client library The client library provides grid mapping from swath data GrADS displays swath through HDF-EOS5 specific client library API calls

Summary of Client Library (Prototype) Finished coding Tested with the GrADS visualization client Documented Working on a demo DAP to HDF5 tool by using the client library prototype Will test with NCL if time allows 10/17/2008HDF and HDF-EOS Workshop XII 21

Caution Our Client Library is ONLY A PROTOTYPE!  It does NOT support all DAP data types  It does NOT support all AURA files  It does NOT support all Visualization clients 22 HDF and HDF-EOS Workshop XII10/17/2008

Summary HDF5 Access via OPeNDAP Is Easy Is used by GES DISC to serve Aura files HDF5-Friendly OPeNDAP Client Library Serves HDF5 better(EOS swath) Benefits visualization clients 23 HDF and HDF-EOS Workshop XII10/17/2008

Future Work HDF5 to DAP2 Mapping Document Release HDF5-friendly OPeNDAP Client Library (Prototype) URL: h ttp://hdfgroup.org/projects/opendap 24 HDF and HDF-EOS Workshop XII10/17/2008

Credits Mike Folk (The HDF Group) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR) 25 HDF and HDF-EOS Workshop XII10/17/2008

Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 26 HDF and HDF-EOS Workshop XII10/17/2008