1. MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)
HDF5 OPeNDAP Project
MuQun Yang and Hyo-Kyung Lee (THG)
James Gallagher (OPeNDAP, Inc.)
Today I’ll present the HDF5 OPeNDAP project.
This is a joint project between our group and James Gallagher at OPeNDAP, Inc.
I’ll present an introduction to OPeNDAP and then present how HDF5 is served in OPeNDAP.

2. Question of the Day How can I get One Sub from SUBWAY?
Chicken teriyaki with no onion
How can I get One Subset of EOS data image from NASA?
Today’s satellite image right above the roof
The problem that we’re going to address in this project is similar to this:
“How can I get one sandwich from Subway?”
For example, foot-long chicken teriyaki sandwich with no onion.
I believe most of you think this problem is very trivial.
But, how about this question?
“How can I get one subset of EOS data image from NASA?”
For example, today’s satellite image right above the roof of this building.

3. Subway Easy: anywhere / accepts Visa|MC Efficient: your own recipe
How can I get One Sub from SUBWAY?
Easy: anywhere / accepts Visa|MC
Efficient: your own recipe
Cool: eat fresh and look good
For the first question, everyone thinks it’s easy for the following reasons:
Subway is everywhere.
Some Subway restaurants are even located inside gas stations and Walmarts.
Also, at Subway, you can make your own sub efficiently through selection.
You can pick the size of bread and what to include or exclude in your bread.
As a bonus, Subway provides a way to eat fresh vegetables and make you look good.

4. Dumb-way Difficult: bring a USB drive and visit NASA
How can I get One Subset of EOS data image from NASA?
Difficult: bring a USB drive and visit NASA
Inefficient: download 10 Gbyte and search for 10 byte dataset
Not cool: wrong or useless dataset
How about the answers to the second question?
Here are some really dumb ways of getting the subset of data from NASA.
First, you bring a USB drive and visit NASA near you.
This is extremely difficult way of getting data unless you have a complete access to NASA and it’s conveniently located nearby.
Here’s a better way than the first but it is very inefficient.
For example, you can download one 10G HDF5 file via FTP and search for the 10 byte that you cares most.
Downloading the entire 10G hdf5 file for a 10 byte of data you need is like going to the Niagara Falls to get a cup of water.
In both cases, the worst scenario is that you may end up finding that the real data is either empty or corrupted due to satellite communication failure.
Is there a better way to get what you want?

5. DAP-way! Easy: anywhere / accepts IE|FF Efficient: WYSIWYG – less fat!
How can I get One Subset of EOS data image from NASA?
Easy: anywhere / accepts IE|FF
Efficient: WYSIWYG – less fat!
Cool: Visualization clients – see fresh!
Yes, there is one and it’s called DAP-way.
The DAP is a short name for Data Access Protocol.
In DAP-way, getting a data is like ordering a pizza online.
Also, it’s quite efficient --- what you want is what you get.
You can browse the table of content before you pick the data.
Since you retrieve the small dataset you want, it’s possible to show it with nice visualization clients as a bonus.
Imagine that you cannot wrap the entire food stored in a local subway restaurant store with the tiny sandwich wrapper.
If the data you retrieve is so small, visualization client wrap the data nicely.
Also, you can see your data instantly and recognize that whether it’s useful or not by looking at it.

6. DAP-way Implementation
OPeNDAP: Free DAP-way Software
Server-Client Model APIs
Widely used by Earth Science Community
Now let’s see how the idea of DAP-way can be implemented.
DAP is a mere protocol like FTP and OPeNDAP is an open and free software implementation of DAP-way.
It is a server-client model and provides APIs for both.
And the biggest advantage of OPeNDAP is that it is very popular among Earth Science Community.

7. OPeNDAP server sites OPeNDAP Market Dominance: http://www.opendap.org
This is a map of OPeNDAP server sites that provide scientific data in DAP-way.
Although it’s not as many as subway restaurant chains,
we can see plenty of sites from east coast to Hawaii.
Courtesy of OPeNDAP, Inc.

8. OPeNDAP Servers Tables SQL Flat Binary General
Data
Matlab
HDF4
JDBC
FreeFrom
FITS
CDF
CEDAR
netCDF
DSP
JGOFS
Tables
SQL
Flat
Binary
CODAR
ESML
General
Here is a list of scientific data file formats that can be served in DAP-way.
Courtesy of OPeNDAP, Inc.

9. OPeNDAP Clients netCDF C netCDF Java Client IDV Ferret GrADS VisAD
ncBrowse
Matlab
Excel
IDL
Access
Client
And here is list of visualization clients that can be used with OPeNDAP.
Courtesy of OPeNDAP, Inc.

10. How OPeNDAP works NetCDF OPeNDAP NetCDF Server HDF4 OPeNDAP Client
HDF4 Server
Here’s how OPeNDAP works.
First thing to note is that OPeNDAP server works for different scientific data storage file formats like NetCDF and HDF4.
What users can really see is only the DAP-way representation of data through OPeNDAP clients.
They don’t have to know how NetCDF or HDF4 store the data internally.

11. Architecture of OPeNDAP Server
Server 3 (CGI)
C/C++ based HTTP Server
Insecure / Inflexible
Server 4 (Hyrax)
Java based TomCat Servlet Engine
Secure / Flexible
Let’s take a look at the inside of OPeNDAP server since this will help us understand how OPeNDAP works.
There are two types of servers.
Server 3 is old and Server 4 is new one.
Server 3 is similar to the traditional C/C++ based web server.
Server 3 will be no longer supported since it is insecure and inflexible.

12. Server 4 (Hyrax) Architecture
Server 3 (CGI) Architecture
Server 4 (Hyrax) Architecture
HTTP Server
OLFS
Java
Servlet Engine
BES Commands
BES
Unix Daemon
DAP
XML, GIS, KML
HTML
XML- encapsulated object
Let’s highlight the evolution of OPeNDA server.
In the old server, a single server does everything --- reading scientific data and deliver the result in HTML or DAP.
In a new server, the role of single server is split into two: front-end and back-end.
OLFS is OPeNDAP lightweight front-end servlet.
BES is back-end server.
By splitting the one server into two, it becomes more secure and boots its performance.
Also, by separating the presentation layer from data retrieval, you can format the final output of response more easily through catalog.
It can help produce output that Google maps or other GIS can understand.
OPeNDAP’s goal is to support as many clients as possible through this catalog method.
(Thematic Realtime Environmental Distributed Data Services)
Data
Store
Optional catalogs:
XML, GIS, KML
File system with data files like HDF, NetCDF and SQL Database, …
Courtesy of OPeNDAP, Inc.

13. Example Usage HDF4 OPeNDAP Client OPeNDAP HDF4 Server
Actual Content of Data
Here’s an example usage of OPeNDAP.
First, here is a HDF4 file that you want to allow people to see.
Once the OPeNDAP HDF4 server is installed and running, this file becomes available world-wide through Data Access Protocol and you can view its content with a standard web browser.
Data Access Protocol provides three ways of viewing data.
First you can view the syntactic structure of data.
Second you can view the semantic meaning of data.
Finally, you can view the real content of data you want to retrieve.
Semantic Meaning of Data
Syntactic Structure of Data
OPeNDAP Visualization Client

14. What about HDF5? NetCDF OPeNDAP NetCDF Server HDF4 OPeNDAP Client
HDF4 Server
So far we’ve seen how data access becomes easy through OPeNDAP.
So, what about HDF5? Let there be OPeNDAP HDF5 server and it’s what we’ve been working on since 2007.
HDF5
OPeNDAP
HDF5 Server

15. Why Important? HDF5: NASA EOS / NPOESS NASA’s MISSION: ACCESS!
Our MISSION: Build DAP-way Bridge
Why is HDF5 part so important?
The main reason is that new satellite data are being stored and will be stored in HDF5 in the future.
And one of the most important mission of NASA is to provide a full access to its huge data center.
So our mission is to bridge the gap between data provider and data user through OPeNDAP.

16. Goals 1st: Transform HDF5 access in DAP-way
2nd: Yet preserve the beauty of HDF5
So, here are our project goals.
First, we want to provide way to access HDF5 in remote location like NASA in DAP-way.
Yet, we want to lose as little as possible during this transformation process.

17. OPeNDAP HDF5 Server Map HDF5 Datatypes in DAP-way
Goal: Transform HDF5 access in DAP-way
Map HDF5 Datatypes in DAP-way
Plus, tweaks for OPeNDAP Clients
Plus, tweaks for HDF-EOS Files
And, we built the HDF5 server to provide a standard DAP-way of accessing HDF5.
Plus, we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

18. Mapping HDF5 in DAP-way Prototype server in 2001
NASA Grant from Nov. 2006
Support for Compound Datatype
Support for Group
Support for References / Links
First Product Release in Mar. 2008
And, we built the HDF5 server to provide a standard DAP-way of accessing HDF5.
Plus, we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

19. OPeNDAP HDF5 Server Map HDF5 Datatypes in DAP-way
Goal: Transform HDF5 access in DAP-way
Map HDF5 Datatypes in DAP-way
Plus, tweaks for OPeNDAP Clients
Plus, tweaks for HDF-EOS Files
And, we built the HDF5 server to provide a standard DAP-way of accessing HDF5.
Plus, we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

20. Subway Customers I want 3 inch-long sub I need Wasabi sauce
I eat only wheat bread
I eat only meat balls
In business, customers are king and knowing customers is a key to success.
Back to subway example, imagine a situation that a customer that says:
What are you going to do?
There’s no choice but to cut the bread and bring the special sauce the customer wants.

21. OPeNDAP Clients I hate foot-long variable names.
Not all OPeNDAP clients are created equal!
I hate foot-long variable names.
I need special attributes on dataset.
I care pre-defined Grid data type.
I care only well-formed attributes.
The same pattern happens in DAP-way clients.
Some clients are very picky in terms of what DAP-server can provide.
They may even ask something that HDF5 doesn’t have in the attribute like A-1 sauce.
They simply reject what the standard DAP protocol allows.

22. Tweaks for OPeNDAP Clients
Two configuration options
--enable-short-name = cut bread
--enable-CF = put Wasabi sauce
Thus, we provided some configuration options during installation.
This can make most picky clients happy.
However, enabling these options are risky due to you may not serve some dataset in HDF5 through server.

23. OPeNDAP HDF5 Server Map HDF5 Datatypes in DAP-way
Goal: Transform HDF5 access in DAP-way
Map HDF5 Datatypes in DAP-way
Plus, tweaks for OPeNDAP Clients
Plus, tweaks for HDF-EOS5 Files
And, we built the HDF5 server to provide a standard DAP-way of accessing HDF5.
Plus, we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

24. Subway Customers I want 3 inch-long I need Wasabi sauce
I eat only wheat bread
I eat only meat balls
So we could handle some strange situation that clients mandate by providing options.
Going back to the Subway customer scenario, bottom two requests from a customer do not seem to have any problems.

25. Subway Suppliers Problem
Subway Customers - OK
I eat only wheat bread
I eat only meat balls
Subway Suppliers Problem
However, what if Subway suppliers have problems in their factories and can’t provide what Subway want? We’re in big trouble.
No breads, only wheat!
No meat balls, only meat!

26. HDF EOS Grid with No Dimension data
Not all HDF5 files are created for DAP-way!
Grid with No Dimension data
Clients expect Grid with Dimension data
Metadata Attribute in Two Strings
Clients expect One Structured Format
This kind of situation happened with NASA HDF-EOS case.
The dataset in HDF EOS files that NASA produce cannot be served directly in DAP-way.
Since HDF5 data producer like NASA doesn’t have to keep DAP-way in mind, they can create HDF5 files in any way they want.

27. Tweaks for HDF-EOS Two more configuration options
--enable-eos-grid = bake bread
--enable-eos-meta = make balls
Thus, we believe it’s our job to make the raw HDF5 data into a new form that clients will like.
--enable-eos-grid processes the raw data into Grid that can client can consume easily.
--enable-eos-meta chops the long string into a better format that can client can handle.

28. OPeNDAP HDF5 Server Map HDF5 Datatypes in DAP-way
Goal: Transform HDF5 access in DAP-way
Map HDF5 Datatypes in DAP-way
Plus, tweaks for OPeNDAP Clients
Plus, tweaks for HDF-EOS5 Files
And, we built the HDF5 server to provide a standard DAP-way of accessing HDF5.
Plus, we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

29. Day After Server Tweaks
Finally, Happy Clients!
ncBrowse
Ferret
MATLAB
NCL
IDV
So after applying the tweaks for clients and hdf-eos, our server can serve many clients. Here’s a live demo and movie.
GrADS
ODC

30. Live Demo Ozone concentration level over the south pole
I’m going to do live demo with one visualization client.
This movie is generated from the real NASA data.

31. Goals 1st: Transform HDF5 access in DAP-way
2nd: Yet Preserve the beauty of HDF5
So we could achieve the first goal.
Achieving the first goal has some side effects and give another challenge of preservation.

32. (Yet Preserve the beauty of HDF5)
HDF5 Served in DAP-way
(Yet Preserve the beauty of HDF5)
There are some things money can’t buy. For everything else, there is MasterCardTM.
Minimize
There are some things DAP can’t serve. For everything else, there is _________?
If you ever watched the “priceless” MasterCard commercial, you’ve heard this famous phrase:
“There are …. “.
We have found a similar situation when we tried to serve HDF5 in DAP-way.
If HDF5 is served in DAP-way, it was inevitable to lose some details that the original HDF5 has simply because DAP doesn’t know how to represent.
However, we think it would be great if we can minimize such things and find a clever solution like MasterCard that can cover and deliver maximum features in HDF5 through DAP-way.
Maximize

33. Some Things
(that OPeNDAP HDF5 server can’t serve)
Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types
Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5
Here are some limitations that the current server can’t handle.
Certain types like Opaque, Bitmap, Enum are not available in the current DAP protocol so it’s impossible to map them.
Another interesting case is that there are some reserved characters in DAP yet they are perfectly legal in HDF5.
In these cases, DAP protocol itself can be changed to have new additional types or use special hex encoding of characters.

34. For Everything Else Let there be HDF5-friendly OPeNDAP client library!
Package and Deliver in DAP-way first.
Then, let the client library handle it.
Concept of Trojan Horse
However, it’s hard to modify the protocol itself since it’s standard so
this is our current solution for everything else in HDF5.
We’d like to make a special HDF5-friendly OPeNDAP client dedicated for our OPeNDAP HDF5 server.
The concept is quite simple. We simply package and deliver them in DAP-way and let the client handle it.

35. Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
Example: Group in HDF5
HDF5-friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
Here’s really one good example.
In DAP, an attribute plays a role like comment in computer programming.
Although DAP doesn’t have any concept of group, the group may play an important role in HDF5 like disambiguation of same variable name.
This entire group structure is being sent as a single attribute in DAP.
Thus, if a user wants to re-construct HDF5 using DAP client, this group attribute is essential and HDF5-friendly OPeNDAP client library should handle it properly.

36. Example: Reference in HDF5
Object / Regional Reference
Map to DAP URL at server
No de-referencing of URL at client library
Important for NPOESS
Another good example is the reference in HDF5.
There are two types of reference in HDF5.
Both object and regional reference can be mapped to a special data type called URL in DAP.
However, the current OPeNDAP client library doesn’t supprot the de-referencing of the URL.
That is, there is no way for a OPeNDAP client can access the dataset that the URL points to.
This is particularly important for NPOESS data since it uses tons of HDF5 reference inside.

37. HDF5-Friendly OPeNDAP Client Library
No Latitude and Longitude
One more reason: Tame Clients
Only ODC can display Swath properly.
Easy but Evil OPeNDAP NC Client Library!
easy: nc style API
evil: fixed dimensional attributes
Our client library must be Easy but Good
There’s another reason that we want to pursue HDF5-friendly OPeNDAP client library.
A swath is a 3-D scan of very small region on the earth.
Earth Scientists are not much interested in seeing 2-D Grid data but a vertical profile of a swath like this.
However, among the six OPeNDAP clients that we tried, only one client could display a vertical profile of swath data properly.
The main reason is that many DAP clients do not use generic OPeNDAP client library.
They use a well-established NetCDF style API of OPeNDAP client library which impose extra unnecessary restrictions in DA Protocol. Visualization tool developers used it because it’s quick and easy to adapt into their existing software.
Thus, our goal is to build a client library that is similar to NetCDF API yet doesn’t put any restrictions.
Courtesy of NASA

38. NC-Friendly OPeNDAP Client Library
HDF5-Friendly OPeNDAP Client Library
View G/R/Swath
???
View NetCDF
Group/Ref/Swath
NetCDF
View HDF5
HDF5
GrADS
GrADS
OPeNDAP
HDF5 Server
OPeNDAP
NC Server
liboc-dap
libnc-dap
dapserver
However, it’s hard to modify the protocol itself since it’s standard so
this is our current solution for everything else.
We’d like to make a special DAP client dedicated for OPeNDAP HDF5 server.
The concept is quite simple. We simply package and deliver them in DAP-way and let the client handle it.
We want to provide a more generic OPeNDAP client library.
libdap

39. Summary DAP-way access of HDF5 Is Easy / Efficient / Cool
Loses some things
Requires HDF5-friendly OPeNDAP Client Library
HDF5-friendly OPeNDAP Client Library
Serves HDF5 better
Tames evil visualization clients
Here’s a summary of our project.
Providing a DAP-way access of HDF5 is easy, efficient and cool.
However, it can lose some information that HDF5 originally has.
This requires either the modification of DAP or the creation of HDF5-friendly client.
Since it’s hard to modify the well-established DAP, we think it’s easier to implement HDF5-friendly OPeNDAP client library.
When it’s done, we believe it will serve HDF5 better and tame the visualization clients that abuse the easiness of NetCDF-friendly OPeNDAP client library.

40. Future Work HDF5 DAP Mapping Document
Finish HDF5-friendly OPeNDAP Client Library prototype
Test it on GrADS and display Swath
Here’s our future work.
First, we’ll give a detailed document on mapping between HDF5 and DAP.
Second, we’ll finish HDF5 client library prototype and test it with one client.
This will be only prototype due to the limited funding availability.
We’ll replace part of GrADS source code that refers libdap-nc so that it can display a swath like ODC did.
According to NASA people, GrADS is most popular.
Also, it’s open source software.

41. Project Website http://hdfdap.hdfgroup.uiuc.edu/joomla Beta preview
Feedbacks are more than welcome!
This is our new project website that everyone is welcome to visit
and give some feedback on us as a sneak previewer.

42. Credits Mike Folk (THG) Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)
Christopher Lynnes, James Johnson, Denis Nadeau (NASA)
Jennifer Adams (GrADS)
Dave Brown (UCAR)
We’d like to thank these people.
They have provided us the right direction in development, an early access to data files and numerous feedback.