1. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO Proposal for adding Named Dimensions to HDF5 Arrays Daniel Kahn Science Systems and Applications, Inc. HDF/HDF-EOS Meeting Oct 15-17 th, Aurora CO

2. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO Motivation: The Ozone Mapping and Profiler Suite (OMPS) data produced by the O 3 PEATE will be using HDF5, but not HDF-EOS5. Features of HDF-EOS5 needed by the PEATE are easily represented in HDF5, with the exception of named array dimensions. This talk describes a proposed method to add this functionality to HDF5 and also shows how named dimensions could be used to describe other types of useful data structures.

3. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO Prior Art: HDF-EOS5 Swaths An HDF-EOS5 Swath consists of a collection of arrays, generally as a result of a satellite remote sensing measurements (i.e., an orbit's worth of data). Rectangular array of ozone data projected on Earth.

4. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO The HDF-EOS5 Swath consists of a 1) list of named dimension (Name-Value pairs) metadata. 2) set of arrays whose axes are defined by the named dimensions. HDF-EOS5 achieves this via a simple API, kept simple by restricting the user from defining HDF5 group hierarchies. File Root Swath 1 Dataset1Dataset2 Swath 2 etc. Dim Metadata HDF-EOS5 Swath Cartoon Diagram Dim Metadata

5. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO Goal: Offer a corresponding facility under HDF5 1) without sacrificing the ability to define arbitrary hierarchies and 2) using publicly defined attributes associated directly with the HDF5 objects they help to describe.

6. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO Named Dimensions vs Numerical Dimensions HDF5 uses numerical dimensions: hid_t H5Screate_simple(int rank, const hsize_t * dims, const hsize_t * maxdims)‏ dims is an array of numbers. A named dimension is a Name-Value pair used to define arrays. The equivalent statement using named dimensions would look like: H5ext_create_simple(rank, char **DimNames)‏

7. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO A challenge in defining named extents in HDF5 is that we need some idea of the “scope” of the names within the HDF5 Group hierarchy. G G GG* G NameValue1 NameValue2 etc. Should these groups see the Extent Names in G*? Presumably not, but some kind of inheritance would be nice. Attributes

8. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO G+G+ G GG* G NameValue1 NameValue2 etc. NameValue3 NameValue4 etc. We can define inheritance in which datasets in G* could use Named Dims from G* and also from extents inherited from G* parent, G +, and so on up the hierarchy. Attributes

9. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO What would the code look like? Group = H5Gcreate(“/”,”G1”)‏ H5ext_CreateNamedExtent(Group,“Name1”, Value)‏ etc. char **Names = {“Name1”,”Name2”,”Name3”} ret= H5ext_CreateDataSet(Group, rank, Names, ArrayPointer)‏ This searches up the hierarchy for names. if(ret == Error) then Dimension Name not found in hierarchy. Normal HDF5 writing routines. (close, close, close, etc.)‏ (next, side benefits)‏

10. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO Additional Benefits: Dimension Scale-like capability built in… Data Scale DimName1 …and the Named Dimension also open the possibility to have scales for incommensurate mappings. Data Scale DimName1 DimName2 DimName1 Index Map Index Map contains indexes of DimName2

11. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO They seem equivalent but.... The Named dimension approach also appears to be isomorphic to compound datatypes. Field 3Field 2Field 1 DimName1 Rows Field 3Field 2Field 1 DimName1 Rows Three value compound datatype DimName1 Measurement Group Three ordinary arrays of basic type One array of compound type

12. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO With the Named Dimension approach fields can be added or deleted trivially and the field elements themselves can easily have other dimensions. The target language can read/write the data without any dependence on the C language implementation of structures. Field 4 DimName1 DimName2 + Field 3Field 2Field 1 DimName1 Rows DimName1

13. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO Implementation Thoughts: Most likely implementation would be to publicly define HDF5 attributes to represent the metadata and write C interface to simplify programing. Ideas or comments are welcome and needed, in particular on any gaps or problems with the proposed design. Potential Pitfalls: Order of Dimension Names is language dependent. Need bi-directional parent-child relationships. others?

14. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO Conclusion: The proposed method, called “Named Dimensions”, is expected to provide equivalent data representation to the named dimensions of HDF-EOS5 in vanilla HDF5. In addition, other benefits may be realized, including full use of the HDF5 group hierarchy, a flexible Dimension Scale-like capability and an alternative to representing data in compound data types without depending on C style structures. The Named Dimensions facility needs to be implemented on top of HDF5 with no changes to the base library. (i.e., No PEATE branch of HDF5)‏ using publicly defined attributes. Acknowledgment: This work was carried out under NASA contract NNG06HX18C, task 614.5-01-06

15. HDF/HDF-EOS Meeting Oct 15-17 th 2008, Aurora CO More Detailed Implementation Thoughts: Group DataSet DimNames: {“Width”,”Height”,”Length”} Group Attributes DataSet Attributes 16 100 40