The HDF Group Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Ruth Duerr, NSIDC Christopher Lynnes, GES DISC Mike Folk, Kent Yang, Peter Cao, The HDF Group November 3-5, 20091HDF/HDF-EOS Workshop XIII

HDF4 files are complex November 3-5, 2009HDF/HDF-EOS Workshop XIII2

November 3-5, 2009HDF/HDF-EOS Workshop XIII3 How do we save HDF users from having to deal with all of the complexity under the hood?

November 3-5, 2009HDF/HDF-EOS Workshop XIII4 Through the HDF software libraries, either by using the HDF APIs directly or by using HDF tools that depend on the HDF libraries. But what about the future…

November 3-5, 2009HDF/HDF-EOS Workshop XIII5 There is a risk in depending solely on HDF libraries to access HDF- formatted data over the long term. It is possible, especially in the distant future, that the libraries may not be available.

November 3-5, 2009HDF/HDF-EOS Workshop XIII6 read HDF data with an independent program that does not rely on the HDF API… If only we could read HDF data with an independent program that does not rely on the HDF API… A possible approach [would be to create] a map of a data file, [and] utilities to find, assemble and write out SDSes and vdatas. Leveraging HDF Utilities Christopher Lynnes HDF Workshop X.

Users view of the HDF4 SD model November 3-5, 2009HDF/HDF-EOS Workshop XIII7

Mapping SDS to file offset/length November 3-5, 2009HDF/HDF-EOS Workshop XIII8 HDF4 file layout

Mapping with chunks November 3-5, 2009HDF/HDF-EOS Workshop XIII9 HDF4 file layout

Recap Problem The complex byte layout of HDF files makes long-term readability of HDF data dependent on long-term availability HDF software. Solution Create a map of the layout of data objects in an HDF file, allowing a simple reader to be written to access the data. November 3-5, HDF/HDF-EOS Workshop XIII

The HDF Group The project – phase November 3-5, 2009HDF/HDF-EOS Workshop XIII11

HDF4 mapping project activities 1.Assess and categorize HDF4 data held by NASA Determine what types of objects to map. Get an idea of the magnitude of the project. 2.Develop prototype for proof of concept Develop markup-language based layout specification. Develop tool to produce layout for an HDF4 file. Develop and test two independent tools to read HDF4 data based solely on the map files November 3-5, HDF/HDF-EOS Workshop XIII

HDF/HDF-EOS Workshop XIII How many HDF4 products? Data CenterHDF4 Products ASF0 GES-DISC236 GHRC54 ASDC63 LP-DAAC67 NSIDC47 ORNL-DAAC2 PO.DAAC22 SDAC0 MrDC95 Total586 November 3-5,

HDF/HDF-EOS Workshop XIII Data characteristics Product Identification Product Name Data Level Archive Location Product Version Whether the product was multi-file For HDF-EOS products HDF-EOS version For point data Number of point data sets Maximum number of levels For swath data Number of swaths Maximum number of dimensions Organized by time, space, both, or other Whether dimension maps were used For gridded data Number of grids Max number of dimensions in a grid Number of projections used Whether any grids were indexed HDF Version For raster data Number of 8-bit rasters Number of 24-bit rasters Number of general rasters Whether any rasters had attributes Whether any rasters were compressed Whether any rasters were chunked Whether there were any palettes For SDS data Number of SDSs Maximum number of dimensions Did any SDS have attributes Was any SDS annotated Were dimension scales used Was compression used and if so what kind Was chunking used For Vdata Number of Vdata structures Did any Vdata have attributes Did any Vdata fields have attributes Was compression used and if so what kind Was chunking used Product Characteristics Examined November 3-5,

HDF4 mapping prototype workflow November 3-5, 2009HDF/HDF-EOS Workshop XIII15 HDF4 File H4.hdf HDF4 File H4.hdf HDF4 Mapping File (XML document) H4.hdf.map.xml HDF4 Mapping File (XML document) H4.hdf.map.xml hmap linked with HDF4 library hmap linked with HDF4 library Reader 1 (C program) Object Data Groups, Data Objects, Structural and Application Metadata; Locations of Object Data Reader 2 (Perl Script) Reader 2 (Perl Script)

The HDF Group Phase 2: Productizing HDF4 Mapping schema and tools for deployment November 3-5, 2009HDF/HDF-EOS Workshop XIII16

Phase 2 tasks Revise schema Investigate integration of mapping schema with existing standards Analyze whats needed to include HDF-EOS 2 Revise the XML schema Implement production quality HDF4 map writer Develop demo HDF4 map reader Deploy Optional tasks Implement general purpose reader Develop validation utilities November 3-5, HDF/HDF-EOS Workshop XIII

How you can help Project page at The HDF Group website: Consider what it might take to implement this for your archive - contact us if youd like support Let us know if you are interested in participating in any capacity. November 3-5, HDF/HDF-EOS Workshop XIII

The HDF Group Thank You! November 3-5, 2009HDF/HDF-EOS Workshop XIII19

Acknowledgements This work was supported by cooperative agreement number NNX08AO77A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. November 3-5, 2009HDF/HDF-EOS Workshop XIII20

The HDF Group Questions/comments? November 3-5, 2009HDF/HDF-EOS Workshop XIII21

Example map fragment November 3-5, HDF/HDF-EOS Workshop XIII