Towards Long-Term Archiving of NASA HDF-EOS and HDF Data Data Maps and the Use of Mark-Up Language Ruth Duerr, Mike Folk, Muqun Yang, Chris Lynnes, Peter Cao

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Outline Background Data Mapping Project Description Plans and Early Results

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Outline Background Data Mapping Project Description Plans and Early Results

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland A Concern The majority of the data from NASA’s Earth Observing System (EOS) have been archived in HDF Version 4 (HDF4) or HDF-EOS 2 format. HDF files have a complex internal byte layout, requiring one to use the API to access HDF data Long-term readability of HDF data depends on long-term allocation of resources to support the API

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland A Proposal from the Workshop Last Year Chris Lynnes noted that  What was needed was a map to the contents of an HDF file  The output of the HDF4 tools (e.g., hdfls, hdp, etc.) already provide much of the information needed  Extending these tools to create a map to the contents of the file might be feasible

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Outline Background Data Mapping Project Description Plans and Early Results

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Data Mapping Project Description Assess and categorize NASA holdings of HDF4 data Investigate methods of mapping HDF4 files Develop requirements for tools to create maps of HDF4 files Create a prototype tool to create maps Test the utility of these maps by developing 2 independent tools that use the maps to read real data

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Data Mapping Project Description (continued) Assess the utility of this approach Document our findings Present results and options for proceeding to the user community Evaluate the effort required for a full solution that meets community needs Submit a proposal for that effort

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Outline Background Data Mapping Project Description Plans and Early Results

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Assess and Categorize NASA Holdings While the volume of NASA data stored in HDF4/HDF-EOS2 format is measured in PB; the fraction of the total number of NASA data sets archived in HDF4/ HDF-EOS2 is “small” NASA provided a starter list of data sets held NASA data centers were requested to provide a list at a project briefing Results from each DAAC being compared to ECHO assessment of data sets using a.hdf extension

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Assess and Categorize NASA Holdings (continued) Examples of each of the hdf4 data sets have been obtained and examined* Information kept summarized below: Product id/name Data Center Product Version Multi-file product? HDF/EOS info (if any)  HDF/EOS version  Point info  Swath info  Grid info HDF info  Version  Raster image info  Palette  SDS info  V data info  Annotation * For the most part

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Assess and Categorize NASA Holdings (continued) Very preliminary findings  Roughly 50/50 split between HDF-EOS and plain HDF  Point data is relatively rare and when found is not accompanied by swath or grid data  No indexes yet  While a few products use the image types, there are no palettes yet

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Investigate Methods of Mapping HDF4 Files NSIDC and GES-DISC have provided THG sample data files Preliminary priorities for capabilities to tackle:  Contiguous SDS  Contiguous SDS with unlimited dimension  Chunked SDS  Compressed SDS  Chunked and compressed SDS  SDS and attributes  Vdata and attributes  Annotation  Vgroup  Raster image and attributes

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Investigate Methods of Mapping HDF4 Files NSIDC and GES-DISC have provided THG sample data files Preliminary priorities for capabilities to tackle:  Contiguous SDS  Contiguous SDS with unlimited dimension  Chunked SDS  Compressed SDS  Chunked and compressed SDS  SDS and attributes  Vdata and attributes  Annotation  Vgroup  Raster image and attributes

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Develop Requirements for Tools to Create Maps Maps will be XML-based A draft of a map format specification has been started

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Create a Prototype Tool to Create Maps An iterative process is being used to create the prototype Each iteration adds the next capability from the prioritized list shown earlier At this point, the tool just creates a text description

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Communications Plan Bi-weekly telecons with our sponsors (may move to monthly) Briefing to NASA Data Center managers held, expect to provide periodic updates Brief community at the HDF-Workshop and other relevant meetings (e.g., AGU) Submit a paper to the special issue of IEEE Transactions of Geoscience and Remote Sensing devoted to Data Archiving and Distribution Public wiki established but not yet populated

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007 Landover, Maryland Summary We’ve started a project to assess and prototype the ability to create maps to the contents of HDF4 files that allow programmers to develop code to read data without using the HDF APIs We welcome community involvement