1. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term

2. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 2 Outline The Long Term Problem A Few Ideas Discussion

3. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 3 The Digital Preservation Challenge “digital objects require constant and perpetual maintenance, and they depend on elaborate systems of hardware, software, data and information models, and standards that are upgraded or replaced every few years” NSF and Library of Congress, August 2003

4. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 4 What is a Good Long-Term Archive Format? Per a recent paper by Mike Folk and Bruce Barkstrom  Ease of archival storage  Ease of archival access  Usability  Data scholarship enablement  Support for data integrity  Maintainability and durability

5. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 5 What Makes a Good File Format? Per Eric Raymond, in “The Art of Unix Programming”, Addison-Wesley, 2004  Transparency  Interoperability  Extensibility  Storage economy He argues that the best general purpose file format is text He also argues that the only good justification for binary data is with very large data sets

6. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 6 Disaster-Proofing Your Data If you can’t keep a data set as text, then at least keep the representational information in a human readable format (preferably right with the data itself)

7. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 7 HDF and HDF-EOS as an Archive Format Neither are optimized for archival  Not a text-based format  Does not enforce inclusion of semantic metadata  Meant to be used with the HDF* libraries

8. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 8 Towards the Future? Discussions with Don Sawyer, Lou Reich, Mike Folk and others at NCSA A Few Ideas  Specifying an HDF-archive format a’la PDF-A  Text wrapper/encoding for HDF  Tools to translate to/from an archive format Mechanisms?  CODATA  International workshop  Proposals

9. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 9 Discussion Any suggestions for moving forward? Questions? Concerns?

10. HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, 2005 10 Contact Info For questions about data preservation, etc. contact me at rduerr@nsidc.orgrduerr@nsidc.org For information about NSIDC data products, programs, etc. see http://nsidc.org or contact our user services at nsidc@nsidc.org http://nsidc.org