1. University of Illinois at Urbana-ChampaignHDF - 1 - Mike Folk HDF-EOS Workshop IV Sept. 19-21, 2000 HDF Update HDF

2. University of Illinois at Urbana-ChampaignHDF - 2 -Topics What is HDF? HDF community HDF4 –past and future work HDF5 –HDF5-HDF5 interoperability –Activities and growth –Tools and utilities –Focus for 2001

3. University of Illinois at Urbana-ChampaignHDF - 3 - NCSA HDF Mission To develop, promote, deploy, and support open and free technologies that facilitate scientific data storage, exchange, access, analysis and discovery.

4. University of Illinois at Urbana-ChampaignHDF - 4 - What is HDF? Format and software for scientific data Stores images, arrays, tables, etc. Emphasis on storage and I/O efficiency Free and commercial software support Emphasis on standards Users from many engineering & scientific fields

5. University of Illinois at Urbana-ChampaignHDF - 5 - HDF People NCSA –QA, maintanance & support –Library development & performance –Tools development EOS & ASCI –Users, applications and tools –Performance Volunteers from all over –Users, applications & tools

6. University of Illinois at Urbana-ChampaignHDF - 6 - Who is supporting HDF? NASA/ESDIS –Earth science applications, instrument data –All aspects of data management DOE/ASCI (Accelerated Strategic Computing Init.) –Simulations on massively parallel machines –Emphasis on parallel I/O performance, functionality NCSA –Grid, Vis, other R&D, user support Others –Applications, support, some R&D

7. University of Illinois at Urbana-ChampaignHDF - 7 -HDF4 User support, tools, maintenance, QA JHV (Java viewer) - two update releases HDF4 dumper (hdp) - many improvements Next HDF4 Release (Oct. 2000) –Bug fixes –Chunking for General Raster (GR) images –Unlimited dimensions for SDS –GIF HDF4 conversion tools

8. University of Illinois at Urbana-ChampaignHDF - 8 - HDF4 Archiving New File Format and Specification Manual –Facilitate long term maintenance of HDF4 –Necessary for bug fixing, adding new features –Decreases learning curve for new developers –Available soon in PDF, HTML and PS formats

9. University of Illinois at Urbana-ChampaignHDF - 9 - HDF5

10. University of Illinois at Urbana-ChampaignHDF - 10 - The growth of HDF5 Users in 27 countries Users include –Government agencies –National labs –Companies –Universities

11. University of Illinois at Urbana-ChampaignHDF - 11 - The growth of HDF5 Scientific fields –Astronomy, astrophysics, aerospace engineering –Geophysics, remote sensing, meteorology, –Oceanography, environmental Science –Informational Science –Medical Research (brain, cancer, biotech) –Product model data

12. University of Illinois at Urbana-ChampaignHDF - 12 - Facilitating interoperability between HDF4 and HDF5 HDF4 to HDF5 mapping specification –“Mapping HDF4 Objects to HDF5 Objects” http://hdf.ncsa.uiuc.edu/HDF5/papers –Rules for mapping high level HDF4 objects to HDF5 How to describe HDF4 objects in HDF5 How to interpret HDF5 objects as HDF4 objects HDF4-to-HDF5 conversion software (later) Space Research, Inc. Explorer 1.1 –reads both HDF4 and HDF5

13. University of Illinois at Urbana-ChampaignHDF - 13 - HDF5 Activities in 2000 HDF5 1.2.2 library release Fortran 90 & C++ API HDF5 Abstract Data Model XML Document Type Definition (DTD) HDF5 tools Support for users and application developers HDF5 Tutorial Searchable, printable documentation

14. University of Illinois at Urbana-ChampaignHDF - 14 - Virtual file I/O (C only)  Perform byte-stream I/O operations (open/close, read/write, seek)  User-implementable (I/O to stdio, network, memory, etc.) Library internals Performs data transformations and other prep for I/OPerforms data transformations and other prep for I/O Configurable transformations (compression, etc.)Configurable transformations (compression, etc.) Structure of HDF5 Library Object API (C, Fortran 90, Java, C++)  Specify objects and transformation properties  Invoke data movement operations and data transformations

15. University of Illinois at Urbana-ChampaignHDF - 15 - Tools & Utilities NCSA –Java wrapper for HDF5 –H5View: Java browser/Editor for HDF5 –H5gen: XML-to-HDF5 file generator (Java) –H5dump & H5ls Others –VisAD data adapter for HDF5 (Java toolkit) –HDF Inspector/Explorer –Open Data Explorer (IBM) –Ensight

16. University of Illinois at Urbana-ChampaignHDF - 16 -H5View

17. University of Illinois at Urbana-ChampaignHDF - 17 -H5View Java-based tool for browsing and editingJava-based tool for browsing and editing Display structure of fileDisplay structure of file Display content of objectsDisplay content of objects Create and delete objectsCreate and delete objects Modify values of objects and attributesModify values of objects and attributes

18. University of Illinois at Urbana-ChampaignHDF - 18 - HDF5 XML DTD A flexible standard language that can describe an HDF5 file in precise detail –Datasets, dataspaces, datatypes –Groups and links, structure of the file –Values of the attributes and data http://hdf.ncsa.uiuc.edu/HDF5/XML

19. University of Illinois at Urbana-ChampaignHDF - 19 - H5Gen: XML  HDF5 Reads an XML description of an HDF5 file Generates the corresponding HDF5 file Validates XML description vs. HDF5 DTD

20. University of Illinois at Urbana-ChampaignHDF - 20 - Some uses of XML View HDF5 files using a web browser XMLtoHTML Java viewers,editors, other tools XML to Java Java to XML Description in XML using HDF5 DTD HDF5 file XML to HDF5 (H5gen) HDF5 to XML Generate, validate, reconstruct HDF5 files Data location services Catalog records in XML Archive XML documentation of metadata & structure

21. University of Illinois at Urbana-ChampaignHDF - 21 - Focus areas for 2001 Support Terra & Aqua Get ready for Aura Support HDF-EOS 3 Enhance HDF5View Refine XML DTD & design tools around it HDF5 converters –HDF5  XML –HDF4-to-HDF5 –Others (e.g. GIF  HDF5)

22. University of Illinois at Urbana-ChampaignHDF - 22 - Focus areas for 2001 Expand list of applications and users Facilitate access to other tools & software Get vendors on board Performance testing and tuning Extend API with ease-of-use functions Clusters and other new environments Implement a thread-safe version HDF5 advanced tutorial

23. University of Illinois at Urbana-ChampaignHDF - 23 - Thank you! HDF website –http://hdf.ncsa.uiuc.edu/ HDF5 Information Center –http://hdf.ncsa.uiuc.edu/HDF5/ HDF Helpdesk –hdfhelp@ncsa.uiuc.edu HDF users mailing list –hdfnews@ncsa.uiuc.eduHDF5