October 15, 2008HDF and HDF-EOS Workshop XII1 What will be new in HDF5?

HDF5 Road Map October 15, 2008HDF and HDF-EOS Workshop XII2 Performance Robustness Ease of use Innovation

October 15, 2008HDF and HDF-EOS Workshop XII3 Outline Performance improvements Fortran 2003 features HDF5 file recover (metadata journaling)

October 15, 2008HDF and HDF-EOS Workshop XII4 Performance Improvements

Performance Improvements in HDF5 Examples of completed work: New implementation of metadata cache to improve I/O performance and memory usage when accessing many objects (HDF ) Faster, more scalable storage and access for large groups (HDF ) October 15, 2008HDF and HDF-EOS Workshop XII5

Performance Improvements in HDF5 Work in progress New implementation of free-space management Affects “dynamic” applications that add/delete/modify existing objects When creating HDF5 objects, space is allocated from available space tracked by the free-space manager When deleting objects, unused space is added to the free-space pool via free-space manager Current implementation uses O(N 2 ) operations for each N allocations or freeing space New implementation O(log 2 N) October 15, 2008HDF and HDF-EOS Workshop XII6

Free-space Management Test: creating/deleting attributes Create first set of attributes Delete odd-numbered attributes from the first set Create second set of attributes Delete all attributes from the second set October 15, 2008HDF and HDF-EOS Workshop XII7 Number of attributes Old implementation New implementation Improvement ratio sec68.2 sec11.5x sec289 sec38x

Performance Improvements in HDF5 Work in progress Fast data append (along slowest changing dimension) Future areas of interest Efficient chunking cache implementation (NetCDF4) Efficient handling of the variable-length data including compression (will affect sizes of NPOESS files) October 15, 2008HDF and HDF-EOS Workshop XII8

October 15, 2008HDF and HDF-EOS Workshop XII9 Fortran 2003 features

Status of the HDF5 Fortran Library HDF5 Fortran library is a part of standard HDF5 distribution First release goes back to 1999 Implemented as Fortran90 wrappers on top of the HDF5 C library Supported on Linux, Windows, Mac Intel, Solaris, VMS, clusters, etc. Compilers Open source gfortran, g95 Vendors (SUN, Intel, PGI, Absoft) 32 and 64-bit versions October 15, 2008HDF and HDF-EOS Workshop XII10

HDF5 Fortran Library Mimics HDF5 C APIs Fortran 90 features used Modules Function overloading Function interfaces Dynamic memory allocation Optional parameters Many Fortran APIs are simpler than their C counterparts October 15, 2008HDF and HDF-EOS Workshop XII11

Current Limitations Supports only “native” Fortran types such as INTEGER REAL CHARACTER DOUBLE PRECISION (obsolete Fortran feature) No support for INTEGER*1, INTEGER*2, INTEGER*8, INTEGER*16, REAL*8, REAL*16 Cannot write/read buffers of those types Fortran types have to match C types No support for –r8 and –r16 flags (Fortran real =/= C float) October 15, 2008HDF and HDF-EOS Workshop XII12

Current Limitations No support for INTEGER(KIND=n) and REAL(KIND=m) Integers n and m are called KIND parameters Returned by selected _int_kind (r) -10 r < n < 10 r selected_real_kind(p,r) p – precision r – decimal exponent range October 15, 2008HDF and HDF-EOS Workshop XII13

Current Limitations Limited support for derived types (compare with C structures and HDF5 compound datatypes) Supports derived types with “native” fields only Doesn’t support complex HDF5 datatypes (e.g., with array member, or nested compound) Writes/reads HDF5 compound datasets by fields only Cannot be used in parallel applications No support for enum types No support for callback functions October 15, 2008HDF and HDF-EOS Workshop XII14

Current Limitations - Summary Any application written according to Fortran 95/2003 standard will struggle using HDF5 Fortran Library Many HDF5 features are not available to Fortran applications October 15, 2008HDF and HDF-EOS Workshop XII15

Fortran 2003 Features Fortran 2003 provides a standardized mechanism for interoperating with C Module ISO_C_BINDING for interoperability of intrinsic types C_PTR and C_FUNCPTR for interoperability with C pointers C_LOC(x) and C_FUNLOC(x) inquiry functions for getting addresses of variables and procedures BIND attribute for interoperability of derived types and C structures and enumerated types October 15, 2008HDF and HDF-EOS Workshop XII16

Fortran 2003 Features and HDF5 New 2003 features allowed us to support Any Fortran INTEGER and REAL type data in HDF5 files Fortran derived types and HDF5 compound datatypes Fortran enumerated types and HDF5 enumerated types HDF5 APIs with callbacks October 15, 2008HDF and HDF-EOS Workshop XII17

Fortran Compilers with 2003 Features CompilerCurrent versions and status of HDF5 Future versions IntelVersions 10.1 and 11 All F2003 functionality works in HDF5 Version 11 will have a fix that will allow us to remove common blocks g95August 2008 and later; All Fortran 2003 functionality works in HDF5 gfortranVersion 4.4 Limited support for C interoperability; HDF5 doesn’t work No plans from compiler developers to improve the support SUN compilersExpress-July 2008 build; HDF5 doesn’t work No timeline for fixes; may be in a year PGIVersion 7.2.1; HDF5 doesn’t work Fixes will be available in Version 8. October 15, 2008HDF and HDF-EOS Workshop XII18

Example October 15, 2008HDF and HDF-EOS Workshop XII19

Example October 15, 2008HDF and HDF-EOS Workshop XII20

Example October 15, 2008HDF and HDF-EOS Workshop XII21 HDF5 "SDScompound.h5" { GROUP "/" { DATASET "ArrayOfStructures" { DATATYPE H5T_COMPOUND { H5T_ARRAY { [13] H5T_STRING { STRSIZE 1; STRPAD H5T_STR_SPACEPAD; CSET H5T_CSET_ASCII; CTYPE H5T_C_S1; } } "chr_name"; H5T_STD_I8LE "a_name"; H5T_IEEE_F64LE "c_name"; H5T_IEEE_F32LE "b_name"; } …..

Information October 15, 2008HDF and HDF-EOS Workshop XII22

23 HDF5 file recovery or Surviving a System Failure through Metadata Journaling October 15, HDF and HDF-EOS Workshop XII

October 15, 2008HDF and HDF-EOS Workshop XII24 Surviving a System Failure in HDF5 Problem: Data in an opened HDF5 files susceptible to corruption in the event of an application or system crash. Corruption possible if an opened HDF5 file has been updated when the crash occurs. Initial Objective: Guarantee an HDF5 file with consistent metadata can be reconstructed in the event of a crash. No guarantee on state of raw data – contains whatever made it to disk prior to crash.

October 15, 2008HDF and HDF-EOS Workshop XII25 Crash Survivability in HDF5 Approach: Metadata Journaling When an HDF5 file is opened with Metadata Journaling enabled, a companion Journal file is created. When an HDF5 API function that modifies metadata is completed, a transaction is recorded in the Journal file. If the application crashes, a recovery program can replay the journal by applying in order all metadata writes until the end of the last completed transaction written to the journal file.

Oct HDF and HDF-EOS Workshop XII26 Application crashed HDF5 Metadata Journaling Recovery Restored HDF5 File h5recover tool liFe Corrupted 5DFH Companion Journal File

October 15, 2008HDF and HDF-EOS Workshop XII27 Implementation Status Serial HDF5 with synchronous write mode Alpha1 released August 2008 User interface (API definition and h5recover tool) and file format may change

October 15, 2008HDF and HDF-EOS Workshop XII28 Metadata Journaling Plans Serial HDF5 with synchronous write mode Finalize User interface definitions and file format Serial HDF5 with asynchronous write mode To improve Journal file write speed More features (need funding) Make raw data operations atomic Allow "super ‐ transactions" to be created by applications Enable journaling for Parallel HDF5

October 15, 2008HDF and HDF-EOS Workshop XII29 Questions?

Acknowledgement This report is based upon work supported in part by a Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA Awards NNX06AC83A and NNX08AO77A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration. October 16, HDF and HDF-EOS Workshop XII