1. Moving applications to HDF5 1.8.0
12/25/2018
HDF and HDF-EOS Workshop XI

2. HDF and HDF-EOS Workshop XI
Outline
Quick overview of new features
Move or not to move?
HDF5 APIs versioning
HDF5 library configuration
HDF5 application configuration
Examples
12/25/2018
HDF and HDF-EOS Workshop XI

3. HDF and HDF-EOS Workshop XI
Goal of the tutorial
Raise awareness about forward/backward compatibility issues with the new release
Help with transition to the new release
Get feedback from you before official release is done
12/25/2018
HDF and HDF-EOS Workshop XI

4. Current status of the HDF5 1.8.0 release
HDF beta4 was released on October 22, 2007
Stable file format and APIs
Has been successfully tested with NetCDF-4 and HDF-EOS5 on many supported platforms
Tasks left before the full release
Bugs fixing
Incorporating contributed features
Performance tuning (more will follow)
Documentation
Thorough backward/forward compatibility testing for APIs and files format
12/25/2018
HDF and HDF-EOS Workshop XI

5. HDF and HDF-EOS Workshop XI
Quick overview of new features in HDF and their affect on current applications
12/25/2018
HDF and HDF-EOS Workshop XI

6. HDF and HDF-EOS Workshop XI
HDF themes
Better handling of HDF5 metadata
Reduced HDF5 file overhead
Efficient metadata caching
Improved internal structures for better handling of variable-length data
Features to support HDF and NetCDF harmonization
New and improved tools
Numerous bug fixes and improvements
Result: More than 200 new APIs and revised file format
12/25/2018
HDF and HDF-EOS Workshop XI

7. Quick overview of the new features
Complete list can be found at
Two types of features
Features that did not require file format and APIs changes
Features that did require file format changes
12/25/2018
HDF and HDF-EOS Workshop XI

8. Quick overview of the new features
Features that did not require file format changes and changes to existing APIs
Improved metadata caching
New filters (scale+offset, n-bit, arithmetic transformation)
Improved error handling mechanism
Support for dimension scales, packet tables and other high-level APIs
Affect on applications:
Applications can be extended (calls added) to use new features
There is no need to modify existing calls
Created files are compatible with the earlier versions of software
12/25/2018
HDF and HDF-EOS Workshop XI

9. Quick overview of the new features
Features that did require file format changes and changes to existing APIs
Creation order of the objects (groups, datasets, attributes)
Support for UTF-8 encoding for strings that describe links to the objects and names of the attributes
Configurable compact or indexed storage for groups
Shared object headers, external links
New file format features to reduce files overhead
Affect on applications:
Applications have to be modified to use new features
Created files are NOT compatible with the earlier versions of software
12/25/2018
HDF and HDF-EOS Workshop XI

10. HDF and HDF-EOS Workshop XI
To move or not to move?
12/25/2018
HDF and HDF-EOS Workshop XI

11. Why applications should move to HDF5 1.8.0?
My application doesn’t need any new features, why should I consider moving to HDF ?
Take advantage of improved library
Bug fixes (e.g. corrupted files)
Better metadata cache performance
Smaller memory footprint
Support for big attributes, new compression filters, datatype conversion, etc.
Availability of new platforms like Windows 64-bit
Better support for different Fortran compilers
Improved tools
First step in taking advantage of the new features in the future
12/25/2018
HDF and HDF-EOS Workshop XI

12. Why applications may not move to HDF5 1.8.0?
May be expensive
Moving to the new release may require
Source code modification
Building scripts modification
No new features are desired
12/25/2018
HDF and HDF-EOS Workshop XI

13. HDF and HDF-EOS Workshop XI
Making decision….
I would like to use new features, how much work it would be?
I would like to use new library but I cannot modify my application, can I do it?
The rest of the talk is about the ways to move applications to the new release without doing much work
12/25/2018
HDF and HDF-EOS Workshop XI

14. Introduction to HDF5 APIs versioning
12/25/2018
HDF and HDF-EOS Workshop XI

15. Example: Four ways to create a group
Old way
H5Gcreate ( loc_id, “New/My old group”, 0 )
H5Gcreate1( loc_id, “New/My old group”, 0 )
No new features can be invoked
New way
H5Gcreate ( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id)
H5Gcreate2( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id)
New features can be invoked
Creation order
Unicode names
Compact storage
Intermediate group creation
12/25/2018
HDF and HDF-EOS Workshop XI

16. HDF and HDF-EOS Workshop XI
New APIs in HDF
Examples of the new APIs
H5Acreate1, H5Aopen1
H5Acreate2, H5Aopen2
H5Gcreate1, H5Gopen1
H5Gcreate2, H5Gopen2, H5G_link_hard
H5Rget_obj_type1
H5Rget_obj_type2
New APIs have more parameters to set up creation and access properties for the objects
Default values for new parameters H5P_DEFAULT will emulate old behavior
H5Gcreate ( loc_id, “New/My old group”, 0 )
H5Gcreate ( loc_id, “New/My new group”,
H5P_DEFAULT,H5P_DEFAULT, H5P_DEFAULT)
12/25/2018
HDF and HDF-EOS Workshop XI

17. HDF and HDF-EOS Workshop XI
Removed APIs in HDF
Examples of the removed APIs
H5Acreate, H5Aopen
H5Gcreate, H5Gopen
H5Rget_obj_type
Still can be called by using API mapping and macros
H5Gcreate ( loc_id, “New/My old group”, 0 )
12/25/2018
HDF and HDF-EOS Workshop XI

18. HDF and HDF-EOS Workshop XI
Deprecated APIs in HDF
Include both public and private APIs
Functionality is replaced by new functions
See files in the src directory
H5Adeprec.c H5Ddeprec.c H5Edeprec.c H5Gdeprec.c H5Rdeprec.c H5Tdeprec.c
Examples of deprecated APIs
H5Acreate1, H5Aopen_name
H5Gcreate1, H5Gopen1, H5G_link_hard
H5Rget_obj_type1
For complete list see Changes.html file, “API Compatibility Macros in HDF5”
Deprecated APIs will be removed with caution (in some cases will not be removed at all)
12/25/2018
HDF and HDF-EOS Workshop XI

19. HDF5 Macros for API compatibility
Name: H5Gcreate
Signature:
hid_t H5Gcreate( hid_t loc_id, const char *name, size_t size_hint )
hid_t H5Gcreate( hid_t loc_id, const char *name, hid_t lcpl_id, hid_t gcpl_id, hid_t gapl_id
Purpose:
Creates a new empty group and links it to a location in the file.
Description:
H5Gcreate is a macro that is mapped to either H5Gcreate1 or
H5Gcreate2, depending on the needs of the application.
12/25/2018
HDF and HDF-EOS Workshop XI

20. HDF5 Macros for API compatibility
Defined in the H5version.h file
Example of the H5Gcreate macro
#ifdef H5_USE_16_API …
#if !defined(H5Gcreate_vers)
#define H5Gcreate_vers 1
#endif /* !defined(H5Gcreate_vers) */
#if !defined(H5Gcreate_vers) || H5Gcreate_vers == 2
#ifndef H5Gcreate_vers
#define H5Gcreate_vers 2
#endif /* H5Gcreate_vers */
#define H5Gcreate H5Gcreate2
#elif H5Gcreate_vers == 1
#define H5Gcreate H5Gcreate1
#else /* H5Gcreate_vers */
#error "H5Gcreate_vers set to invalid value"
12/25/2018
HDF and HDF-EOS Workshop XI

21. HDF5 Library configuration
Mapping of APIs is set at configuration time
Several configuration flags are provided to enable different mappings
Macro name is mapped to old API, e.g. H5Gcreate is mapped to H5Gcreate1
Macro name is mapped to new API, e.g. H5Gcreate is mapped to H5Gcreate2 (will have different parameters from 1.6 H5Gcreate)
Disable old APIs completely, e.g. H5Gcreate1 is not available in the library
12/25/2018
HDF and HDF-EOS Workshop XI

22. HDF5 Library configuration
Configure flag
(global settings)
Public symbols are mapped to
Are deprecated symbols available?
(default)
--with-default-api-version=v18
1.8
(e.g. H5Gcreate is mapped to H5Gcreate2, old H5Gcreate is H5Greate1)
Yes
--disable-deprecated-symbols
(e.g. H5Gcreate is mapped to H5Gcreate2, H5Gcreate1 is not available) No
--with-default-api-version=v16
1.6
(e.g. H5Gcreate is mapped to H5Gcreate1, H5Gcreate2 is available)
12/25/2018
HDF and HDF-EOS Workshop XI

23. HDF5 Library configuration
Library configuration information is available in the libhdf5.settings file
Examples
…….
Default Version of Public Symbols: v18
With Deprecated Public Symbols: Yes
Default Version of Public Symbols: v16
With Deprecated Public Symbols: No
12/25/2018
HDF and HDF-EOS Workshop XI

24. HDF5 Application configuration options
Global mapping
HDF5 APIs mapping can be done by application
Assuming both deprecated and new symbols are available in the library:
h5cc my_program.c
Both H5Gcreate1, H5Gcreate2 and H5Gcreate may be used
h5cc -DH5_NO_DEPRECATED_SYMBOLS my_program.c
Only new symbols are available for application; H5Gcreate is mapped to H5Gcreate2; application may use both H5Gcreate2 and H5Gcreate; cannot use H5Gcreate1
h5cc -DH5_USE_16_API my_program.c
H5Gcreate is mapped to H5Gcreate1; all three H5Gcreate1, H5Gcreate2 and H5Gcreate can be used
12/25/2018
HDF and HDF-EOS Workshop XI

25. HDF5 Application configuration options
Per-function scale
Version and mapping can be set per function
Assuming both deprecated and new symbols are available in the library:
h5cc -D H5Gcreate_vers=1 -D H5Acreate_vers=2 my_program.c
Maps H5Gcreate to H5Gcreate1
Maps H5Acreate to H5Acreate2
both H5Gcreate1 and H5Gcreate2 may be used; the same for H5Acreate1 and H5Acreate2
h5cc -D H5Gcreate_vers=2 my_program.c
Maps H5Gcreate to H5Gcreate2
Both H5Gcreate1 and H5Gcreate2 may be used
12/25/2018
HDF and HDF-EOS Workshop XI

26. HDF5 Fortran and C++ applications
HDF5 Fortran and C++ libraries use new APIs H5<name>2 using default parameters for new arguments
Backward/forward compatible
New features are not available yet
12/25/2018
HDF and HDF-EOS Workshop XI

27. Example: --with-default-api-version=v18
hid_t file_id, group_id; /* identifiers */
...
/* Open “file.h5” */
file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT);
/* Create several groups in a file */
grp1_id = H5Gcreate (file_id, ”New/larry",
H5P_DEAFULT, gcpt, gapt);
grp2_id = H5Gcreate1(file_id,"/larry",0); …
grp3_id = H5Gcreate2(file_id,”New/larry",
12/25/2018
HDF and HDF-EOS Workshop XI

28. Example: --with-default-api-version=v16
hid_t file_id, group_id; /* identifiers */
...
/* Open “file.h5” */
file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT);
/* Create several groups in a file */
grp1_id = H5Gcreate (file_id, "/larry",0);
grp2_id = H5Gcreate1(file_id,"/larry1",0);
grp3_id = H5Gcreate2(file_id,”New/larry",
H5P_DEAFULT, gcpt, gapt);
12/25/2018
HDF and HDF-EOS Workshop XI

29. Example: --disable-deprecated-symbols
hid_t file_id, group_id; /* identifiers */
...
/* Open “file.h5” */
file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT);
/* Create several groups in a file */
grp1_id = H5Gcreate (file_id, ”New/larry",
H5P_DEAFULT, gcpt, gapt);
/* Compilation will fail */
grp2_id = H5Gcreate1(file_id,"/larry",0);
grp3_id = H5Gcreate2(file_id,”New/larry",
12/25/2018
HDF and HDF-EOS Workshop XI

30. Example: new version of the file format
hid_t file_id, group_id; /* identifiers */
...
/* Open “file.h5” and create objects using new
format*/
fapl = H5Pcreate(H5P_FILE_ACCESS);
ret = H5Pset_latest_format(fapl, 1);
fid = H5Fcreate(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT,
fapl);
/* Compose group creation property list */
gcpl = H5Pcreate(H5P_GROUP_CREATE);
ret = H5Pset_link_creation_order(gcpl, (H5P_CRT_ORDER_TRACKED | H5P_CRT_ORDER_INDEXED));
/* Create a group */
gid = H5Gcreate(fid, GNAME, H5P_DEFAULT, gcpl, H5P_DEFAULT);
12/25/2018
HDF and HDF-EOS Workshop XI

31. Example: h5dump --group=1 tordergr.h5
HDF5 "tordergr.h5" {
GROUP "1" {
GROUP "a" {
GROUP "a1" { }
GROUP "a2" {
GROUP "a21" {
GROUP "a22" {
GROUP "b" {
GROUP "c" {
12/25/2018
HDF and HDF-EOS Workshop XI

32. Example: h5dump … --sort_by=creation_order … tordergr.h5
HDF5 "tordergr.h5" {
GROUP "1" {
GROUP "c" { }
GROUP "b" {
GROUP "a" {
GROUP "a1" {
GROUP "a2" {
GROUP "a22" {
GROUP "a21" {
12/25/2018
HDF and HDF-EOS Workshop XI

33. Example: creation of intermediate groups
/* Create link creation property list */
lcr_id = H5Pcreate(H5P_LINK_CREATE);
/* Set flag for intermediate group creation */
status = H5Pset_create_intermediate_group(lcr_id,
TRUE);
grp_id = H5Gcreate (file_id, "/A/B/C", lcr_id,
H5P_DEFAULT, access_id);
Groups A and B will be created automatically
12/25/2018
HDF and HDF-EOS Workshop XI

34. HDF and HDF-EOS Workshop XI
Thank you!
Questions ?
12/25/2018
HDF and HDF-EOS Workshop XI