1. www.hdfgroup.org The HDF Group Data Interoperability The HDF Group Staff Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV1

2. www.hdfgroup.org Interoperability Interoperability is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation. (From Wikipedia) Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV2

3. www.hdfgroup.org Background for data interoperability Focus on Earth Sciences Data needs to be shared across communities Self-described data format Many different types of data Grids, moving-sensor multidimensional fields, time series, profiles, trajectories, geospatial framework data, points Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV3

4. www.hdfgroup.org Why data interoperability? Data format Several data formats available HDF4/5, netCDF3/4, GRIB, BUFR, Binary Data are created with some data formats by following different physical models End users would like to use familiar tools to access data via different formats Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV4

5. www.hdfgroup.org Why data interoperability? Semantic/content interoperability An example: the missing value of a physical variable Tools or applications need to know the fill value of a physical variable How can they identify? Normally via an attribute that stores the fill value The name can be “Fill_value, _FillValue, badValue,……” Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV5

6. www.hdfgroup.org Goals Discuss a few cases about data interoperability Hear opinions or issues about data interoperability from audience. Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV6

7. www.hdfgroup.org The HDF Group Data Format Interoperability Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV7

8. www.hdfgroup.org Case 1 One set of APIs to handle multiple data formats Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV8

9. www.hdfgroup.org Case 1 netCDF4 Combine powerful model and simplicity of netCDF with features of HDF5 netCDF interface on top of HDF5 CDM(Common Data Model) Harmonize netCDF, HDF4, HDF5, OPeNDAP, GRIB and others Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV9

10. www.hdfgroup.org Discussions Any comments/sharings about the experiences with netCDF4 and CDM? Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV10

11. www.hdfgroup.org Case 2 Format conversions and translations Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV11

12. www.hdfgroup.org Data format Conversions/Translations Format conversions The HDF4 to HDF5 conversion tool The HDF-EOS2 to netCDF3/4 conversion tool HDF-EOS to GeoTIFF conversion tool …… Data format Translations netCDF tools to access HDF4/5 files via OPeNDAP …… Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV12

13. www.hdfgroup.orgSep. 28-30, 2010HDF/HDF-EOS Workshop XIV Use netCDF tools to access HDF via OPeNDAP Users OPeNDAP Clients IDV OPeNDAP Servers HDF4/5 Files AIRS/OMI DAP libnc-dap Visualization Tools HDF4/5 Handlers Aqua/Aura 13 Translation Layers

14. www.hdfgroup.orgSep. 28-30, 2010HDF/HDF-EOS Workshop XIV14 netCDF4 to access HDF-EOS5 files Augmentation One file can be used for both EOS5 and NetCDF-4. Note that EOS5 users are not affected at all. HDF5 HDF-EOS5 file HDF5 NetCDF4 NetCDF-4 file Augmented HDF-EOS5 file Augmentation

15. www.hdfgroup.org Discussions Any comments/sharings about the experiences with data format conversions and translations Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV15

16. www.hdfgroup.org The HDF Group Semantic/Content Interoperability Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV16

17. www.hdfgroup.org An example Degree of latitude and longitude Champaign’s latitude is 40°6´38" N It can be represented in two formats: 400638 in DDDMMSS format 40.1105556 in decimal format How can the applications know which format the file is used? Better to have a common standard to facilitate the exchange this kind of information Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV17

18. www.hdfgroup.org Semantic/Content Interoperability CF conventions become such a standard Many applications/tools follow CF conventions to interpret the metadata Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV18

19. www.hdfgroup.orgSep. 28-30, 2010HDF/HDF-EOS Workshop XIV Use netCDF tools to access HDF via OPeNDAP Users OPeNDAP Clients IDV OPeNDAP Servers HDF4/5 Files AIRS/OMI DAP libnc-dap Visualization Tools HDF4/5 Handlers Aqua/Aura 19 Not follow CF conventions

20. www.hdfgroup.org Discussions Any comments about semantic/content interoperability Any comments about the usage of CF conventions Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV20

21. www.hdfgroup.org The HDF Group HDF5 and netCDF-4 Libraries Interoperability Two sources of interoperability problems and how to deal with them Sep. 28-30, 2010HDF/HDF-EOS Workshop XIV21

22. www.hdfgroup.org netCDF-4 library and arbitrary HDF5 files netCDF-4 is built on top of HDF5 netCDF-4 files are HDF5 files with specific characteristics 1.Tracked creation order of objects and their attributes 2.Absence of datasets with complex compound, and region and object references 3.Presence of dimension scales netCDF-4 expects to find properties 1 and 2 in an HDF5 file and fails to open or read the file when those are not present (e.g., the NPOESS files) Solution: TB discussed with the netCDF-4 folks 12/13/2015Sep. 28- 30, 2010HDF/HDF-EOS Workshop XIV22

23. www.hdfgroup.org Programming models and libraries settings Use case 1 (libraries settings): Simultaneous access to an HDF5 file via HDF5 and netCDF-4 One of the libraries will not be able to open the file due to the different access properties used by each library (H5F_CLOSE_WEAK in HDF5 vs. H5F_CLOSE_ STRONG in netCDF-4) Solution: Design improvements in HDF5 New APIs to detect situation Automatic detection and correction of the access properties Document the best practices for applications 12/13/2015Sep. 28- 30, 2010HDF/HDF-EOS Workshop XIV23

24. www.hdfgroup.org Programming models and libraries settings Use case 2 (programming models): Usage of HDF5 wrapper libraries and netCDF-4 library in the same application C++ H5Topen call fails due to the netCDF-4 library shutting down the HDF5 library underneath by calling H5close Solution: Resolved in the newest netCDF-4 Document the best practices for applications For more detailed discussion see “NetCDF-4/HDF5 Libraries Interoperability Issues” at http://www.hdfgroup.uiuc.edu/RFC/HDF5/netCDF4-HDF5/ Let us know your use case to improve testing! 12/13/2015Sep. 28- 30, 2010HDF/HDF-EOS Workshop XIV24

25. www.hdfgroup.orgSep. 28-30, 2010HDF/HDF-EOS Workshop XIV25 Thank you !