1. HDF Update Mike Folk, Kent Yang, Elena Pourmal The HDF Group March 31st, 2009 March 31, 2009Annual HDF Briefing to ESDIS1

2. HDF Project Update Helpdesk HDF Releases Library release highlights Quality Assurance Java Product Highlights Other Tools Activities Outreach Related projects and activities EOS Support OPeNDAP netCDF Support Other projects of interest Platform support issues

3. HDF Project Update March 31, 2009Annual HDF Briefing to ESDIS3

4. Helpdesk statistics March 31, 2009Annual HDF Briefing to ESDIS4

5. Number of HDF4 and HDF5 issues since 2000 March 31, 20095Annual HDF Breifing to ESDIS

6. HDF4 issues by category March 31, 20096Annual HDF Breifing to ESDIS

7. HDF5 issues by category March 31, 20097Annual HDF Breifing to ESDIS

8. HDF issues by domain March 31, 20098Annual HDF Breifing to ESDIS

9. HDF issues resolution time March 31, 20099Annual HDF Breifing to ESDIS

10. HDF open and closed bugs March 31, 200910Annual HDF Breifing to ESDIS

11. HDF5 open and closed bugs March 31, 200911Annual HDF Breifing to ESDIS

12. HDF4 open and closed bugs March 31, 200912Annual HDF Breifing to ESDIS

13. QUESTIONS? March 31, 2009Annual HDF Breifing to ESDIS13

14. HDF RELEASES March 31, 2009Annual HDF Breifing to ESDIS14

15. HDF Releases in 2008 - 2009 HDF4 HDF4r2.4 (February 2009) HDF5 HDF5-1.8.1 (May 2008) HDF5-1.6.8 (November 2008) HDF5-1.8.2 (November 2008) HDF- Java Products Version 2.5 (February 2009) H4toH5 Version 2.0 (June 2008) Version 2.1 (February 2009) March 31, 200915Annual HDF Breifing to ESDIS

16. HDF LIBRARIES RELEASE HIGHLIGHTS March 31, 2009Annual HDF Breifing to ESDIS16

17. HDF4r2.4 Fixed a flaw in the HDF4 design exposed by applications that use HDF-NetCDF interface to access both HDF4 and NetCDF-3 files with datasets/variables with unlimited dimensions. Added several new APIs including API to get compressed size of data. Updated hdp and hrepack to display compression information. Added support for Fedora 10, Ubuntu 8.10, and OpenSuse 11.1 with GNU C and Fortran. Better support for Mac OSX (GNU compilers and 32-bit Intel). http://www.hdfgroup.org/release4/obtain.html March 31, 200917Annual HDF Breifing to ESDIS

18. HDF5-1.6.8 Many improvements in tools to address ESDIS requests Availability of compression information in h5dump Files comparison with h5diff Support for strings in h5import Support for user block in h5repack Added compatibility macros (e.g., H5Gcreate1) for applications to work the same with 1.6.* and 1.8.* Support for object and region references in C++ Bug fixes http://www.hdfgroup.org/HDF5/release/obtain516. html http://www.hdfgroup.org/HDF5/release/obtain516. html March 31, 200918Annual HDF Breifing to ESDIS

19. HDF5-1.8.1&2 Tools improvements Support for external links Support for a simplified way to produce binary output by h5dump (ESDIS) Better support for user block with h5repack Improved performance for free space tracking in a file Fixed bugs with file mounting Many improvements in configuration, installation and testing Better support for shared C, Fortran and C++ libraries http://www.hdfgroup.org/HDF5/release/obtain5.ht mlhttp://www.hdfgroup.org/HDF5/release/obtain5.ht ml March 31, 200919Annual HDF Breifing to ESDIS

20. HDF Java Products New released is based on JDK 1.6.x, HDF4r2.4 and HDF5-1.8.2 with 1.6 compatibility Java products (i.e. HDFView) can read files created by HDF5 1.8.2 New HDFView: Supports access to remote HDF5 files stored at the iRODS server (Windows and Linux only). For information on HDF5- iRODS module, see http://www.hdfgroup.org/projects/irods/index.html. http://www.hdfgroup.org/projects/irods/index.html Shows large dataset (e.g. 1,000,000 x 100) in a table viewer Converts JPEG, GIF, PNG, and BMP images to HDF5 or HDF4 Saves HDF4 or HDF5 images to JPEG, GIF, PNG, and BMP images Writes variable length string data Filters fill values for images March 31, 200920Annual HDF Breifing to ESDIS

21. H4toH5 tools and library Based on HDF4r2.4 and HDF5-1.8.2 h4toh5 utility Recognizes HDF-EOS2 files (--with-hdfeos2 configuration option) Can generate HDF5 files that can be read by netCDF-4 h4toh5 library Bug fixes Performance improvements March 31, 200921Annual HDF Breifing to ESDIS

22. QUALITY ASSURANCE Some work of interest to ESDIS March 31, 2009Annual HDF Breifing to ESDIS22

23. QA Efforts We test at least 250 different configurations for HDF software (OSs x compilers x HDF configurations) every day Regression testing of HDF5 released branches (1.6. and 1.8) and development branch (1.9), and HDF4 Platforms Linux 32 and 64, Solaris 32 and 64, Free BSD 32 and 64, AIX 32 and 64 (at LLNL), Windows 32 and 64, Linux Clusters, SGI Altix Backward/forward compatibility testing Third part software testing March 31, 200923Annual HDF Breifing to ESDIS

24. Backward/forward compatibility testing Goal: assure interoperability between 1.6 and later versions of the HDF5 library used to create AURA files Regression tests with and without strict format checks using known problem files (currently MLS Cloud series with corrupted headers) Download sample from subdirectories of OMI, MLS, HIRDL, TOMS and TES files. Sample consists of files from the current day of the year for available years in two of the subdirectories listed for the above instruments. March 31, 200924Annual HDF Breifing to ESDIS

25. Backward/forward compatibility testing FTP sites (about 540 000 files) OMI Level 2 files ftp://aurapar2u.ecs.nasa.gov OMI Level2G, MLS and HIRDL files ftp://acdisc.gsfc.nasa.gov TOM files ftp://acdisc.sci.gsfc.nasa.gov TES files ftp://l4ftl01.larc.nasa.gov March 31, 200925Annual HDF Breifing to ESDIS

26. Backward/forward compatibility testing Each known subdirectory will be visited ~5 times per year. hdf5 tools from versions 1.6.9, 1.8.3, and 1.9.0 are checked for proper operation with downloaded files. Downloaded files are copied with h5copy and repacked with h5repack. The resulting files are checked again with all 3 versions of the tools. March 31, 200926Annual HDF Breifing to ESDIS

27. Third party software testing NetCDF-4 Daily tests of the netCDF4 snapshot Built against current HDF5 v1.8.* Tested on Linux 32/64 bit and solaris HDF-EOS2 and HDF-EOS5 testing HDFEOS2 with h4cc, 32/64 bit Linux, 32 bit solaris HDFEOS5 with hdf5 v1.6 and v1.9, 32/64 bit Linux, 32/64 bit solaris IDL wrappers Created IDL/HDF module binaries based on HDF5 1.8.2 and HDF4.2r3 March 31, 200927Annual HDF Breifing to ESDIS

28. QUESTIONS? March 31, 2009Annual HDF Breifing to ESDIS28

29. JAVA PRODUCT HIGHLIGHTS March 31, 2009Annual HDF Briefing to ESDIS29

30. HDF Java Products Highlights V2.5 released in February 2009 50 bug fixes/enhancements Support for HDF5 1.8-with-1.6 compatibility (Does not support new HDF5 1.8 features) March 31, 200930Annual HDF Briefing to ESDIS

31. HDFView Highlights Support for iRODS Show large dataset in table Write variable length string Convert image to/from HDF4/5 JPEG GIF PNG BMP March 31, 200931Annual HDF Briefing to ESDIS

32. Future work for Java 26 bugs/enhancements hdf-java 2.6 release Include latest HDF4 and HDF5 1.8 libraries Fix bugs and add new enhancements Full support for HDF5 1.8 (next slide) March 31, 200932Annual HDF Briefing to ESDIS

33. Future work for Java Full support for HDF5 1.8 Add HDF5 1.8 APIs to the HDF5 Java wrapper Initial implementation (done) Review and revision is in progress Implement and test HDF5 JNI C layer Implementation (200+ APIs), 1 month fte Testing, 1 month fte Support new features in HDF5 1.8 Revise hdf-java object package Add new features to HDFView 33Annual HDF Briefing to ESDIS April 18-19, 2007

34. March 31, 2009Annual HDF Briefing to ESDIS34 Project Plan for 1.8 support

35. OTHER TOOLS ACTIVITIES March 31, 2009Annual HDF Briefing to ESDIS35

36. Improvements in HDF4 tools Display compression information in hdf4 tools Preserve unlimited dimensions in hrepack Fix bugs in hdp and hdiff reported by DAACs. March 31, 200936Annual HDF Briefing to ESDIS

37. HDF5 tools highlights 35 bug fixes/enhancements to HDF5 tools All bug fixes/enhancements are applied to both HDF5 1.6 and 1.8 release March 31, 200937Annual HDF Briefing to ESDIS

38. Highlights of HDF5 Tools enhancements h5dump Display compression information Follow external links Add ability to specify precision output Use MEMORY option for default binary output h5diff Display the precision the differences Handle NaN values h5repack Keep the original user block information Multiple filters can be invoked March 31, 200938Annual HDF Briefing to ESDIS

39. Outreach March 31, 2009Annual HDF Briefing to ESDIS39

40. March 31, 2009Annual HDF Briefing to ESDIS40 Earth Science Related Outreach IEEE Transactions on Geosciences and Remote Sensing, vol. 47, issue 1 Paper: “Ensuring Long Term Access to Remotely Sensed Data with Layout Maps” by Duerr, Cao, Folk, Lynnes, Yang. ESIP Federation Summer meeting, July 2008 Presented a poster about OPeNDAP work Meeting with ITTVis, Oct. 2008 Presentation about HDF Worked with them to test the IDL-HDF5 module with a new HDF5 release 7th NASA ESDSWG conference, Oct. 2008 Reported on the OPeNDAP project, participated in several working groups AGU Fall meeting, Dec. 2008 Presentation: “Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps,” by Folk and Duerr American Meteorological Society Annual Meeting, Jan. 2009 Presentations on OPeNDAP and the HDF-EOS2 to netCDF-4 conversion tool April 1-2, 2008Annual HDF Briefing to ESDIS40

41. HDF-EOS Workshop XII Tutorials Introduction to HDF5 Data and Programming Models Advanced HDF5 Features Migrating from HDF5 1.6 to 1.8 Introduction to HDF5 Tools HDFView and HDF Java Products What will be new in HDF5 Status HDF Status and Developments HDF Group Support for NPP/NPOESS Applications and discussions NOAA Science Data Stewardship project update HDF5-OPeNDAP Project Update and Demo Independent HDF4 mapping project update HDF5 IRODS Users: What are your needs No programming language left behind March 31, 200941Annual HDF Briefing to ESDIS

42. Other outreach - sample Presentation “HDF5: Toward a Universal Platform for Science and Engineering Data,” Folk National Renewable Energy Lab Scientific and Technical Data Management and Analysis Workshop ’08, May 2008 “HDF Experiences with I/O Bottlenecks,” Folk NSF/NARA Collaborative Expedition Workshop: Toward Scalable Data Management – Overcoming I/O Bottlenecks, June 10, 2008 HDF5 Tutorial, Pourmal and Cheng 37th SPEEDUP Workshop on HPC, Zurich, Sept. 2008 “HDF5: Past, Present, and Future” “Tuning Parallel HDF5 for HPCApplications,” Koziol, Aydt National Energy Research Scientific Computing Center HDF5 Workshop on Scalable Performance March 31, 200942Annual HDF Briefing to ESDIS

43. Related projects and activities March 31, 2009Annual HDF Briefing to ESDIS43

44. EOS SUPPORT March 31, 2009Annual HDF Briefing to ESDIS44

45. EOS support HDF-EOS2 and HDF-EOS5 Automatic configuration with szip enabled/disabled EOS2 and EOS5 are tested daily with HDF4 and HDF5 development code HDF-EOS website workshop coordination EOS product release March 31, 200945Annual HDF Briefing to ESDIS 10/11/2015The HDF Group for 2008 annual briefing45

46. OPENDAP March 31, 2009Annual HDF Briefing to ESDIS46

47. OPeNDAP – current status HDF5-OPeNDAP handler Served OMI and TES data HDF5 –awareness OPeNDAP client library HDF4-OPeNDAP handler Tested with AIRS data and some MODIS data More information in the afternoon session March 31, 200947Annual HDF Briefing to ESDIS 10/11/2015The HDF Group for 2008 annual briefing47

48. NETCDF SUPPORT March 31, 2009Annual HDF Briefing to ESDIS48

49. NetCDF4 support Help Unidata NetCDF4 developers Frequent communications with NetCDF4 developers Test NetCDF4 daily with the HDF5 development code NetCDF4 performance benchmark report http://www.hdfgroup.org/pubs/papers/2008 -06_netcdf4_perf_report.pdf March 31, 200949Annual HDF Briefing to ESDIS 10/11/2015The HDF Group for 2008 annual briefing49

50. LONG TERM ACCESS TO REMOTELY SENSED HDF4 DATA WITH LAYOUT MAPS Discussed later March 31, 2009Annual HDF Briefing to ESDIS50

51. Other projects of Interest March 31, 2009Annual HDF Briefing to ESDIS51

52. March 31, 2009Annual HDF Briefing to ESDIS52 NOAA SDS April 1-2, 2008Annual HDF Briefing to ESDIS52 10/11/2015The HDF Group for 2008 annual briefing52 Enhanced HDF4 to HDF5 conversion tool An HDF-EOS2 file can be converted to a netCDF-4 file Accumulated knowledge and experience to handle HDF-EOS2 data

53. Other projects “NPOESS HDF5 Development & Support” “BioHDF – HDF5 for next gen sequencing” Human Genome Research Institute SBIR with Geospiza, Inc. “HDF support for urban military operations” Army Engineering Research and Development Center – Topographic Engineering Center March 31, 200953Annual HDF Briefing to ESDIS

54. March 31, 2009Annual HDF Briefing to ESDIS54 HDFView ERDC Concept Map Plugin

55. Platform support issues March 31, 2009Annual HDF Briefing to ESDIS55

56. HDF4 Platforms Supported Systems AIX 5.2 (32-bit, 64-bit) Free BSD 6.2 (32-bit, 64- bit) IRIX 64 v6.5 (32-bit, 64-bit) Linux 2.4, 2.6 (RH3) Linux ia64 Linux x86_64 (SUSE) Sun OS 5.8, 5.10 (32-bit, 64-bit) Windows XP, Vista Windows XP 64-bit* Mac OS X Intel * - new platform Compilers IBM C and Fortran compilers GNU gcc 3.4.* and GNU Fortran GNU gcc 3.4 and 4.* PGI v 7.* compilers Intel C and Fortran versions 9.1 and 10.00 SUN WorkShop C and Fortran Visual Studio.NET and 2005 and Intel Fortran Visual Studio 2005 (no fortran) GNU gcc 4.0.1 with gfortran and g95 March 31, 2009Annual HDF Briefing to ESDIS56

57. HDF4 Platforms to drop/add Drop Windows XP with MSVC++ 6.0 Linux 2.4 IRIX64 6.5 SunOS 5.8, 5.9 HP 11.23 (no access) Sun on Intel (no access) Add ? XT3 Blue Gene Blue Water Linux flavors? ? March 31, 2009Annual HDF Briefing to ESDIS57

58. HDF5 Platforms Supported Systems AIX 5.2 (32-bit, 64-bit) Free BSD 6.2 (32-bit, 64-bit) IRIX 64 v6.5 (32-bit, 64-bit) Linux 2.4, 2.6 (RH3) Linux ia64 Linux x86_64 (SUSE) Sun OS 5.8, 5.10 (32-bit, 64- bit) Cray XT3 Windows XP, Vista Windows XP 64-bit* Mac OS X Intel Alpha Open VMS* * - new platform Compilers IBM C and Fortran compilers GNU gcc 3.4.* and GNU Fortran GNU gcc 3.4 and 4.* PGI v 7.* compilers Intel C and Fortran versions 9.1 and 10.00 SUN WorkShop C and Fortran Visual Studio.NET and 2005 and Intel Fortran Visual Studio 2005 (no fortran) GNU gcc 4.0.1 with gfortran and g95 Compaq C and C++ compilers, HP Fortran compiler March 31, 2009Annual HDF Briefing to ESDIS58

59. HDF5 Platforms to drop/add Drop Windows XP with MSVC++ 6.0 Linux 2.4 IRIX64 6.5 SunOS 5.8, 5.9 HP 11.23 (no access) Sun on Intel (no access) Add ? XT3 Blue Gene Blue Water Linux flavors? ? March 31, 2009Annual HDF Briefing to ESDIS59

60. Platform support questions Platform to support for HDF Java products Windows XP/Vista Solaris Linux Mac Intel Mac PowerPC Linux x86_64 Linux ia64 Solaris 64-bit Sparc Any to add? Any to drop? March 31, 200960Annual HDF Briefing to ESDIS