1. Mike Folk, Elena Pourmal, Kent Yang
HDF Project Update
Mike Folk, Elena Pourmal, Kent Yang
The HDF Group
Briefing to ESDIS
October 16-17, 2012
10/16/2012
HDF Briefing to NASA

2. Project Goals
Support for NASA HDF data providers and data users, and the EOSDIS Core System (ECS)
Maintenance and quality assurance for HDF4 and HDF5 libraries and utilities
Evolving the HDF4 and HDF5 libraries and utilities to meet new NASA EOS requirement
Integration with complementary technologies and application domains
Prototype study of web technologies to access HDF data
On this slide you see the five goals for the the current project. In this first presentation I will mostly talk about the activities we perform in support of the first three goals: Support for NASA HDF data providers and data users, maintenance and quality assurance for the HDF products and how we evolve the products to meet NASA EOS requirements. Kent will talk about the work we do to achieve two other goals.
We have two major objectives today: first, to report on our work for the period of time since the last briefing and to get new requirements from NASA and new priorities for the next year.
I will make a pause in a particular places in this talk and ask for your feedback, but please feel free to ask questions and give suggestions as we go.
For support activities:
- Provide timely and good workmanship help to NASA corresponding stakeholders
Improve usability and accessibility of NASA HDF4, HDF-EOS2, HDF5 and HDF-EOS5 data
Maintenance and QA
Feature changes and quality assurance to address NASA EOSDIS requirements.
Evolution:  Ditto
Integration:
Strive to make NASA HDF4,HDF-EOS2, HDF5 and HDF-EOS5 interoperable with netCDF and OPeNDAP.
Prototype study
Investigate other web technologies that can benefit the access of NASA HDF data. 
10/16/2012
HDF Briefing to NASA

3. Support activities for NASA HDF data providers and users and ECS
10/16/2012
HDF Briefing to NASA

4. Activities Help Desk Outreach EOS support
Mailing lists, Web site
Conferences, Workshops, Other
EOS support
netCDF support and collaboration
Later today:
HDF OPeNDAP support
Support for NASA data products
SMAP, ICESat-2, MEaSURES)
We use different avenues to reach and work with out customers: we provide helpdesk, support mailing lists and Web site. We go to the conferences and Workshops to meet with out users. We also pay a special attention to the projects like HDF-EOS and netCDF-4. Later today Kent will talk about our support for OPeNDAP and about the work we do with SMAP, ICESAT and Measures teams.
10/16/2012
HDF Briefing to NASA

5. Help desk
10/16/2012
HDF Briefing to NASA

6. The HDF Group Help Desk High-priority support for ESDIS users
Fast response
Problem investigation and resolution
Performance tuning
Access to HDF developers
Basic support for entire HDF user community, as resources permit
Available to all:
Problems compiling and executing an application
Accessing information from the HDF web pages
Adding user's application information to the tools web page
Building the software
Documentation concerns
Downloading software
Java HDF questions
Really ANY problems/concerns regarding HDF, HDF5, the Java Products, and anything else related to HDF.
10/16/2012
HDF Briefing to NASA

7. Help Desk issues resolution time
For the helpdesk we track resolution time , # of issues for different products, type of issues. We also try to keep track of our users and projects they are working on.
This slide shows the issues resolution time for the last two years and for 2012 including August. As you can see we more than 40% of all issues in less than 4 hours, and most of the issues are resolved in two weeks.
I should also mention here that we have been working on improving our helpdesk ticketing system to make it less dependent on one person (Barbara). Currently we are using JIRA (an issue tracking system from Atlassian). With this new tool it is much easier to pass an issue to developers if necessary and keep exchange log and a history. The tool is open source and cab be extended if necessary.
* Number of messages up do August 31, 2012
10/16/2012
HDF Briefing to NASA

8. Help Desk issues by years
This slide shows the number of the HDF5 and HDF4 questions received during the years. It is constantly declining for HDF4 and had two spikes for HDF5 at the time of the major releases 1.6 and 1.8.0
In 2011 we had 285 issues
EOS – 35
JPSS -18
In 2012 up to 10/14/12
505 issues
EOS - 66
JPSS 34
What will be parameters that NASA/ESDIS would like us to track?
* Number of messages up do August 31, 2012
10/16/2012
HDF Briefing to NASA

9. outreach
10/16/2012
HDF Briefing to NASA

10. Web pages improvements
Outreach
Web pages improvements
FAQ support for HDF4 and HDF5
Later today: Detailed talk on support for EOS (Kent)
Organizing material based on the ESDIS survey results and feedback from the users
Example: Tools by category (work in progress)
How do I edit, copy, import/export data with HDF?
We work on improving the content of our Web pages. We support FAQ for both HDF4 and HDF5. Later today Kent will talk about HDF-EOS support work.
In April we received results from the ESDIS users survey. One of the funding was that ESDIS users had hard time finding the right tools. Even all information about HDF tools is available on our Webpages, users couldn’t easily find tools, to manipulate data stored in HDF, for example, how to edit, copy, export/import data.
10/16/2012
HDF Briefing to NASA

11. Tools by category
This is work in progress. We now provide pages for HDF tools where one can find tools under different categories with the links to usage and Tutorials or more materials when available . Our next step will be to add missing Tutorials for tools that show how to perform most common tasks, for example, how to get binary or ascii data from HDF files, or how to modify an attribute.
What kind of topics will be the most useful for the users of EOS products?
10/16/2012
HDF Briefing to NASA

12. Outreach Mailing list and archives news@hdfgroup.org
Information about events are sent to the news ML, discussion happens on hdf-forum
10/16/2012
HDF Briefing to NASA

13. Monitored by the members of the HDF group Number of messages:
Started in Summer 2007
Monitored by the members of the HDF group
Number of messages:
Most discussed topics:
Parallel HDF5
Advanced HDF5 features (drivers, complex types, performance)
.NET and C++ wrappers, HL libraries
2012 to Sep 30
2011
2010
703
1,334
1,721
When we see EOS or NPOESS users on the FORUM we jump in and work with them via helpdesk or directly.
10/16/2012
HDF Briefing to NASA

14. Members of the FORUM help with
Members of the FORUM help with
Releases testing on a regular basis
Maintaining CMake build systems on platforms beyond Windows
Answering questions
The Help Desk focuses on ESDIS and other paying customers while referring others to the FORUM for topics that require domain knowledge or specific HDF5 usage
Reporting defects
Securing funding for The HDF Group
10/16/2012
HDF Briefing to NASA

15. Conferences, Workshops, Other
2011 IDV/THREDDS Workshop, July 2011
Learned how to use IDV and THREDDS
10th NASA ESDSWG Conference, Nov 2011
Poster on HDF and netCDF interoperability
2012 ESIP Summer Meeting, July 2012
Presentations
HDF/HDF-EOS Data for Dummies
What does Open Source mean for HDF?
Poster
What’s new at hdfgroup.org?
10/16/2012
HDF Briefing to NASA

16. Conferences, Workshops, Other
15th HDF and HDF-EOS Workshop, April 2012
Five tutorials, nine presentations and lightning talks, and one special session
Materials:
10/16/2012
HDF Briefing to NASA

17. Conferences, Workshops, Other
Advised ICESat-2, MOPITT and several MEaSURES projects on interoperability with netCDF-4
Participated in the SMAP ISO work
Participated in ESDSWG SPG telecons
Joined several ESDSWG mailing lists related to HDF
10/16/2012
HDF Briefing to NASA

18. netCDF support and collaboration
10/16/2012
HDF Briefing to NASA

19. Help Unidata netCDF-4 developers
NetCDF-4 support
Help Unidata netCDF-4 developers
Frequent communications with netCDF-4 developers
Provided a fix for ncdump critical to the ESDIS users
Test netCDF-4 daily with the HDF5 development code
Plan to participate in Unidata Advanced netCDF Workshop on October 26, 2012
10/16/2012
HDF Briefing to NASA

20. EOS support
10/16/2012
HDF Briefing to NASA

21. Submitted several bugs and the corresponding fixes to the HDF-EOS team
EOS Support
Submitted several bugs and the corresponding fixes to the HDF-EOS team
EOS2 and EOS5 are tested daily with HDF4 and HDF5 development code
HDF-EOS website
Actively maintain the website
Improve the quality of the example codes and plots to access NASA HDF and HDF-EOS data via MATLAB, IDL and NCL
More information in a separate talk
10/16/2012
HDF Briefing to NASA

22. Maintenance and quality assurance for HDF4 and HDF5 libraries and utilities
10/16/2012
HDF Briefing to NASA

23. Activities Manage and resolve issues
Perform maintenance releases of HDF software
Support OSs and compilers identified by ESDIS
Address issues and software evolution requests from ESDIS
Perform SQE and SQA activities
Put in place missing processes
Improve existing processes
Build in software quality by performing testing and using the best software development practices
We focus in three areas: manage and resolve issues; we perform maintenance releases of the HDF software and support them on OSs and with compilers as requested by ESDIS. We also implement new features requested by ESDIS.
Special attention in the past year and half was paid to the Software Quality Engineering and Software Quality Assurance. We have been working on putting in place missing processes, improving existing processes, improving our testing and using the best software practices.
10/16/2012
HDF Briefing to NASA

24. HDF issues
10/16/2012
HDF Briefing to NASA

25. Issues and their priorities
Must Fix
Data corruption
Portability
Backward and Forward Compatibility
Funded Requests
Fix after “Must Fix”
Power User Requests
Tools
Library issues
Build Infrastructure
When resources permit
Wrappers
HL Libraries
Other
Need your input on priorities!
10/16/2012
HDF Briefing to NASA

26. HDF4 opened and closed issues
Spike is due to the H4mapping project
10/16/2012
HDF Briefing to NASA

27. HDF5 opened and closed issues
Several factors for 2011 spike: moved to JIRA – more issues were entered, better tracking of all issues including documentation issues
2008 spike – release
10/16/2012
HDF Briefing to NASA

28. SQE and SQA at the HDF group
10/16/2012
HDF Briefing to NASA

29. Activities
CMMI appraisal and process improvement action plan (Dec. 2011)
Created SQE position
Focus on
Implementing CMMI best practices and procedures
Enforcing current practices and procedures
Focused on improving software testing
10/16/2012
HDF Briefing to NASA

30. Activities
Worked on establishing a culture of code reviews for the HDF libraries and tools
Improved releases planning and monitoring for HDF5
Used static analysis tools to identify defects
Coverity (used at THG)
Klocwork (used at LLNL)
Report from LLNL
Compared Open Source software packages used at LLNL
HDF5
MOAB (Mesh-Oriented datBASE) from Argonne
AMP (Advanced Muti-Physics code) from ORNL
Visit (parallel visualization and graphical analysis tool) from LLNL
LOC Density
HDF ,
MOAB 104,
AMP ,
Visit 1,195,
10/16/2012
HDF Briefing to NASA

31. HDF Maintenance releases
10/16/2012
HDF Briefing to NASA

32. General maintenance for 2011-2012
Implemented CMake for all HDF software
Worked on closing reported issues
Focused on performance enhancements
Improved HDF documentation
Explored the needs for internal documentation
Preventive measures:
Coverity static analysis; Rational tools; valgrind
Have been exploring new testing tools to sustain testing for HDF products (CDash, Bamboo, Cloud)
New documentation includes
Chunking document
Data flow document
Identifiers
Metadata
Things one need to know for better performance
10/16/2012
HDF Briefing to NASA

33. Maintenance Releases 2011 – 2012
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
HDF4
HDF5
1.8.7
1.8.8
HDF-Java
2.8
h4h5 tools
2012
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
HDF4
4.2.7
patch1
4.2.8
HDF5
1.8.9 CF
1.8.10
HDF-Java
2.9
h4h5 tools
2.2.1
HDF – Fortran 2003 support, support for Fortran dimension scales
HDF4 releases in support of the H4 mapping project
Support for Powerpc64 platform (big-endian)
Java – addressed all ESDIS requests
Based on the latest available HDF4 and HDF5
H4h5tools – updated to 18 APIs, no 18 features were added
CF – code freeze
6 weeks for HDF5
4 weeks for other products
10/16/2012
HDF Briefing to NASA

34. HDF4 maintenance releases
HDF released in February 2012
Extra functions to support H4 mapping project
Support for Linux PPC64 with IBM XL Fortran
Minor bug fixes and documentation improvements
HDF release in August 2012
Support for H4mapping project
Information about releases available from
10/16/2012
HDF Briefing to NASA

35. HDF5 maintenance releases
Many improvements in tools
ESDIS request to handle “/” in attributes names
Fortran 2003 support
Fortran wrappers for Dimension Scales APIs
Information about releases available from
HDF November 15, 2012
Many improvements in h5diff, minor bug fixe
10/16/2012
HDF Briefing to NASA

36. H4h5tools maintenance releases
2.2.1 released February, 15th, 2012
Based on HDF and HDF
Fixed bugs reported by ESDIS users
Upgraded to HDF5 1.8 APIs
Updated documents
We had 120 downloads in 2011
10/16/2012
HDF Briefing to NASA

37. Java maintenance releases
2.8 release December 2011
Addressed all issues reported by ESDIS users
2.9 release December 2012
Based on HDF and HDF
ESDIS requests:
Show groups/attributes in creation order
Export data to a binary/ascii file without having to open the object in the TableView
Here is the list
Adding an option to HDFView to apply bitmask instead of extracting bits
Allowing users to choose 0-based or 1-based indices
Importing/exporting image palettes
Adding an option to allow users to choose the origin of an image
Show index for 2D strings in the text viewer
Dealing with NaN values in XY plots and images
Adding capability to install HDFView in silent mode (no GUI)
Adding tests to hdf-java to make sure all objects are serializable, which is required for server/client applications.
10/16/2012
HDF Briefing to NASA

38. future directions
10/16/2012
HDF Briefing to NASA

39. General maintenance for 2013-2014
Document and improve maintenance processes and procedures
Focus on documentation for HDF4 and HDF5
Improve quality
Establish code review procedures for all HDF products
Expand static analysis to all HDF software packages
Enhance testing framework
Enhance testing framework
Daily regression testing
Release binaries testing
Performance testing
Backward/forward compatibility testing
10/16/2012
HDF Briefing to NASA

40. Future Maintenance Releases
2013
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
HDF4 CF
4.2.9
HDF5
1.8.11
1.8.12
HDF-Java
2.10
h4h5 tools
2.2.2
2014
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
HDF4 CF
4.2.10
HDF5
1.8.13
1.8.14
HDF-Java
2.11
h4h5 tools
2.2.3
CF – code freeze
6 weeks for HDF5
4 weeks for other products
10/16/2012
HDF Briefing to NASA

41. Future HDF4 maintenance releases
HDF planned for February 2013
Close as many issues as possible
HDF planned for February 2014
Complete “self-configuration” work for library maintainability and portability
ESDIS priorities?
Code clean-up
Make it portable as much as possible like HDF5
10/16/2012
HDF Briefing to NASA

42. Future HDF5 maintenance releases
HDF May 2013
Features sponsored by European Synchrotron Community (PSI/DECTRIS and DESY)
Writing compressed chunks directly to HDF5 file bypassing library internals
Finished prototype; achieved order of magnitude boost in I/O
Support for dynamically loaded third party libraries (compression and other filters)
Preliminary studies soon; several prototypes are already available
10/16/2012
HDF Briefing to NASA

43. Future HDF5 maintenance releases
How soon will MATLAB and IDL be able to support ?
ESDIS priorities?
10/16/2012
HDF Briefing to NASA

44. Future h4h5tools maintenance releases
2.2.2 release March 2013
Use latest HDF4 and HDF5 releases
Address any ESDIS issues if arise
2.2.3 release March 2014
Possible improvements
Conversion of dimension scales
Leveraging HDF5 1.8 features
ESDIS priorities?
We had 120 downloads in 2011
We know that there is a lot of data in HDF4, new data for NPOESS is coming in HDF5? One would think that there will be a need to work with both formats and converting from one to another.
10/16/2012
HDF Briefing to NASA

45. Future Java maintenance releases
2.10 release December 2013
We are gathering requirements (next slides)
10/16/2012
HDF Briefing to NASA

46. Tools update
10/16/2012
HDF Briefing to NASA

47. HDFView User survey
10/16/2012
HDF Briefing to NASA

48. Background
HDFView was created for browsing and editing HDF files on a local machine
The most popular HDF tool
August 2012 downloads
HDFView – 2,805
HDF5 binaries for all platforms – 1,073
Twelve releases since the first release in 1998
Time to reevaluate its capabilities and find out users’ expectations
10/16/2012
HDF Briefing to NASA

49. HDFView Survey Goals
Understand users needs and future directions for of HDFView maintenance and evolution
New requirements
Current deficiencies
Most desired improvements
Help with tasks prioritization
10/16/2012
HDF Briefing to NASA

50. HDFView Survey
Conducted a survey at
Over 40 questions in 8 different areas
About 200 responses
10/16/2012
HDF Briefing to NASA

51. Requested Improvements Areas
Import/export capabilities
Binary, text, Excel, tree-shape (XML, JSON), GeoTIFF, TIFF, KML (Google Earth), PNG, netCDF
Access
Cloud, OPeNDAP, mobile devices
Data discovery/search
Values, metadata, file structure
Data presentation
Plots, images, data with dimension scales
10/16/2012
HDF Briefing to NASA

52. Requested Improvements Areas
Usability
HDF5 functionality
Performance with large files
Support for all HDF5 datatypes
Support for custom filters
Data handling
Displaying, copying, selecting, exporting, importing
GUI features
One working window
Java vs. new technologies (QT, C++, ?)
UNIX remote display
Platform-specific requests
Mac OS X (installation, shortcuts)
Usability:
GUI:
Most respondents found that it is easy or moderately easy to use HDFView GUI, but several improvements were requested
10/16/2012
HDF Briefing to NASA

53. HDFView Survey
Report with raw data is available from
10/16/2012
HDF Briefing to NASA

54. HDF tools plan
10/16/2012
HDF Briefing to NASA

55. Plans for HDFView
Plan for and address high-priority items for ESDIS from the survey
Quality improvements
Clean issues database (37 right now)
Finish unit testing work (HDF4 JNI)
Set up GUI automatic testing
Performance improvements
Address large files issues
Evolution
Add support for dimension scales
Add support for misc. HDF5 features
10/16/2012
HDF Briefing to NASA

56. HDFView Survey
The ESDIS 2013 priority list as identified by the ESDSWG Executive Council available from
Topics relevant to HDFView
ESDIS standards: GeoTIFF and KML
Cloud and mobile computing
ESDS conventions for HDF5
CF conventions
netCDF interoperability
What are ESDIS priorities for HDFView?
10/16/2012
HDF Briefing to NASA

57. Plans for command-line tools
Regular maintenance
Clean up issues database
Focus on quality and performance
Focus on tools to close a gap between HDF5 and XML
Settle on new XML schema
H5gen to generate HDF5 from XML
Enhance XML output generated by h5dump
Possible open source experiment?
Small Apache project to learn to create tools that are developed and maintained primarily by the open source community.
ESDIS priorities?
10/16/2012
HDF Briefing to NASA

58. Open Source Initiative Action Plan
Mission Statement
Conduct a small Apache project that will enable us to learn to create tools for earth science data users that is developed and maintained primarily by the open source community.
Stakeholders
Key groups, organizations, individuals that will be impacted
Earth science data users.
EOS projects, Decadal Survey Missions, other NASA Earth Science data activities
All HDF communities
Who needs to be involved in developing deliverables.
Developers and end users who work with NASA earth science data.
Approach
Achieve approval from NASA to make this project a part of the ESDIS project (EED Task 2 Rev 8).
Identify a need that is likely to have community support. For example, a need for tools to convert HDF files to certain other formats.
Learn to create and lead an Apache Incubator Project.
Create an Apache Incubator project.
Develop tools.
Graduate the project from incubation.
Outcomes, Deliverables, Milestones
Specific outcome: the company learns to create an open source project in which HDF software is developed and used by the community
Work products
An Apache Incubator project
An Apache project supporting the software
Tools
Recommendation report
Milestones
Periodic reports on steps in the approach.
Incubator project established.
Apache project achieved.
Final report
10/16/2012
HDF Briefing to NASA

59. Platform support issues
10/16/2012
HDF Briefing to NASA

60. HDF4 platforms supported
Systems
Linux 2.6 (CentOS-5) Linux 2.6 x86_64
Linux PowerPC64
Solaris 5.10
32/64bit Windows XP, 7 and Cygwin
Mac OS Intel 10.6.* and 10.7.* 32/64-bit
Compilers
IBM C and Fortran cmp
GNU C and Fortran up to 4.6.2, g95 (4.0.3)
Intel C and Fortran v12
PGI C and Fortran v11
Sun C 5.10 and Fortran
VS 2008 Intel Fortran 10
VS 2010 Intel Fortran 10
For other flavors of Linux like Open SuSe, Ubuntu, Fedora we have test software but fixing bugs has a very low priority.
10/16/2012
HDF Briefing to NASA

61. HDF5 platforms supported
Systems
Same as for HDF4
AIX 5.3
Compilers
Same as for HDF4
IBM compilers
10/16/2012
HDF Briefing to NASA

62. HDF4 and HDF5 platforms to drop
Systems
Mac 10.6.*
Windows XP
Last releases HDF and HDF
Compilers
VS2008 with Intel 11 and MSVS build environment
We will use “CMake only” for Windows for all HDF products starting in Spring 2013
10/16/2012
HDF Briefing to NASA

63. HDF4 and HDF5 platforms to add
Systems
Mac 10.8.*
How important is 32-bit on 64-bit Mac OS X?
Windows 8
Spring 2013
Flavors of Linux?
Compilers
Clang?
VS 2012
Spring 2013
We are using virtualization very successfully.
Can add any Linux or Windows flavors.
Just let us know!
10/16/2012
HDF Briefing to NASA

64. Integration with complementary technologies and application domains
10/16/2012
HDF Briefing to NASA

65. opendap
10/16/2012
HDF Briefing to NASA

66. OPeNDAP – current status
HDF5-OPeNDAP handler
Re-engineered to support more NASA HDF5 and HDF-EOS5 products
HDF4-OPeNDAP handler
Enhanced the support of many NASA HDF-EOS2 and HDF4 products
More information in a separate talk
10/16/2012
HDF Briefing to NASA

67. Tools to improve interoperability between HDF and netcdf
10/16/2012
HDF Briefing to NASA

68. HDF and netCDF interoperabilty tools
HDF-EOS5 augmentation tool
HDF-EOS2 dumper tool
HDF-EOS5 to netCDF-4 conversion tool
HDF4/HDF-EOS2 to CF conversion library
More information in a separate talk
10/16/2012
HDF Briefing to NASA

69. HDF4 Layout Maps
10/16/2012
HDF Briefing to NASA

70. HDF4 Layouts Map
Provide tools and support to ensure long-term access to EOS data via HDF4 File Content Maps.
H4mapwriter release in June 2012
More information in a separate talk
10/16/2012
HDF Briefing to NASA

71. Thank You!
10/16/2012
HDF Briefing to NASA

72. Acknowledgements
This work was supported by Subcontract number under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.
10/16/2012
HDF Briefing to NASA