Presentation is loading. Please wait.

Presentation is loading. Please wait.

Www.hdfgroup.org The HDF Group HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group Briefing to ESDIS October 16-17, 2012 10/16/2012HDF.

Similar presentations


Presentation on theme: "Www.hdfgroup.org The HDF Group HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group Briefing to ESDIS October 16-17, 2012 10/16/2012HDF."— Presentation transcript:

1 www.hdfgroup.org The HDF Group HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group Briefing to ESDIS October 16-17, 2012 10/16/2012HDF Briefing to NASA 1

2 www.hdfgroup.org Project Goals Support for NASA HDF data providers and data users, and the EOSDIS Core System (ECS) Maintenance and quality assurance for HDF4 and HDF5 libraries and utilities Evolving the HDF4 and HDF5 libraries and utilities to meet new NASA EOS requirement Integration with complementary technologies and application domains Prototype study of web technologies to access HDF data 10/16/2012HDF Briefing to NASA2

3 www.hdfgroup.org The HDF Group Support activities for NASA HDF data providers and users and ECS 10/16/2012HDF Briefing to NASA 3

4 www.hdfgroup.org Activities Help Desk Outreach Mailing lists, Web site Conferences, Workshops, Other EOS support netCDF support and collaboration Later today: HDF OPeNDAP support Support for NASA data products SMAP, ICESat-2, MEaSURES) 10/16/2012HDF Briefing to NASA4

5 www.hdfgroup.org HELP DESK 10/16/2012HDF Briefing to NASA5

6 www.hdfgroup.org The HDF Group Help Desk High-priority support for ESDIS users Fast response Problem investigation and resolution Performance tuning Access to HDF developers Basic support for entire HDF user community, as resources permit help@hdfgroup.org 10/16/2012HDF Briefing to NASA6

7 www.hdfgroup.org Help Desk issues resolution time * Number of messages up do August 31, 2012 10/16/2012HDF Briefing to NASA7

8 www.hdfgroup.org Help Desk issues by years * Number of messages up do August 31, 2012 10/16/2012HDF Briefing to NASA8

9 www.hdfgroup.org OUTREACH 10/16/2012HDF Briefing to NASA9

10 www.hdfgroup.org Outreach Web pages improvements FAQ support for HDF4 and HDF5 http://www.hdfgroup.org/HDF5-FAQ.html http://www.hdfgroup.org/HDF-FAQ.html Later today: Detailed talk on support for EOS (Kent) Organizing material based on the ESDIS survey results and feedback from the users Example: Tools by category (work in progress) How do I edit, copy, import/export data with HDF? 10/16/2012HDF Briefing to NASA10

11 www.hdfgroup.org Tools by category 10/16/2012HDF Briefing to NASA11

12 www.hdfgroup.org Outreach Mailing list and archives news@hdfgroup.org http://www.hdfgroup.org/news/ hdf-forum@hdfgroup.org http://mail.hdfgroup.org/pipermail/hdf- forum_hdfgroup.org/ 10/16/2012HDF Briefing to NASA12

13 www.hdfgroup.org hdf-forum@hdfgroup.org Started in Summer 2007 Monitored by the members of the HDF group Number of messages: Most discussed topics: Parallel HDF5 Advanced HDF5 features (drivers, complex types, performance).NET and C++ wrappers, HL libraries 2012 to Sep 3020112010 7031,3341,721 10/16/2012HDF Briefing to NASA13

14 www.hdfgroup.org hdf-forum@hdfgroup.org Members of the FORUM help with Releases testing on a regular basis Maintaining CMake build systems on platforms beyond Windows Answering questions The Help Desk focuses on ESDIS and other paying customers while referring others to the FORUM for topics that require domain knowledge or specific HDF5 usage Reporting defects Securing funding for The HDF Group 10/16/2012HDF Briefing to NASA14

15 www.hdfgroup.org Conferences, Workshops, Other 2011 IDV/THREDDS Workshop, July 2011 Learned how to use IDV and THREDDS 10 th NASA ESDSWG Conference, Nov 2011 Poster on HDF and netCDF interoperability 2012 ESIP Summer Meeting, July 2012 Presentations HDF/HDF-EOS Data for Dummies What does Open Source mean for HDF? Poster Whats new at hdfgroup.org? 10/16/2012HDF Briefing to NASA15

16 www.hdfgroup.org Conferences, Workshops, Other 15 th HDF and HDF-EOS Workshop, April 2012 Five tutorials, nine presentations and lightning talks, and one special session Materials: http://www.hdfeos.org/workshops/ws15/agenda.php http://www.hdfeos.org/workshops/ws15/agenda.php 10/16/2012HDF Briefing to NASA16

17 www.hdfgroup.org Conferences, Workshops, Other Advised ICESat-2, MOPITT and several MEaSURES projects on interoperability with netCDF-4 Participated in the SMAP ISO-19115 work Participated in ESDSWG SPG telecons Joined several ESDSWG mailing lists related to HDF 10/16/2012HDF Briefing to NASA17

18 www.hdfgroup.org NETCDF SUPPORT AND COLLABORATION 10/16/2012HDF Briefing to NASA18

19 www.hdfgroup.org NetCDF-4 support Help Unidata netCDF-4 developers Frequent communications with netCDF-4 developers Provided a fix for ncdump critical to the ESDIS users Test netCDF-4 daily with the HDF5 development code Plan to participate in Unidata Advanced netCDF Workshop on October 26, 2012 10/16/2012HDF Briefing to NASA19

20 www.hdfgroup.org EOS SUPPORT 10/16/2012HDF Briefing to NASA20

21 www.hdfgroup.org EOS Support Submitted several bugs and the corresponding fixes to the HDF-EOS team EOS2 and EOS5 are tested daily with HDF4 and HDF5 development code HDF-EOS website Actively maintain the website Improve the quality of the example codes and plots to access NASA HDF and HDF-EOS data via MATLAB, IDL and NCL More information in a separate talk 10/16/2012HDF Briefing to NASA21

22 www.hdfgroup.org The HDF Group Maintenance and quality assurance for HDF4 and HDF5 libraries and utilities 10/16/2012HDF Briefing to NASA 22

23 www.hdfgroup.org Activities Manage and resolve issues Perform maintenance releases of HDF software Support OSs and compilers identified by ESDIS Address issues and software evolution requests from ESDIS Perform SQE and SQA activities Put in place missing processes Improve existing processes Build in software quality by performing testing and using the best software development practices 10/16/2012HDF Briefing to NASA23

24 www.hdfgroup.org HDF ISSUES 10/16/2012HDF Briefing to NASA24

25 www.hdfgroup.org Issues and their priorities Must Fix Data corruption Portability Backward and Forward Compatibility Funded Requests Fix after Must Fix Power User Requests Tools Library issues Build Infrastructure When resources permit Wrappers HL Libraries Other Need your input on priorities! 10/16/2012HDF Briefing to NASA25

26 www.hdfgroup.org HDF4 opened and closed issues 10/16/2012HDF Briefing to NASA26

27 www.hdfgroup.org HDF5 opened and closed issues 10/16/2012HDF Briefing to NASA27

28 www.hdfgroup.org SQE AND SQA AT THE HDF GROUP 10/16/2012HDF Briefing to NASA28

29 www.hdfgroup.org Activities CMMI appraisal and process improvement action plan (Dec. 2011) Created SQE position Focus on Implementing CMMI best practices and procedures Enforcing current practices and procedures Focused on improving software testing 10/16/2012HDF Briefing to NASA29

30 www.hdfgroup.org Activities Worked on establishing a culture of code reviews for the HDF libraries and tools Improved releases planning and monitoring for HDF5 Used static analysis tools to identify defects Coverity (used at THG) Klocwork (used at LLNL) 10/16/2012HDF Briefing to NASA30

31 www.hdfgroup.org HDF MAINTENANCE RELEASES 10/16/2012HDF Briefing to NASA31

32 www.hdfgroup.org General maintenance for 2011-2012 Implemented CMake for all HDF software Worked on closing reported issues Focused on performance enhancements Improved HDF documentation http://www.hdfgroup.org/HDF5/doc/Advanced.html Explored the needs for internal documentation Preventive measures: Coverity static analysis; Rational tools; valgrind Have been exploring new testing tools to sustain testing for HDF products (CDash, Bamboo, Cloud) 10/16/2012HDF Briefing to NASA32

33 www.hdfgroup.org 10/16/2012HDF Briefing to NASA33 2011MayJunJulAugSepOctNovDec HDF4 HDF51.8.71.8.8 HDF-Java2.8 h4h5 tools CF – code freeze 6 weeks for HDF5 4 weeks for other products 2012JanFebMarAprMayJunJulAugSepOctNovDec HDF44.2.7 patch1 4.2.8 HDF51.8.9CF1.8.10 HDF- Java CF2.9 h4h5 tools 2.2.1 Maintenance Releases 2011 – 2012

34 www.hdfgroup.org HDF4 maintenance releases HDF 4.2.7 released in February 2012 Extra functions to support H4 mapping project Support for Linux PPC64 with IBM XL Fortran Minor bug fixes and documentation improvements HDF 4.2.8 release in August 2012 Support for H4mapping project Information about releases available from http://www.hdfgroup.org/ftp/HDF/prev-releases/ReleaseFiles/ http://www.hdfgroup.org/ftp/HDF/prev-releases/ReleaseFiles/ 10/16/2012HDF Briefing to NASA34

35 www.hdfgroup.org HDF5 maintenance releases HDF5 1.8.7-1.8.9 Many improvements in tools ESDIS request to handle / in attributes names Fortran 2003 support Fortran wrappers for Dimension Scales APIs Information about releases available from http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/ http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/ HDF5 1.8.10 November 15, 2012 Many improvements in h5diff, minor bug fixe 10/16/2012HDF Briefing to NASA35

36 www.hdfgroup.org H4h5tools maintenance releases 2.2.1 released February, 15 th, 2012 Based on HDF 4.2.7 and HDF5 1.8.8 Fixed bugs reported by ESDIS users Upgraded to HDF5 1.8 APIs Updated documents 10/16/2012HDF Briefing to NASA36

37 www.hdfgroup.org Java maintenance releases 2.8 release December 2011 Addressed all issues reported by ESDIS users 2.9 release December 2012 Based on HDF 4.2.8 and HDF5 1.8.10 ESDIS requests: Show groups/attributes in creation order Export data to a binary/ascii file without having to open the object in the TableView 10/16/2012HDF Briefing to NASA37

38 www.hdfgroup.org FUTURE DIRECTIONS 10/16/2012HDF Briefing to NASA38

39 www.hdfgroup.org General maintenance for 2013-2014 Document and improve maintenance processes and procedures Focus on documentation for HDF4 and HDF5 Improve quality Establish code review procedures for all HDF products Expand static analysis to all HDF software packages Enhance testing framework 10/16/2012HDF Briefing to NASA39

40 www.hdfgroup.org Future Maintenance Releases 10/16/2012HDF Briefing to NASA40 2013JanFebMarAprMayJunJulAugSepOctNovDec HDF4CF4.2.9 HDF5CF1.8.11CF1.8.12 HDF- Java CF2.10 h4h5 tools CF2.2.2 CF – code freeze 6 weeks for HDF5 4 weeks for other products 2014JanFebMarAprMayJunJulAugSepOctNovDec HDF4CF 4.2.10 HDF5CF1.8.13CF1.8.14 HDF- Java CF2.11 h4h5 tools CF2.2.3

41 www.hdfgroup.org Future HDF4 maintenance releases HDF 4.2.9 planned for February 2013 Close as many issues as possible HDF 4.2.10 planned for February 2014 Complete self-configuration work for library maintainability and portability ESDIS priorities? 10/16/2012HDF Briefing to NASA41

42 www.hdfgroup.org Future HDF5 maintenance releases HDF5 1.8.11 May 2013 Features sponsored by European Synchrotron Community (PSI/DECTRIS and DESY) Writing compressed chunks directly to HDF5 file bypassing library internals Finished prototype; achieved order of magnitude boost in I/O Support for dynamically loaded third party libraries (compression and other filters) Preliminary studies soon; several prototypes are already available 10/16/2012HDF Briefing to NASA42

43 www.hdfgroup.org Future HDF5 maintenance releases How soon will MATLAB and IDL be able to support 1.8.11? ESDIS priorities? 10/16/2012HDF Briefing to NASA43

44 www.hdfgroup.org Future h4h5tools maintenance releases 2.2.2 release March 2013 Use latest HDF4 and HDF5 releases Address any ESDIS issues if arise 2.2.3 release March 2014 Possible improvements Conversion of dimension scales Leveraging HDF5 1.8 features ESDIS priorities? 10/16/2012HDF Briefing to NASA44

45 www.hdfgroup.org Future Java maintenance releases 2.10 release December 2013 We are gathering requirements (next slides) 10/16/2012HDF Briefing to NASA45

46 www.hdfgroup.org TOOLS UPDATE 10/16/2012HDF Briefing to NASA46

47 www.hdfgroup.org HDFVIEW USER SURVEY 10/16/2012HDF Briefing to NASA47

48 www.hdfgroup.org Background HDFView was created for browsing and editing HDF files on a local machine The most popular HDF tool August 2012 downloads HDFView – 2,805 HDF5 binaries for all platforms – 1,073 Twelve releases since the first release in 1998 Time to reevaluate its capabilities and find out users expectations 10/16/2012HDF Briefing to NASA48

49 www.hdfgroup.org HDFView Survey Goals Understand users needs and future directions for of HDFView maintenance and evolution New requirements Current deficiencies Most desired improvements Help with tasks prioritization 10/16/2012HDF Briefing to NASA49

50 www.hdfgroup.org HDFView Survey Conducted a survey at https://www.surveymonkey.com/s/PLGRLJY https://www.surveymonkey.com/s/PLGRLJY Over 40 questions in 8 different areas About 200 responses 10/16/2012HDF Briefing to NASA50

51 www.hdfgroup.org Requested Improvements Areas Import/export capabilities Binary, text, Excel, tree-shape (XML, JSON), GeoTIFF, TIFF, KML (Google Earth), PNG, netCDF Access Cloud, OPeNDAP, mobile devices Data discovery/search Values, metadata, file structure Data presentation Plots, images, data with dimension scales 10/16/2012HDF Briefing to NASA51

52 www.hdfgroup.org Requested Improvements Areas Usability HDF5 functionality Performance with large files Support for all HDF5 datatypes Support for custom filters Data handling Displaying, copying, selecting, exporting, importing GUI features One working window Java vs. new technologies (QT, C++, ?) UNIX remote display Platform-specific requests Mac OS X (installation, shortcuts) 10/16/2012HDF Briefing to NASA52

53 www.hdfgroup.org HDFView Survey Report with raw data is available from http://www.hdfgroup.org/hdf-java-html/hdfview/ http://www.hdfgroup.org/hdf-java-html/hdfview/ 10/16/2012HDF Briefing to NASA53

54 www.hdfgroup.org HDF TOOLS PLAN 10/16/2012HDF Briefing to NASA54

55 www.hdfgroup.org Plans for HDFView Plan for and address high-priority items for ESDIS from the survey Quality improvements Clean issues database (37 right now) Finish unit testing work (HDF4 JNI) Set up GUI automatic testing Performance improvements Address large files issues Evolution Add support for dimension scales Add support for misc. HDF5 features 10/16/2012HDF Briefing to NASA55

56 www.hdfgroup.org HDFView Survey The ESDIS 2013 priority list as identified by the ESDSWG Executive Council available from http://earthdata.nasa.gov/wiki/main/index.php/The_ESDIS_2013_Priority_List Topics relevant to HDFView ESDIS standards: GeoTIFF and KML Cloud and mobile computing ESDS conventions for HDF5 CF conventions netCDF interoperability What are ESDIS priorities for HDFView? 10/16/2012HDF Briefing to NASA56

57 www.hdfgroup.org Plans for command-line tools Regular maintenance Clean up issues database Focus on quality and performance Focus on tools to close a gap between HDF5 and XML Settle on new XML schema H5gen to generate HDF5 from XML Enhance XML output generated by h5dump Possible open source experiment? Small Apache project to learn to create tools that are developed and maintained primarily by the open source community. ESDIS priorities? 10/16/2012HDF Briefing to NASA57

58 www.hdfgroup.org 58 Open Source Initiative Action Plan Mission Statement Conduct a small Apache project that will enable us to learn to create tools for earth science data users that is developed and maintained primarily by the open source community. Stakeholders Key groups, organizations, individuals that will be impacted Earth science data users. EOS projects, Decadal Survey Missions, other NASA Earth Science data activities All HDF communities Who needs to be involved in developing deliverables. Developers and end users who work with NASA earth science data. Approach Achieve approval from NASA to make this project a part of the ESDIS project (EED Task 2 Rev 8). Identify a need that is likely to have community support. For example, a need for tools to convert HDF files to certain other formats. Learn to create and lead an Apache Incubator Project. Create an Apache Incubator project. Develop tools. Graduate the project from incubation. Outcomes, Deliverables, Milestones Specific outcome: the company learns to create an open source project in which HDF software is developed and used by the community Work products An Apache Incubator project An Apache project supporting the software Tools Recommendation report Milestones Periodic reports on steps in the approach. Incubator project established. Apache project achieved. Final report 10/16/2012HDF Briefing to NASA

59 www.hdfgroup.org PLATFORM SUPPORT ISSUES 10/16/2012HDF Briefing to NASA59

60 www.hdfgroup.org HDF4 platforms supported Systems Linux 2.6 (CentOS-5) Linux 2.6 x86_64 Linux PowerPC64 Solaris 5.10 32/64bit Windows XP, 7 and Cygwin Mac OS Intel 10.6.* and 10.7.* 32/64-bit Compilers IBM C and Fortran cmp GNU C and Fortran up to 4.6.2, g95 (4.0.3) Intel C and Fortran v12 PGI C and Fortran v11 Sun C 5.10 and Fortran 95 8.4 VS 2008 Intel Fortran 10 VS 2010 Intel Fortran 10 10/16/2012HDF Briefing to NASA60

61 www.hdfgroup.org HDF5 platforms supported Systems Same as for HDF4 AIX 5.3 Compilers Same as for HDF4 IBM compilers 10/16/2012HDF Briefing to NASA61

62 www.hdfgroup.org HDF4 and HDF5 platforms to drop Systems Mac 10.6.* Windows XP Last releases HDF 4.2.9 and HDF5 1.8.10 Compilers VS2008 with Intel 11 and MSVS build environment We will use CMake only for Windows for all HDF products starting in Spring 2013 10/16/2012HDF Briefing to NASA62

63 www.hdfgroup.org HDF4 and HDF5 platforms to add Systems Mac 10.8.* How important is 32-bit on 64-bit Mac OS X? Windows 8 Spring 2013 Flavors of Linux? Compilers Clang? VS 2012 Spring 2013 We are using virtualization very successfully. Can add any Linux or Windows flavors. Just let us know! 10/16/2012HDF Briefing to NASA63

64 www.hdfgroup.org The HDF Group Integration with complementary technologies and application domains 10/16/2012HDF Briefing to NASA 64

65 www.hdfgroup.org OPENDAP 10/16/2012HDF Briefing to NASA65

66 www.hdfgroup.org OPeNDAP – current status HDF5-OPeNDAP handler Re-engineered to support more NASA HDF5 and HDF-EOS5 products HDF4-OPeNDAP handler Enhanced the support of many NASA HDF- EOS2 and HDF4 products More information in a separate talk 10/16/2012HDF Briefing to NASA66

67 www.hdfgroup.org TOOLS TO IMPROVE INTEROPERABILITY BETWEEN HDF AND NETCDF 10/16/2012HDF Briefing to NASA67

68 www.hdfgroup.org HDF and netCDF interoperabilty tools HDF-EOS5 augmentation tool HDF-EOS2 dumper tool HDF-EOS5 to netCDF-4 conversion tool HDF4/HDF-EOS2 to CF conversion library More information in a separate talk 10/16/2012HDF Briefing to NASA68

69 www.hdfgroup.org The HDF Group HDF4 Layout Maps 10/16/2012HDF Briefing to NASA 69

70 www.hdfgroup.org HDF4 Layouts Map Provide tools and support to ensure long-term access to EOS data via HDF4 File Content Maps. H4mapwriter release 1.0.3 in June 2012 More information in a separate talk 10/16/2012HDF Briefing to NASA70

71 www.hdfgroup.org The HDF Group Thank You! 10/16/2012HDF Briefing to NASA 71

72 www.hdfgroup.org Acknowledgements 10/16/2012HDF Briefing to NASA This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration. 72


Download ppt "Www.hdfgroup.org The HDF Group HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group Briefing to ESDIS October 16-17, 2012 10/16/2012HDF."

Similar presentations


Ads by Google