1. www.hdfgroup.org The HDF Group November 3-5, 2009 HDF-OPeNDAP Project Update HDF/HDF-EOS Workshop XIII1 Joe Lee and Kent Yang The HDF Group James Gallagher OPeNDAP, Inc. The 13 th HDF and HDF-EOS Workshop November 3-5, 2009

2. www.hdfgroup.orgNovember 3-5, 2009 Outline Introduction to OPeNDAP What’s New? o HDF5-OPeNDAP Handler (minor) o HDF4-OPeNDAP Handler (major) HDF/HDF-EOS Workshop XIII2

3. www.hdfgroup.orgNovember 3-5, 2009 Access HDF via FTP HDF/HDF-EOS Workshop XIII3 Users FTP Clients IDL FTP Servers HDF Files AIRS/OMI FTP Local HDF Files Aqua/Aura

4. www.hdfgroup.orgNovember 3-5, 2009 Client Library (libnc_dap) Data Access Protocol (DAP) via Internet Server (Hyrax) Handler (hdf4/5_handler) Remote Data (HDF4/5) View Data (User) OPeNDAP Concept 4 Visualization Tools (gradsdap) Typical Remote Access FTP/HTTP Visualization tools (e.g., grads) HDF/HDF-EOS Workshop XIII

5. www.hdfgroup.orgNovember 3-5, 2009 Access HDF via OPeNDAP HDF/HDF-EOS Workshop XIII5 Users OPeNDAP Clients IDV OPeNDAP Servers HDF Files Hyrax AIRS/OMI DAP libnc-dap Visualization Tools HDF HANDLER Aqua/Aura

6. www.hdfgroup.orgNovember 3-5, 2009 What’s New? HDF 5 -OPeNDAP Handler Work HDF/HDF-EOS Workshop XIII6

7. www.hdfgroup.orgNovember 3-5, 2009 HDF5 Handler Update HDF/HDF-EOS Workshop XIII7 Users OPeNDAP Clients IDV OPeNDAP Servers HDF5 Files Hyrax OMI Grid DAP libnc-dap Aura OMI Swath HDF5 HANDLER Update Visualization Tools

8. www.hdfgroup.orgNovember 3-5, 2009 HDF5-OPeNDAP Handler Update Support Aura HDF-EOS5 OMI Swath Access some files that follow HDF5 Dimension Scale Specification Change in Short Name Option HDF/HDF-EOS Workshop XIII8 RadiativeCloudFraction A3RadiativeClou

9. www.hdfgroup.orgNovember 3-5, 2009 Summary: Updated HDF5 Handler Official Release: October 6, 2009 http://opendap.org/hdf5_handler HDF/HDF-EOS Workshop XIII9

10. www.hdfgroup.orgNovember 3-5, 2009 What’s New? HDF 4 -OPeNDAP Handler Work HDF/HDF-EOS Workshop XIII10

11. www.hdfgroup.orgNovember 3-5, 2009 HDF4-OPeNDAP Handler Work Problems with the old HDF4 Handler Our Work Limitations Future Work HDF/HDF-EOS Workshop XIII11

12. www.hdfgroup.orgNovember 3-5, 2009 Old HDF4 Handler Problem OPeNDAP Visualization Tools CAN NOT display most NASA HDF-EOS2 and HDF4 data HDF/HDF-EOS Workshop XIII12

13. www.hdfgroup.orgNovember 3-5, 2009 Old HDF4 Handler Problem HDF/HDF-EOS Workshop XIII13 Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax AIRS Grid DAP libnc-dap Aqua Old HDF4 HANDLER Visualization Tools

14. www.hdfgroup.orgNovember 3-5, 2009 Our Work Let Visualization Clients Access HDF- EOS2 and HDF4 files HDF/HDF-EOS Workshop XIII14

15. www.hdfgroup.orgNovember 3-5, 2009 What’s Missing? My house costs 1. Data – Computer Science int my_house = 1; // Use boolean to save? HDF/HDF-EOS Workshop XIII15 My house in Hawaii costs $1M. Information – Earth Science Geo-Location, Geo-Location, Geo-Location Unit, Scale factor

16. www.hdfgroup.orgNovember 3-5, 2009 What’s Missing? Clients expect certain data structures o DAP Grids and Shared dimensions Clients expect some key attributes o CF conventions HDF/HDF-EOS Workshop XIII16

17. www.hdfgroup.orgNovember 3-5, 2009 Solution: Metadata Parser HDF/HDF-EOS Workshop XIII17 Goal: Meet the Client’s Requirements Parse the structMetadata to o generate the DAP Grids and Shared Dims o generate the CF-compliant attributes

18. www.hdfgroup.orgNovember 3-5, 2009 New HDF4 Handler (parser) HDF/HDF-EOS Workshop XIII18 Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax AIRS Grid DAP libnc-dap Aqua Old HDF4 HANDLER Visualization Tools New HDF4 HANDLER (parser)

19. www.hdfgroup.orgNovember 3-5, 2009 Demo: Parser Solution HDF/HDF-EOS Workshop XIII19 Problem: Only a few HDF-EOS2 files work! AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10 MODIS Grid MOD08_D3.A2008284.005.2008289105354.hdf http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08 331121849.hdf MODIS Swath MOD06_L2.A2001222.0025.004.2003127111841.hdf http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html

20. www.hdfgroup.orgNovember 3-5, 2009 New Challenges for HDF4 Handler Hundreds of HDF-EOS2 and HDF4 products o Grid Projections o Swath Dimension Map HDF/HDF-EOS Workshop XIII20

21. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII21 HDF-EOS2 Grid projections Geographic Sinusoidal Polar Stereographic etc. New Challenges HDF-EOS2 Swath Dimension Map

22. www.hdfgroup.orgNovember 3-5, 2009 Solution: Use HDF-EOS2 Library HDF/HDF-EOS Workshop XIII22 Goal: Support more HDF-EOS2 products Use HDF-EOS2 Library to o handle different Grid projections o handle Swath dimension map

23. www.hdfgroup.orgNovember 3-5, 2009 New HDF4 Handler (EOS2 library) HDF/HDF-EOS Workshop XIII23 Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax MODIS Swath DAP libnc-dap Aura MODIS Swath w/ Dim. Map New HDF4 HANDLER (parser)New HDF4 HANDLER (EOS2 library) Visualization Tools New HDF4 Handler (Parser)

24. www.hdfgroup.orgNovember 3-5, 2009 Demo: HDF-EOS2 Library Solution HDF/HDF-EOS Workshop XIII24 Swath with Dimension Map

25. www.hdfgroup.orgNovember 3-5, 2009 Two Solutions Metadata Parser Pro: Simple installation Con: Limited products HDF-EOS2 Library Pro: More products Con: HE2 library installation HDF/HDF-EOS Workshop XIII25

26. www.hdfgroup.orgNovember 3-5, 2009 Two Solutions - Implementation Metadata Parser hdf5_handler HDF-EOS2 Library h4toh5 tool HDF/HDF-EOS Workshop XIII26 Adapted the implementation of previous software packages

27. www.hdfgroup.orgNovember 3-5, 2009 Solution Comparison HDF/HDF-EOS Workshop XIII27

28. www.hdfgroup.orgNovember 3-5, 2009 Live Demo Tutorial Example o http://hdfeos.org/software/idv.php Demo Servers o http://hdfgroup.org/projects/opendap o http://hdfgroup.org/projects/opendap4 HDF/HDF-EOS Workshop XIII28

29. www.hdfgroup.orgNovember 3-5, 2009 Limitations 7 Reasons for Visualization Failures Sample Files for Visualization Failures o http://hdfgroup.org/projects/opendap4/limits.html HDF/HDF-EOS Workshop XIII29

30. www.hdfgroup.orgNovember 3-5, 2009 The Tip of the Iceberg HDF/HDF-EOS Workshop XIII30 The current HDF4 handler CAN NOT visualize most of NASA HDF-EOS2 / HDF4 files

31. www.hdfgroup.orgNovember 3-5, 2009 More Challenges for HDF4 Handler Tackle limitations and other challenges. Handle NASA HDF4 (e.g.,CERES) files. In 2010 HDF/HDF-EOS Workshop XIII31

32. www.hdfgroup.orgNovember 3-5, 2009 Summary: Enhanced HDF4 Handler Official Release on Oct. 6, 2009. http://opendap.org/hdf4_handler Support some HDF-EOS2 Grid / Swath files Use HDF-EOS2 library Future work: support more NASA HDF- EOS2 / HDF4 files HDF/HDF-EOS Workshop XIII32

33. www.hdfgroup.org The HDF Group November 3-5, 2009 Thank You! HDF/HDF-EOS Workshop XIII33

34. www.hdfgroup.orgNovember 3-5, 2009 Acknowledgements This work was supported by cooperative agreement number NNX06AC83A, NNX08AO77A and NNX06AG75A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. HDF/HDF-EOS Workshop XIII34

35. www.hdfgroup.org The HDF Group November 3-5, 2009 Questions/comments? HDF/HDF-EOS Workshop XIII35