1. www.hdfgroup.org The HDF Group HDF/HDF-EOS Workshop XV1 HDF-OPeNDAP Project Update Joe Lee and Kent Yang The HDF Group April 18, 2012

2. www.hdfgroup.org Outline Introduction to OPeNDAP HDF5 Handler Update HDF4 Handler Update Live Demo HDF/HDF-EOS Workshop XV2April 18, 2012

3. www.hdfgroup.org Introduction to OPeNDAP Goal: Background on Update Why OPeNDAP? What is HDF Handler? How important CF conventions? April 18, 2012HDF/HDF-EOS Workshop XV3

4. www.hdfgroup.org Access HDF via FTP HDF/HDF-EOS Workshop XV4 Users FTP Clients IDL FTP Servers HDF Files AIRS/OMI FTP Local HDF Files Aqua/Aura April 18, 2012

5. www.hdfgroup.org Client Library (libnc_dap) Data Access Protocol (DAP) via Internet Server (Hyrax) Handler (hdf4/5_handler) Remote Data (HDF4/5) View Data (User) OPeNDAP concept 5 Visualization Tools (gradsdap) Typical remote access problem FTP/HTTP Visualization tools (e.g., GrADS) HDF/HDF-EOS Workshop XVApril 18, 2012

6. www.hdfgroup.org Access remote data easily April 18, 2012HDF/HDF-EOS Workshop XV6

7. www.hdfgroup.org without worrying about data plans. April 18, 2012HDF/HDF-EOS Workshop XV7

8. www.hdfgroup.org Visualize remote data easily, too. April 18, 2012HDF/HDF-EOS Workshop XV8

9. www.hdfgroup.org HDF handler is key HDF/HDF-EOS Workshop XV9 Users OPeNDAP Clients IDV OPeNDAP Servers HDF Files Hyrax AIRS/OMI DAP NetCDF Java Visualization Tools HDF HANDLER Aqua/Aura April 18, 2012

10. www.hdfgroup.org because if handler doesn’t follow CF April 18, 2012HDF/HDF-EOS Workshop XV10 No Visualization Bonus!

11. www.hdfgroup.org If handler gives up extra bonus April 18, 2012HDF/HDF-EOS Workshop XV11 HDF / HDF-EOS DAP Server DAP handler development is easier.

12. www.hdfgroup.org What’s involved in extra bonus April 18, 2012HDF/HDF-EOS Workshop XV12 DAP Server CF Conventions DAP DAP Client Visualization Earth Scientist HDF / HDF-EOS

13. www.hdfgroup.org How handler developers feel April 18, 2012HDF/HDF-EOS Workshop XV13

14. www.hdfgroup.org Because we care all NASA products. April 18, 2012HDF/HDF-EOS Workshop XV14 ? AURA NPP AQUA TERRA MISR

15. www.hdfgroup.org Developers’ Dream April 18, 2012HDF/HDF-EOS Workshop XV15 … all data are created equal, that they shall be visualized by all client tools easily…

16. www.hdfgroup.org What if all people agreed… April 18, 2012HDF/HDF-EOS Workshop XV16 UDF (Imaginary) DAP Server DAP DAP Client Visualization Earth Scientist development might have been much easier. Check once.

17. www.hdfgroup.org Lessons from handler development April 18, 2012HDF/HDF-EOS Workshop XV17 Communicate and work together among different groups. Standardize everything from data products to visualization tools. Follow conventions and standards.

18. www.hdfgroup.org Outline Introduction to OPeNDAP HDF5-OPeNDAP Handler Update HDF4-OPeNDAP Handler Update Live Demo HDF/HDF-EOS Workshop XV18April 18, 2012

19. www.hdfgroup.org Redesign of HDF5 handler Goal: Support all NASA HDF5 products. Modularize for future product support. Follow CF conventions robustly. April 18, 2012HDF/HDF-EOS Workshop XV19 Product

20. www.hdfgroup.org Why did we redesign? April 18, 2012HDF/HDF-EOS Workshop XV20 In 2007, was enough for AURA Grid HDF5 handler

21. www.hdfgroup.org Why did we redesign? April 18, 2012HDF/HDF-EOS Workshop XV21 In 2011, wasn’t enough for AURA Grid AURA Swath AURA ZA GSSTF

22. www.hdfgroup.org Why did we redesign? April 18, 2012HDF/HDF-EOS Workshop XV22 In 2011, wasn’t enough for AURA Grid AURA Swath AURA ZA GSSTF

23. www.hdfgroup.org and more… so we gave up! April 18, 2012HDF/HDF-EOS Workshop XV23 GOSAT ICESat-2 Aquarius SMAP

24. www.hdfgroup.org We rebuilt one that can scale up April 18, 2012HDF/HDF-EOS Workshop XV24

25. www.hdfgroup.org by modularizing the design April 18, 2012HDF/HDF-EOS Workshop XV25 HDF5 handlerCFHDF5GOSATAquariusSMAP HDF- EOS5 GRIDSWATHZAGeneric with a big room for future products.

26. www.hdfgroup.org Key updates on HDF5 handler April 18, 2012HDF/HDF-EOS Workshop XV26 version1.5.x1.6.x CF-convention Support PrimitiveRobust Range of NASA Data Products LimitedAll Mulitiple Grids/Swath/ZA NoYes Configuration Options Static BuildRun-time Support for Future Products RigidFlexible

27. www.hdfgroup.org HDF-EOS5 products supported OMI Swath, Grid MLS Swath HIRDLS Swath, ZA TES Swath, Grid GSSTF April 18, 2012HDF/HDF-EOS Workshop XV27

28. www.hdfgroup.org HDF5 products supported Aquarius Grid MEaSUREs SeaWiFS DeepBlue SBUV2 Ozone (O3) Profile SMAP (Simulation) April 18, 2012HDF/HDF-EOS Workshop XV28

29. www.hdfgroup.org HIRDLS Zonal Average April 18, 2012HDF/HDF-EOS Workshop XV29

30. www.hdfgroup.org New TES Swath April 18, 2012HDF/HDF-EOS Workshop XV30

31. www.hdfgroup.org MEaSUREs GSSTF April 18, 2012HDF/HDF-EOS Workshop XV31

32. www.hdfgroup.org Aquarius Grid (Simulation) April 18, 2012HDF/HDF-EOS Workshop XV32

33. www.hdfgroup.org MEaSUREs SeaWiFS DeepBlue April 18, 2012HDF/HDF-EOS Workshop XV33

34. www.hdfgroup.org Ozone Profile (BUV) April 18, 2012HDF/HDF-EOS Workshop XV34

35. www.hdfgroup.org SMAP Swath (Simulation) April 18, 2012HDF/HDF-EOS Workshop XV35

36. www.hdfgroup.org SMAP Swath (Simulation) April 18, 2012HDF/HDF-EOS Workshop XV36

37. www.hdfgroup.org No CF magic for some products Tool / CF Limitation GOSAT/acos: 1-D trajectory ICESat-2 OMI L2G: Lat/Lon with 3-D Java-NetCDF Client Limitation Dataset w/ long string April 18, 2012HDF/HDF-EOS Workshop XV37

38. www.hdfgroup.org It doesn’t mean handler failure April 18, 2012HDF/HDF-EOS Workshop XV38 because other tools like can still access and visualize data.

39. www.hdfgroup.org MATLAB and IDL will work too. April 18, 2012HDF/HDF-EOS Workshop XV39 HDF5 Examples under zoo will work. Visit http://hdfeos.org/zoo Replace local file access with OPeNDAP URL access. MATLAB: Try new OPeNDAP functions. The rest will be same.

40. www.hdfgroup.org Read if you can’t see April 18, 2012HDF/HDF-EOS Workshop XV40 http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/ Aura_OMI_Level3/OMTO3e.003/2012/OMI-Aura_L3- OMTO3e_2012m0416_v003- 2012m0418t020608.he5.ascii?lon[1029:1:1029],lat[513:1:513],_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[513:1:513][1029:1:1029] Dataset: OMI-Aura_L3-OMTO3e_2012m0416_v003- 2012m041 8t020608.he5 lon, 77.375 lat, 38.375 _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3.lon, 77.375 _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3._HDFEOS_GRIDS_O MI_Column_Amount_O3_Data_Fields_ColumnAmountO3[_HDFEOS_GRIDS_OMI_Column_Amount_ O3_Data_Fields_ColumnAmount O3.lat=38.375], 309.6

41. www.hdfgroup.org or hear from Siri? April 18, 2012HDF/HDF-EOS Workshop XV41

42. www.hdfgroup.org Outline Introduction HDF5-OPeNDAP Handler Update HDF4-OPeNDAP Handler Update Live Demo HDF/HDF-EOS Workshop XV42April 18, 2012

43. www.hdfgroup.org HDF4 OPeNDAP handler update Corrected the handling the scale/offset for MODIS products. Added the support for Lambert azimuthal equal-area (LAMAZ). Added the support for Space-oblique Mercator projection (SOM). Other minor bug fixes. HDF/HDF-EOS Workshop XV43April 18, 2012

44. www.hdfgroup.org Key updates on HDF4 handler April 18, 2012HDF/HDF-EOS Workshop XV44 version3.9.33.9.4 Some MODIS Scale/offset WrongCorrected LAMAZ projection support NoYes SOM projection support NoYes

45. www.hdfgroup.org MODIS scale / offset correction HDF/HDF-EOS Workshop XV45April 18, 2012

46. www.hdfgroup.org Support for LAMAZ projection HDF/HDF-EOS Workshop XV46April 18, 2012

47. www.hdfgroup.org Support for SOM projection HDF/HDF-EOS Workshop XV47April 18, 2012 Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[XDimD2 = 128][YDimD3 = 512]; … } Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; … }

48. www.hdfgroup.org HDF-EOS2 products supported GES DISC AIRS / MERRA / TOMS LAADS / LP DAAC / NSIDC Many MODIS products NSIDC AMSR_E / NISE / MODIS LaRC MISR / MOPITT / some CERES (TRMM) HDF/HDF-EOS Workshop XV48April 18, 2012

49. www.hdfgroup.org HDF4 products supported GES DISC TRMM L1B, L2B Swath TRMM L3 Grid 42B & 43B products OBPG (Ocean Color) SeaWiFS / MODIST / MODISA CZCS / OCTS HDF/HDF-EOS Workshop XV49April 18, 2012

50. www.hdfgroup.org HDF4 products supported CERES CER_AVG CER_ES4 CER_ISCCP-D2like-Day CER_ISCCP-D2like-GEO CER_SRBAVG CER_SYN CER_ZAVG HDF/HDF-EOS Workshop XV50April 18, 2012

51. www.hdfgroup.org HDF4 handler limitations HDF-EOS2 + additional HDF4 objects In some products, some additional HDF4 objects will be ignored. Other HDF4 products not listed in previous slides are not tested and may not be visualized by Java OPeNDAP clients. HDF/HDF-EOS Workshop XV51April 18, 2012

52. www.hdfgroup.org Other limitations CF conventions Limitation _FillValue inside latitude and longitude Java-NetCDF Limitation Java Heap Memory HDF/HDF-EOS Workshop XV52April 18, 2012

53. www.hdfgroup.org Outline Introduction HDF5-OPeNDAP Handler HDF4-OPeNDAP Handler Live Demo HDF/HDF-EOS Workshop XV53April 18, 2012

54. www.hdfgroup.org We’ve updated handlers HDF/HDF-EOS Workshop XV54April 18, 2012 OPeNDAP HDF Handlers HDF- EOS2 HDF5 HDF- EOS5 HDF4 … and will continue.

55. www.hdfgroup.org GES DISC Uses Handlers HDF/HDF-EOS Workshop XV55April 18, 2012

56. www.hdfgroup.org Please try HDF4/HDF5 handlers HDF/HDF-EOS Workshop XV56April 18, 2012 New hdf5_handler will be released in July. Source codes and RPMs: http://opendap.org Demo HDF5/HDF4 servers: http://hdfeos.org/zoo/hdf4_handler/ http://hdfeos.org/zoo/hdf5_handler/

57. www.hdfgroup.org and give us feedback. HDF/HDF-EOS Workshop XV57April 18, 2012

58. www.hdfgroup.org The HDF Group HDF/HDF-EOS Workshop XV58 Thank you! April 18, 2012

59. www.hdfgroup.org Acknowledgements HDF/HDF-EOS Workshop XV59April 18, 2012 This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

60. www.hdfgroup.org The HDF Group Questions/comments? help@hdfgroup.org HDF/HDF-EOS Workshop XV60April 18, 2012