1. www.hdfgroup.org The HDF Group ESIP 2012 Summer Meeting1 NASA HDF/HDF-EOS Data for Dummies (and Developers) Joe Lee The HDF Group July 19, 2012

2. www.hdfgroup.org How many dummies in this room? ESIP 2012 Summer Meeting2July 19, 2012 I’ve never used NASA HDF data. (What is HDF…?)

3. www.hdfgroup.org I was and I am still a dummy ESIP 2012 Summer Meeting3July 19, 2012 because NASA HDF is complex! Even after 5 years of work at The HDF Group, I still have something to learn.

4. www.hdfgroup.org I still remember the day one ESIP 2012 Summer Meeting4July 19, 2012 as a NASA HDF dummy so I’d like to share common mistakes very useful tips for handling NASA HDF data products from the perspective of new users like you.

5. www.hdfgroup.org Let’s feel the pain of a new user. ESIP 2012 Summer Meeting5July 19, 2012 Analyze Drought in Madison,WI using NASA EOS data. July 16 – Images of the Day – earthobservatory.org (6/25-7/10)

6. www.hdfgroup.org Possible tasks for a new user ESIP 2012 Summer Meeting6July 19, 2012 Compare the current drought period (6/25-7/10) data against the average data of the same period in last 10 years in terms of: Rainfall Ozone Normalized Difference Vegetation Index (NDVI) Wildfire and visualize the data! Can a new user do it easily with NASA satellite data?

7. www.hdfgroup.org The answer will be no ESIP 2012 Summer Meeting7July 19, 2012 because the user should master the skill of handling many different NASA data products in HDF.

8. www.hdfgroup.org Rainfall task example ESIP 2012 Summer Meeting8July 19, 2012 A simple internet search will eventually lead to GES DISC TRMM web page that distributes data in HDF.

9. www.hdfgroup.org Same for Ozone, NDVI, and Wildfire ESIP 2012 Summer Meeting9July 19, 2012 A simple internet search will eventually lead to various NASA data centers web page that distributes data in HDF. HDF is simply indispensable...

10. www.hdfgroup.org So what is HDF? ESIP 2012 Summer Meeting10July 19, 2012 I downloaded the.hdf file but no application can open. PDF HDF It’s a file format for scientific data. It needs an application like HDFView to open it.

11. www.hdfgroup.org Why does NASA use HDF? ESIP 2012 Summer Meeting11July 19, 2012 Why didn’t NASA use these well-known file formats? ASCII Spreadsheet (MS Excel) Database (MySQL, Oracle) Image (JPEG, GeoTIFF, GIF) XML because HDF is efficient data management solution in terms of space (storage) and time (performance).

12. www.hdfgroup.org Is HDF also efficient for users? ESIP 2012 Summer Meeting12July 19, 2012 Not necessarily because users may need to learn new tools and APIs how to process the data correctly Yet, there’s no “NASA HDF for Dummies” book. Instead, help is always available through hdfgroup.org and hdfeos.org.

13. www.hdfgroup.org What’s inside NASA HDF? ESIP 2012 Summer Meeting13July 19, 2012 A typical NASA HDF file has attributes (metadata) describing the file contents itself coordinate variables (time, location – lat/lon/height) data variable (fire, water, air, land)

14. www.hdfgroup.org NASA HDF has four faces ESIP 2012 Summer Meeting14July 19, 2012 NASA data centers provide files in HDF4 (.h4,.hdf) HDF-EOS2 (he2, he) HDF5 (.h5) HDF-EOS5 (he5) They are all different although they share same HDF prefix.

15. www.hdfgroup.org HDF4 vs HDF5 ESIP 2012 Summer Meeting15July 19, 2012 HDF4 is old. File starts with ^N^C^S^A. HDF5 is new. File starts with \211HDF. They are not compatible. You need different tools and APIs to access them. HDF4: Hopen() HDF5: H5open()

16. www.hdfgroup.org NASA HDF vs. HDF-EOS ESIP 2012 Summer Meeting16July 19, 2012 HDF- EOS is HDF + GCTP. GCTP: General Cartographic Transformation Package Library HDF: has latitude/longitude dataset. HDF-EOS: has no latitude/longitude dataset.

17. www.hdfgroup.org Why does HDF-EOS includes GCTP? ESIP 2012 Summer Meeting17July 19, 2012 Save space for geo-location (map) information data by storing only a few parameters in the special StructMetadata file attribute.

18. www.hdfgroup.org HDF-EOS is based on HDF API (1) ESIP 2012 Summer Meeting18July 19, 2012 so HDF APIs can read all HDF-EOS objects. However, there’s no easy way to figure out geo-location information with HDF APIs alone. exception: geographic projection / regular grid

19. www.hdfgroup.org HDF-EOS API vs. HDF API ESIP 2012 Summer Meeting19July 19, 2012 Thus, use HDF-EOS APIs for HDF-EOS products instead of HDF APIs to resolve lat/lon. HDF-EOS2: GDopen() HDF4: Hopen() HDF-EOS5:HE5GDopen() HDF5:H5Open() HDF-EOS APIs start with (HE5) GD/SW.

20. www.hdfgroup.org What are GD/SW in HDF-EOS APIs? ESIP 2012 Summer Meeting20July 19, 2012 GD: Grid Level 3 data global SW: Swath Level 2 data regional HDF-EOS API is specific to Earth Science domain. HDF API is for any domain such as finance and biology.

21. www.hdfgroup.org HDF-EOS API is based on HDF API (2) ESIP 2012 Summer Meeting21July 19, 2012 so HDF APIs can modify all HDF-EOS objects. Data producers can add new data using HDF API later. HDF-EOS APIs cannot access such data.

22. www.hdfgroup.org NASA HDF format summary ESIP 2012 Summer Meeting22July 19, 2012 If the NASA file ends with Use.he2HDF-EOS2 API.hdf,.h4HDF4 API.he5HDF-EOS5 API.h5HDF5 API

23. www.hdfgroup.org NASA HDF format review ESIP 2012 Summer Meeting23July 19, 2012 Quiz: If file ends with.he2 but has some additional objects that HDF-EOS2 cannot access, what a user should do? Answer: Open and read twice. First, with HDF-EOS2 API. Second, with HDF4 API for additional objects.

24. www.hdfgroup.org So far, we talked about formats only ESIP 2012 Summer Meeting24July 19, 2012 Show me the real example.

25. www.hdfgroup.org #include int main(int argc, char **argv) { /* Open 'MCD43C4.A2000049.005.2006271205532.hdf' using grid API */ if ((gridfile1 = GDopen("MCD43C4.A2000049.005.2006271205532.hdf", DFACC_RDONLY)) == -1) { } /* Open a grid named 'MCD_CMG_BRDF_0.05Deg' */ if ((grid1 = GDattach(gridfile1, "MCD_CMG_BRDF_0.05Deg")) == -1) { } /* Retrieve information about 'Nadir_Reflectance_Band1' datafield */ if ((GDfieldinfo(grid1, "Nadir_Reflectance_Band1", &datafield1rank, datafield1dimsize, &datafield1type, datafield1dimname)) == -1) { } /* Allocate buffer for 'Nadir_Reflectance_Band1' */ if ((datafield1data = malloc(sizeof(int16) * 3600 * 7200)) == NULL) { } /* Read data from 'Nadir_Reflectance_Band1' */ if ((GDreadfield(grid1, "Nadir_Reflectance_Band1", NULL, NULL, NULL, datafield1data)) == -1) { } /* Close the grid named 'MCD_CMG_BRDF_0.05Deg' */ if ((GDdetach(grid1)) == -1) { } /* Close 'MCD43C4.A2000049.005.2006271205532.hdf' */ if ((GDclose(gridfile1)) == -1) { } return 0; } Read HDF-EOS2 Grid data in C ESIP 2012 Summer Meeting25July 19, 2012

26. www.hdfgroup.org Read HDF-EOS5 Swath data in FORTRAN ESIP 2012 Summer Meeting26July 19, 2012 c Open 'OMI-Aura_L2-OMAERO_2004m1001t0003-o01132_v003.he5' using swath c API swathfile1 = he5_swopen $ ('OMI-Aura_L2-OMAERO_2004m1001t0003-o01132_v003.he5', $ HE5F_ACC_RDONLY) c Open a swath named 'ColumnAmountAerosol' swath1 = he5_swattach(swathfile1, 'ColumnAmountAerosol') c Retrieve information about 'AerosolIndexUV' datafield status = he5_swfldinfo(swath1, 'AerosolIndexUV', datafield1rank, $ datafield1dimsize, datafield1type, datafield1dimname, $ datafield1maxdimname) c Select necessary range of elements datafield1start(1) = 0 datafield1stride(1) = 1 datafield1edge(1) = 60 datafield1start(2) = 0 datafield1stride(2) = 1 datafield1edge(2) = 1644 c Read data from 'AerosolIndexUV' status = he5_swrdfld(swath1, 'AerosolIndexUV', datafield1start, $ datafield1stride, datafield1edge, datafield1data) c Dump data from 'AerosolIndexUV' do i = 1, 16 do j = 1, 6 write(*,*), datafield1data(j,i) enddo write(*,*) enddo

27. www.hdfgroup.org Complete Examples for Dummies ESIP 2012 Summer Meeting27July 19, 2012 Complete and more examples: http://hdfeos.org/examples/library.php Don’t miss how to read lat/lon : http://hdfeos.org/examples/c_grid_lonlat.php

28. www.hdfgroup.org HDF/HDF-EOS API look too difficult ESIP 2012 Summer Meeting28July 19, 2012 Is there a simple API like netCDF?

29. www.hdfgroup.org H4CF Conversion Library APIs // open the example HDF file h4cf_open("geo.hdf"); // HDF file attributes are obtained in a C++ STL map object const map file_attrs = h4cf_get_file_attrs(); // HDF file variables are obtained in a C++ STL list object const list pvars = h4cf_get_vars(); // close the HDF file h4cf_close(); July 19, 2012ESIP 2012 Summer Meeting29

30. www.hdfgroup.org H4CF Conversion Library works on both HDF4 and HDF-EOS2. Variables (datasets) list Attributes (metadata) list All Vars & Attrs follow CF conventions Develop your own CF application! July 19, 2012ESIP 2012 Summer Meeting30

31. www.hdfgroup.org What is CF conventions? Metadata conventions for earth science data. Sharing of files created with the NetCDF APIs, but not specifically to netCDF. The CF conventions are now increasingly gaining acceptance. URL: http://cf-pcmdi.llnl.gov/http://cf-pcmdi.llnl.gov/ Apr. 17-19, 2012HDF/HDF-EOS Workshop XV31

32. www.hdfgroup.org Don’t follow CF conventions blindly. July 19, 2012ESIP 2012 Summer Meeting32 [1] CF: scale * value + offset [2] MODIS: scale * (value – offset) Scale = 0.01 Value = 5000 Offset = -15000.0 [1] 0.01 * 5000 + (-15000) = -14950 [2] 0.01 * (5000 – (-15000)) = 200

33. www.hdfgroup.org -14950 Kelvin? July 19, 2012ESIP 2012 Summer Meeting33

34. www.hdfgroup.org MODIS Scale / Offset Correction ESIP 2012 Summer Meeting34July 19, 2012

35. www.hdfgroup.org Limitation – No Support Yet July 19, 2012ESIP 2012 Summer Meeting35 MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc.

36. www.hdfgroup.org Is H5CF Conversion Library available? July 19, 2012ESIP 2012 Summer Meeting36 Unfortunately, no. Compared to HDF4/HDF-EOS2, HDF5/HDF-EOS5 products are not diverse. HDF-EOS5 products follow NASA AURA file guidelines and use only geographic projections. New HDF5 products will follow CF- conventions.

37. www.hdfgroup.org How about programming dummies? ESIP 2012 Summer Meeting37July 19, 2012 What is FORTRAN? What is C? What is C++? …

38. www.hdfgroup.org Why do I need APIs to read data? July 19, 2012ESIP 2012 Summer Meeting38 Do you use PDF APIs to read PDF document? For most users, PDF reader application is enough. Same for most HDF users. HDF reader application is enough.

39. www.hdfgroup.org Read HDF data in ASCII July 19, 2012ESIP 2012 Summer Meeting39 Use generic HDF “dump” tools hdp for HDF-EOS2 and HDF4 files h5dump for HDF-EOS5 and HDF5 files They are only good for attributes and data. Cannot extract lat/lon from HDF-EOS Grid

40. www.hdfgroup.org eos2 dump HDF-EOS2 Dumper Tool Useful for tools that can’t handle some projections. July 19, 2012ESIP 2012 Summer Meeting40 eos2 ASCII MATLAB IDL NCL http://hdfeos.org/zoo

41. www.hdfgroup.org Important Tool for Zoo Maintenance July 19, 2012ESIP 2012 Summer Meeting41

42. www.hdfgroup.org Why Tools Need HDF-EOS2 Dumper July 19, 2012ESIP 2012 Summer Meeting42 Lat / Lon in ASCII MATLAB IDL NCL HDF-EOS2 dumper LaRC MISR SOM Projection HDF-EOS2 Lat / Lon

43. www.hdfgroup.org HDF-EOS2 Dumper Options Usage: eos2dump options filename.hdf What: 1=lat / 2 =lon Format: m=5 values per line July 19, 2012ESIP 2012 Summer Meeting43

44. www.hdfgroup.org eos2dump Latitude Example eos2dump –c 1 filename.hdf > latitude.txt What: lat Format: one value per line (default) July 19, 2012ESIP 2012 Summer Meeting44 90 80 70 60 50 40 30 20 10 0 -10 -20 -30

45. www.hdfgroup.org eos2dump Longitude Example eos2dump –c2 m filename.hdf > longitude.txt What: lon Format: 5 values per line MATLAB cannot read it. July 19, 2012ESIP 2012 Summer Meeting45 180 175 170 165 160 155 150 145 140 135 130 125 120 115 110

46. www.hdfgroup.org How about eos5dump? July 19, 2012ESIP 2012 Summer Meeting46 It’s not necessary because all HDF-EOS5 Grid use geographic projection. You can easily generate lat/lon using the parameters stored in StructMetadata. E.g., 360 / x-dimension, 180 / y-dimension

47. www.hdfgroup.org How about visualization? ESIP 2012 Summer Meeting47July 19, 2012 0000111020230232 3234234299234234 2342343292343493 … ASCII data is boring!

48. www.hdfgroup.org HDFView July 19, 2012ESIP 2012 Summer Meeting48 THE GUI Tool equivalent to Acrobat Reader. It can visualize data as image. It can even edit some data. Critical drawback: no map! Why? Again, it’s a general purpose tool. Not for Earth science users only.

49. www.hdfgroup.org HDFView HDF-EOS Plug-in July 19, 2012ESIP 2012 Summer Meeting49 An additional download for HDFView. It can visualize HDF-EOS2 and HDF-EOS5. It supports map.

50. www.hdfgroup.org HDFView HDF-EOS Plug-in screenshot July 19, 2012ESIP 2012 Summer Meeting50 Close but no cigar – it works only for HDF-EOS! There are many generic HDF data products.

51. www.hdfgroup.org Then, what should dummies do? ESIP 2012 Summer Meeting51July 19, 2012 Any visualization tool that works for both HDF and HDF-EOS?

52. www.hdfgroup.org Use a script language tool ESIP 2012 Summer Meeting52July 19, 2012 such as NCL IDL MATLAB with eos2dump tool

53. www.hdfgroup.org Rich Dummy Poor Dummy ESIP 2012 Summer Meeting53July 19, 2012 MATLAB and IDL licenses are expensive!

54. www.hdfgroup.org Solution for Poor Dummy ESIP 2012 Summer Meeting54July 19, 2012 Try NCL. Free yet powerful. http://hdfeos.org/zoo page has complete codes for visualizing most NASA data products in NCL.http://hdfeos.org/zoo ncl-talk@ucar.edu mailing list will rock you.ncl-talk@ucar.edu

55. www.hdfgroup.org Solution for Rich Dummy ESIP 2012 Summer Meeting55July 19, 2012 Visit Comprehensive Examples page. http://hdfeos.org/zoo page has complete codes for visualizing most NASA data products in IDL and MATLAB. http://hdfeos.org/zoo

56. www.hdfgroup.org Poor and Lazy Dummy ESIP 2012 Summer Meeting56July 19, 2012 I hate programming. Any free visualization tool that works for both HDF/HDF-EOS?

57. www.hdfgroup.org Recall: H4CF Conversion Library works on both HDF4 and HDF-EOS2. Variables (datasets) list Attributes (metadata) list All Vars & Attrs follow CF conventions Develop your own CF application! July 19, 2012ESIP 2012 Summer Meeting57

58. www.hdfgroup.org Solution: h4tonccf Usage: h4tonccf filename.hdf [filename.nc] generates either netcdf-3 or netcdf-4 file. follows netcdf classic model. is based on H4CF Conversion Library. July 19, 2012ESIP 2012 Summer Meeting58 Use free netcdf visualization tool like Panoply and IDV after conversion.

59. www.hdfgroup.org h4tonccf Demo July 19, 2012ESIP 2012 Summer Meeting59

60. www.hdfgroup.org Converted File Size Comparison July 19, 2012ESIP 2012 Summer Meeting60 HDF-EOS2 Netcdf-4 Netcdf-3 9X

61. www.hdfgroup.org Same Limitation – No Support Yet July 19, 2012ESIP 2012 Summer Meeting61 MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc.

62. www.hdfgroup.org How about h5tonccf? July 19, 2012ESIP 2012 Summer Meeting62 Recall: no H5CF Conversion Library exists. The latest IDV and Panoply may open and visualize HDF-EOS5 directly. However, no luck with HDF5.

63. www.hdfgroup.org No hope for Poor and Lazy Dummy? ESIP 2012 Summer Meeting63July 19, 2012 I hate programming. I want free visualization tool that works for both HDF/HDF-EOS. Am I hopeless?

64. www.hdfgroup.org Solution for the hopeless: OPeNDAP ESIP 2012 Summer Meeting64July 19, 2012 Why OPeNDAP? What is HDF Handler? How important CF conventions?

65. www.hdfgroup.org Access HDF via FTP ESIP 2012 Summer Meeting65 Users FTP Clients IDL FTP Servers HDF Files AIRS/OMI FTP Local HDF Files Aqua/Aura July 19, 2012

66. www.hdfgroup.org Client Library (libnc_dap) Data Access Protocol (DAP) via Internet Server (Hyrax) Handler (hdf4/5_handler) Remote Data (HDF4/5) View Data (User) OPeNDAP concept 66 Visualization Tools (gradsdap) Typical remote access problem FTP/HTTP Visualization tools (e.g., GrADS) ESIP 2012 Summer MeetingJuly 19, 2012

67. www.hdfgroup.org Access remote data easily July 19, 2012ESIP 2012 Summer Meeting67

68. www.hdfgroup.org without worrying about data plans. July 19, 2012ESIP 2012 Summer Meeting68

69. www.hdfgroup.org Visualize remote data easily, too. July 19, 2012ESIP 2012 Summer Meeting69

70. www.hdfgroup.org HDF handler is key ESIP 2012 Summer Meeting70 Users OPeNDAP Clients IDV OPeNDAP Servers HDF Files Hyrax AIRS/OMI DAP NetCDF Java Visualization Tools HDF HANDLER Aqua/Aura July 19, 2012

71. www.hdfgroup.org because if handler doesn’t follow CF July 19, 2012ESIP 2012 Summer Meeting71 No Visualization Bonus!

72. www.hdfgroup.org If handler gives up extra bonus July 19, 2012ESIP 2012 Summer Meeting72 HDF / HDF-EOS DAP Server DAP handler development is easier.

73. www.hdfgroup.org What’s involved in extra bonus July 19, 2012ESIP 2012 Summer Meeting73 DAP Server CF Conventions DAP DAP Client Visualization Earth Scientist HDF / HDF-EOS

74. www.hdfgroup.org How handler developers feel July 19, 2012ESIP 2012 Summer Meeting74

75. www.hdfgroup.org Because we care all NASA products. July 19, 2012ESIP 2012 Summer Meeting75 ? AURA NPP AQUA TERRA MISR

76. www.hdfgroup.org Developers’ Dream July 19, 2012ESIP 2012 Summer Meeting76 … all data are created equal, that they shall be visualized by all client tools easily…

77. www.hdfgroup.org What if all people agreed… July 19, 2012ESIP 2012 Summer Meeting77 UDF (Imaginary) DAP Server DAP DAP Client Visualization Earth Scientist development might have been much easier. Check once.

78. www.hdfgroup.org Lessons from handler development July 19, 2012ESIP 2012 Summer Meeting78 Communicate and work together among different groups. Standardize everything from data products to visualization tools. Follow conventions and standards.

79. www.hdfgroup.org Redesign of HDF5 handler Goal: Support all NASA HDF5 products. Modularize for future product support. Follow CF conventions robustly. July 19, 2012ESIP 2012 Summer Meeting79 Product

80. www.hdfgroup.org Why did we redesign? July 19, 2012ESIP 2012 Summer Meeting80 In 2007, was enough for AURA Grid HDF5 handler

81. www.hdfgroup.org Why did we redesign? July 19, 2012ESIP 2012 Summer Meeting81 In 2011, wasn’t enough for AURA Grid AURA Swath AURA ZA GSSTF

82. www.hdfgroup.org Why did we redesign? July 19, 2012ESIP 2012 Summer Meeting82 In 2011, wasn’t enough for AURA Grid AURA Swath AURA ZA GSSTF

83. www.hdfgroup.org and more… so we gave up! July 19, 2012ESIP 2012 Summer Meeting83 GOSAT ICESat-2 Aquarius SMAP

84. www.hdfgroup.org We rebuilt one that can scale up July 19, 2012ESIP 2012 Summer Meeting84

85. www.hdfgroup.org by modularizing the design July 19, 2012ESIP 2012 Summer Meeting85 HDF5 handlerCFHDF5GOSATAquariusSMAP HDF- EOS5 GRIDSWATHZAGeneric with a big room for future products.

86. www.hdfgroup.org Key updates on HDF5 handler July 19, 2012ESIP 2012 Summer Meeting86 version1.5.x2.0.0 CF-convention Support PrimitiveRobust Range of NASA Data Products LimitedAll Mulitiple Grids/Swath/ZA NoYes Configuration Options Static BuildRun-time Support for Future Products RigidFlexible

87. www.hdfgroup.org HDF-EOS5 products supported OMI Swath, Grid MLS Swath HIRDLS Swath, ZA TES Swath, Grid GSSTF July 19, 2012ESIP 2012 Summer Meeting87

88. www.hdfgroup.org HDF5 products supported Aquarius Grid MEaSUREs SeaWiFS DeepBlue SBUV2 Ozone (O3) Profile SMAP (Simulation) July 19, 2012ESIP 2012 Summer Meeting88

89. www.hdfgroup.org HIRDLS Zonal Average July 19, 2012ESIP 2012 Summer Meeting89

90. www.hdfgroup.org New TES Swath July 19, 2012ESIP 2012 Summer Meeting90

91. www.hdfgroup.org MEaSUREs GSSTF July 19, 2012ESIP 2012 Summer Meeting91

92. www.hdfgroup.org Aquarius Grid (Simulation) July 19, 2012ESIP 2012 Summer Meeting92

93. www.hdfgroup.org MEaSUREs SeaWiFS DeepBlue July 19, 2012ESIP 2012 Summer Meeting93

94. www.hdfgroup.org Ozone Profile (BUV) July 19, 2012ESIP 2012 Summer Meeting94

95. www.hdfgroup.org SMAP Swath (Simulation) July 19, 2012ESIP 2012 Summer Meeting95

96. www.hdfgroup.org SMAP Swath (Simulation) July 19, 2012ESIP 2012 Summer Meeting96

97. www.hdfgroup.org No CF magic for some products Tool / CF Limitation GOSAT/acos: 1-D trajectory ICESat-2 OMI L2G: Lat/Lon with 3-D Java-NetCDF Client Limitation Dataset w/ long string July 19, 2012ESIP 2012 Summer Meeting97

98. www.hdfgroup.org It doesn’t mean handler failure July 19, 2012ESIP 2012 Summer Meeting98 because other tools like can still access and visualize data.

99. www.hdfgroup.org MATLAB and IDL will work too. July 19, 2012ESIP 2012 Summer Meeting99 HDF5 Examples under zoo will work. Visit http://hdfeos.org/zoo Replace local file access with OPeNDAP URL access. MATLAB: Try new OPeNDAP functions. The rest will be same.

100. www.hdfgroup.org Read if you can’t see July 19, 2012ESIP 2012 Summer Meeting100 http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/ Aura_OMI_Level3/OMTO3e.003/2012/OMI-Aura_L3- OMTO3e_2012m0416_v003- 2012m0418t020608.he5.ascii?lon[1029:1:1029],lat[513:1:513],_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[513:1:513][1029:1:1029] Dataset: OMI-Aura_L3-OMTO3e_2012m0416_v003- 2012m041 8t020608.he5 lon, 77.375 lat, 38.375 _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3.lon, 77.375 _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3._HDFEOS_GRIDS_O MI_Column_Amount_O3_Data_Fields_ColumnAmountO3[_HDFEOS_GRIDS_OMI_Column_Amount_ O3_Data_Fields_ColumnAmount O3.lat=38.375], 309.6

101. www.hdfgroup.org or hear from Siri? July 19, 2012ESIP 2012 Summer Meeting101

102. www.hdfgroup.org HDF4 OPeNDAP handler update Corrected the handling the scale/offset for MODIS products. Added the support for Lambert azimuthal equal-area (LAMAZ). Added the support for Space-oblique Mercator projection (SOM). Other minor bug fixes. ESIP 2012 Summer Meeting102July 19, 2012

103. www.hdfgroup.org Key updates on HDF4 handler July 19, 2012ESIP 2012 Summer Meeting103 version3.9.33.9.4 Some MODIS Scale/offset WrongCorrected LAMAZ projection support NoYes SOM projection support NoYes

104. www.hdfgroup.org MODIS scale / offset correction ESIP 2012 Summer Meeting104July 19, 2012

105. www.hdfgroup.org Support for LAMAZ projection ESIP 2012 Summer Meeting105July 19, 2012

106. www.hdfgroup.org Support for SOM projection ESIP 2012 Summer Meeting106July 19, 2012 Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[XDimD2 = 128][YDimD3 = 512]; … } Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; … }

107. www.hdfgroup.org HDF-EOS2 products supported GES DISC AIRS / MERRA / TOMS LAADS / LP DAAC / NSIDC Many MODIS products NSIDC AMSR_E / NISE / MODIS LaRC MISR / MOPITT / some CERES (TRMM) ESIP 2012 Summer Meeting107July 19, 2012

108. www.hdfgroup.org HDF4 products supported GES DISC TRMM L1B, L2B Swath TRMM L3 Grid 42B & 43B products OBPG (Ocean Color) SeaWiFS / MODIST / MODISA CZCS / OCTS ESIP 2012 Summer Meeting108July 19, 2012

109. www.hdfgroup.org HDF4 products supported CERES CER_AVG CER_ES4 CER_ISCCP-D2like-Day CER_ISCCP-D2like-GEO CER_SRBAVG CER_SYN CER_ZAVG ESIP 2012 Summer Meeting109July 19, 2012

110. www.hdfgroup.org HDF4 handler limitations HDF-EOS2 + additional HDF4 objects In some products, some additional HDF4 objects will be ignored. Other HDF4 products not listed in previous slides are not tested and may not be visualized by Java OPeNDAP clients. ESIP 2012 Summer Meeting110July 19, 2012

111. www.hdfgroup.org Other limitations CF conventions Limitation _FillValue inside latitude and longitude Java-NetCDF Limitation Java Heap Memory ESIP 2012 Summer Meeting111July 19, 2012

112. www.hdfgroup.org Summary: we’ve updated handlers ESIP 2012 Summer Meeting112July 19, 2012 OPeNDAP HDF Handlers HDF- EOS2 HDF5 HDF- EOS5 HDF4 … and will continue.

113. www.hdfgroup.org GES DISC Uses Handlers ESIP 2012 Summer Meeting113July 19, 2012

114. www.hdfgroup.org Please try HDF4/HDF5 handlers ESIP 2012 Summer Meeting114July 19, 2012 New hdf5_handler 2.0.0 is released! Source codes and RPMs: http://opendap.org Demo HDF5/HDF4 servers: http://hdfeos.org/zoo/hdf4_handler/ http://hdfeos.org/zoo/hdf5_handler/

115. www.hdfgroup.org and give us feedback. ESIP 2012 Summer Meeting115July 19, 2012

116. www.hdfgroup.org Got HDF? July 19, 2012ESIP 2012 Summer Meeting116 I’m not a dummy any more.

117. www.hdfgroup.org Enjoy observing the Earth! July 19, 2012ESIP 2012 Summer Meeting117

118. www.hdfgroup.org Topics for advanced dummies ESIP 2012 Summer Meeting118July 19, 2012 That’s it? How about netCDF-4 interoperability?

119. www.hdfgroup.org Clarification –netCDF format netCDF-3 format Simple self-describing data format based on netCDF classic data model netCDF-4 format Uses HDF5 as a storage layer Exploits Compression, chunking, parallel-IO Group hierarchy, user-defined data types, etc. Supports both netCDF enhanced and netCDF classic data models Interoperability with netCDF-4 format in this talk Apr. 17-19, 2012HDF/HDF-EOS Workshop XV119

120. www.hdfgroup.org Clarification – netCDF packages netCDF software packages netCDF-C Support both netCDF-3 and netCDF-4 formats C++/Fortran Wrappers netCDF-Java Support both netCDF-3 and netCDF-4 formats The implementation of the Common Data Model netCDF version 4 Generally mean the version 4 package of the netCDF-C library (Not only support netCDF-4 format) Apr. 17-19, 2012HDF/HDF-EOS Workshop XV120

121. www.hdfgroup.org Why netCDF-4 Big user community User-friendly data models Tools Home-grown and third-party visualization and analysis tools - ncdump, ncgen, IDV, Panoply, Ferret etc. Apr. 17-19, 2012HDF/HDF-EOS Workshop XV121

122. www.hdfgroup.org NetCDF Classic Model Apr. 17-19, 2012HDF/HDF-EOS Workshop XV122

123. www.hdfgroup.org NetCDF enhanced model Apr. 17-19, 2012HDF/HDF-EOS Workshop XV123

124. www.hdfgroup.org Conversion July 19, 2012ESIP 2012 Summer Meeting124 Creating a new netCDF-4 file from HDF5 file.

125. www.hdfgroup.org eos52nc4 converts hdf-eos5 to netcdf-4. follows netcdf enhanced model. does not follow CF conventions. Usage: eos52nc4 test.he5 test.nc http://hdfeos.org/software/convert_hdfeos5.php July 19, 2012ESIP 2012 Summer Meeting125 eos5 NETCDF-4 eos52nc4

126. www.hdfgroup.org Augmentation July 19, 2012ESIP 2012 Summer Meeting126 Making an HDF5 file look like a netCDF-4 file.

127. www.hdfgroup.org aug_eos5 augments hdf-eos5 to netcdf-4. Both HDF-EOS5 and NetCDF4 can access it. Usage: aug_eos5 [-i|-f map] test.h5 modifies the input file directly. does not follow CF conventions. http://hdfeos.org/software/aug_hdfeos5.php July 19, 2012ESIP 2012 Summer Meeting127 eos5 NETCDF-4 aug_eos5

128. www.hdfgroup.org hdf5 netcdf4 Augmentation Concept July 19, 2012ESIP 2012 Summer Meeting128 hdf-eos5 HDF-EOS5 API NETCDF-4 API aug_eos5 augmentation hdf-eos5 HDF5 API

129. www.hdfgroup.org Why NetCDF-4 Doesn’t Work July 19, 2012ESIP 2012 Summer Meeting129 [nLevels][nTimes] NETCDF-4 API hdf-eos5

130. www.hdfgroup.org What aug_eos5 Does July 19, 2012ESIP 2012 Summer Meeting130 [nLevels][nTimes] NETCDF-4 API hdf-eos5

131. www.hdfgroup.org Options in aug_eos5 1.Default option -i option 2.-f map_file option July 19, 2012ESIP 2012 Summer Meeting131

132. www.hdfgroup.org Default option in aug_eos5 Grid XDim, YDim: calculated Other dims: default fill value Swath / ZA All dims: default fill value July 19, 2012ESIP 2012 Summer Meeting132

133. www.hdfgroup.org Better -i option in aug_eos5 Grid XDim, YDim: calculated Other dims: 1,2,…,n Swath / ZA All dims: 1,2,…,n July 19, 2012ESIP 2012 Summer Meeting133

134. www.hdfgroup.org Best -f option in aug_eos5 Needs an input map file. Input file specifies 3 rules : 1.What to be filled in 2.How to fill in 3.Where to get the values Most useful and flexible option July 19, 2012ESIP 2012 Summer Meeting134

135. www.hdfgroup.org aug_eos5 Input File Format Each line in input file contains 1.How: 0,1,2 2.What: dim name 3.Where: data file or existing variable in file. 4.(Optional) Multiple grid/swath/za case only: grid/swath/za name: July 19, 2012ESIP 2012 Summer Meeting135

136. www.hdfgroup.org aug_eos5 Input File Example 0 nTimes Time 1 nLevels data.txt 2 nCandidates July 19, 2012ESIP 2012 Summer Meeting136 Copy the data Read file data.txt Fill 1,2,…, n HowWhatWhere 95% cases – Copy the data

137. www.hdfgroup.org Multiple Swath Case swath name: BrO 0 nTimes Time 1 nLevels data1.txt 2 nCandidates July 19, 2012ESIP 2012 Summer Meeting137 HowWhatWhere swath name: BrO column 2 nTimes 1 nLevels data2.txt Multiple

138. www.hdfgroup.org aug_eos5 vs eos52nc4 July 19, 2012ESIP 2012 Summer Meeting138 aug_eos5eos52nc4 hdfeos-5YesNo netcdf-4Yes

139. www.hdfgroup.org HIRDLS/MLS Team Use aug_eos5 July 19, 2012ESIP 2012 Summer Meeting139 So You Can!

140. www.hdfgroup.org The HDF Group ESIP 2012 Summer Meeting140 Thank you! July 19, 2012

141. www.hdfgroup.org Acknowledgements ESIP 2012 Summer Meeting141July 19, 2012 This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

142. www.hdfgroup.org The HDF Group Questions/comments? help@hdfgroup.org ESIP 2012 Summer Meeting142July 19, 2012