1. HDF4 OPeNDAP Project Progress Report MuQun Yang and Hyo-Kyung Lee 1 HDF Developers' Meeting11/24/2015

2. OPeNDAP Review A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community 2 HDF Developers' Meeting11/24/2015

3. Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax AIRS DAP libnc-dap Visualization Tools OPeNDAP Reivew

4. Client Library (libnc-dap) DAP Protocol (via http) Server (apache) Handler (hdf4_handler) Remote Data (HDF4) View Data (User) OPeNDAP Concept 4 HDF Developers' Meeting11/24/2015 Visualization Tools (gradsdap) Typical Remote Access FTP/HTTP Visualization tools (e.g. grads)

5. Better Access for HDF-EOS2 via DAP Faster download through selection Quicker analysis through visualization 5 HDF Developers' Meeting11/24/2015 Goals

6. You bought a HDTV but... No HD signal Needs more Channels in HD Cable Service Provider Problem 6 HDF Developers' Meeting11/24/2015 HDTV Problems

7. You downloaded a viewer but... No CF convention Needs more HDF4 data in CF Data Service Provider Problem 7 HDF Developers' Meeting11/24/2015 HDF4 Problems

8. Challenges: HDF-EOS2 CF convention (e.g. NetCDF model) Syntax Clients expect a certain data structure Semantics Clients expect some key attributes 8 HDF Developers' Meeting11/24/2015 HD Signal Challenge

9. Challenges: HDF-EOS2 DVD/VHS/VCD/LD >>> Blue-Ray Can all titles be delivered in HD? HDF4/HDF-EOS2 >>> HDF5/HDF-EOS5 Can all HDF-EOS2 be delivered in CF? 9 HDF Developers' Meeting11/24/2015 More Channels in HD Challenge

10. Solution: New HDF4 Handler 10 HDF Developers' Meeting11/24/2015 HD Signal Solution Syntax  Generate the correct DAP data type.  Add shared geo-location information. Semantics  Insert additional key attributes.

11. Solution: New HDF4 Handler Use Metadata parser Pro: simple Con: limited channels Use HDF-EOS2 library Pro: maximum channels Con: install HE2 library first 11 HDF Developers' Meeting11/24/2015 More Channels in HD Solution

12. Solution Comparison 12 HDF Developers' Meeting11/24/2015

13. Quick Implementation Parse Metadata – hdf5_handler Use HDF-EOS2 library – h4toh5 13 HDF Developers' Meeting11/24/2015

14. Current Status MODIS / AIRS Grid and Swath Other Projections (Polar, Sinusoidal) Testing 14 HDF Developers' Meeting11/24/2015

15. HDF-EOS2 in CF via DAP Finally, Happy Data Consumer! 15 HDF Developers' Meeting11/24/2015 AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10 MODIS Grid MOD08_D3.A2008284.005.2008289105354.hdf http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08 331121849.hdf MODIS Swath MOD06_L2.A2001222.0025.004.2003127111841.hdf http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html

16. Limitations – Missing Attributes MODIS Grid – Rain fall 16 HDF Developers' Meeting11/24/2015 No Unit! AURA OMI Grid - Ozone

17. Limitations – Missing Names How to Match Dimension Names? 17 HDF Developers' Meeting11/24/2015 HDF4 Handler NetCDF4 Handler pressureStd → Level

18. Limitations – Missing Values How to Supply Missing Dimension Values? 18 HDF Developers' Meeting11/24/2015 MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf HTTP/1.0 200 OK XDODS-Server: 3.7.9 XOPeNDAP-Server: 3.7.9 XDAP: 3.1 Content-Type: text/plain Content-Description: dods_error Cache-Control: no-cache Error { code = 1002; message = "An internal error was encountered: This HDF-EOS2 file has multiple Grids with different dimension sizes. Please report this to support@opendap.org "; };

19. Solution – Customization Provide an option for handler customization to generate fully CF- compliant output Create a separate XML input file for handler and feed it into the handler. Insert attributes, renaming rules, and missing values by putting them in the XML input. This requires a knowledge about the characteristics of data in advance. 19 HDF Developers' Meeting11/24/2015

20. Solution – Customization Example: Attributes, Renaming, Values 20 HDF Developers' Meeting11/24/2015 inches 0,1,2,3,4,5,6,7,8,9

21. Summary HDF4 Access via OPeNDAP Is Easy, Efficient and Fast Is Cool like HDTV Adding CF is not straightforward Use HDF-EOS2 library Is Essential Maximize the data coverage 21 HDF Developers' Meeting11/24/2015

22. Future Work (Short Term) Expand test-suite with data pool Release through OPeNDAP website Support for Swath via Parsing Support for Point data type via HDF- EOS2 library 22 HDF Developers' Meeting11/24/2015

23. Future Work (Long Term) General HDF4 Files Support Customization for CF option HE2 to DAP Mapping Documentation 23 HDF Developers' Meeting11/24/2015