Improving the usability of HDF-EOS2 data Kent Yang, Joe Lee, Choonghwan Lee The HDF Group March 31 st, /26/2016Annual briefing to ESDIS1

5/26/2016Annual briefing to ESDIS2 EOSDIS Evolution 2015 Vision Tenets Vision Tenet Vision 2015 Goals Archive Management  NASA will ensure safe stewardship of the data through its lifetime.  The EOS archive holdings are regularly peer reviewed for scientific merit. EOS Data Interoperability  Multiple data and metadata streams can be seamlessly combined.  Research and value added communities use EOS data interoperably with other relevant data and systems.  Processing and data are mobile. Future Data Access and Processing  Data access latency is no longer an impediment.  Physical location of data storage is irrelevant.  Finding data is based on common search engines.  Services invoked by machine-machine interfaces.  Custom processing provides only the data needed, the way needed.  Open interfaces and best practice standard protocols universally employed. Data Pedigree  Mechanisms to collect and preserve the pedigree of derived data products are readily available. Cost Control  Data systems evolve into components that allow a fine-grained control over cost drivers. User Community Support  Expert knowledge is readily accessible to enable researchers to understand and use the data.  Community feedback directly to those responsible for a given system element. IT Currency  Access to all EOS data through services at least as rich as any contemporary science information system. Feb 3, 2005

Purpose of the talk Identify problems to access HDF-EOS2 data Share our plan to help improve the EOS2 usability 5/26/20163Annual briefing to ESDIS

The success of EOS mission 5/26/2016Annual briefing to ESDIS4 MODIS HDF-EOS data to detect sand storm from Gobi china Courtesy NASA Earth Observatory

The success of EOS mission 5/26/2016Annual briefing to ESDIS5 MODIS HDF-EOS data to detect fires in Africa Courtesy NASA

Identify problems : technical part 5/26/2016Annual briefing to ESDIS6

Beauty of HDF-EOS2 Very close to physical model Convenient to handle geo-location projections Files are generally compact  No needs to store latitude and longitude for grid  Reduce the size of latitude and longitude for swath 5/26/20167Annual briefing to ESDIS

5/26/2016Annual briefing to ESDIS8 A compact HDF-EOS2 Grid MODIS (MYD10A2) Grid file Two 2400 X 2400 grid data fields with gzip compression File size: 75 KB If lat/lon are kept in the file File size may become 44 MB, 500 times bigger than the original file size October 15-18, 20088Workshop Name (edit footer to change)

A compact HDF-EOS2 swath Small file size – Swath 5/26/20169 Annual briefing to ESDIS cross along cross along coarse_cross coarse_along with dimension maps CloudCover(along, cross) Latitude(along, cross) Longitude(along, cross) CloudCover(along, cross) Latitude(along, cross) Longitude(along, cross) CloudCover(along, cross) Latitude(coarse_along, coarse_cross) Longitude(coarse_along, coarse_cross) CloudCover(along, cross) Latitude(coarse_along, coarse_cross) Longitude(coarse_along, coarse_cross) with dimension maps w/o dimension maps along cross Longitude 0.5 MB (406 X 271 X 4) 42 MB (4060 X 2708 X 4)

There are issues to access some HDF-EOS2 files Aims to improve the usability 5/26/2016Annual briefing to ESDIS10

5/26/2016Annual briefing to ESDIS11 First of all, there should be more communications among providers, tool developers, users and data storage software developers. Focus on HDF-EOS2 and HDF4 in the next few slides.

Schematic of Grid 5/26/2016Annual briefing to ESDIS12 YDim XDim StructMetadata.0 = “… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0) … Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS: Float32 Ozone(YDim, XDim) StructMetadata.0 = “… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0) … Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS: Float32 Ozone(YDim, XDim)

Issue 1: lack of strict checking in HDF-EOS2 library 5/26/2016Annual briefing to ESDIS13

Lack of strict checking in HDF-EOS2 library XDim and YDim exist and have values .e.g, OMI (OMUVBL3)  Generally, XDim and YDim do not exist because they are implied.  XDim and YDim do not need to have values because they can be calculated. Values are just redundant. 5/26/ Annual briefing to ESDIS

Inconsistency in the number of elements HDF- EOS2 informs and HDF-EOS2 actually reads  e.g., MODIS (MAC021S0, MAC02QS0, MAC02QS1, MAC021S1)  We believe that this data field had four elements once and it was appended later. Lack of strict checking in HDF-EOS2 library 5/26/ Annual briefing to ESDIS StructMetadata … Band_1KM_RefSB[4] … StructMetadata … Band_1KM_RefSB[4] … Data Field Band_1KM_RefSB 4 elements Data Field Band_1KM_RefSB 4 elements Data Field Band_1KM_RefSB 15 elements Data Field Band_1KM_RefSB 15 elements consistent INCONSISTENT!

Lack of strict checking in HDF-EOS2 library Wrong grid parameters  e.g., MISR (MI3DAER)  Projection is defined by code, projection parameters, etc.  For Lambert Azimuth projection, the fifth parameter specifies “longitude of center of projection”.  Its value is entered in DDDMMMSSS.SS format.  /26/201616Annual briefing to ESDIS degrees -92 seconds 0 minutes 970?

Issue 2: lack of standard way to handle some geolocation fields 5/26/2016Annual briefing to ESDIS17

Schematic of Swath 5/26/2016Annual briefing to ESDIS18 cross along StructMetadata.0 = “ … DimensionName=”along” size= 11 DimensionName=”cross” size= 3 …” Data fields: Float32 CloudCover(along,cross) Geolocation fields: Float32 lat(along,cross) Float32 lon(along,cross)

No standard way to handle geolocation fields No standard way to figure out the vertical coordinate in an HDF-EOS2 file  A data field TAirStd[GeoTrack][GeoXTrack]  StdPressureLev is a dimension  Additional data field supplies values, probably pressStd  How can a user know for sure that the field pressStd provides the vertical coordinate information? 5/26/201619Annual briefing to ESDIS [StdPressureLev]

Issue 3: No enforcement for the use of standard names 5/26/2016Annual briefing to ESDIS20

Schematic of Grid 5/26/2016Annual briefing to ESDIS21 YDim XDim StructMetadata.0 = “… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0) … Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS: Float32 Ozone(YDim, XDim) StructMetadata.0 = “… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0) … Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS: Float32 Ozone(YDim, XDim)

No enforcement for the use of standard names Use of non-standard dimensions in grid  e.g., MOPITT (MOP03)  To be interpreted properly, each data field must make use of the two predefined dimensions: XDim and YDim  However, none of data fields in that file refers to XDim or YDim. Instead, they refer to nlon and nlat, which are not standard way. 5/26/201622Annual briefing to ESDIS

No enforcement for the use of standard names Object names are not restrictive  e.g., MODIS (MAC03S0)  Name of a data field called Land/SeaMask  A dimension called nscans*10  These names may not be safe in other APIs. Especially, HDF5 does not allow slash(/). 5/26/201623Annual briefing to ESDIS

Other bad practices Extreme long attributes Chunk size much bigger than dimensional size for fixed size SDS These are not enforced by HDF4 library 5/26/201624Annual briefing to ESDIS

Other technical difficulties Hybrid HDF-EOS2 files  Some data cannot be accessed by HDF-EOS2 library Need to help users access these files 5/26/201625Annual briefing to ESDIS

Identify problems : service part 5/26/2016Annual briefing to ESDIS26

A message from DAAC User Survey 5/26/201627Annual briefing to ESDIS I am a student and I need to read MODIS HDF-EOS data. But it is hard for me to extract the data. I need a fortran program to extract the data. I wish DAAC can provide this.

DAAC tool sites Provide most information users need 5/26/201628Annual briefing to ESDIS

What’s left? Some tools are out of date On some sites, no C/fortran examples to access the data Information on the tool page out of date 5/26/201629Annual briefing to ESDIS

What’s the problem? A small percentage of EOS2 users needs help.  Access “difficult” files  Compensate tools/examples for DAACs 5/26/201630Annual briefing to ESDIS

The HDF Group’s plan to DAACs Compensate fortran/C example code for DAACs C/fortran examples for hybrid files and “difficult” files Evaluate/update tools listed at hdfeos.org Evaluate tools at DAACs EOS-forum  Web forum?  EOS2 user and developer mailing lists? 5/26/201631Annual briefing to ESDIS

Thank You! 5/26/2016Annual briefing to ESDIS32

Acknowledgements This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 5/26/201633Annual briefing to ESDIS

Questions/comments? 5/26/2016Annual briefing to ESDIS34