HDF-EOS 2/5 to netCDF Converter Bob Bane, Richard Ullman, Jingli Yang Data Usability Group NASA/Goddard Space Flight Center

Introduction Status report Properties of netCDF and HDF-EOS Conversion strategy

Status Report Last year - hdfeos52netcdf –HDF-EOS 5 -> netCDF –COARDS compatible Current –Uses he25 interoperability library, so does both HDF-EOS 2 and 5 –CF compatible

Data Formats and Conventions Generic data containers –HDF, netCDF Conventions for domain-specific metadata –HDF-EOS, COARDS/CF HDF -> HDF-EOS netCDF -> COARDS/CF

netCDF netCDF files contain: –Variables multi-dimensional arrays of basic data types (character/integer/float) –Dimensions named sizes for dimensions of variables –Attributes named one-dimensional arrays properties of variables

netCDF Conventions Metadata is stored in attributes –Conventions for names: “units” Coordinate vector –Variable with the same name as a dimension –Value is a vector of same size as the dimension –Is a mapping between (0,1,2…) dimension indexing and physical quantities for dimension

COARDS Conventions Cooperative Ocean/Atmospheric Research Data Service –Conventions for use of netCDF Order of dimensions for variables Names of attributes (“Units”, “_FillValue”) Coordinate variables – _profile.html

CF Conventions Climate and Forecast –Follow-on to COARDS –Tighter Many attributes optional in COARDS are required in CF –More capable Multi-dimensional geolocation support – metadata/

HDF Hierarchical Data Format HDF files contain: –Datasets multi-dimensional arrays of basic data types –Dimensions Named sizes of dataset dimensions –Groups Named groups of datasets (and groups) –Attributes Named properties of datasets and groups, similar to netCDF

HDF-EOS Conventions and API for HDF HDF-EOS files contain: –Fields (datasets) –Points Individually geolocated measurements –Swaths Groups of data and geolocation fields, and mappings between them –Grids Groups of data fields with rectilinear geolocation

HDF-EOS (cont) HDF-EOS 2 over HDF4 HDF-EOS 5 over HDF5 –HDF5 very different from HDF4 –HDF-EOS 2/5 near identical API –Our he25 library allows uniform access to HDF-EOS 2/5, so converter works for both Looks/works like HDF-EOS 5 On HDF-EOS 4 files, translates in/out

Observations HDF-EOS is “bigger” than netCDF –Additional structured metadata (ODL) –HDF-EOS API calls for geolocation netCDF file ~= HDF-EOS Swath/Grid –Both are groups of related datasets

Conversion Strategies One HDF-EOS file -> one netCDF file –Alternative is one Swath/Grid -> one file COARDS/CF - if original HDF-EOS followed conventions, converted netCDF will also –Most HDF-EOS data producers are aware of COARDS/CF Skip HDF-EOS Point datasets –Reconsider this if real world Point data emerges

Conversion Strategies (cont) Convert data to enable future processing –Geolocation data, attributes (units) –Other metadata less important Could transfer ODL metadata as a string, but why? –Can always go back to the original file and use good HDF-EOS tools

Conversion in General Swath s1 Dimensions(lat,lon,time) Datafield f1(lat,lon,time) Geofield f2(lat,lon,time) Swath s2 Dimensions(lat,lon,time) Datafield f3(lat,lon,time) Geofield f4(lat,lon.time) Dimensions(lat,lon,time,s2_time) Variable s1_f1(lat,lon,time) Variable s1_GEO_f2(lat,lon,time) Variable s2_f3(lat,lon,s2_time) Variable s2_GEO_f4(lat,lon,s2_time) HDF-EOS netCDF Flatten HDF-EOS hierarchy Encode names, types in variable names

Swaths Swath s2 Dimensions(lat, glat,lon, glon, time) DimensionMap(lat, glat, 0, 1) DimensionMap(lon, glon, 0, 1) Datafield f3(lat,lon,time) Geofield f4(glat,glon.time) Dimensions(lat,glat,lon,glon,time,s2_time) Attributes: s2_DimensionMap: “lat/glat, lon/glon” s2_DMOffsets: (0,0) s2_DMIncrements: (1,1); Variable s2_f3(lat,lon,s2_time) Attributes: coordinates: “s2_GEO_f3” Variable s2_GEO_f4(glat,glon,s2_time) HDF-EOS netCDF Swath name, geofield type encoded in variable names Record dimension map in global attributes

Grids Grid g1 Dimensions(lat,lon,time) Corners(upleft, upright, lowleft, lowright) Datafield f1(lat,lon,time) Dimensions(lat,lon,time) Variable lat(lat) = (lowright,…upright) Variable lon(lon) = (lowleft, … upleft) Variable g1_f1(lat,lon,time) HDF-EOS netCDF Grid geolocation becomes coordinate variables

Converter C command-line application –hdfeos2netcdf HDF_file netCDF_file Should be portable to all HDF- EOS5/netCDF platforms –Naturally uses all libraries

Where is the Software? –‘Tools’ category –System ‘hdfeos2netcdf’

Big Picture File Attributes fa1: “fa value” Swath s1 Attributes: sa1: “sa value” Dimensions(lat,lon,time) Datafield f1(lat,lon,time) Geofield f2(lat,lon,time) Swath s2 Dimensions(lat,lon,time) Datafield f3(lat,lon,time) Geofield f4(lat,lon.time) File Attributes: fa1: “fa value” s1_sa1: “sa value” Dimensions(lat,lon,time,s2_time) Variable s1_f1(lat,lon,time) Variable s1_GEO_f2(lat,lon,time) Variable s2_f3(lat,lon,s2_time) Variable s2_GEO_f4(lat,lon,s2_time) HDF-EOS netCDF