The HDF Group ESIP 2012 Summer Meeting1 NASA HDF/HDF-EOS Data for Dummies (and Developers) Joe Lee The HDF Group July 19, 2012

How many dummies in this room? ESIP 2012 Summer Meeting2July 19, 2012 I’ve never used NASA HDF data. (What is HDF…?)

I was and I am still a dummy ESIP 2012 Summer Meeting3July 19, 2012 because NASA HDF is complex! Even after 5 years of work at The HDF Group, I still have something to learn.

I still remember the day one ESIP 2012 Summer Meeting4July 19, 2012 as a NASA HDF dummy so I’d like to share common mistakes very useful tips for handling NASA HDF data products from the perspective of new users like you.

Let’s feel the pain of a new user. ESIP 2012 Summer Meeting5July 19, 2012 Analyze Drought in Madison,WI using NASA EOS data. July 16 – Images of the Day – earthobservatory.org (6/25-7/10)

Possible tasks for a new user ESIP 2012 Summer Meeting6July 19, 2012 Compare the current drought period (6/25-7/10) data against the average data of the same period in last 10 years in terms of: Rainfall Ozone Normalized Difference Vegetation Index (NDVI) Wildfire and visualize the data! Can a new user do it easily with NASA satellite data?

The answer will be no ESIP 2012 Summer Meeting7July 19, 2012 because the user should master the skill of handling many different NASA data products in HDF.

Rainfall task example ESIP 2012 Summer Meeting8July 19, 2012 A simple internet search will eventually lead to GES DISC TRMM web page that distributes data in HDF.

Same for Ozone, NDVI, and Wildfire ESIP 2012 Summer Meeting9July 19, 2012 A simple internet search will eventually lead to various NASA data centers web page that distributes data in HDF. HDF is simply indispensable...

So what is HDF? ESIP 2012 Summer Meeting10July 19, 2012 I downloaded the.hdf file but no application can open. PDF HDF It’s a file format for scientific data. It needs an application like HDFView to open it.

Why does NASA use HDF? ESIP 2012 Summer Meeting11July 19, 2012 Why didn’t NASA use these well-known file formats? ASCII Spreadsheet (MS Excel) Database (MySQL, Oracle) Image (JPEG, GeoTIFF, GIF) XML because HDF is efficient data management solution in terms of space (storage) and time (performance).

Is HDF also efficient for users? ESIP 2012 Summer Meeting12July 19, 2012 Not necessarily because users may need to learn new tools and APIs how to process the data correctly Yet, there’s no “NASA HDF for Dummies” book. Instead, help is always available through hdfgroup.org and hdfeos.org.

What’s inside NASA HDF? ESIP 2012 Summer Meeting13July 19, 2012 A typical NASA HDF file has attributes (metadata) describing the file contents itself coordinate variables (time, location – lat/lon/height) data variable (fire, water, air, land)

NASA HDF has four faces ESIP 2012 Summer Meeting14July 19, 2012 NASA data centers provide files in HDF4 (.h4,.hdf) HDF-EOS2 (he2, he) HDF5 (.h5) HDF-EOS5 (he5) They are all different although they share same HDF prefix.

HDF4 vs HDF5 ESIP 2012 Summer Meeting15July 19, 2012 HDF4 is old. File starts with ^N^C^S^A. HDF5 is new. File starts with \211HDF. They are not compatible. You need different tools and APIs to access them. HDF4: Hopen() HDF5: H5open()

NASA HDF vs. HDF-EOS ESIP 2012 Summer Meeting16July 19, 2012 HDF- EOS is HDF + GCTP. GCTP: General Cartographic Transformation Package Library HDF: has latitude/longitude dataset. HDF-EOS: has no latitude/longitude dataset.

Why does HDF-EOS includes GCTP? ESIP 2012 Summer Meeting17July 19, 2012 Save space for geo-location (map) information data by storing only a few parameters in the special StructMetadata file attribute.

HDF-EOS is based on HDF API (1) ESIP 2012 Summer Meeting18July 19, 2012 so HDF APIs can read all HDF-EOS objects. However, there’s no easy way to figure out geo-location information with HDF APIs alone. exception: geographic projection / regular grid

HDF-EOS API vs. HDF API ESIP 2012 Summer Meeting19July 19, 2012 Thus, use HDF-EOS APIs for HDF-EOS products instead of HDF APIs to resolve lat/lon. HDF-EOS2: GDopen() HDF4: Hopen() HDF-EOS5:HE5GDopen() HDF5:H5Open() HDF-EOS APIs start with (HE5) GD/SW.

What are GD/SW in HDF-EOS APIs? ESIP 2012 Summer Meeting20July 19, 2012 GD: Grid Level 3 data global SW: Swath Level 2 data regional HDF-EOS API is specific to Earth Science domain. HDF API is for any domain such as finance and biology.

HDF-EOS API is based on HDF API (2) ESIP 2012 Summer Meeting21July 19, 2012 so HDF APIs can modify all HDF-EOS objects. Data producers can add new data using HDF API later. HDF-EOS APIs cannot access such data.

NASA HDF format summary ESIP 2012 Summer Meeting22July 19, 2012 If the NASA file ends with Use.he2HDF-EOS2 API.hdf,.h4HDF4 API.he5HDF-EOS5 API.h5HDF5 API

NASA HDF format review ESIP 2012 Summer Meeting23July 19, 2012 Quiz: If file ends with.he2 but has some additional objects that HDF-EOS2 cannot access, what a user should do? Answer: Open and read twice. First, with HDF-EOS2 API. Second, with HDF4 API for additional objects.

So far, we talked about formats only ESIP 2012 Summer Meeting24July 19, 2012 Show me the real example.

#include int main(int argc, char **argv) { /* Open 'MCD43C4.A hdf' using grid API */ if ((gridfile1 = GDopen("MCD43C4.A hdf", DFACC_RDONLY)) == -1) { } /* Open a grid named 'MCD_CMG_BRDF_0.05Deg' */ if ((grid1 = GDattach(gridfile1, "MCD_CMG_BRDF_0.05Deg")) == -1) { } /* Retrieve information about 'Nadir_Reflectance_Band1' datafield */ if ((GDfieldinfo(grid1, "Nadir_Reflectance_Band1", &datafield1rank, datafield1dimsize, &datafield1type, datafield1dimname)) == -1) { } /* Allocate buffer for 'Nadir_Reflectance_Band1' */ if ((datafield1data = malloc(sizeof(int16) * 3600 * 7200)) == NULL) { } /* Read data from 'Nadir_Reflectance_Band1' */ if ((GDreadfield(grid1, "Nadir_Reflectance_Band1", NULL, NULL, NULL, datafield1data)) == -1) { } /* Close the grid named 'MCD_CMG_BRDF_0.05Deg' */ if ((GDdetach(grid1)) == -1) { } /* Close 'MCD43C4.A hdf' */ if ((GDclose(gridfile1)) == -1) { } return 0; } Read HDF-EOS2 Grid data in C ESIP 2012 Summer Meeting25July 19, 2012

Read HDF-EOS5 Swath data in FORTRAN ESIP 2012 Summer Meeting26July 19, 2012 c Open 'OMI-Aura_L2-OMAERO_2004m1001t0003-o01132_v003.he5' using swath c API swathfile1 = he5_swopen $ ('OMI-Aura_L2-OMAERO_2004m1001t0003-o01132_v003.he5', $ HE5F_ACC_RDONLY) c Open a swath named 'ColumnAmountAerosol' swath1 = he5_swattach(swathfile1, 'ColumnAmountAerosol') c Retrieve information about 'AerosolIndexUV' datafield status = he5_swfldinfo(swath1, 'AerosolIndexUV', datafield1rank, $ datafield1dimsize, datafield1type, datafield1dimname, $ datafield1maxdimname) c Select necessary range of elements datafield1start(1) = 0 datafield1stride(1) = 1 datafield1edge(1) = 60 datafield1start(2) = 0 datafield1stride(2) = 1 datafield1edge(2) = 1644 c Read data from 'AerosolIndexUV' status = he5_swrdfld(swath1, 'AerosolIndexUV', datafield1start, $ datafield1stride, datafield1edge, datafield1data) c Dump data from 'AerosolIndexUV' do i = 1, 16 do j = 1, 6 write(*,*), datafield1data(j,i) enddo write(*,*) enddo

Complete Examples for Dummies ESIP 2012 Summer Meeting27July 19, 2012 Complete and more examples: Don’t miss how to read lat/lon :

HDF/HDF-EOS API look too difficult ESIP 2012 Summer Meeting28July 19, 2012 Is there a simple API like netCDF?

H4CF Conversion Library APIs // open the example HDF file h4cf_open("geo.hdf"); // HDF file attributes are obtained in a C++ STL map object const map file_attrs = h4cf_get_file_attrs(); // HDF file variables are obtained in a C++ STL list object const list pvars = h4cf_get_vars(); // close the HDF file h4cf_close(); July 19, 2012ESIP 2012 Summer Meeting29

H4CF Conversion Library works on both HDF4 and HDF-EOS2. Variables (datasets) list Attributes (metadata) list All Vars & Attrs follow CF conventions Develop your own CF application! July 19, 2012ESIP 2012 Summer Meeting30

What is CF conventions? Metadata conventions for earth science data. Sharing of files created with the NetCDF APIs, but not specifically to netCDF. The CF conventions are now increasingly gaining acceptance. URL: Apr , 2012HDF/HDF-EOS Workshop XV31

Don’t follow CF conventions blindly. July 19, 2012ESIP 2012 Summer Meeting32 [1] CF: scale * value + offset [2] MODIS: scale * (value – offset) Scale = 0.01 Value = 5000 Offset = [1] 0.01 * (-15000) = [2] 0.01 * (5000 – (-15000)) = 200

Kelvin? July 19, 2012ESIP 2012 Summer Meeting33

MODIS Scale / Offset Correction ESIP 2012 Summer Meeting34July 19, 2012

Limitation – No Support Yet July 19, 2012ESIP 2012 Summer Meeting35 MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc.

Is H5CF Conversion Library available? July 19, 2012ESIP 2012 Summer Meeting36 Unfortunately, no. Compared to HDF4/HDF-EOS2, HDF5/HDF-EOS5 products are not diverse. HDF-EOS5 products follow NASA AURA file guidelines and use only geographic projections. New HDF5 products will follow CF- conventions.

How about programming dummies? ESIP 2012 Summer Meeting37July 19, 2012 What is FORTRAN? What is C? What is C++? …

Why do I need APIs to read data? July 19, 2012ESIP 2012 Summer Meeting38 Do you use PDF APIs to read PDF document? For most users, PDF reader application is enough. Same for most HDF users. HDF reader application is enough.

Read HDF data in ASCII July 19, 2012ESIP 2012 Summer Meeting39 Use generic HDF “dump” tools hdp for HDF-EOS2 and HDF4 files h5dump for HDF-EOS5 and HDF5 files They are only good for attributes and data. Cannot extract lat/lon from HDF-EOS Grid

eos2 dump HDF-EOS2 Dumper Tool Useful for tools that can’t handle some projections. July 19, 2012ESIP 2012 Summer Meeting40 eos2 ASCII MATLAB IDL NCL

Important Tool for Zoo Maintenance July 19, 2012ESIP 2012 Summer Meeting41

Why Tools Need HDF-EOS2 Dumper July 19, 2012ESIP 2012 Summer Meeting42 Lat / Lon in ASCII MATLAB IDL NCL HDF-EOS2 dumper LaRC MISR SOM Projection HDF-EOS2 Lat / Lon

HDF-EOS2 Dumper Options Usage: eos2dump options filename.hdf What: 1=lat / 2 =lon Format: m=5 values per line July 19, 2012ESIP 2012 Summer Meeting43

eos2dump Latitude Example eos2dump –c 1 filename.hdf > latitude.txt What: lat Format: one value per line (default) July 19, 2012ESIP 2012 Summer Meeting

eos2dump Longitude Example eos2dump –c2 m filename.hdf > longitude.txt What: lon Format: 5 values per line MATLAB cannot read it. July 19, 2012ESIP 2012 Summer Meeting

How about eos5dump? July 19, 2012ESIP 2012 Summer Meeting46 It’s not necessary because all HDF-EOS5 Grid use geographic projection. You can easily generate lat/lon using the parameters stored in StructMetadata. E.g., 360 / x-dimension, 180 / y-dimension

How about visualization? ESIP 2012 Summer Meeting47July 19, … ASCII data is boring!

HDFView July 19, 2012ESIP 2012 Summer Meeting48 THE GUI Tool equivalent to Acrobat Reader. It can visualize data as image. It can even edit some data. Critical drawback: no map! Why? Again, it’s a general purpose tool. Not for Earth science users only.

HDFView HDF-EOS Plug-in July 19, 2012ESIP 2012 Summer Meeting49 An additional download for HDFView. It can visualize HDF-EOS2 and HDF-EOS5. It supports map.

HDFView HDF-EOS Plug-in screenshot July 19, 2012ESIP 2012 Summer Meeting50 Close but no cigar – it works only for HDF-EOS! There are many generic HDF data products.

Then, what should dummies do? ESIP 2012 Summer Meeting51July 19, 2012 Any visualization tool that works for both HDF and HDF-EOS?

Use a script language tool ESIP 2012 Summer Meeting52July 19, 2012 such as NCL IDL MATLAB with eos2dump tool

Rich Dummy Poor Dummy ESIP 2012 Summer Meeting53July 19, 2012 MATLAB and IDL licenses are expensive!

Solution for Poor Dummy ESIP 2012 Summer Meeting54July 19, 2012 Try NCL. Free yet powerful. page has complete codes for visualizing most NASA data products in NCL. mailing list will rock

Solution for Rich Dummy ESIP 2012 Summer Meeting55July 19, 2012 Visit Comprehensive Examples page. page has complete codes for visualizing most NASA data products in IDL and MATLAB.

Poor and Lazy Dummy ESIP 2012 Summer Meeting56July 19, 2012 I hate programming. Any free visualization tool that works for both HDF/HDF-EOS?

Recall: H4CF Conversion Library works on both HDF4 and HDF-EOS2. Variables (datasets) list Attributes (metadata) list All Vars & Attrs follow CF conventions Develop your own CF application! July 19, 2012ESIP 2012 Summer Meeting57

Solution: h4tonccf Usage: h4tonccf filename.hdf [filename.nc] generates either netcdf-3 or netcdf-4 file. follows netcdf classic model. is based on H4CF Conversion Library. July 19, 2012ESIP 2012 Summer Meeting58 Use free netcdf visualization tool like Panoply and IDV after conversion.

h4tonccf Demo July 19, 2012ESIP 2012 Summer Meeting59

Converted File Size Comparison July 19, 2012ESIP 2012 Summer Meeting60 HDF-EOS2 Netcdf-4 Netcdf-3 9X

Same Limitation – No Support Yet July 19, 2012ESIP 2012 Summer Meeting61 MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc.

How about h5tonccf? July 19, 2012ESIP 2012 Summer Meeting62 Recall: no H5CF Conversion Library exists. The latest IDV and Panoply may open and visualize HDF-EOS5 directly. However, no luck with HDF5.

No hope for Poor and Lazy Dummy? ESIP 2012 Summer Meeting63July 19, 2012 I hate programming. I want free visualization tool that works for both HDF/HDF-EOS. Am I hopeless?

Solution for the hopeless: OPeNDAP ESIP 2012 Summer Meeting64July 19, 2012 Why OPeNDAP? What is HDF Handler? How important CF conventions?

Access HDF via FTP ESIP 2012 Summer Meeting65 Users FTP Clients IDL FTP Servers HDF Files AIRS/OMI FTP Local HDF Files Aqua/Aura July 19, 2012

Client Library (libnc_dap) Data Access Protocol (DAP) via Internet Server (Hyrax) Handler (hdf4/5_handler) Remote Data (HDF4/5) View Data (User) OPeNDAP concept 66 Visualization Tools (gradsdap) Typical remote access problem FTP/HTTP Visualization tools (e.g., GrADS) ESIP 2012 Summer MeetingJuly 19, 2012

Access remote data easily July 19, 2012ESIP 2012 Summer Meeting67

without worrying about data plans. July 19, 2012ESIP 2012 Summer Meeting68

Visualize remote data easily, too. July 19, 2012ESIP 2012 Summer Meeting69

HDF handler is key ESIP 2012 Summer Meeting70 Users OPeNDAP Clients IDV OPeNDAP Servers HDF Files Hyrax AIRS/OMI DAP NetCDF Java Visualization Tools HDF HANDLER Aqua/Aura July 19, 2012

because if handler doesn’t follow CF July 19, 2012ESIP 2012 Summer Meeting71 No Visualization Bonus!

If handler gives up extra bonus July 19, 2012ESIP 2012 Summer Meeting72 HDF / HDF-EOS DAP Server DAP handler development is easier.

What’s involved in extra bonus July 19, 2012ESIP 2012 Summer Meeting73 DAP Server CF Conventions DAP DAP Client Visualization Earth Scientist HDF / HDF-EOS

How handler developers feel July 19, 2012ESIP 2012 Summer Meeting74

Because we care all NASA products. July 19, 2012ESIP 2012 Summer Meeting75 ? AURA NPP AQUA TERRA MISR

Developers’ Dream July 19, 2012ESIP 2012 Summer Meeting76 … all data are created equal, that they shall be visualized by all client tools easily…

What if all people agreed… July 19, 2012ESIP 2012 Summer Meeting77 UDF (Imaginary) DAP Server DAP DAP Client Visualization Earth Scientist development might have been much easier. Check once.

Lessons from handler development July 19, 2012ESIP 2012 Summer Meeting78 Communicate and work together among different groups. Standardize everything from data products to visualization tools. Follow conventions and standards.

Redesign of HDF5 handler Goal: Support all NASA HDF5 products. Modularize for future product support. Follow CF conventions robustly. July 19, 2012ESIP 2012 Summer Meeting79 Product

Why did we redesign? July 19, 2012ESIP 2012 Summer Meeting80 In 2007, was enough for AURA Grid HDF5 handler

Why did we redesign? July 19, 2012ESIP 2012 Summer Meeting81 In 2011, wasn’t enough for AURA Grid AURA Swath AURA ZA GSSTF

Why did we redesign? July 19, 2012ESIP 2012 Summer Meeting82 In 2011, wasn’t enough for AURA Grid AURA Swath AURA ZA GSSTF

and more… so we gave up! July 19, 2012ESIP 2012 Summer Meeting83 GOSAT ICESat-2 Aquarius SMAP

We rebuilt one that can scale up July 19, 2012ESIP 2012 Summer Meeting84

by modularizing the design July 19, 2012ESIP 2012 Summer Meeting85 HDF5 handlerCFHDF5GOSATAquariusSMAP HDF- EOS5 GRIDSWATHZAGeneric with a big room for future products.

Key updates on HDF5 handler July 19, 2012ESIP 2012 Summer Meeting86 version1.5.x2.0.0 CF-convention Support PrimitiveRobust Range of NASA Data Products LimitedAll Mulitiple Grids/Swath/ZA NoYes Configuration Options Static BuildRun-time Support for Future Products RigidFlexible

HDF-EOS5 products supported OMI Swath, Grid MLS Swath HIRDLS Swath, ZA TES Swath, Grid GSSTF July 19, 2012ESIP 2012 Summer Meeting87

HDF5 products supported Aquarius Grid MEaSUREs SeaWiFS DeepBlue SBUV2 Ozone (O3) Profile SMAP (Simulation) July 19, 2012ESIP 2012 Summer Meeting88

HIRDLS Zonal Average July 19, 2012ESIP 2012 Summer Meeting89

New TES Swath July 19, 2012ESIP 2012 Summer Meeting90

MEaSUREs GSSTF July 19, 2012ESIP 2012 Summer Meeting91

Aquarius Grid (Simulation) July 19, 2012ESIP 2012 Summer Meeting92

MEaSUREs SeaWiFS DeepBlue July 19, 2012ESIP 2012 Summer Meeting93

Ozone Profile (BUV) July 19, 2012ESIP 2012 Summer Meeting94

SMAP Swath (Simulation) July 19, 2012ESIP 2012 Summer Meeting95

SMAP Swath (Simulation) July 19, 2012ESIP 2012 Summer Meeting96

No CF magic for some products Tool / CF Limitation GOSAT/acos: 1-D trajectory ICESat-2 OMI L2G: Lat/Lon with 3-D Java-NetCDF Client Limitation Dataset w/ long string July 19, 2012ESIP 2012 Summer Meeting97

It doesn’t mean handler failure July 19, 2012ESIP 2012 Summer Meeting98 because other tools like can still access and visualize data.

MATLAB and IDL will work too. July 19, 2012ESIP 2012 Summer Meeting99 HDF5 Examples under zoo will work. Visit Replace local file access with OPeNDAP URL access. MATLAB: Try new OPeNDAP functions. The rest will be same.

Read if you can’t see July 19, 2012ESIP 2012 Summer Meeting100 Aura_OMI_Level3/OMTO3e.003/2012/OMI-Aura_L3- OMTO3e_2012m0416_v m0418t he5.ascii?lon[1029:1:1029],lat[513:1:513],_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[513:1:513][1029:1:1029] Dataset: OMI-Aura_L3-OMTO3e_2012m0416_v m041 8t he5 lon, lat, _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3.lon, _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3._HDFEOS_GRIDS_O MI_Column_Amount_O3_Data_Fields_ColumnAmountO3[_HDFEOS_GRIDS_OMI_Column_Amount_ O3_Data_Fields_ColumnAmount O3.lat=38.375], 309.6

or hear from Siri? July 19, 2012ESIP 2012 Summer Meeting101

HDF4 OPeNDAP handler update Corrected the handling the scale/offset for MODIS products. Added the support for Lambert azimuthal equal-area (LAMAZ). Added the support for Space-oblique Mercator projection (SOM). Other minor bug fixes. ESIP 2012 Summer Meeting102July 19, 2012

Key updates on HDF4 handler July 19, 2012ESIP 2012 Summer Meeting103 version Some MODIS Scale/offset WrongCorrected LAMAZ projection support NoYes SOM projection support NoYes

MODIS scale / offset correction ESIP 2012 Summer Meeting104July 19, 2012

Support for LAMAZ projection ESIP 2012 Summer Meeting105July 19, 2012

Support for SOM projection ESIP 2012 Summer Meeting106July 19, 2012 Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[XDimD2 = 128][YDimD3 = 512]; … } Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; … }

HDF-EOS2 products supported GES DISC AIRS / MERRA / TOMS LAADS / LP DAAC / NSIDC Many MODIS products NSIDC AMSR_E / NISE / MODIS LaRC MISR / MOPITT / some CERES (TRMM) ESIP 2012 Summer Meeting107July 19, 2012

HDF4 products supported GES DISC TRMM L1B, L2B Swath TRMM L3 Grid 42B & 43B products OBPG (Ocean Color) SeaWiFS / MODIST / MODISA CZCS / OCTS ESIP 2012 Summer Meeting108July 19, 2012

HDF4 products supported CERES CER_AVG CER_ES4 CER_ISCCP-D2like-Day CER_ISCCP-D2like-GEO CER_SRBAVG CER_SYN CER_ZAVG ESIP 2012 Summer Meeting109July 19, 2012

HDF4 handler limitations HDF-EOS2 + additional HDF4 objects In some products, some additional HDF4 objects will be ignored. Other HDF4 products not listed in previous slides are not tested and may not be visualized by Java OPeNDAP clients. ESIP 2012 Summer Meeting110July 19, 2012

Other limitations CF conventions Limitation _FillValue inside latitude and longitude Java-NetCDF Limitation Java Heap Memory ESIP 2012 Summer Meeting111July 19, 2012

Summary: we’ve updated handlers ESIP 2012 Summer Meeting112July 19, 2012 OPeNDAP HDF Handlers HDF- EOS2 HDF5 HDF- EOS5 HDF4 … and will continue.

GES DISC Uses Handlers ESIP 2012 Summer Meeting113July 19, 2012

Please try HDF4/HDF5 handlers ESIP 2012 Summer Meeting114July 19, 2012 New hdf5_handler is released! Source codes and RPMs: Demo HDF5/HDF4 servers:

and give us feedback. ESIP 2012 Summer Meeting115July 19, 2012

Got HDF? July 19, 2012ESIP 2012 Summer Meeting116 I’m not a dummy any more.

Enjoy observing the Earth! July 19, 2012ESIP 2012 Summer Meeting117

Topics for advanced dummies ESIP 2012 Summer Meeting118July 19, 2012 That’s it? How about netCDF-4 interoperability?

Clarification –netCDF format netCDF-3 format Simple self-describing data format based on netCDF classic data model netCDF-4 format Uses HDF5 as a storage layer Exploits Compression, chunking, parallel-IO Group hierarchy, user-defined data types, etc. Supports both netCDF enhanced and netCDF classic data models Interoperability with netCDF-4 format in this talk Apr , 2012HDF/HDF-EOS Workshop XV119

Clarification – netCDF packages netCDF software packages netCDF-C Support both netCDF-3 and netCDF-4 formats C++/Fortran Wrappers netCDF-Java Support both netCDF-3 and netCDF-4 formats The implementation of the Common Data Model netCDF version 4 Generally mean the version 4 package of the netCDF-C library (Not only support netCDF-4 format) Apr , 2012HDF/HDF-EOS Workshop XV120

Why netCDF-4 Big user community User-friendly data models Tools Home-grown and third-party visualization and analysis tools - ncdump, ncgen, IDV, Panoply, Ferret etc. Apr , 2012HDF/HDF-EOS Workshop XV121

NetCDF Classic Model Apr , 2012HDF/HDF-EOS Workshop XV122

NetCDF enhanced model Apr , 2012HDF/HDF-EOS Workshop XV123

Conversion July 19, 2012ESIP 2012 Summer Meeting124 Creating a new netCDF-4 file from HDF5 file.

eos52nc4 converts hdf-eos5 to netcdf-4. follows netcdf enhanced model. does not follow CF conventions. Usage: eos52nc4 test.he5 test.nc July 19, 2012ESIP 2012 Summer Meeting125 eos5 NETCDF-4 eos52nc4

Augmentation July 19, 2012ESIP 2012 Summer Meeting126 Making an HDF5 file look like a netCDF-4 file.

aug_eos5 augments hdf-eos5 to netcdf-4. Both HDF-EOS5 and NetCDF4 can access it. Usage: aug_eos5 [-i|-f map] test.h5 modifies the input file directly. does not follow CF conventions. July 19, 2012ESIP 2012 Summer Meeting127 eos5 NETCDF-4 aug_eos5

hdf5 netcdf4 Augmentation Concept July 19, 2012ESIP 2012 Summer Meeting128 hdf-eos5 HDF-EOS5 API NETCDF-4 API aug_eos5 augmentation hdf-eos5 HDF5 API

Why NetCDF-4 Doesn’t Work July 19, 2012ESIP 2012 Summer Meeting129 [nLevels][nTimes] NETCDF-4 API hdf-eos5

What aug_eos5 Does July 19, 2012ESIP 2012 Summer Meeting130 [nLevels][nTimes] NETCDF-4 API hdf-eos5

Options in aug_eos5 1.Default option -i option 2.-f map_file option July 19, 2012ESIP 2012 Summer Meeting131

Default option in aug_eos5 Grid XDim, YDim: calculated Other dims: default fill value Swath / ZA All dims: default fill value July 19, 2012ESIP 2012 Summer Meeting132

Better -i option in aug_eos5 Grid XDim, YDim: calculated Other dims: 1,2,…,n Swath / ZA All dims: 1,2,…,n July 19, 2012ESIP 2012 Summer Meeting133

Best -f option in aug_eos5 Needs an input map file. Input file specifies 3 rules : 1.What to be filled in 2.How to fill in 3.Where to get the values Most useful and flexible option July 19, 2012ESIP 2012 Summer Meeting134

aug_eos5 Input File Format Each line in input file contains 1.How: 0,1,2 2.What: dim name 3.Where: data file or existing variable in file. 4.(Optional) Multiple grid/swath/za case only: grid/swath/za name: July 19, 2012ESIP 2012 Summer Meeting135

aug_eos5 Input File Example 0 nTimes Time 1 nLevels data.txt 2 nCandidates July 19, 2012ESIP 2012 Summer Meeting136 Copy the data Read file data.txt Fill 1,2,…, n HowWhatWhere 95% cases – Copy the data

Multiple Swath Case swath name: BrO 0 nTimes Time 1 nLevels data1.txt 2 nCandidates July 19, 2012ESIP 2012 Summer Meeting137 HowWhatWhere swath name: BrO column 2 nTimes 1 nLevels data2.txt Multiple

aug_eos5 vs eos52nc4 July 19, 2012ESIP 2012 Summer Meeting138 aug_eos5eos52nc4 hdfeos-5YesNo netcdf-4Yes

HIRDLS/MLS Team Use aug_eos5 July 19, 2012ESIP 2012 Summer Meeting139 So You Can!

The HDF Group ESIP 2012 Summer Meeting140 Thank you! July 19, 2012

Acknowledgements ESIP 2012 Summer Meeting141July 19, 2012 This work was supported by Subcontract number under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

The HDF Group Questions/comments? ESIP 2012 Summer Meeting142July 19, 2012