1. Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009

2. Overview The NCGEN4 Utility NCGEN4 Command Synopsis netCDF4 in CDL: Types netCDF4 in CDL: Typed Attributes netCDF4 in CDL: Groups Scope Rules Specifying Data Constants Rules for Using Braces Special Attributes Debugging Note: The Cycle Building and Installing NCGEN4 Extended Example

3. The NCGEN4 Utility NCGEN4 extends the CDL language Includes all of the netCDF-4 data model True inverse of ncdump Includes the special attributes (chunking, etc) Supports binary, C, NOT FORTRAN (yet)  Experimental: Java and NcML Can produce 4 kinds of binary netCDF files:  netcdf-3 (classic) 32 bit  netcdf-3 (classic) 64 bit  netcdf-3 (classic, but stored in netcdf-4 file format)  netcdf-4 (supports full netcdf-4 model)

4. NCGEN4 Command Synopsis Primary Command Line Options [-b] Create a (binary) netCDF file (default) [-o ] Name for the binary netCDF file created [-k ] Format of the file to be created 1 => classic 32 bit 2 => classic 64 bit 3 => netcdf-4/CDM 4 => classic, but stored in an enhanced file format [-x] Don't initialize data with fill values [-l ] Specify output language to use when generating source code to create or define a netCDF file matching the CDL specification. Input CDL file

5. netCDF4 in CDL: Types New section called “types:” Consistent with the output of ncdump Supports the new primitive data types: ubyte, ushort, uint, string, int64 (LL), uint64 (ULL) Supports the new user-defined types: int enum enum_t {off=0,on=1,unknown=2}; opaque(11) opaque_t; compound cmpd_t { vlen_t f1; enum_t f2;}; int(*) vlen_t;

6. netCDF4 in CDL: Typed Attributes vlen_t v:attr = {17, 18, 19}; Attribute typing is optional (=> type inferred) Warning! x:attr = “abc”; is inferred to be type char, not string Instead say string x:attr = “abc”; Why? for backward compatibility with ncgen Good practice to add “_t” to the end of all type names Why? Because X :attr = … might be interpreted incorrectly; is X a type or variable?

7. netCDF4 in CDL: Groups group: g {…} A group can itself contain dimensions, types, variables, and groups Name prefixing allows references to types and dimensions that reside in other groups Example: /g/cmpd_t => Do not use ‘/’ in your names Pretty much like the Unix file system Or Windows, but using forward slashes

8. Scope Rules Scope rules determine how references to a dimension or type without a prefix are interpreted General rule: 1.Look in immediately enclosing group 2.Look in the parent of the immediately enclosing group and so on up the enclosing groups For dimensions, if not found => error For types, continue to search the whole group tree to find a unique match, then error if not found

9. Specifying Data Constants Constants for user defined types require the use of braces {…} in certain places. dimensions: d=2; types: int(*) vlen_t; compound cmpd_t { int64 f1; string f2;}; variables: vlen_t v1(d); cmpd_t v2(d); data: v1 = {7, 8, 9}, {17,18,19}; v2 = {107LL, “abc”}, {1234567LL, “xyz”};

10. Rules for Using Braces The top level is automatically assumed to be a list of items, so it should not be inside {...}  Different than C constants lists Instances of UNLIMITED dimensions (other than as the first dimension) must be surrounded by {...} in order to specify the size. Instances of VLENs must be surrounded by {...} in order to specify the size. Compound instances must be embedded in {...} Compound fields may optionally be embedded in {...}. No other use of braces is allowed.

11. Special Attributes Special attributes specified in an ncgen4 CDL file will be properly handled Consistent with ncdump -s Global special attributes  “_Format” – specify the netCDF file format “classic” “64-bit offset” “netCDF-4” “netCDF-4 classic model”  Overridden by the -k flag

12. Special Attributes (cont.) Per-variable special attributes  “_ChunkSizes” – list of chunk sizes 1 per dimension  “_DeflateLevel” – compression level: integer (0-9)  “_Endianness” – “big” or “little”  “_Fletcher32” – “true” or “false” to set check summing  “_NoFill” – “true” or “false” to set persistent NoFill property  “_Shuffle” – “true” or “false” to set shuffle filter  “_Storage” – “contiguous” or “chunked” to set storage mode

13. Debugging Note Use the “Cycle”, Luke Use ncgen/ncgen4 to convert your.cdl to.nc Then use ncdump to convert your.nc to.cdl Compare.cdl to.cdl Watch out for UNLIMITED! dimensions: u = unlimited; variables: v1(u); v2(u); data: v1 = {1,2,3,4}; v2 = {7,8}; Ncdump produces v2 = {7,8,_,_};

14. Building and Installing NCGEN4 Easy: add --enable-ncgen4 to your list of./configure flags Ncgen4 will be installed along with ncdump and the original ncgen

15. Extended Example netcdf foo { types: ubyte enum enum_t {Clear = 0, Cumulonimbus = 1, Stratus = 2}; opaque(11) opaque_t; int(*) vlen_t; dimensions: lat = 10; lon = 5; time = unlimited ; variables: long lat(lat), lon(lon), time(time); float Z(time,lat,lon), t(time,lat,lon); double p(time,lat,lon); long rh(time,lat,lon); string country(time,lat,lon); ubyte tag;

16. Extended Example (cont.) // variable attributes lat:long_name = "latitude"; lat:units = "degrees_north"; lon:long_name = "longitude"; lon:units = "degrees_east"; time:units = "seconds since 1992-1-1 00:00:00"; // typed variable attributes string Z:units = "geopotential meters"; float Z:valid_range = 0., 5000.; double p:_FillValue = -9999.; long rh:_FillValue = -1; vlen_t :globalatt = {17, 18, 19};

17. Extended Example (cont.) data: lat = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90; lon = -140, -118, -96, -84, -52; group g { types: compound cmpd_t { vlen_t f1; enum_t f2;}; } // group g group h { variables: /g/cmpd_t compoundvar; data: compoundvar = { {3,4,5}, Stratus } ; } // group h }

18. Questions?