Utilities for netCDF-4 Dr. Dennis Heimbigner Unidata Advanced netCDF Workshop July 25, 2011

Overview Extending CDL to support netCDF-4 Model netCDF4 in CDL: Types netCDF4 in CDL: Typed Attributes netCDF4 in CDL: Groups Scope Rules Specifying Data Constants Special Attributes NCGEN for netCDF4 NCGEN Command Synopsis Extended Example

Extending CDL to Support netCDF-4 Model ncdump and ncgen both support CDL extended to cover the netCDF-4 model. CDL now has the following new elements:  Extended set of atomic types  User defined types  Typed Attributes  Groups  Special attributes Extended Example

netCDF4 in CDL: Types Supports the new atomic data types: ubyte, ushort, uint, string, int64, uint64 New section called “types:” for user-defined types  int enum enum_t {t=0,f=1,other=2};  opaque(11) opaque_t;  compound cmpd_t { int64 f1; enum_t f2;};  int(*) vlen_t;

netCDF4 in CDL: Typed Attributes vlen_t v:attr = {17, 18, 19}; Attribute typing is optional (=> type inferred) Warning! x:attr = “abc”; is inferred to be type char, not string  Instead say string x:attr = “abc”;  Why? for backward compatibility with ncgen3 Good practice to add “_t” to the end of all type names Why? Because e.g. X :attr = … is ambiguous  Is X a type for a global attribute or a variable?

netCDF4 in CDL: Groups Syntax: group: g {…} [Note the colon] A group can itself contain types, dimensions, variables, and (nested) groups Name prefixing allows references to types and dimensions that reside in other groups Example: /g/cmpd_t => Do not use ‘/’ in your names Pretty much like the Unix file system  Or Windows, but using forward slashes

Scope Rules Scope rules determine how references to a dimension or type without a prefix are interpreted General rule: 1.Look in immediately enclosing group 2.Look in the parent of the immediately enclosing group and so on up the enclosing groups For dimensions, if not found => error For types, continue to search the whole file to find a unique match, then error if not found

Specifying Data Constants Constants for user defined types require the use of braces {…} in certain places. dimensions: d=2; types: int(*) vlen_t; compound cmpd_t { int64 f1; string f2;}; variables: vlen_t v1(d); cmpd_t v2(d); data: v1 = {7, 8, 9}, {17,18,19, 20, 21}; v2 = {107, “abc”}, { , “xyz”};

Rules for Using Braces The top level is automatically assumed to be a list of items, so it should not be inside {...} ( Different than C constants lists). UNLIMITED dimension instances (other than the outer dimension) must be surrounded by {...} in order to specify the size. Vlen instances must be surrounded by {...} in order to specify the size. Compound instances must be embedded in {...} Non-scalar fields of compound instances must be embedded in {...}. Datalists associated with attributes are implicitly a vector (i.e., a list) of values of the type of the attribute and the above rules must apply with that in mind. No other use of braces is allowed (esp. for arrays)

Special Attributes Special attributes specified in an ncgen CDL file will be properly handled Consistent with ncdump -s Global special attributes  “_Format” – specify the netCDF file format “classic” (equivalent to –k1) “64-bit offset” (-k2) “netCDF-4” (-k3) “netCDF-4 classic model” (-k4)  _Format is overridden by command line -k flag

Special Attributes (cont.) Per-variable special attributes _Storage – constant string “contiguous” or “chunked” to set storage mode _ChunkSizes – 1-d array of integer chunk sizes;1 per dimension _DeflateLevel – (integer range 0-9) compression level _Endianness – constant strings “big” or “little” _Fletcher32 – boolean to set check summing _NoFill – boolean sets persistent NoFill property _Shuffle – boolean to enable/disable shuffle filter Note: boolean value can be “true”, “1”, “false”, or “0”

NCGEN Command Synopsis Primary Command Line Option Extensions [-k ] Format of the file to be created 1 => classic 32 bit 2 => classic 64 bit 3 => netcdf-4/CDM 4 => classic, but stored in netcdf-4 file format [-l ] Currently C (full netcdf-4 support); Java, F77 (netcdf-3 only); default => binary output with full netcdf-4 support Input CDL file

Extended Example An extended example is included in the ncgen man page

Questions?

Debugging Note Use the “Cycle” Use ncgen to convert your.cdl to.nc Then use ncdump to convert your.nc to.cdl Compare.cdl to.cdl Watch out for UNLIMITED! CDL: dimensions: u = unlimited; variables: v1(u); v2(u); data: v1 = {1,2,3,4}; v2 = {7,8}; Ncdump produces v2 = {7,8,_,_};