SRNWP Interoperability Workshop GRIB2 vs NetCDF: Evaluation of the Technical Aspects Bruce Wright (IT Architect), ECMWF, Reading, 14-15 January 2008) © Crown copyright Met Office

Contents Background Data Format Overview GRIB vs NetCDF Summary This presentation covers the following areas Background Data Format Overview GRIB vs NetCDF Summary © Crown copyright Met Office

Background © Crown copyright Met Office

Background Activity Diagram showing Met Office high-level gridded data processes: Key processes Distinct data formats: <<datastore>> Data flows between them Note, that this shows: Only some of the core processes At least 9 different gridded data formats: Variants of Fieldsfile/PP (6) NetCDF GRIB1 Nimrod format Large amount of format conversion that is undertake © Crown copyright Met Office

Background (2) Activity: Aim: Recommendations: Review and evaluate the options for managing, storing and exchanging multi-dimensional meteorological gridded datasets within the Met Office Aim: Change the way in which we manage our gridded data through a change in standard data format Recommendations: Adopt CF-compliant NetCDF as the future standard format for (multi-dimensional) gridded data, replacing the current bespoke solution (mainly Fieldsfile / PP) Use GRIB as part of dissemination process for products (primarily for operational meteorology); new products use GRIB2, staged migration from GRIB1 © Crown copyright Met Office

Data Format Overview © Crown copyright Met Office

GRIB Developed by WMO for the exchange of gridded data Allowing detailed description of a huge variety of grids, parameters, processes, represented by codes that reference external tables Portable, implemented as octets (groups of 8 bits): Section Name Length Notes Indicator 16 bytes "GRIB" initially 1 Identification 21 bytes 2 * Local Use Optional section - may contain anything 3 ** Grid definition 52 bytes? 4 *** Product definition 14-50 bytes? 5 *** Data representation typically 21 bytes Dependent on compression 6 *** Bit-map 6 bytes + bitmap Optional section 7 *** Data 5 bytes + data 8 End 4 bytes "7777" © Crown copyright Met Office *, **, *** - can be nested

NetCDF Developed by Unidata to facilitate the access and sharing of array-orientated data in a form that was self-describing and portable Simple, but flexible, data model based around variables, dimensions and attributes. Climate and Forecast (CF) convention (developed for sharing climate model and NWP forecast data) offers a widely-used standard for metadata (esp. through standard names) Users interact with the data and metadata in a file through an API that allows things like querying the number of variables (data objects) in a file, reading or writing slices of data to the variables, defining the variables in a file after creation, etc. Unidata provide a library of routines for interacting with NetCDF files © Crown copyright Met Office

GRIB vs NetCDF © Crown copyright Met Office

Interoperability GRIB NetCDF Standard WMO standard Used by NMSs and other areas of operational meteorology (e.g. Aviation) De facto standard Used by wider fluid earth sciences community (esp. research & academia) Machine Independent Yes Self-Describing No – interpretation requires: - WMO-agreed tables - Local tables Yes – low-level definition of dimensions, variables and attributes COTS Support Limited Mainly WMO-sponsored More extensive ERSI ArcView 9.2 OGC web service standard Transmission (& Archiving) Sequential – record-based access Direct Access – whole file required © Crown copyright Met Office

Metadata GRIB NetCDF (CF) Yes Comprehensive Yes Standard range of descriptors agreed out of considerable experience Less so than GRIB for operational meteorology …but potential to be extremely comprehensive Flexible Yes, but… Use of local tables can be ‘clumsy’ Highly Use of global and variable attributes Human Readable No Self-Describing No – refers to external tables Extensible WMO process slow Local table limitations CF managed through active community mailing list © Crown copyright Met Office

Interface GRIB NetCDF (CF) Standard Interface No, but… ECMWF GRIB API NCEP software… Yes (from Unidata) NetCDF is really an interface, rather than a format Languages ECMWF – C, FORTRAN 77, Java NCEP – C, FORTRAN 90, (Tcl/Tk GUI, MS-Windows degrib) C, C++, FORTRAN 77, FORTRAN 90 Java Bindings to Perl, python, ruby, IDL, Tcl, (PV-Wave), MATLAB, R, OpenDX Platforms ECMWF has tested on Linux, AIX, HP-UX, IRIX Most platforms Installation & Use ? Easy Linux install, learning curve but not severe © Crown copyright Met Office

Compression GRIB NetCDF (CF) Packing Good Simple packing Complex, with different algorithms Crude packing only Lossless Compression Proposals to include LZW, Weather-Huffman, JPEG2000 Fraunhofer Institute developed compression (factor 2.5 on 16 bit data) No for NetCDF3, but Cornell developed extension NetCDF4 (HDF5) supports various Licencing Issues None known, except for specialist solutions Possible limitations for some algorithms © Crown copyright Met Office

General GRIB NetCDF 2D fields only Dimensions 2D fields only GRIB2 allows these to be grouped for higher dimensions N-dimensional e.g. 3 spatial dimensions and time in a single array Small Reads Whole file must be parsed to index file Efficient, as ‘header’ provides direct access Initial Write Sequential format allow efficient streaming File has to be constructed using API © Crown copyright Met Office

Summary © Crown copyright Met Office

Summary CF-compliant NetCDF (CF-NetCDF) GRIB2 NetCDF interface developed by Unidata to facilitate the access and sharing of array-orientated data in a form that was self-describing and portable (the format being machine-independent) Highly flexible data management solution for multi-dimensional gridded data Very widely used in atmospheric and oceanographic sciences community CF (Climate-Forecast) metadata convention offers, arguably, the best metadata standard available within this community GRIB2 Developed as WMO standard to provide an efficient, machine-independent format for the exchange of gridded data by National Met Services No standard interface, although several have been developed Metadata is code-based, needing to cross-reference external tables, with a highly specified metadata ‘vocabulary’ and layout and no indexing of the data © Crown copyright Met Office

Questions and answers © Crown copyright Met Office