1. SRNWP Interoperability Workshop
GRIB2 vs NetCDF: Evaluation of the Technical Aspects
Bruce Wright (IT Architect), ECMWF, Reading, January 2008)
© Crown copyright Met Office

2. Contents Background Data Format Overview GRIB vs NetCDF Summary
This presentation covers the following areas
Background
Data Format Overview
GRIB vs NetCDF
Summary
© Crown copyright Met Office

3. Background
© Crown copyright Met Office

4. Background
Activity Diagram showing Met Office high-level gridded data processes:
Key processes
Distinct data formats: <<datastore>>
Data flows between them
Note, that this shows:
Only some of the core processes
At least 9 different gridded data formats:
Variants of Fieldsfile/PP (6)
NetCDF
GRIB1
Nimrod format
Large amount of format conversion that is undertake
© Crown copyright Met Office

5. Background (2) Activity: Aim: Recommendations:
Review and evaluate the options for managing, storing and exchanging multi-dimensional meteorological gridded datasets within the Met Office
Aim:
Change the way in which we manage our gridded data through a change in standard data format
Recommendations:
Adopt CF-compliant NetCDF as the future standard format for (multi-dimensional) gridded data, replacing the current bespoke solution (mainly Fieldsfile / PP)
Use GRIB as part of dissemination process for products (primarily for operational meteorology); new products use GRIB2, staged migration from GRIB1
© Crown copyright Met Office

6. Data Format Overview
© Crown copyright Met Office

7. GRIB Developed by WMO for the exchange of gridded data
Allowing detailed description of a huge variety of grids, parameters, processes, represented by codes that reference external tables
Portable, implemented as octets (groups of 8 bits):
Section
Name
Length
Notes
Indicator
16 bytes
"GRIB" initially 1
Identification
21 bytes
2 *
Local Use
Optional section - may contain anything
3 **
Grid definition
52 bytes?
4 ***
Product definition
14-50 bytes?
5 ***
Data representation
typically 21 bytes
Dependent on compression
6 ***
Bit-map
6 bytes + bitmap
Optional section
7 ***
Data
5 bytes + data 8
End
4 bytes
"7777"
© Crown copyright Met Office
*, **, *** - can be nested

8. NetCDF
Developed by Unidata to facilitate the access and sharing of array-orientated data in a form that was self-describing and portable
Simple, but flexible, data model based around variables, dimensions and attributes.
Climate and Forecast (CF) convention (developed for sharing climate model and NWP forecast data) offers a widely-used standard for metadata (esp. through standard names)
Users interact with the data and metadata in a file through an API that allows things like querying the number of variables (data objects) in a file, reading or writing slices of data to the variables, defining the variables in a file after creation, etc.
Unidata provide a library of routines for interacting with NetCDF files
© Crown copyright Met Office

9. GRIB vs NetCDF
© Crown copyright Met Office

10. Interoperability GRIB NetCDF Standard WMO standard
Used by NMSs and other areas of operational meteorology (e.g. Aviation)
De facto standard
Used by wider fluid earth sciences community (esp. research & academia)
Machine Independent
Yes
Self-Describing
No – interpretation requires:
- WMO-agreed tables
- Local tables
Yes – low-level definition of dimensions, variables and attributes
COTS Support
Limited
Mainly WMO-sponsored
More extensive
ERSI ArcView 9.2
OGC web service standard
Transmission (& Archiving)
Sequential – record-based access
Direct Access – whole file required
© Crown copyright Met Office

11. Metadata GRIB NetCDF (CF) Yes
Comprehensive
Yes
Standard range of descriptors agreed out of considerable experience
Less so than GRIB for operational meteorology
…but potential to be extremely comprehensive
Flexible
Yes, but…
Use of local tables can be ‘clumsy’
Highly
Use of global and variable attributes
Human Readable No
Self-Describing
No – refers to external tables
Extensible
WMO process slow
Local table limitations
CF managed through active community mailing list
© Crown copyright Met Office

12. Interface GRIB NetCDF (CF) Standard Interface No, but… ECMWF GRIB API
NCEP software…
Yes (from Unidata)
NetCDF is really an interface, rather than a format
Languages
ECMWF – C, FORTRAN 77, Java
NCEP – C, FORTRAN 90, (Tcl/Tk GUI, MS-Windows degrib)
C, C++, FORTRAN 77, FORTRAN 90
Java
Bindings to Perl, python, ruby, IDL, Tcl, (PV-Wave), MATLAB, R, OpenDX
Platforms
ECMWF has tested on Linux, AIX, HP-UX, IRIX
Most platforms
Installation & Use ?
Easy Linux install, learning curve but not severe
© Crown copyright Met Office

13. Compression GRIB NetCDF (CF) Packing Good Simple packing
Complex, with different algorithms
Crude packing only
Lossless Compression
Proposals to include LZW, Weather-Huffman, JPEG2000
Fraunhofer Institute developed compression (factor 2.5 on 16 bit data)
No for NetCDF3, but Cornell developed extension
NetCDF4 (HDF5) supports various
Licencing Issues
None known, except for specialist solutions
Possible limitations for some algorithms
© Crown copyright Met Office

14. General GRIB NetCDF 2D fields only
Dimensions
2D fields only
GRIB2 allows these to be grouped for higher dimensions
N-dimensional
e.g. 3 spatial dimensions and time in a single array
Small Reads
Whole file must be parsed to index file
Efficient, as ‘header’ provides direct access
Initial Write
Sequential format allow efficient streaming
File has to be constructed using API
© Crown copyright Met Office

15. Summary
© Crown copyright Met Office

16. Summary CF-compliant NetCDF (CF-NetCDF) GRIB2
NetCDF interface developed by Unidata to facilitate the access and sharing of array-orientated data in a form that was self-describing and portable (the format being machine-independent)
Highly flexible data management solution for multi-dimensional gridded data
Very widely used in atmospheric and oceanographic sciences community
CF (Climate-Forecast) metadata convention offers, arguably, the best metadata standard available within this community
GRIB2
Developed as WMO standard to provide an efficient, machine-independent format for the exchange of gridded data by National Met Services
No standard interface, although several have been developed
Metadata is code-based, needing to cross-reference external tables, with a highly specified metadata ‘vocabulary’ and layout and no indexing of the data
© Crown copyright Met Office

17. Questions and answers
© Crown copyright Met Office