WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 An Introduction to GRIB2 Simon Elliott EUMETSAT

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Saturday Afternoon Agenda Background (i.e. my background!) Overview (codes, GRID, GRIB, GRIB2) Code Structure (Sections like others, iterations of sections) GRIB2 Code Templates (what kinds are there … examples) Tables (Template tables, code and flag tables, examples) Identification of parameters (?) GRIB2 compression (methods, explanations of some)

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Sunday Morning Agenda Procedure for code form update (addns to tables, addition of tables/templates, new editions) Changes not requiring code change (adding table/template entries for decoders … down stream code change) Changes requiring code change (always for data provider, plus new editions) Case studies of Cloud Mask and Precipitation Data (bespoke Encoder and Decoder, example product) GRIB2 Interface for NMC (example of NCEP code) Review/Discussion

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Overview Intention is to share information efficiently and unambiguously Data should be accessible by anyone, not encrypted Format should allow all information to be included (not rounded or skipped) Format should allow addition of new data types WMO codes developed per data type, TEMP, SYNOP, SATOB, et c. FM 13 SHIP FM 42 AMDAR FM 41 CODAR FM 35 TEMP FM 88 SATOB FM 87 SARAD FM 86 SATEM FM 85 SAREP

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 GRID, GRIB, GRIB2 GRID was developed for gridded data (forecast, analysis) –ASCII -> human readable? –ASCII -> big files, slow transfer –Symbolic letters and code tables: F 1 F 2 NNN in Section 0 F 1 F 2 is originating centre as per C-1, NNN is catalogue number of grid used by centre GRIB introduced for binary exchange (storage and transmission efficiency). –Edition 0, 1985 –Edition 1, 1990 GRIB Edition 2 (i.e. GRIB2) because Code Table 2 was full, EPS data,...

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 GRIB uses the concept of Template: « Description of the standardized layout of a set of data items » GRIB1 structure Section 0Indicator section: GRIB Section 1Product definition section: unique standard template Section 2Grid description section: one of the standard templates describing a type of grid Section 3Bit map section Section 4Binary data section Section 5End section: 7777

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 GRIB usage On the GTS By the WAFS Centres (« ICAO » products) For archiving of fields on MDD Main limitation is: One parameter on one level for one grid per GRIB field

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 GRIB1 weaknesses Transmission and archiving of: Spectral data Multi-dimension data Long-range and climate products Ensemble products (EPS) Also: No convention for missing data IEEE not used for floating point data No support for small time steps No cross-sections, no time-sections No Hovmöller Diagrams (ex. Time-longitude) Limited support for images

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Need for object oriented GRIB2 (more modules = modularity) Modularity: code and parameter tables referred to through templates Flexibility: new tables and templates can be added ALSO: –All GRIB1 fields can be described in GRIB2 –More compression schemes (e.g. introduction of JPEG 2000 and PNG)

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 GRIB2 calendar Experimental GRIB2 presented at CBS 98 Finalized in Spring 2000 Approved by CBS in Autumn 2000 Operational in November 2001 First products in 2003

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Generic structure Identification: GRIB/BUFR/CREX Header: Date, time, originator, table versions... Optional section: Metadata (potentially XML), private data … Data description: What sort of data follows Actual data: here Closure: 7777 Table driven codes generally have this structure

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 GRIB2 structure Section 0Indicator section: GRIB Section 1Identification section: data characteristics Section 2Optional (local) section: anything or nothing Section 3Grid definition section: geometry of grid used, potential reference to pre-defined grid Section 4Product definition section: description of data type, e.g. satellite data from spectral bands xx, yy, zz Section 5Data representation section: packing method used for data, reference value, scale factor et c. Section 6Bit map section: data present indicators (if used) Section 7Data section: data values themselves Section 8End section: 7777 Note: Sections 1 to 7 start with section length and number

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Section 0 Section 0:Indicator section –Octets 1 - 4:GRIB in ASCII (i.e ) –Octets 5 - 6:Reserved (normally set to 0) –Octet 7:Discipline (CT 0.0, e.g. 3 for space) –Octet 8:GRIB edition number (2) –Octets :Total length in octets Messages of length up to ~18x10 18 (or ) bytes, i.e. ~18,000,000 Tb

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Section 1 Section 1Identification section –Octets 1 - 4:Section length –Octet 5:Section number (1) –Octets 6 - 9:Originating centre and sub-centre –Octets :Master and local table version numbers –Octet 12:Time significance (CT 1.2, e.g. 3 for observation time) –Octets :Date / time –Octet 20:Production status of data (CT 1.3, e.g. 0 for operational data) –Octet 21:Type of data (CT 1.4, e.g. 7 for processed radar observations)

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Section repetition GRIB2 allows some groups of sections to be repeated for efficiency (not in GRIB1, one field per message). Non-repeated sections stay in effect Section 0 (Indicator) Section 1 (Identifier) Section 2 (Local use) Section 3 (Grid) Section 4 (Product) Section 5 (Data Rep n ) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) Repeatable

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Section repetition (example 1) Forecast from same centre on same grid for same parameter at same level but from but for different validity times Validity time is in Section 4 (Product Definition) Section 4 to 7 repeated in message Section 0 (Indicator) Section 1 (Identifier) Section 3 (Grid) Section 4 (Product) Section 5 (Data Rep n ) Section 6 (Bit map) Section 7 (Data) Section 4 (Product) Section 5 (Data Rep n ) Section 6 (Bit map) Section 7 (Data) Section 4 (Product) Section 5 (Data Rep n ) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) 0600Z 1200Z 1800Z

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Section 0 (Indicator) Section 1 (Identifier) Section 3 (Grid) Section 4 (Product) Section 5 (Data Rep n ) Section 6 (Bit map) Section 7 (Data) Section 3 (Grid) Section 4 (Product) Section 5 (Data Rep n ) Section 6 (Bit map) Section 7 (Data) Section 3 (Grid) Section 4 (Product) Section 5 (Data Rep n ) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) Section repetition (example 2) Forecast from same centre for same validity time for same parameter at same level but from but on different grids Grid is defined in Section 3 (Grid Definition Section) Section 3 to 7 repeated in message GRID A GRID B GRID C

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Grid definition templates Grid definition is in Section 3 (GRIB1, Section 2) Can refer to centres own grid or use Grid Definition Template (GDT) to specify details Many GDTs have been developed, e.g. –3.0: Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) –3.1: Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) –3.2: Stretched Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) –3.3: Stretched and Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) –3.20: Polar Stereographic –3.90: Space view perspective, or orthographic

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 A grid definition template example GDT Space view perspective or orthographic Floating point numbers stored in 5 bytes as scale factor (one byte) and scaled value (4 byte IEEE floating point)

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 GDT example continued Repeating sections 4 to 7 means this information need not be repeated for multiple fields in one GRIB2 message

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Product definition templates Product definition is in Section 4 (GRIB1, Section 1) Many PDTs have been developed, e.g. –4.0: Analysis or forecast on horizontal level or layer at a point in time –4.1 to 4.4: Various information pertaining to ensemble forecast systems –4.5: Probability forecast on horizontal level or layer at a point in time –4.7: Analysis or forecast error on horizontal level or layer at a point in time –4.20: Radar products –4.30: Satellite products Hybrid sigma levels can be specified in Section 4

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 A product definition template example PDT Satellite products

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 PDT example continued Note: Octets 25 to 24 are given for band 1, but are repeated for each contributing band up to NB, as specified in octet 14

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Code table 4.3: Type of generating process Code figure Meaning Analysis Initialisation Forecast Bias corrected forecast Ensemble forecast Probability forecast Forecast error Analysis error Observation Reserved for local use Missing

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Data representation templates Data representation is in Section 5 (GRIB1, within Section 4) Some DRTs have been developed, e.g. –5.0: Grid point data - simple packing –5.1: Matrix values at grid point - simple packing –5.2: Grid point data - complex packing –5.3: Grid point data - complex packing and spatial differencing –5.50: Spectral data - simple packing –5.51: Spherical harmonics data - complex packing Section 5 also gives total number of data values to be found in Section 7

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 A data representation template example DRT Grid point data - simple packing IEEE floating point numbers in 32 bits: seeeeeee emmmmmmm mmmmmmmm mmmmmmmm

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Simple and complex packing Simple packing –Y x 10 D = R + (X1 + X2) x 2 E –Y is original value, D is decimal scale, R is reference, E is binary scale, X1 is 0, and X2 is encoded value –In template R, E, D and bits per value are stored –Field values follow sequentially Complex packing (intended to decrease message size) –Y x 10 D = R + (X1 + X2) x 2 E –R, E, D are as for simple packing –X1 is reference for group, X2 is scaled value in group (X1 removed) –Data are split into groups with similar values per group

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Use of D and E D E Increment D E Increment D D - 1 Increment Increment x 10 Here R = 0, X1 = 0, X2 = 1 Increment = Y = original value

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 R, E and D selection example Temperature on balcony: to 35.0, ±0.2 Set D to give increments in range –D = 1, i.e scale by 10 –data are now: -150 to 350, ±2 Set R to minimum –R = (IEEE floating point) –data are now: 0 to 500, ±2 Set E to give required precision –E = 1, i.e. data are in increments of 2 Set bit width to give required range –Required range = scaled range / increment –Required range = 500 / 2 = 250 so 8 bits R = -150, E = 1, D = 1, bit width = 8

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Spatial differencing Saves space for data which vary smoothly (order 1), or data whose variation varies smoothly (order 2), et c. For order 1, data values, f, define new values, g: –g 1 = f 1, g 2 = f 2 - f 1, …, g n = f n - f n-1 –1, 2, 3, 5, 7, 8, 9, 11 becomes 1, 1, 1, 2, 2, 1, 1, 2 For order 2, g values are replaced by h: –h 1 = f 1, h 2 = f 2, h 3 = g 3 - g 2, …, h n = g n - g n-1 –1, 2, 4, 10, 17, 27, 40, 57 becomes 1, 2, 1, 4, 1, 3, 3, 4 Minimum value is subtracted to keep values positive Order 1 Complex packing 4 groups Order 2 Complex packing 3 groups

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Section 6 Section 6:Bit map section –Octets 1 - 4:Length of section –Octet 5:Section number (6) –Octet 6:Bit map indicator (CT 6.0) –Octets 7 - xx:Bit map (if present) Code figure Meaning Bit map is present and follows in this section Use bit map predefined by originating centre Re-use previously defined bit map Missing (no bit map present) Code Table 6.0:Bit map section

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Bit map case study Conclusion: bit map may not always save space, but if more than two bits per data point (typically true) it usually will … consider cloud top height data, space corners, et c No bit map 2000 x 2000 x 8 = 32,000,000 bits No data Data (8 bits per pixel) With bit map bit map (2000 x 2000 x 1) = 4,000,000 bits data ( x 1000 x 1000 x 8) 25,130,000 bits total 29,130,000 bits

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Updates not requiring code change Type of changes concerned –Addition of code and or flag table entries Agreement and implementation –Fast Track can be used –Table entries can be approved for pre-operational use within a few months of consideration and successful validation –Full formal CBS approval follows, available around 2 years later Application updates –Encoder needs new code / flag table available –Encoder software needs updating to insert field –Decoder needs new code / flag table available –Down-stream application (beyond decoder) needs to be updated to process new field –Decoder software can remain unchanged

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Updates requiring code change Type of changes concerned –Changes to code structure (new edition), or new GDT, PDT or DRT Agreement and implementation –ET/DRC considers a request and proposes update –Multilateral validation of proposed update –Full formal CBS approval follows, available around 2 to 4 years later Application updates –Encoder needs new software and /or template available –Encoder software needs updating accordingly –Decoder needs new software and /or template available –Down-stream application (beyond decoder) needs to be updated to process new data Long lead time necessary … consider PUMA example

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Interfacing to third party software Use of third party generic software simplifies implementation NOAA provides GRIB2 software for free download (look at ECMWF provides GRIBEX software - will be / is being updated for GRIB2 (look at Typically software package is set of functions compiled into a library Users application sets up required parameters and calls function from library, which are referenced at run time Specific (not generic) encoders and decoders are quite simple but less flexible

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Some GRIB2 data from EUMETSAT Cloud mask, CLM, every 15 minutes, ~3km resolution, via EUMETCast or archive request. Cloud top height, CTH, every 60 minutes, ~9km resolution, via EUMETCast or archive request. Cloud analysis image, CLAI, every 180 minutes, ~9km resolution, via EUMETCast or archive request. Clear sky reflectance map, CRM, twice per week, ~3km resolution, via EUMETCast or archive request. Multi-sensor precipitation estimate, MPE, every 30 minutes, ~5km resolution, via web site. Fire detection, FIR, every 15 minutes, ~3km resolution, via anonymous FTP (evolving algorithm).

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Case study A, precipitation data

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Case study A, precipitation data GRIB2 encoded file Specific decoder program for GRIB2 precipitation data Decoder output Dump of GRIB2 file

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Case study B, cloud mask data

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Case study B, cloud mask data GRIB2 encoded file Specific en/decoder program for EUMETSAT cloud mask data Decoder output Dump of GRIB2 file Original data to be encoded

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Case study C, cloud analysis image data GRIB2 Cloud analysis image

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Case study C, cloud analysis image data GRIB2 encoded file Specific en/decoder program for EUMETSAT cloud mask data Bit map used

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Case study D, cloud top height data GRIB2 Cloud top height

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Case study D, cloud top height data GRIB2 encoded file Specific en/decoder program for EUMETSAT cloud mask data Bit map used Two repeats of Sections 4, 5, 6 and 7

WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, December, 2005 Sources of GRIB2 information The information used in preparing this presentation is based on: Guide to WMO Table Driven Code Form Used for the Representation and Exchange of Regularly Spaced Data in Binary Form: FM 92-XII GRIB Written by Dr Cliff Dey WMO Manual on Codes, WMO Publication No. 306, Vol. 1, Part B &