1. WHY TABLE DRIVEN CODE FORMS?
Langen, 17 April 2007
(Joël Martellet,
WMO, World Weather Watch, Data Processing and Forecasting Systems)

2. Why table driven codes?: Plan
The past: fixed alphanumeric codes
and the present: fast evolution of science and technology
Solution: table driven codes
Objectives of this seminar
What is different?

3. WHY WMO CODES?
A WMO “code” is a way of representing data which are exchanged internationally between countries = a data representation form
The Purpose is global exchange of meteorological (or associated to meteorology or hydrology, e.g. marine, oceanography) data in an unambiguous and most efficient way for use by many different applications in different countries
FUNDAMENTAL: IT IS THE RAW MATTER MAKING METEOROLOGY
The design of the Codes should take into account operational constraints

4. WMO DATA REPRESENTATION
The design of a data representation system
for real time exchange and processing
has to take into account operational constraints
THE PAST: the Constraints decades ago:
Telecommunication lines with low bandwidth (50 Bits/s) led to:
character type
abbreviated coding
parameter representation often translated by code rather than the exact value
Manual operation:
human readability
not too frequent
Solution:
FIXED ALPHANUMERIC CHARACTER CODES

5. Teletype Paper Manual plotting
CONSIDER THE WORLD WEATHER WATCH DATA FLOW (OLD!): Before the meteorological codes were seen! at all stages of the data flow!
Teletype Paper Manual plotting
Forecaster
Observerand
encoding on Teletype

6. And they have a «fixed» format.
The old WMO “codes” were developed by data type, e.g. TEMP, SYNOP, SATOB, etc.
And they have a «fixed» format.
FM 42 AMDAR
FM 41 CODAR
FM 35 TEMP
FM 13 SHIP
FM 88 SATOB
FM 87 SARAD
FM 86 SATEM
FM 85 SAREP

7. Nowadays: faster technical evolution
Expanding requirements for applications: e.g. NWP and Global Climate Studies
Increasing volume of data (e.g. satellite) and complexity of observations
Higher accuracy of the measurements
Higher resolution required for observations (e.g. Radiosonding):
In time (more frequent)
In space (vertical resolution)
All radiosonde observations required (flight of the sonde, like an aircraft) for very high resolution non-hydrostatic models.)
New types of data observed and exchanged (e.g. ozone, radiology, sea and water level , etc…)
More frequent requested changes for new data types

8. And how to modify a traditional “fixed” code like inserting a new group:
Such a modification would require a corresponding update to all software programs that encode or decode such reports, if not: the software would either give incorrect values or fail completely.
The reason is that the fixed format and the coding conventions which define the data have to be translated and built fully into the coding and decoding programs. It is this fact that renders the traditional alphanumeric code forms incapable of accommodating new types of data.

9. Difficulties, if not impossibilities:
Changing requirements but fixed codes
Change implies long time scale and costs, because automation came!:
software fixes needed at all stage of the processing chain
equipment made by manufacturers
training required, etc.
Approval of code changes by WMO official body: CBS, and then implementation by Member Countries take from 2 to 4 years, or even more
This is incompatible with the modern fast rate of evolution of science and technology

10. Then blocking !
Current (but old) Traditional Alphanumeric Codes (TAC) prevent the exchange of critical environmental information to NMHSs and their customers
There is an unnecessarily inefficient costly use of resources for information exchange
They will increasingly constrain NMHS capabilities to exchange more accurate and timely information
They will become increasingly more costly to sustain expansion and growing scientific needs.

11. WHY TABLE DRIVEN CODES? Operational meteorology evolves:
More automation:
more computer processing
Higher bandwidth (9.6kb/s, 64kb/s, 1 Mb/s)
More frequently: new parameters (e.g. satellite, oceanography)
Higher resolution required in time and space
Higher accuracy- more precision required

12. Automatic decoding, displaying
Today the meteorological code does not need to be seen at all! CONSIDER THE WORLD WEATHER WATCH DATA FLOW (ALL AUTOMATED):
Automatic decoding, displaying
program
Forecaster
Automatic
Weather Station (AWS)
automatic encoding

13. Different functions within the data flow:
Not to confuse the functions of:
Data acquisition
Data collection
Data transmission
Data reception
Data visualization
The format used to represent data could be different at each stage, if it is more efficient.

14. The format used to represent data could be different at each stage, if it is more efficient. Not to confuse the functions of:
Data acquisition: AWS sensors value format (or observer reporting data)
Data collection: Encoding in a data representation form the observer report (or the AWS sensors values) for national collection
Data transmission: Keeping format or Encoding with a view to perform international transmission
International transmission process: If possible do not change the data representation, just move a mail “envelope”
Data reception: Decode the data representation form to feed data base or other processing applications
Data visualization: Visualize the data values for a human reader = convert to the most appropriate display format for the user

15. The WMO concern is the International Exchange and the agreed formats to transmit data between countries.
National formats could be different than WMO recommended standards if more appropriate.
But we need new efficient formats for international exchanges.

16. SOLUTION: Need for Data Representations which offer:
EXPANDABILITY (ability to add easily new parameters, to change accuracy)
SELF DESCRIPTION (auto description of content)
FLEXIBILITY (ability to vary the content)
SUSTAINABILITY (old archives readable)
COMPRESSION (for binary digital exchange)

17. SOLUTION: Need for Data Representations which offer:
EXPANDABILITY (ability to add easily new parameters, to change accuracy):
BUFR, CREX, GRIB Edition 2 (GRIB2)
SELF DESCRIPTION (auto description of content)
BUFR, CREX, GRIB Edition 1 (GRIB1), GRIB2
FLEXIBILITY (ability to vary the content)
BUFR, CREX, GRIB2
SUSTAINABILITY (old archives readable)
BUFR, CREX, GRIB1, GRIB2
COMPRESSION (for binary digital exchange)
BUFR, GRIB1, GRIB2

18. BUFR: Binary Universal Form for the Representation of meteorological data designed in early eighties- approved for operation implementation on November Used for archives of all data types and operational exchange of satellites data, ASDAR, AMDAR ,wind profilers, tropical cyclone data, ARGOS data: buoy, XBT, XCTD, sub-surface floats, and starts to be used for translating Traditional Alphanumeric Codes (TAC: SYNOP, SHIP, TEMP, CLIMAT, etc..).
CREX:
Character form for the Representation and Exchange of data designed in the nineties (it is the image of BUFR in character) - approved for operational implementation on May designed for data types which do not have Traditional Alphanumeric Codes (TAC) and where BUFR cannot be used - Used for operational exchange of ozone, radiological, tide gauge, hydrological and soil temperature data , squall lines (Africa) and tropical cyclone information (also it is a good tool to understand BUFR - could be an interim solution before BUFR, if binary connection and binary processing not possible)

19. Objectives of the seminar
To understand fully the structure of BUFR
To understand how to convert Traditional Alphanumeric Codes (TAC) data into BUFR
To understand how to initiate and have approved additions to the Table Driven Code Forms
To understand how to use and implement in the data processing chains some of the existing decoder and encoder software
To understand how to organize the migration internationally and therefore nationally (migration plan)
To understand what to do to start operationally the migration within the World Weather Watch System

20. TABLE-DRIVEN CODES FORMAT
In a table driven code form, like in a Traditional Alphanumeric Codes, there is also a fixed structure, but it applies only to the shape of the «container» (or code layout or structure) rather than to the content of the «container». These structure rules are the same for whatever the type of data: “only one physical format for all data types”. The presence and form of the data are described within the «container» itself. This is the concept of:
SELF-DESCRIPTION.
In order to accomplish it, there is a section: THE DATA DESCRIPTION SECTION in a BUFR, CREX or GRIB message, where the type and form of the data, contained within the message, are defined.
Once this section is read, the following content: the transmitted data (in the DATA SECTION) can be understood (and decoded).

21. THE DATA DESCRIPTION SECTION
This section will in fact contain a set of “pointers” (called “descriptors”) which refer to elements which are listed in predefined and internationally agreed TABLES (kept in the official WMO Manual on Codes and on the WMO web server). In the TABLES, the list of descriptors (elements) defines how every “datum” is and SHALL BE coded.
Hence: the name:
“TABLE DRIVEN CODE FORMS”.
All the “datum” to be transmitted must be defined in the tables of the WMO Manual (e.g. name, unit (m/s, °K,…), size: 6 bits, 9 bits,…).

22. Descriptors in the Data Description Section
In the MESSAGE:
The Data Description Section:
I see the List of Descriptors
In the
WMO MANUAL:
Set of TABLES
containing
set of Descriptors
Each Descriptor defines how a parameter (an element)
Shall be coded: if I know that, I can decode the data in the
Data Section (indeed, the data must be listed in the same
order as the descriptors in the Description Section) and the
Tables must be readable by the decoder (program or human)

23. General structure of a TDCF message
Indicator: GRIB/BUFR/CREX
Identification: Date, time, originator, edition number, table version number ...
Optional section: e.g. more Metadata, private data …
Data description section: What sort of data follows (pointers to WMO tables identifying the data to be transmitted)
Data section: Actual data here
Closure: “7777”
Table driven codes generally have this structure 

24. When there is a requirement for transmission of new parameters or new data types, items are simply added to the WMO defined tables (to be agreed by CBS) and new pointers (descriptors) will be included in the data description section to reference the new items, if they must be exchanged. In principle, table driven codes can transmit an “infinity of information”. There is total FLEXIBILITY.
Definition of new «codes » as such, is no longer necessary. Expansion of WMO tables is sufficient = EXPANDABILITY.
Edition number of the format (physical structure of the message) and version number of the WMO tables recorded in the identification section at a fixed for ever place enable the safe processing of archived data = SUSTAINABILITY.

25. BUFR versus CREX IN THE DATA SECTION:
IN BUFR, AN ITEM (PARAMETER CORRESPONDING TO A TRANSMITTED DATUM IN A REPORT) WILL BE TRANSLATED INTO A SET OF BITS.
IN CREX AN ITEM WILL BE TRANSLATED INTO A SET OF CHARACTERS (BYTES). CREX IS THE IMAGE IN CHARACTERS OF BUFR BIT FIELDS. THUS, IT IS ALSO VERY SIMPLE TO READ A CREX MESSAGE.
IN BUFR AND CREX THE PARAMETERS ARE SIMPLY LISTED AS THE USER REQUIRES. THE DATUM ARE LAYED OUT ONE AFTER THE OTHER.

26. BUFR versus CREX Compression(1)
BUFR has an advantage over CREX: it offers condensation (packing) or compression, therefore voluminous data (ex. satellites, ACARS, wind profilers) will require less resources for transmission and stocking.
But the big disadvantage is that man cannot read it!

27. BUFR versus CREX Compression(2)
BUFR provides efficient data packing via:
“BUFR compression” part of BUFR (and GRIB) specifications, like run length compression. Compression is performed by an algorithm which is defined within the code regulations of BUFR (and GRIB).
And somehow data condensation by:
Use of combined data descriptors, called “common sequences”
Use of scale and reference values for descriptors of datum.
CREX is not designed for packing efficiency, obviously not fit for bulky data

28. BUFR versus CREX CREX, provides flexibility and human readability.
It requires for the transmission of big or many reports a substantial amount of characters. (However, if considered as a bit stream, and by applying compression algorithms available today, it can be compressed somehow, like any computer file) .
CREX tables will have the same parameters as BUFR tables, and similar rules, but CREX will be simpler.
There is no packing, no associated data (flags, substituted values).
CREX is the image of BUFR and provides a standard (but not the best) for visualisation (read and decode ) of BUFR information.

29. VERY SIMPLE CREX EXAMPLE
T A000 B11001 B
7777
Data Description Section:
CREX WMO Master Table 00, Edition1, Version 4, Surface data, wind direction, wind speed
Data Section:
wind 1: direction 260 degrees, speed 24.6 m/s
wind 2: direction 30 degrees, speed 13.7 m/s

30. DATA DESCRIPTION SECTION:
CODING OF SQUALL LINES IN WEST AFRICA IN CREX: Observations (3 points) and forecasted trajectory and evolution:
CREX++
T // A P U00 S001 Y H1830 D
– – –
7777
DATA DESCRIPTION SECTION:
T // = 00 = Master table for meteorology
02 = CREX edition number
04 = BUFR edition number
12 = version table of BUFR/CREX
// = No local table
A = 007 = Synoptic features
001 = Squall Line
P = = Originating Centre = Dakar
000 = No sub-centre
U00 = 00 = Sequence number of message, = 0 = first
S001 = 001 = number of sub-sets in the report = 1
Y = Date of the message = year, month, day
H1830 = Hour of the message = hour, minute
D = Common sequence defining Squall line (type 1)

31. CODING OF SQUALL LINES IN WEST AFRICA IN CREX: Observations (3 points) and forecasted trajectory and evolution:
CREX++
T // A P U00 S001 Y H1830 D
– – –
7777
DATA SECTION (Part 1):
Time of observation:
D01011 Date:
2005 = Year
08 = Month
23 = Day
D01012 Hour:
17 = Hour
50 = Minute
Position of Squall Line Centre:
B05002 Latitude:
1500 = 15 deg. North
B06002 Longitude:
= 10 deg. West
B19005 Direction of moving feature:
070 = 070 degrees
B19006 Speed of moving feature:
= 10 m/s

32. CODING OF SQUALL LINES IN WEST AFRICA IN CREX: Observations (3 points) and forecasted trajectory and evolution:
CREX++
T // A P U00 S001 Y H1830 D
– – –
7777
DATA SECTION (Part 2):
Amplitude of feature, from most external point to centre point: North side:
B05002 Latitude:
1900 = 19 deg. North
B06002 Longitude:
= 8 deg. 40 West
South side:
1100 = 11 deg. North
= 12 deg. 20 West
B20048 Evolution of feature:
02 = Intensification
B11041 maximum burst expected:
0038 = 38 m/s
B13055 intensity of rain expected:
0300 = 300 mm/h
CODE TABLE
Evolution of feature
Code figure
 0 Stability
1 Diminution
2 Intensification
3 Unknown
4-14 Reserved
14-99 Not used

33. EXAMPLE FOR SYNOP: A SYNOP message from an RA VI high altitude station
- the station VAN (Turkey):
SMTU10 LTAA
AAXX 23064
333  / =
KSMDii LTAA
CREX++
T A000 D
/////
////
7777
Note:
The underlined groups in the data section
represent additional information that is not included in the SYNOP report.

34. The Table Driven Codes Forms (TDCF) BUFR/CREX
Offer great advantages compared to the traditional alphanumeric codes (TAC) (1):
Self-description, flexibility, expandability, sustainability, packing (BUFR), readability (CREX= image of BUFR in alphanumeric code )
For new parameters or new data types, no need to change software, just additional table entries (this is fundamental in light of the fast evolution of science and technology: there are regular requests for representation of NEW DATA TYPES, METADATA, HIGHER RESOLUTION (TIME AND SPACE) AND HIGHER ACCURACY)

35. The Table Driven Codes Forms (TDCF) BUFR/CREX
Offer great advantages compared to the traditional alphanumeric codes (TAC) (2):
The systematic passing of metadata including geographical coordinates (latitude, longitude, height) in every report, which can be easily performed with the table driven codes, would alleviate the notorious WMO Volume A problems.
The Volume A, containing stations coordinates, is updated with too much delay, the WMO secretariat receiving sometimes with considerable delay and other times not at all, the updates that the Countries should send. The use of BUFR or CREX would solve the majority of the cases where there is a problem of wrong coordinates for a station.
The reliability of binary data transmission leads to expectation for an increase in data quality and data quantity received in meteorological centres.
More data and of better quality leading to better data assimilation, and consequently better products: better forecasts, better climate studies and generation of better products by data processing centres

36. SUMMARY: TABLE DRIVEN CODES, LIKE BUFR, GRIB 2 AND CREX OFFER:
SELF DESCRIPTION
FLEXIBILITY
EXPANDABILITY
SUSTAINABILITY
COMPRESSION (PACKING) FOR BUFR
EASY READABILITY FOR CREX
DATA OF BETTER QUALITY LEADING TO BETTER PRODUCTS
IT IS A MUST TO USE THEM IN THE 21ST CENTURY FOR THE SAKE OF SCIENCE EVOLUTION AND PROGRESS

37. THANK YOU FOR YOUR ATTENTION
Questions???