WMO BUFR &CREX Gil Ross, UK Met Office

WMO BUFR Standard BUFR –Binary Universal Form for the Representation of meteorological data –Operational 1988 CREX –Character form for the Representation and EXchange of data –Semi-operational 1994 Difference between BUFR and GRIB (generally) –GRIB is for uniform (multidimensional) array coverages –BUFR is for keyword pairs (hashes in perl, dictionaries in python) Codes (keyword pairs) are "self-descriptive“ –the description and content of the data are both contained within the BUFR or CREX message itself –Unlike XML elements (or hashes and dictionaries), keyword pairs are split into descriptor groups and data groups allowing independent compaction techniques Same philosophy and the same era as SGML (1986) predates XML (late 1990s) Major difference –BUFR and CREX formats have brevity as a major design requirement. –XML philosophy Terseness in Markup is of minimal importance”

What is WMO standard: BUFR Domain Specific Language and Schema –Specific application is Meteorology in general, also Oceanography –Domain is any data type fully definable in formal table structure –Everything is predefined in references Even operators which redefine the references Data model and data instance –Metadata – standard and locally defined metadata –Local metadata allows foreign key structures enabling BUFR to be used as RDBMS objects. –Data definition structure Every instance contains its own “DTD” –Composed of full or partial specification –or reference(s) to predefined components of the “DTD” Dynamic data specification operations –effectively allows a dynamic “feature catalogue”, –defining new features in an instance –Dynamic specification not achievable in ISO 191xx Controlled vocabulary, reference and code specifications Decoded/expanded BUFR is equivalent to a full markup specification

BUFR Data model Descriptor definition structure –Simple descriptor classes and elements –Operators –Coding operators – of course - but also –Definition operators (coverage operators, generalised coordinates) Grouping-hierarchy operator Descriptor (Feature) attribute operator Descriptor (Feature) context modifier Data specification –Highly defined and modifiable data precision specification Registry table structure –WMO standard versions –Local versions –Complete independent registries (oceanography)

BUFR Standard Terseness of BUFR and CREX –Predefine all characteristics –Refer to published tables for all descriptors, operations and data codes. –BUFR is formed from main tables and subsidiary code tables However BUFR standard mixes a number of ISO concepts together –One difficulty in converting from BUFR to ISO/GML like XML is deciding which ISO concept to use When the concepts are separated –BUFR realises most of the basic ISO modelling. –BUFR uses generalised coordinates to do all three ISO/XML things

BUFR in operation Exchange container –BUFR –millions of BUFR files exchanged each day –BUFR can even encode its own registry tables for exchange. –Several compression mechanisms The design decisions allow conflicting requirements of “full specification” and high “compactness”. –Everything is a reference.

BUFR – ISO Concepts Main classification table A – hierarchy of 20 feature catalogues. Table D sequences (around 300 sequences with 4000 elements) –collections of features or collections of collections of features. Table B elements (around 1200) –800 simple features in 30 classes, –8 classes (with 400 members using 1500 code values) define generalised coverage grids –others define operations on the data values. –refers to around 370 code tables, flag tables and enumerations for classifications of discrete data features. Table C defines data descriptors and operations on the data descriptors –compression, sequencing and repetition. –operators which redefine the nature of the features mutable features. –also defines operations on the data which include data compression. –ISO has feature operations BUT they change the VALUE of the feature, –BUFR generalised Coordinates - particularly the significance coordinates in group 8, change the MEANING and CONTEXT of the feature. BUFR message –feature instance –Millions of new BUFR messages are exchanged each day between and within WMO Member States

Table A – 20 Feature Catalogues Table D – Feature collections 300 collections 4000 elements in sequence Table B – 8 Coverage GRID Definition Classes 1200 simple elements Table B references 370 Code Tables code tables – flag tables - enumerations Table C – operators defining coding Changing table B attributes Changing descriptors Table C also defines further compression BUFR Message – Feature Instance

BUFR Feature Catalogue

Table A - Feature Catalogue 20 sub catalogues

Table B simple Feature Classes

Coordinates - Coverage Grids

Code Tables discrete Coverages

Continuous Coverage

Table C Descriptor coding operators

BUFR Message –Feature Instance