1. Slide 1 WMO BUFR training course Langen 17-20 April 2007 Slide 1 BUFR software Milan Dragosavac

2. Slide 2 WMO BUFR training course Langen 17-20 April 2007 Slide 2 ECMWF Data Formats Standard WMO data formats ASCI and binary FM 94-XII Ext. BUFR FM 92-XII Ext. GRIB BUFR for observations and verification scores GRIB for fields Archiving formats BUFR and GRIB only

3. Slide 3 WMO BUFR training course Langen 17-20 April 2007 Slide 3 Requirements for BUFR software Suitable for all observations data handling Building blocks for other applications Encoding Decoding Low maintenance costs Portable User friendly

4. Slide 4 WMO BUFR training course Langen 17-20 April 2007 Slide 4 Programming languages used Fortran 77 - All BUFR subroutines and functions C - Reading/writing BUFR data - gbyte, sbyte routines to do bit manipulations

5. Slide 5 WMO BUFR training course Langen 17-20 April 2007 Slide 5 Ecmwf BUFR software How to get the software? The software can be downloaded from Ecmwf web site http://www.ecmwf.int/products/data/software/ Go to Free download and get bufr_000320.tar.gz file gunzip bufr_000320.tar.gz cd bufr_000320 Read README file for explanations Execute make command which will create libbufr.a cd examples Running make will create number of executables

6. Slide 6 WMO BUFR training course Langen 17-20 April 2007 Slide 6 Ecmwf BUFR software Content of bufr_000320 directory -rwxr--r-- 1 maa ma 462 Jul 19 2006 Makefile -rwxr--r-- 1 maa ma 467 Jul 19 2006 Makefile.in -rwxr--r-- 1 maa ma 11201 Jul 19 2006 README drwxr-xr-x 2 maa ma 4096 Jan 22 17:13 bufr_main_c_program drwxr-xr-x 2 maa ma 4096 Feb 14 08:35 bufrdc drwxr-xr-x 2 maa ma 8192 Feb 14 08:35 bufrtables -rwxr--r-- 1 maa ma 5327 Jul 19 2006 build_library drwxr-xr-x 2 maa ma 4096 Feb 7 07:52 config

7. Slide 7 WMO BUFR training course Langen 17-20 April 2007 Slide 7 Ecmwf BUFR software drwxr-xr-x 2 maa ma 4096 Jan 5 11:42 data drwxr-xr-x 2 maa ma 4096 Jul 19 2006 doc drwxr-xr-x 2 maa ma 4096 Feb 14 08:28 examples -rwxr--r-- 1 maa ma 1330 Jul 19 2006 install -rwxr--r-- 1 maa ma 1196 Jul 19 2006 licence drwxr-xr-x 2 maa ma 4096 Jan 22 16:35 options drwxr-xr-x 2 maa ma 4096 Feb 14 08:35 pbio

8. Slide 8 WMO BUFR training course Langen 17-20 April 2007 Slide 8 Ecmwf BUFR software config directory contains various configuration files for different platforms and compilers options directory contains various options for different platforms You can make additional configurations and options if needed

9. Slide 9 WMO BUFR training course Langen 17-20 April 2007 Slide 9 Ecmwf BUFR software pbio pbio directory contain all c and fortran routines needed for BUFR and CREX IO all routines are part of bufr and crex libraries bufrdc bufrdc directory contain all bufr software source code routines which are written in fortran 77

10. Slide 10 WMO BUFR training course Langen 17-20 April 2007 Slide 10 Ecmwf BUFR software examples decode_bufr.F tdexp.F bufr2crex.F create_bufr.F doc doc directory contain bufr software Users Guide and Reference Manual in pdf format

11. Slide 11 WMO BUFR training course Langen 17-20 April 2007 Slide 11 Bufr format Binary universal form for the representation of meteorological data Indicator section Identification section Optional section Data description section Data section End section

12. Slide 12 WMO BUFR training course Langen 17-20 April 2007 Slide 12 Bufr indicator section Octet numberContent 1 - 4BUFR four letters in CCITT IA5 5 - 7Total length of Bufr message in bytes 8Bufr edition number ( currently 4 )

13. Slide 13 WMO BUFR training course Langen 17-20 April 2007 Slide 13 Identification section Octet numberContent 1 - 3Length of section 1 4Bufr master table (zero if WMO FM 94-XII used ) 5 - 6Originating / generating centre 7 - 8Originating / generating sub-centre

14. Slide 14 WMO BUFR training course Langen 17-20 April 2007 Slide 14 Identification section continued Octet numberContent 9 Update sequence number (zero for original BUFR message; incremented by one for updates) 10 Bit 1 = 0 no optional section Bit 1 = 1 optional section follows Bit 2-8 set to zero ( reserved ) 11Data category ( Bufr table A ) 12International data sub-category (common Tab 13) 13Local sub-category

15. Slide 15 WMO BUFR training course Langen 17-20 April 2007 Slide 15 Identification section continued Octet numberContent 14Version number of master table used 15 Version number of local tables used ( set to 0 or 255 if no local tables used 16 - 17Year ( 4 digits ) 18Month 19Day 20Hour 21Minute 22Second 23-Reserved for local use by ADP centres

16. Slide 16 WMO BUFR training course Langen 17-20 April 2007 Slide 16 Optional section Octet numberContent 1 - 3Length of section in bytes 4Set to zero ( reserved ) 5-Reserved for local used by ADP centres

17. Slide 17 WMO BUFR training course Langen 17-20 April 2007 Slide 17 Data description section Octet numberContent 1 - 3Length of section in bytes 4Set to zero ( reserved ) 5-6Number of data subsets 7 Bit 1 = 1 observed data Bit 1 = 0 other data Bit 2 = 1 compressed data Bit 2 = 0 non compressed data Bits 3-8 set to zero ( reserved ) 8Collection of descriptors defining the data

18. Slide 18 WMO BUFR training course Langen 17-20 April 2007 Slide 18 Data section Octet numberContent 1-3Length of section in bytes 4Set to zero ( reserved ) 5-Binary data as defined by sequence descriptors

19. Slide 19 WMO BUFR training course Langen 17-20 April 2007 Slide 19 End section Octet numberContent 1 - 47777 ( coded according to CCITT IA5 )

20. Slide 20 WMO BUFR training course Langen 17-20 April 2007 Slide 20 Ecmwf Bufr software Bufr is table driven code form - no need to transfer element names, units … - no need to change bufr software for new observations Bufr table B – classification elements - reference number - element name - element unit - scale - reference value - data width in bits

21. Slide 21 WMO BUFR training course Langen 17-20 April 2007 Slide 21 Ecmwf Bufr software Bufr table D – list of common sequences to describe the data - table B elements - operators - other table D descriptors Bufr table C – text and meaning of code/flag tables

22. Slide 22 WMO BUFR training course Langen 17-20 April 2007 Slide 22 Ecmwf Bufr software Bufr table naming convention Bssswwwwwxxxxxyyyzzz.TXT Cssswwwwwxxxxxyyyzzz.TXT Dssswwwwwxxxxxyyyzzz.TXT sss - Master table number ( zero for WMO meteorological tables wwwww - Originating sub-centre xxxxx - Originating centre yyy - Version number of master tables used zzz - Version number of local tables used

23. Slide 23 WMO BUFR training course Langen 17-20 April 2007 Slide 23 Ecmwf Bufr software PATH for Bufr tables export BUFR_TABLES=/xxx/ setenv BUFR_TABLES /xxx/ The path must end with back slash / Defaults Integer missing value indicator NVIND=2147483647 Real missing value indicator RVIND= 1.7E38

24. Slide 24 WMO BUFR training course Langen 17-20 April 2007 Slide 24 Decoding Bufr data There are four steps in bufr data processing Open bufr input file Read in one bufr message at a time Expand bufr message Use the data Repeat last three steps until the end of file is reached

25. Slide 25 WMO BUFR training course Langen 17-20 April 2007 Slide 25 PBIO routines FORTRAN routines to handle products in an unblocked binary file subroutine PBOPEN ( kunit, filename, mode, kerr ) Input arguments: filename - character variable mode - character variable r read, w write, a append, r+ read and write Output arguments: kunit - integer unit number kerr - integer status return code

26. Slide 26 WMO BUFR training course Langen 17-20 April 2007 Slide 26 PBIO routines kerr - 0, successful -1, file does not exist -2, invalid file name -3, invalid open mode specified

27. Slide 27 WMO BUFR training course Langen 17-20 April 2007 Slide 27 PBIO routines To read Bufr product subroutine PBBUFR (kunit, karray, kinlen, koutlen, kret) Input parameters are integers: kunit - unit number obtained from pbopen karray - array big enough to hold the product kinlen - size in bytes of the array Output parameters are integers: koutlen - actual size in bytes of the product read into karray kret - status return code

28. Slide 28 WMO BUFR training course Langen 17-20 April 2007 Slide 28 PBIO routines kret from pbbufr subroutine: 0Product has been successfully read -1End-of-file is hit before a product is read. (Note that EOF does not cause a program fail. This value must be explicitly caught by the caller to avoid looping at EOF) -2There has been an error in the file-handling (e.g. the file contains a truncated product) -3The size of karray is not sufficient for the product

29. Slide 29 WMO BUFR training course Langen 17-20 April 2007 Slide 29 PBIO routines subroutine PBCLOSE (kunit, kret) Input argument is an integer: kunit - unit number from PBOPEN Output argument is an integer: kret – status return code 0 - OK -1 - error in handling the file

30. Slide 30 WMO BUFR training course Langen 17-20 April 2007 Slide 30 PBIO routines File handling FORTRAN subroutines subroutine PBSEEK (kunit, koffset, kstart, kret) Input arguments are integers: kunit - unit number from PBOPEN kstart - 0, if koffset is an absolute count from the beginning of the file 1, if koffset is a relative offset from the current byte position in the file 2, if koffset is an absolute offset from the end of file

31. Slide 31 WMO BUFR training course Langen 17-20 April 2007 Slide 31 PBIO routines Output argument is an integer: kret - status return code >= 0 Byte offset from the start of file after positioning -1 End-of-file is hit before the product is read -2 Error in handling the file

32. Slide 32 WMO BUFR training course Langen 17-20 April 2007 Slide 32 PBIO routines subroutine PBREAD (kunit, karray, kount, kret) Input arguments are integers: kunit - unit number from PBOPEN kount - number of bytes to read from the file Output arguments: karray - an integer array to accept the bytes from the read kret – status return code >=0 number of bytes read -1 EOF reached before the product is read -2 Error in file handling

33. Slide 33 WMO BUFR training course Langen 17-20 April 2007 Slide 33 PBIO routines subroutine PBWRITE( kunit, karray, kount, kerr) Input arguments are integers: kunit – unit number from PBOPEN karray- an integer array holding bytes for write kount – number of bytes to write to the file Output arguments: kret – status return code >= 0 number of bytes written to the file -1 Error in writing to the file

34. Slide 34 WMO BUFR training course Langen 17-20 April 2007 Slide 34 Bufr software FORTRAN routines Bufr expansion subroutine BUFREX( kbufl, kbuff, ksup, ksec1, ksec2, ksec3, ksec4, kelem, cnames, cunits, kvals, values, cvals, kerr) Input arguments: kbufl - length of bufr message in words kbuff - an integer array containing bufr message kelem - expected number of expanded elements kvals - the size of values array in words

35. Slide 35 WMO BUFR training course Langen 17-20 April 2007 Slide 35 Bufr software FORTRAN routines Output arguments: ksup - an array containing supplementary information ksec0 - an array containing section 0 information ksec1 - an array containing section 1 information ksec2 - an array containing section 2 information ksec3 - an array containing section 3 information ksec4 - an array containing section 4 information cnames - character array containing element names cunits - character array containing element units values - real*8 array containing element values kerr - status return code

36. Slide 36 WMO BUFR training course Langen 17-20 April 2007 Slide 36 Bufr software FORTRAN routines ksec0: an integer array ( size of 3 ) Array index Content 1Length of section 0 in bytes 2Total length of bufr message in bytes 3Bufr edition number ( currently 4 )

37. Slide 37 WMO BUFR training course Langen 17-20 April 2007 Slide 37 Bufr software FORTRAN routines ksec1: an integer array ( size of 40 ) Array index Content 1Length of section 1 in bytes 2Bufr edition number ( currently 4 ) 3Origination centre 4Update sequence number 5Flag ( presence of section 2 in the message ) 6Bufr message type ( Bufr table A )

38. Slide 38 WMO BUFR training course Langen 17-20 April 2007 Slide 38 Bufr software FORTRAN routines ksec1 continued Array index Content 7Bufr message sub-type ( local use) 8Version number of local tables used 9Year 10Month 11Day 12Hour 13Minute

39. Slide 39 WMO BUFR training course Langen 17-20 April 2007 Slide 39 Bufr software FORTRAN routines ksec1 continued Array index Content 14Bufr master tables used 15Version number of master table used 16Originating sub-centre 17International sub-category 18Second 19-Local ADP centre information (byte by byte)

40. Slide 40 WMO BUFR training course Langen 17-20 April 2007 Slide 40 Bufr software FORTRAN routines Array index Content 1Length of section 2 in bytes 2-RDB key in packed form ksec2

41. Slide 41 WMO BUFR training course Langen 17-20 April 2007 Slide 41 Bufr software FORTRAN routines ksec3 Array index Content 1Length of section 3 in bytes 2Reserved 3Number of subsets 4Flag ( data type, compression)

42. Slide 42 WMO BUFR training course Langen 17-20 April 2007 Slide 42 Bufr software FORTRAN routines Array index Content 1Length of section 4 in bytes 2-Reserved ksec4

43. Slide 43 WMO BUFR training course Langen 17-20 April 2007 Slide 43 Bufr software FORTRAN routines ksup – an integer array (size 9) containing supplementary information Array indexContent 1Dimension ksec1 array 2Dimension ksec2 array 3Dimension ksec3 array 4Dimension ksec4 array 5Total number of expanded elements 6Number of subsets 7Total number of elements in cvals array 8Total bufr message length in bytes 9Dimension of ksec0 array

44. Slide 44 WMO BUFR training course Langen 17-20 April 2007 Slide 44 Bufr software FORTRAN routines Values array is of type real*8 If bufr table B element is CCITTIA5, corresponding element contain a real number, when truncated to an integer represents index*1000+length where index – subscript of the element in CVALS array where character strings are stored length – number of characters in the string

45. Slide 45 WMO BUFR training course Langen 17-20 April 2007 Slide 45 Bufr software FORTRAN routines In the case of multi-subset data, one dimensional array VALUES contains all subsets. Index to values array of i-th element of observation is: index=i+(nsub-1)*kelem

46. Slide 46 WMO BUFR training course Langen 17-20 April 2007 Slide 46 BUFR SOFTWARE fortran ROUTINES Bufr encoding subroutine BUFREN (ksec0, ksec1, ksec2, ksec3, ksec4, ktdlen, ktdlst, kdlen, kdata, kelem, kvals, values, cvals, kbufl, kerr) Input arguments : ksec0 - an integer array containing section 0 information ksec1 - an integer array containing section 1 information ksec2 - an integer array of 4096 words containing section 2 information ksec3 - an integer array containing section 3 header information

47. Slide 47 WMO BUFR training course Langen 17-20 April 2007 Slide 47 Bufr software FORTRAN routines ksec4 - an integer array containing section4 header information ktdlen - an integer, number of data descriptors to be packed in section 3 of bufr message ktdlst - an integer array containing ktdlen descriptors kdlen - an integer, dimension of kdata array kdata - an integer array containing delayed replication values and possibly new reference value if required kelem - an integer containing expected number of expanded elements kvals - an integer containing expected number of data values

48. Slide 48 WMO BUFR training course Langen 17-20 April 2007 Slide 48 Bufr software FORTRAN routines values - real*8 array of kvals words containing element values cvals - character*80 array of kvals words containing CCITTIA5 elements Output arguments: kbufl - an integer containing the length of bufr message in words kbuff - an integer array containing packed bufr message kerr - integer, status return code

49. Slide 49 WMO BUFR training course Langen 17-20 April 2007 Slide 49 Bufr software FORTRAN routines Printing routines Subroutine BUPRS0 (ksec0) Subroutine BUPRS1 (ksec1) Section 2 is for local use – no general printing routines Subroutine buukey (ksec1,ksec2,key,ksup,kerr) Subroutine BUPRS2 (ksup, key)

50. Slide 50 WMO BUFR training course Langen 17-20 April 2007 Slide 50 Bufr software FORTRAN routines To print section3 Subroutine BUSEL (ktdlen, ktdlst, ktdexl, ktdexp, kerr) Or Subroutine BUSEl2(ksubset, kelem, ktdlen, ktdlst, ktdexl, ktdexp, cnames, cunits, kerr) Subroutine BUPRS3 (ksec3, ktdlen, ktdlst, ktdexl, ktdexp, kelem, cnames)

51. Slide 51 WMO BUFR training course Langen 17-20 April 2007 Slide 51 Bufr software FORTRAN routines To print data Subroutine BUPRT (k, ksub1,ksub2,kelem,cnames,cunits, cvals, kvals, values, ksup, ksec1, kerr) Useful subroutines Subroutine BUS012 (kbufl, kbuff, ksup, ksec0, ksec1, ksec2, kerr) Subroutine BUS0123 (kbufl, kbuff, ksup, ksec0, ksec1, ksec2, ksec3, kerr)

52. Slide 52 WMO BUFR training course Langen 17-20 April 2007 Slide 52 Bufr software FORTRAN routines Bufr template design Basic idea is to know one to one correspondence between data descriptors expanded and the data values To achieve that use: subroutine BUXDES ( k, ksec1, ktdlen, ktdlst, kdlen, kdata, kelem, ktdexl, ktdexp, cnames, cunits, kerr) Input arguments: k - an integer, 0 no print; 1 – print

53. Slide 53 WMO BUFR training course Langen 17-20 April 2007 Slide 53 Bufr software FORTRAN routines ksec1 - an integer array containing section 1 information. Set the following words ksec1(2) – Bufr edition number ksec1(3) – Originating centre ksec1(8) – Version number of local tables used ksec1(15) – version number of master table used ktdlen – number of data descriptors ktdlst- list of data descriptors to be packed in section 3 kdlen – dimension of kdata array kdata - an integer array containing delayed replications kelem – expected number of expanded elements

54. Slide 54 WMO BUFR training course Langen 17-20 April 2007 Slide 54 |Bufr software FORTRAN routines Output arguments: ktdexl – number of expanded elements ktdexp – integer array containing list of expanded elements cnames – character *64 array containing element names cunits – character*24 array containing element units kerr – status return code

55. Slide 55 WMO BUFR training course Langen 17-20 April 2007 Slide 55 Bufr software FORTRAN routines Program to use for template design …/bufr_000320/examples/tdexp.f or Bufrviewer application

56. Slide 56 WMO BUFR training course Langen 17-20 April 2007 Slide 56 Bufr software FORTRAN routines Resolving bit maps - Bit maps are very useful bufr features which associate quality control, first order statistics, departures, retained/replaced and substituted values with original values - One application might have many bitmaps referring backward to the data

57. Slide 57 WMO BUFR training course Langen 17-20 April 2007 Slide 57 subroutine BUBOX (ksub, ksup, kelem, kwtr, cnames, cunits, kvals, values, kbox, kapp, klen, kboxr, vals, cboxn, cboxu, kerr) ksub – subset number ksup – array containing supplementary information kelem – expected number of elements, it must be the same as used in call to bufrex kwtr – an integer array containing list of bufr table B reference numbers ( ktdexp, output from busel routine)

58. Slide 58 WMO BUFR training course Langen 17-20 April 2007 Slide 58 Bufr software FORTRAN routines cnames – array containing elements names cunits - array containing element units kvals – expected number of data values values – array containing expanded values Output arguments: kbox – number of elements in the first column of box kapp - number of applications klen - max index for number of rows. The next column starts at KLEN +1 element or index=i +(KAPP-1)*klen

59. Slide 59 WMO BUFR training course Langen 17-20 April 2007 Slide 59 Bufr software FORTRAN routines kboxr – array containing bufr table B reference numbers vals – a real*8 array containing boxed values cboxn – character*64 array, boxed element names cboxu – character*24 array, boxed element units kerr – status return code

60. Slide 60 WMO BUFR training course Langen 17-20 April 2007 Slide 60 Bufr software FORTRAN routines subroutine buprtbox(kbox, kapp, klen, kboxr, vals, cboxn, cboxu) Input arguments: kbox - an integer containing number of elements in the first column of the box kapp - an integer containing number of applications klen - an integer containing max index for number of rows. The next column starts at klen+1 element or index =i+(kapp-1)*klen

61. Slide 61 WMO BUFR training course Langen 17-20 April 2007 Slide 61 Bufr software FORTRAN routines kboxr- an integer array containing bufr table B reference numbers vals- a real*8 array containing boxed values cboxn- a character*64 array containing boxed names cboxu- a character*24 array containing boxed units

62. Slide 62 WMO BUFR training course Langen 17-20 April 2007 Slide 62 Bufr software FORTRAN routines Example output 1 OPERATOR ************** 222000.0 2 GENERATING CENTRE( CODE TABLE 00 ************** 98.0 3 GENERATING APPLICATION (CODE TAB ************** 1.0 4 STATISTICS (008024/008023) ************** ******** 5 INCREMENTAL UPDATE NUMBER ************** ******** 6 MINIMISATION SIMULATION NUMBER ************** ******** 7 WMO BLOCK NUMBER 10.0 70.0 8 WMO STATION NUMBER 320.0 70.0

63. Slide 63 WMO BUFR training course Langen 17-20 April 2007 Slide 63 Some Bufr operators Quality control information in Bufr - class 33 element - associated field - Operator 222000 + bit map - 033220 variation analysis report event, valid for whole report( observation)

64. Slide 64 WMO BUFR training course Langen 17-20 April 2007 Slide 64 Some Bufr operators Associated field 204001 031021 011001 204000 After expansion 031021 associate field significance 999999 associated field 011001 wind direction

65. Slide 65 WMO BUFR training course Langen 17-20 April 2007 Slide 65 Some Bufr operators Quality control information using 222000 operator 222000 101005 031031 001031 001032 101005 033007

66. Slide 66 WMO BUFR training course Langen 17-20 April 2007 Slide 66 Some Bufr operators First order statistics 224000 237000 Use previously define bit map 001031 001032 008023 First order statistics 101005 224255 First order statistics marker

67. Slide 67 WMO BUFR training course Langen 17-20 April 2007 Slide 67 Some Bufr operators Difference statistical values 225000 237000 Use previously define bit map 001031 001032 008024 Difference statistics 101005 225255 Difference statistics marker

68. Slide 68 WMO BUFR training course Langen 17-20 April 2007 Slide 68 Bufr software and pre-processing What happens when bufr data reach DPC How to organize observation Pre-process observations - unpacking - reorganization of data - possible quality control - grouping of observations which belong to particular analysis cycle - creating RDB key for further more efficient processing - packing the data into bufr - writing data into RDB

69. Slide 69 WMO BUFR training course Langen 17-20 April 2007 Slide 69 Bufr software and pre-processing Alternatively - unpack bufr message - write unpacked values into DB - make decisions about archiving