Slide 1 WMO BUFR training course Langen April 2007 Slide 1 Ecmwf observations data handling Milan Dragosavac

Slide 2 WMO BUFR training course Langen April 2007 Slide 2 MDBRDBPDBFDB ODB IFS MARS General applications AnalysisForecast Display Print Plot Data transport Data transport Data acquisition

Slide 3 WMO BUFR training course Langen April 2007 Slide 3 Data acquisition Acquire data Process data Archive data ( not MARS) Outgoing ( send data outside Ecmwf)

Slide 4 WMO BUFR training course Langen April 2007 Slide 4 Acquisition process GTS data ftp Other data

Slide 5 WMO BUFR training course Langen April 2007 Slide 5 Acquisition outgoing Qscat satellite data in BUFR form SSMI brightness temperatures TOVS 1B calibrated and converted into BUFR format

Slide 6 WMO BUFR training course Langen April 2007 Slide 6 MEB feed Restores any old data Join segmented bulletins Write bulletins in MEB statistics

Slide 7 WMO BUFR training course Langen April 2007 Slide 7 Message data base “STREAM”_YYYYMMDD AIRC_ AIRS_ AMAP_ GRIB_ streams in total One stream per day

Slide 8 WMO BUFR training course Langen April 2007 Slide 8 Message data base attributes TIMEcharacter( 4,1) OBSTYPEcharacter( 4,1) TTAAiicharacter( 6,1) BBBcharacter( 3,1) CCCCcharacter( 4,1) _SOURCEcharacter(80,1) _CHECKSUMlonginteger _TIMESTAMPcharacter(20,1)

Slide 9 WMO BUFR training course Langen April 2007 Slide 9 Message data base attributes continued _data_sizelonginteger _bulk_sizelonginteger _ownercharacter( 3,1) _last_accesstime(0) _access_numberlonginteger _compresscharacter( 1,1) _databulk(20,0,512,1)

Slide 10 WMO BUFR training course Langen April 2007 Slide 10 Message data base indices UNIQUE I_timestamp_SYNO_ on (_TIMESTAMP) UNIQUE I_comp_SYNO_ on (TIME, TTAAii, BBB, _CHECKSUM) NORMAL (2,62) I_OBSTYPE_SYNO_ on (OBSTYPE)

Slide 11 WMO BUFR training course Langen April 2007 Slide 11 Message data base 100 Gbytes allocated Gbytes for METOP data Over 22 Gbytes / day Saved on ECFS according to OBSTYPE All data can be reprocessed if Data base is lost HP cluster 4 CPU 64 bit processors 4 operational machines + 1 for development Raid disks 1.2 Tbytes

Slide 12 WMO BUFR training course Langen April 2007 Slide 12 MEB data arrival Monitoring arrival times of different data types Analysis cycle Average number of days

Slide 13 WMO BUFR training course Langen April 2007 Slide 13 Message data base data availability Data availability according to synoptic hour

Slide 14 WMO BUFR training course Langen April 2007 Slide 14 MEB x-interface Allows MEB search Find corresponding BUFR observation in REB Expand BUFR and GRIB data Write data into file

Slide 15 WMO BUFR training course Langen April 2007 Slide 15 MEB x-interface REB BUFR observations corresponding to MEB bulletin

Slide 16 WMO BUFR training course Langen April 2007 Slide 16 Preprocessing suite MEB – message data base feed task deqc – decoding, quality control and BUFR creation, writing into REB ex00 – extraction for analysis bc00 – extraction for boundary condition suite wmo – ftp WMO station list

Slide 17 WMO BUFR training course Langen April 2007 Slide 17 Decoding Quality control BUFR creation Global grib fields creation Writing into REB

Slide 18 WMO BUFR training course Langen April 2007 Slide 18 REB extraction for analysis 14 files created Conventional observations Satellite data Sea surface temperatures Ice analysis

Slide 19 WMO BUFR training course Langen April 2007 Slide 19 REB – report data base One table per observation type per day AIRC_ ATOV_ B002_ CREX_ TEMP_ SSBT_ QSCA_ statistic

Slide 20 WMO BUFR training course Langen April 2007 Slide 20 REB attributes TIMEcharacter(6,1) IDENTcharacter(9,1) _BUFR_TYPEinteger _BUFR_SUBTYPEinteger _LATITUDE1float(6) _LONGITUDE1float(6) _COR1integer _QCinteger

Slide 21 WMO BUFR training course Langen April 2007 Slide 21 REB attributes continued _TIMESTAMPcharacter(20,1) NR_SUBSETSinteger _data_sizelonginteger _bulk_sizelonginteger _ownercharacter(3,1) _last_accesstime(0) _access_numberlonginteger _databulk(20,0,512,1)

Slide 22 WMO BUFR training course Langen April 2007 Slide 22 REB indices UNIQUE I_ts_SYNO_ on (_TIMESTAMP) UNIQUE I_comp_SYNO_ on (TIME, IDENT, _BUFR_SUBTYPE, _LATITUDE1, _LONGITUDE1 NORMAL (2,15) I_extract_SYNO_ on (TIME, _BUFR_SUBTYPE, IDENT)

Slide 23 WMO BUFR training course Langen April 2007 Slide 23 REB reports data base Just bellow 22 Gbytes / day 100 Gbytes allocated Gbytes for METOP data All data can be re-decoded at any time Data can be accessed through MARS request and Metview Archived in MARS

Slide 24 WMO BUFR training course Langen April 2007 Slide 24 REB x-interface Search REB Find original bulletin in MEB Expand BUFR and GRIB data Write data to file Access WMO station list data base

Slide 25 WMO BUFR training course Langen April 2007 Slide 25 BUFR x-interface Expands BUFR message Any subset can be viewed Filtering features

Slide 26 WMO BUFR training course Langen April 2007 Slide 26 WMO Data Base WMO_a_master WMO_a_master_manual WMO_a_master_tape

Slide 27 WMO BUFR training course Langen April 2007 Slide 27 WMO_a_master attributes Region_countryinteger not null Blocksmallint not null Station_indexinteger not null Station_subindexsmallint not null Principal _stationcharacter(1,1) Station_nametext(20,20,30,1) Latitudereal Longitudereal

Slide 28 WMO BUFR training course Langen April 2007 Slide 28 WMO_a_master attributes continued Pressure_elevationinteger Pressure_elevation_flagcharacter(1,1) Ground_elevationinteger Ground_elevation_flagcharacter(1,1) Pressure_levelsmallint Surf_prog_00character(2,1) Surf_prog_03character(2,1) Surf_prog_06character(2,1) Surf_prog_09character(2,1) Surf_prog_12character(2,1)

Slide 29 WMO BUFR training course Langen April 2007 Slide 29 WMO_a_master attributes continued Surf_prog_15character(2,1) Surf_prog_18character(2,1) Surf_prog_21character(2,1) Ua_type_00smallint Ua_type_06smallint Ua_type_12smallint Ua_type_18smallint Modify_flagcharacter(1,1) not null

Slide 30 WMO BUFR training course Langen April 2007 Slide 30 WMO_a_master indices UNIQUE I_wmo_a_master ON (block,station_index, station_subindex)

Slide 31 WMO BUFR training course Langen April 2007 Slide 31 WMO station list

Slide 32 WMO BUFR training course Langen April 2007 Slide 32 ODB Observational data base Interface to analysis Data base contains unpacked bufr data and original bufr data Loading done in parallel with load balance achieved on supercomputer due to bufr features Analysis updates ODB Analysis bufr feedback files created Archiving of feedback files in MARS

Slide 33 WMO BUFR training course Langen April 2007 Slide 33 Loading ODB with observations Tasks running in parallel 2-16 processors used per task Load balancing achieved

Slide 34 WMO BUFR training course Langen April 2007 Slide 34 ODB to BUFR odb2bufr family

Slide 35 WMO BUFR training course Langen April 2007 Slide 35 Odbviewer

Slide 36 WMO BUFR training course Langen April 2007 Slide 36 Odbviewer

Slide 37 WMO BUFR training course Langen April 2007 Slide 37 Odbviewer

Slide 38 WMO BUFR training course Langen April 2007 Slide 38 Odbviewer

Slide 39 WMO BUFR training course Langen April 2007 Slide 39 Odbviewer

Slide 40 WMO BUFR training course Langen April 2007 Slide 40 Odbviewer

Slide 41 WMO BUFR training course Langen April 2007 Slide 41 Odbviewer

Slide 42 WMO BUFR training course Langen April 2007 Slide 42 Future Sharp increase in volume of observations expected IASI data with 8641 channels per observation Data bases have crucial role in observations handling Binary data representation allows easy handling of observations as bulk objects