Presentation on theme: "WMO STRATEGY FOR MIGRATION TO TABLE DRIVEN CODE FORMS"— Presentation transcript:
1 WMO STRATEGY FOR MIGRATION TO TABLE DRIVEN CODE FORMS 18 April 2007(Joël Martellet,WMO, World Weather Watch, Data Processing and Forecasting Systems)
2 OUTLINE THE SITUATION IN 2007 A MIGRATION WITH GOALS AND CONSTRAINTS IMPACTS ON WORLD WEATHER WATCH DATA FLOW AND ASSOCIATED PROGRAMMESMIGRATION PLANIMPACT ON MEMBER STATESMIGRATION SCHEDULESOFTWARE AVAILABLESTATUSISSUES
3 Observations data flow in 2007 For the operation of the World Weather Watch, traditional meteorological and marine in-situ observations are still exchanged between Countries in Traditional Alphanumeric Codes (TAC). Although their total volume is only several tenth of megabytes (in comparison to about 1.5 GB for BUFR satellite data) per day, it is still (for many years) a required input for all meteorological applications - the number of observations for some types exchanged in one day (1 March 2007) is indicated:FM 12 SYNOP (65077), FM 13 SHIP (7825), FM 18 BUOY (22016) FM 32 PILOT (855), FM 35 TEMP (1260), AIREP (23207)(from ECMWF Global Data Monitoring Report)
4 DATA in BUFR or CREX already on GTS (e.g. 26/09/2005) Data typeNumber of bulletinsVolume in bytesAMDAR4371ATOVS retransmitted by EUMETSAT7411ATOVS dataAVHRR radiances12 395Buoy data3 505Wind profiler data6 557TRMM radiances97 101ACAR aircraft data429Geostationary radiances and winds23 427Tropical storm data144 594ERS2 wind and wave982Ice concentration data2411 012HD winds Terra satellite764CREX ozone data1181 040MSG radiances, winds10 581EUMETSAT HD winds348Goes HD winds2033EUMETSAT radiances1 350Jason radiances68 322GPS surface pressure309TAMDAR aircraft data1724Total
5 ESTIMATE OF DATA VOLUME DAILY ON THE GTS OF TRADITIONAL OBSERVATIONS IF ALL MIGRATED TO BUFR: ABOUT 60 À 65 MBYTESData typeSingle observation sizeNumber of obs/dayVolume in bytesSYNOP33265077PILOT400855TEMP1500-1260CLIMAT SYNOP?CLIMAT TEMP600370AWSSAREP146SATEM19030 047SATOBMETAR17842000TAFAMDAR104BUOYUp to 39822016SHIPUp to 3327825Up to
6 WMO decisions on Table Driven Code Forms WMO Executive Council approved as operational codes: BUFR in 1988, CREX in 2000, GRIB Edition 2 in 2001The Commission for Basic Systems (CBS XII - December 2000) agreed to generate a PLAN for the migration from Traditional Alphanumeric Codes to Table Driven Codes and established an Expert Team on Migration to Table Driven Codes.The Team developed a PLAN.CBS Ext. (2002) endorsed the PLAN.The XIV WMO Congress approved the PLAN.CBS XIII (March 2005) established the Coordination Team on Migration to Table Driven Code Forms (MTDCF).
7 MIGRATION GOALULTIMATELY ALL OBSERVATIONS TO BE EXCHANGED IN BUFR (WHICH OFFERS MORE FEATURES THAN CREX, E.G. QUALITY FLAGS, ASSOCIATED VALUES, PACKING).FOR SOME COUNTRIES, HOWEVER, CREX CAN BE AN INTERIM SOLUTION.
8 CBS defined constraints for the MIGRATION to Table Driven Codes BUFR/CREX CBS REQUESTED:IT SHOULD BE A SMOOTH TRANSITION WITHOUT NEGATIVE IMPACTS ON THE WORLD WEATHER WATCH OPERATIONSNOT ALL AT ONCE, BUT PROGRESSIVE AND VOLUNTARILY, BY COUNTRY OR PRODUCER, BY DATA TYPE AND EVEN BY STATION OR PLATFORM.THE PLAN FOR THE MIGRATION TAKES INTO ACCOUNT THESE PRINCIPLES.
9 THE BASIC PRINCIPLES OF THE PLAN ARE: THE PLAN SHOULD ALLOW FOR EVERY WMO MEMBER TO MIGRATE.ALL WMO MEMBER STATES SHOULD HAVE THE FREEDOM TO “SWITCH” INTERNATIONAL TRANSMISSION TO BUFR (OR CREX) WHEN THEY WANT AND WHEN THEY ARE READY TO DO SO.THE DATA PRODUCER, NOT THE USER, IS THE INITIATOR OF THE MIGRATION PROCESS.THE MIGRATION WOULD BE A LONG-TERM PROCESS WITH CONSIDERABLE FLEXIBILITY.INCENTIVES FOR MEMBER STATES TO MIGRATE TO TABLE DRIVEN CODE FORMS SHOULD EXIST. THE BENEFITS OF THE MIGRATION HAVE TO BE CLEARLY EXPLAINED TO MEMBERS.MANUFACTURERS OF OBSERVING SYSTEMS AS WELL AS DEVELOPERS OF PROCESSING SOFTWARE (INCLUDING WORKSTATIONS) SHOULD BE MADE WELL AWARE OF THE PURPOSE OF THE MIGRATION AND OF ITS BENEFITS.
11 EACH WMO MEMBER STATE should find: What is the level of automation? Teletype Paper Manual plottingForecasterObserverencoding on Teletype
12 Automation for data processing: Automatic decoding, plotting,display programForecasterObserverencoding on Teletype
13 Automation for data processing and some at observing level Automatic decoding, displayingprogramForecasterEncodingprogramObserverTyping through anencoding program
14 Automatic decoding, displaying Total automation:Automatic decoding, displayingprogramForecasterAutomaticWeather Station (AWS)automatic encoding
15 Different functions within the data flow: Not to confuse the functions of:Data acquisitionData collectionData transmissionData receptionData visualizationThe format used to represent data could be different at each stage, if it is more efficient.
16 The format used to represent data could be different at each stage, if it is more efficient. Not to confuse the functions of:Data acquisition: AWS sensors value format (or observer reporting data)Data collection: Encoding in a data representation form the observer report (or the AWS sensors values) for national collectionData transmission: Keeping format or Encoding with a view to perform international transmissionInternational transmission process: If possible do not change the data representation, just move a mail “envelope”Data reception: Decode the data representation form to feed data base or other processing applicationsData visualization: Visualize the data values for a human reader = convert to the most appropriate display format for the user
17 WMO OBSERVATION DATA PRODUCERS (Global Observation System) 188 NMCs PRODUCE TRADITIONAL OBSERVATIONS:SYNOP, TEMP, PILOT, CLIMAT (also METAR; SPECI; TAF)A CERTAIN NUMBER OF SPECIALISED COLLECTING CENTRES PRODUCE:SATELLITE DATA (THE MAJORITY ALREADY IN BUFR)AIRCRAFT DATA (AMDAR, ACARS (MANY ALREADY IN BUFR), AIREP)SHIP DATABUOY DATAXBT/CTDSUB-SURFACE PROFILING FLOATS DATAPRODUCERS WILL HAVE THE FREEDOM TO SWITCH TO BUFR WHEN THEY NEED (INTEREST OF THEMSELVES OR SOME OF THEIR USERS FOR NEW PARAMETERS, DATA TYPES, METADATA). HOWEVER, ONE HAS TO ENSURE THEIR OTHER USERS WILL STILL HAVE ACCESS TO THE DATA (ONE HAS TO ANALYSE DATA FLOW AND USER REQUIREMENTS)
18 The Global Observing System (GOS) and the concept of temporary double-dissemination (or dual transmission) for the migration : BUFR and CREX, or BUFR and TAC
23 in Traditional Alphanumeric Codes (TAC) in CREX: in BUFR: Impact of the migration for a Regional Telecommunication Hub (RTH) or DCPC or GISC which receives data (routine operational transmission: store and forward or push system):in Traditional Alphanumeric Codes (TAC)retransmits in TACin CREX:retransmits in CREXin BUFR:retransmits in BUFRConversion BUFR to TAC or BUFR to CREX (only if possible and easy and if associated NMCs really need it, that is: only if there is no double dissemination)Message Switching Directories will have to be updated (Same MSS for BUFR or CREX bulletins as for TAC. Hence, new bulletin header system has been developed by WMO telecommunications experts (eg. TEMP, PILOT BUFR: 2 messages (1st) up to 100 hPa, (2nd) all measurements):
24 Real time operational DATA USERS The Global Data Processing System :16 Centres running operationally global models80 Centres running operationally regional or mesoscale models188 (all NMCs) Centres performing meteorological applications (about 30 are not automated!)Who has a BUFR decoder? A universal one?Who has a CREX decoder? A universal one?:First priority of the migration:universal BUFR and CREX decoder for all!
26 GDPS CENTRES RUNNING MODELS GLOBAL DATA PROCESSING AND FORECASTING SYSTEMS
27 The migration requires preparatory corrective actions at GDPS Centres to avoid missing data: Some Centres will have to ensure that their BUFR and CREX decoders in their processing chain is fully universal, to avoid missing observations.Other Centres will have to implement from scratch a universal BUFR and CREX decoders, to be able to receive observations in BUFR or CREX.Manufacturers of "turn key" work-stations inputting GTS data would need to be approached so that they include in their software universal BUFR and CREX decoders, either in the existing systems, or for the new systems currently in development or to be developed.Centres, which are currently operating manually (about 30 over 188), will have to seriously consider automation with software including a universal BUFR and CREX decoder. Before automation is implemented, or before they have a “binary” connection, they could receive and understand CREX messages with training, and also encode and transmit observation reports in CREX.For some Centres the Internet could allow access to data in binary formats that are not available over their GTS link.
28 ACTIONS ALREADY UNDERTAKEN (1) CBS (Ext. 2006) HAS APPROVED TEMPLATES AND REGULATIONS FOR TRANSMITTING ALL TRADITIONAL OBSERVATIONS IN BUFR (Work of Dr Eva Cervena)ESSENTIAL TO HAVE DESCRIPTORS FOR ALL PARAMETERS THAT CAN BE ENCODED IN A TRADITIONAL CODE FORM INCLUDING REGIONAL AND NATIONAL PRACTICESAND TEMPLATES = LAYOUT OF THE FORMAT, LIST OF DESCRIPTORS TO BE TRANSMITTED (COMMON SEQUENCES)TEMPLATES DEFINED FOR: SYNOP, SHIP, SYNOP MOBIL, PILOT, PILOT SHIP, PILOT MOBIL, TEMP, TEMP SHIP, TEMP MOBIL, TEMPDROP, CLIMAT, CLIMAT SHIP, CLIMAT TEMP AND CLIMAT TEMPSHIPCBS has approved amendments for GTS exchanges to cope with BUFR and CREX bulletins, in particular Rules for transmitting NIL BUFR/CREX bulletins or reports for monitoring purpose>>>>>>>>>>>>>>>>>>>>>>>>>>>>
29 BUFR/CREX TEMPLATES have been defined and some are used for: BUOY, Sub-surface float, TESAC and BATHY dataTRACKOB data (approved by CBS)Single level Standard AMDAR (approved by CBS)More complex Single level AMDAR data (approved by CBS)Ascent/descent AMDAR profile data (approved by CBS)Old AIREPSAREP data (Part A: Information on tropical cyclone) (approved by CBS)RADOB data (Part A: Information on tropical cyclone) (approved by CBS)AUTOMATIC WEATHER STATION DATA OF:- N-MINUTE PERIOD- ONE-HOUR PERIODFOR TROPICAL CYCLONE TRACKS DERIVED FROM EPSEPS GRAMSBuoy data including directional and non-directional wave data (not validated)VOS SHIP (not validated)METAR/SPECI (not validated) and TAF (not validated)
30 CREX TEMPLATES have been defined and some are used for: Bloemhof Flood Monitoring Code for HydrologyTide Gauge dataTotal Ozone MeasurementOzone Sounding Coupled to a Brewer SpectrophotometerOzone Sounding not Coupled to a Brewer SpectrophotometerTropical Cyclone informationMonitoring InformationCREX sequences for coding of Squall Lines in West Africa (approved by CBS)CREX Template for synoptic reports from fixed land stations suitable for manual encoding (in validation)New CREX templates for Tide Elevation (in validation)
31 ACTIONS ALREADY UNDERTAKEN (2) THE WMO SECRETARIAT HAS DESIGNED A WEB PAGE WITH MUCH INFORMATION ON CODES AND MIGRATION TO TDCF. IT INCLUDES, AMONG OTHER INFORMATION, THE INTERNATIONAL MIGRATION PLAN AND A NEW GUIDE ON BUFR/CREX WITH 3 LAYERS :For training 3 levels have been defined:LAYER 1: FOR GENERAL PHILOSOPHYLAYER 2: FOR APPLICATION INTERFACING USERS, DATA AND TELECOMMUNICATION MANAGERSLAYER 3: FOR ENCODER/DECODER PROGRAMMERSWMO ORGANISED TRAINING SEMINARS (Level 1 and 2) : 2003 in RA1 (English), in RA III/IV, 2004 in RA1 (French), in East RA II/West RA V, 2005 in West RA II/East RA VI, 2007 in West RA VI (also a bit Level 3) (about 130 countries have been covered so far)
32 ACTIONS EXPECTED FROM ALL WMO MEMBER STATES DEFINE MIGRATION CONTACT POINTS (national focal point on Codes matters) (105 over 188 so far) does your country has one?NOMINATE A NATIONAL STEERING GROUP ON MIGRATION TO TDCF (TO INCLUDE THE NATIONAL FOCAL POINT)DEFINE A NATIONAL PLAN FOR MIGRATION TO TDCF FOLLOWING THE INTERNATIONAL PLAN - FOUND IN WMO WEB SITE:IDENTIFY IMPACTS OF MIGRATION ON NATIONAL OPERATIONIDENTIFY SOME NECESSARY (BUT USUALLY LIMITED) FINANCIAL AND STAFF RESOURCESDEFINE and RUN NATIONAL TRAINING PROGRAMME ON TDCFDEFINE AND PERFORM SOFTWARE ADJUSTMENTS:1)process BUFR/CREX bulletins in MSS2)introduce BUFR/CREX processing with decoder in PROCESSING CHAIN:RECEPTION, DECODING, DISPLAY, ARCHIVING3)introduce BUFR/CREX encoder for the concerned data type at: OBSERVATION or/and CONCENTRATION site (test and check format)4) transmit BUFR/CREX Bulletins (with proper headers after notification to WMO Secretariat)
33 MIGRATION: WHAT ARE THE KEYS TO SUCCESS? (1) MIGRATION TO TDCF WILL GO WITH AUTOMATIONCANNOT BE UNDERTAKEN WITHOUTCAREFUL PLANNINGTRAININGSOME RESOURCES:COMPETENT STAFFFINANCIALINTERNATIONAL COOPERATION (GROUPING FOR PROJECTS)INTERNATIONAL ASSISTANCE
34 MIGRATION: WHAT ARE THE KEYS TO SUCCESS? (2) TRAINING ON BUFR/CREX - WHAT IS IT?:LEVEL 1: FOR GENERAL PHILOSOPHYLEVEL 2: FOR APPLICATION INTERFACING USERS, DATA AND TELECOMMUNICATION MANAGERSLEVEL3: FOR ENCODER/DECODER PROGRAMMERSWMO SEMINARS:SEMINAR FOR WMO REGIONSTRAIN ALSO THE TRAINERS (WMO RMTCs, METEOROLOGICAL SCHOOLS, COLLEGES, UNIVERSITIES)TRAINING WITHIN COUNTRIES: NATIONAL TRAINING PROGRAMMESSOFTWARE PROJECT:SOFTWARE HOUSE: A CENTRE DISTRIBUTING FREE BUFR/CREX (GRIB 2) DECODER and ENCODER SOFTWARE AND PROVIDING REMOTE ASSISTANCE FOR IMPLEMENTATION, DOCUMENTATION, ANSWERING QUERIESECMWF: encoder/decoder BUFR, CREX and GRIB1/2, transcoder CREX-BUFR (FORTRAN, C languages and UNIX, LINUX operating systems, on Windows if emulator UNIX CYGWIN)NOAA/NCEP: software under UNIXUNIX For WINDOWS: Brazil (INPE software), German software (BUFR), Slovak software (private company),
36 ACTIONS BY ASSOCIATED PROGRAMMES THERE ARE OTHER PRODUCERS AND USERSSATELLITE PRODUCERS PRODUCE ALREADY IN BUFROTHER PRODUCING CENTRES WILL START TO SWITCH TO BUFRMARINE (JCOMM), NEED FOR BUFR ENCODERS, DOUBLE DISSEMINATION FOR USERS:Service ARGOS has started dissemination in BUFR for buoys, XBT/CTD and sub-surface profiling floatsICAO: ACARS, AMDAR TRANSMITTED IN BUFR, SIG WEATHER DATA TRANSMITTED IN BUFR, SIGMET IN BUFR BUT FAR AWAY FOR METAR/SPECI/TAF !USERS WILL HAVE TO BE PROVIDED WITH BUFR/CREX DECODERS
37 Schedule ORGANIZED BY DATA CATEGORIES WHICH HAVE BEEN DEFINED. DIFFERENT STEPS:START OF EXPERIMENTAL EXCHANGE: data will be made available in BUFR (CREX) but not operationally, i.e. in addition to the current alphanumeric codes, which are still operational.Relevant conditions to be satisfied before experimental exchange may start:Corresponding BUFR/CREX-tables and templates are available;Training of concerned testing parties has been completed;Required software of testing parties (encoding, decoding, viewing) is implemented;START OF OPERATIONAL EXCHANGE: data will be made available in BUFR (CREX) whereby some (but not all) Members rely on them operationally. Still the current alphanumeric codes will be distributed (double dissemination or dual transmission ).Relevant conditions to be satisfied before operational exchange may start:Corresponding BUFR/CREX-tables and templates are fully validated;Training of all concerned parties has been completed;All required software (encoding, decoding, viewing) is operational.MIGRATION COMPLETE: at this stage the BUFR (CREX) exchange becomes the standard WMO practice. Double dissemination or dual transmission is terminated. For archiving purposes and at places where BUFR (CREX) exchange still causes problems, the alphanumeric codes may be used and disseminated on a local basis only.
38 satisfied, to start operational exchange as soon as possible. MIGRATION SCHEDULEAll dates above are meant as "not later than". However, Members and Organizationsare encouraged to start experimental exchange, and, if all relevant conditions aresatisfied, to start operational exchange as soon as possible.MIGRATION COMPLETE: For archiving purposes and at places where BUFR (CREX)exchange still causes problems, the alphanumeric codes may be used on a local basis only.
39 Status of the migration to BUFR/CREX (information on 3/2007) BUFR is used for archives of all data types and operational exchange of satellites data, ASDAR, AMDAR ,wind profilers, tropical cyclone data, ARGOS data: buoy, XBT, XCTD, sub-surface floats, RADAR data and starts to be used for translating Traditional Alphanumeric Codes (TAC).RA I: Some African countries are working seriously on the migration: e.g. Botswana, Ethiopia, Morocco, Mozambique, Senegal, Tanzania. Some are considering to use CREX (more metadata more precision and more flexibility compare to traditional SYNOP code) – Senegal produces squall line data in CREXRA II: Japan Meteorological Agency is disseminating since November 2005 in BUFR all TAC data (SYNOP, SHIP, PILOT; TEMP, CLIMAT, etc..), BUOY, BATHY, TESAC; TRACKOB, tropical cyclone, wind profiler data and RADAR data - Hong-Kong produces AMDAR - China will produce SYNOP in BUFR in 2007.RA III: Brazil and Argentina work seriously on development for implementation.RA IV: USA produces internally upper-air data in BUFR, and sea level data in CREX. Mexico tested encoding of SYNOP, TEMP, CLIMAT and CLIMAT TEMP. Costa Rica works on implementation.RA V: Australia produces tropical cyclone data in BUFR and in CREX will produce soon SYNOP in BUFR for Australia and Papua New Guinea and works for other data types - Fiji produces tropical cyclone data in CREXRA VI: Czech Republic: SYNOP, TEMP and AWS data, Israel: TEMP, SYNOP and CLIMAT data, Jordan: SYNOP data, Germany: AWS data, Netherlands: AWS and TEMP data (Note: “TEMP data in BUFR” means radiosonde data including time and position of the radiosonde at each level) - Météo-France disseminates JASON 1 satellite data in BUFR.
40 ACTIONS PLANNEDDevelopment (2007) of a data verification service to help testing encoding of BUFR messagesDevelopment (2007) of a migration guidance document explaining how to start migrationManufacturers are preparing: for radiosondes and for AWS data in BUFRTraining material (2007) to be put into the WMO web site
41 ISSUESMisunderstanding: migration is not a drastic brutal change (as previously for TAC codes), but slow and smooth with dual dissemination (TAC and BUFR in parallel), TARGET DATES in the plan, Countries should plan migration targeting those datesLack of visibility for the migration, for its necessityInitial purpose: for operational real-time exchange of data between NMHS (not for external users: e.g. OPMET)Because it affects the whole meteorological observation data flow: it requires a lot of attention and commitment by NMHSMisunderstandings- Confusions:between visualization and transport format:Exchange process = several steps:producing (encoding), transport, decoding, DB storage, visualizing, archivingbetween physical representation (or format) and envelop (wrapper) (e.g. XML and NetCDF are wrappers, BUT standards of physical representation within the general standard have to be defined).XML and Net CDF are not alternative to BUFR and GRIB 2: the physical meteorological standards within XML and Net CDF would have to be entirely defined. Many years of work would be needed (it would be again other migration processes!).THESE CONFUSIONS SLOW THE PROCESS FOR MIGRATION TO TDCF!
42 Summary: ACTIONS NEEDED Countries not sufficiently informed, Secretariat needs to:To send more circular letters with information on migrationImproved and make up-to-date WMO web siteCountries should develop their national migration plan and implement it (including national training) – The FIRST ACTION, if NMC is automated is to implement a universal BUFR/CREX decoderHelp for developing countries still needed (Migration goes with automation):Training workshopsAssistance by expert missionsPilot projects to demonstrate implementationDevelopment of encoding/decoding software under Windows would helpAdvanced countries should be models and leaders for the migrationBring more considerations on TDCF migration by WMO constituent body meetings (CBS, RAs, EC, Congress)- WMO BUDGET REQUIRED!