Presentation on theme: "Slide 1 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 1 TIGGE-LAM: Archiving at ECMWF Manuel Fuentes Data and Services Section ECMWF."— Presentation transcript:
Slide 1 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM: Archiving at ECMWF Manuel Fuentes Data and Services Section ECMWF
Slide 2 TIGGE-LAM Workshop Bologna Jan The TIGGE-Global database Three archive centres: CMA, NCAR and ECMWF Ten data providers: -ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil) Exchanges using UNIDATA LDM, HTTP and FTP Operational since 1st of October TB, growing by ~ 1.5 TB/week -1.5 million fields/day
Slide 3 TIGGE-LAM Workshop Bologna Jan Archive Centre Data Providers NCAR NCEP CMC UKMO ECMWF MeteoFrance JMA KMA CMA BoM CPTEC TIGGE Archive Centres and Data Providers
Slide 4 TIGGE-LAM Workshop Bologna Jan Strong governance Precise definition of: -Which products: list of parameters, levels, steps, units,… -Which format: GRIB2 -Which transport protocol: UNIDATAs LDM -Which naming convention: WMO file name convention Only exception: the grid and resolution -Choice of the data provider. Data provider to provide interpolation to regular lat/lon -Best possible model output Many tools and examples: -Sample dataset available -Various GRIB2 tools, tigge_check validator, … -Scripts that implement exchange protocol Web site with documentation, sample data set, tools, news….
Slide 5 TIGGE-LAM Workshop Bologna Jan Using SMS to handle TIGGE flow
Slide 6 TIGGE-LAM Workshop Bologna Jan Quality assurance: homogeneity Homogeneity is paramount for TIGGE to succeed -The more consistent the archive the easier it will be to develop applications There are three aspects to homogeneity: -Common terminology (parameter names, file names,…) -Common data format (format, units, …) -Definition of an agreed list of products (Parameters, Steps, levels, …) What is not homogeneous: -Resolution -Base time (although most provider have a run at 12 UTC) -Forecast length -Number of ensemble
Slide 7 TIGGE-LAM Workshop Bologna Jan QA: Checking for homogeneity E.g. Cloud-cover: instantaneous or six hourly?
Slide 8 TIGGE-LAM Workshop Bologna Jan QA: Completeness The objective is to have 100% complete datasets at the Archive Centres Completeness may not be achieved for two reasons: -The transfer of the data to the Archive Centre fails -Operational activities at a data provider are interrupted and back filling past runs is impractical Incomplete datasets are often very difficult to use Most of the current tools (e.g. epsgrams) used for ensemble forecasts assume a fixed number of members from day to day -These tools will have to be adapted
Slide 9 TIGGE-LAM Workshop Bologna Jan QA: Checking completeness
Slide 10 TIGGE-LAM Workshop Bologna Jan Access to TIGGE data Data is available to Research and Education after 48 hours -Self registration by agreeing to the terms and conditions Portals at CMA, NCAR and ECMWF ECMWF portal offers: -Access to offline data -Aggregation along any axis (date, level, parameter, origin, ensemble, …) -Provision of multi-model data on a single grid (regridding to any lat/lon grid) -Sub-area selection -Reduces volumes to be downloaded by many order of magnitude -Batch access via the portal
Slide 11 TIGGE-LAM Workshop Bologna Jan TIGGE portal at ECMWF
Slide 12 TIGGE-LAM Workshop Bologna Jan TIGGE portal at ECMWF (using Ajax)
Slide 18 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: Active users
Slide 19 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: Users by country
Slide 20 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: Data volumes
Slide 21 TIGGE-LAM Workshop Bologna Jan TIGGE: Lessons learnt Strong commitment required: -During development: encoding, units -Day to day operations: failed jobs need to be re-run Robust suites and scripts -Problems with transmission or data corruption -Servers may not available (retry) -Keep data for at least 3 days Technical contacts point
Slide 22 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM Archiving at ECMWF Re-use TIGGE-Global technology TIGGE-LAM Archiving meeting in September Un-rotated lat/lon, 0.1 0, virtually on (0,0) -HP parameters, addition of LSP, ORO and LSM -Specific name for each LAM EPS configuration -Data format: GRIB2
Slide 23 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM: Suite names for Europe LAM EPS SystemSuite Name at ECMWFInstitute -Consortium MOGREPSMOGREPS-MO- EUAUK Met Office - UK SREPSAEMet-SREPS-MM-EUATAEMET- Spain HIRLAM SRNWP PEPSSRNWP-PEPS COSMO-LEPSCOSMOLEPS-ARPASIMC-EUARPA-SIM - Italy COSMO NORLAMEPS Met.NO ALADIN LAEFALADIN-LAEFZAMG / Austria COSMO DE EPSCOSMO-DE EPSDWD - Germany SRNWP COSMO-SREPSCOSMO-SREPS-BO-EUARPA-SIM - Italy COSMO GLAMEPS DNMI/Univ Oslo âNorway HIRLAM ALADIN PEARCE Meteo-France / France DMI - HIRLAM DMI OMSZ ALADIN EPSOMSZ- ALADIN-EPS Hungary,Czech Republic, Croatia
Slide 24 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM: Data format Encoding in GRIB2 -Support for all HP parameters Possibly add TIGGE-LAM GRIB2 local extension -Differentiate between different systems -Preserve Data Providers own local extensions For Data Providers that cannot provide GRIB2, ECMWF can help convert from GRIB1 using grib_api Examples from COSMO/LEPS and SREPS Send sample data to:
Slide 25 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM: Data Transmission & Archiving No need for LDM -FTP, push of pull or other Filenames can be agreed between Data Provider and ECMWF -Date & time required to avoid overlapping ECMWF will archive data -Consistency checks performed after archiving
Slide 26 TIGGE-LAM Workshop Bologna Jan Conclusion TIGGE-Global has been running for more than 2 years -Robust infrastructure in place Basis to ensure TIGGE-LAM success Thank you!