1. Slide 1 GO-ESSP Paris. June 2007 Slide 1 (TIGGE and) the EU Funded BRIDGE project Baudouin Raoult Head of Data and Services Section ECMWF

2. Slide 2 GO-ESSP Paris. June 2007 Slide 2 THORPEX was established in May 2003 by the 14th WMO Congress as a ten-year international global atmospheric research and development programme THORPEX is under the auspices of the WMO Commission for Atmospheric Sciences (CAS) THORPEX Goals: -To reduce and mitigate natural disasters -To fully realise the societal and economic benefits of improved weather forecasts

3. Slide 3 GO-ESSP Paris. June 2007 Slide 3 TIGGE THORPEX Interactive Grand Global Ensemble Key objectives of TIGGE: -Enhanced collaboration on development of ensemble prediction, internationally and between operational centres and universities -New methods of combining ensembles from different sources and of correcting for systematic errors -A deeper understanding of the feasibility of interactive ensemble system responding dynamically to changing uncertainty -The development of a prototype future Global Interactive Forecasting System (GIFS)

4. Slide 4 GO-ESSP Paris. June 2007 Slide 4 The TIGGE core dataset Global ensemble forecasts to around 14 days generated routinely at different centres around the world Outputs collected in near real time and stored in a common format for access by the research community Easy access to long series of data is necessary for applications such as bias correction and the optimal combination of ensembles from different sources

5. Slide 5 GO-ESSP Paris. June 2007 Slide 5 Homogeneity of the TIGGE database Homogeneity is paramount for TIGGE to succeed -The more consistent the archive the easier it will be to develop applications There are three aspects to homogeneity: -Common terminology (parameters names, file names,…) -Common data format -Definition of an agreed list of products (Parameters, Steps, levels, …) NCEPECMWF precipitationprecipTP Mean sea level pressureprmslMSL Temperature at 2 meterst2m2T

6. Slide 6 GO-ESSP Paris. June 2007 Slide 6 Completeness The objective is to have 100% complete datasets at the Archive Centres Completeness may not be achieved for two reasons: -The transfer of the data to the Archive Centre fails -Operational activities at a data provider are interrupted and back filling past runs is impractical Incomplete datasets are often very difficult to use Most of the current tools used for ensemble forecasts assume a fixed number of members from day to day -These tools will have to be rewritten

7. Slide 7 GO-ESSP Paris. June 2007 Slide 7 Strong governance Precise definition of: -Which products: list of parameters, levels, steps, units,… -Which format: GRIB2 -Which transport protocol: UNIDATA’s LDM -Which naming convention: WMO file name convention Only exception: the grid and resolution -Choice of the data provider -Best possible model output One must do what one requires from others -Sample dataset available -Various GRIB2 tools, “tigge_check” validator, … -Scripts that implement exchange protocol

8. Slide 8 GO-ESSP Paris. June 2007 Slide 8 Phased implementation of the archive Phase 1: multiple instances, low development effort ECMWFNCARCMA Phase 1 Archive Centres

9. Slide 9 GO-ESSP Paris. June 2007 Slide 9 Building the TIGGE database Three archive centres: CMA, NCAR and ECMWF Ten data providers: -Already sending data routinely: ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA) -Coming soon: BOM (Australia), CPTEC (Brazil), KMA (Korea), MSC (Canada), Météo-France (France) Exchanges using UNIDATA LDM, HTTP and FTP -Firewall issues with China Operational since 1st of October 2006 43 TB, growing by ~ 1 TB/week

10. Slide 10 GO-ESSP Paris. June 2007 Slide 10 Quality Assurance

11. Slide 11 GO-ESSP Paris. June 2007 Slide 11 Accessing TIGGE Portals at NCAR and ECMWF. CMA portal to be developed ECMWF portal offers: -Access to offline data -Aggregation along any axis (date, level, parameter, origin, ensemble, …) -Provision of multi-model data on a single grid (regridding to any lat/lon grid) -Sub-area selection -Reduces volumes to be downloaded by many order of magnitude

12. Slide 12 GO-ESSP Paris. June 2007 Slide 12 Phased implementation of the archive Phase 2: distributed approach, higher development effort

13. Slide 13 GO-ESSP Paris. June 2007 Slide 13 BRIDGE Bridge is a 2 years project funded by the EC under the FP6-IST programme. It will demonstrate the benefits of GRID technology for international cooperation, in particular between Europe and China This work focuses on the development of interoperable Grid infrastructures (CNGrid, GRIA) Three applications: Aircraft Design, Meteorology, Drug Design

14. Slide 14 GO-ESSP Paris. June 2007 Slide 14 Meteo scenario: exploring TIGGE phase 2 Distributed processing on distributed data, across the two GRID middleware Each site hosts only part of the data Each site offers basic operations on the data (e.g. computing an average) Strategy: minimize data transfers -Run operations at data location -Decompose operations in simpler ones -Most of the time intermediate results are much smaller

15. Slide 15 GO-ESSP Paris. June 2007 Slide 15 EPS Products Examples -Ensemble mean -Standard deviation -Clustering -Probability of weather events -Extreme Forecast Index -EPSgram Some products can be decomposed in simpler operations on a subset of members -Ensemble means Some products need all the members -Clustering

16. Slide 16 GO-ESSP Paris. June 2007 Slide 16 Example 1: ensemble mean Data requests describes 10 fields 6 are available from ECMWF 4 are available from CMA S1 = sum(6 ECMWF fields) performed at ECMWF S2 = sum(4 CMA fields) performed at CMA Intermediate results and associated metadata moved to site were user invoked request: (S1,6) and (S2,4) Final result A=(S1 + S2)/(6+4) computed locally and returned to user

17. Slide 17 GO-ESSP Paris. June 2007 Slide 17 Example 2: EPSGram Send lat/lon location at each sites Receive list of values back Compute distributions Generate plot

18. Slide 18 GO-ESSP Paris. June 2007 Slide 18 Non-decomposable operation

19. Slide 19 GO-ESSP Paris. June 2007 Slide 19 Decomposable operation

20. Slide 20 GO-ESSP Paris. June 2007 Slide 20 Operations as Workflows Study a selection of EPS products Classify them into decomposable and non- decomposable For decomposable operations -Implement sub-operations as web services -Deploy several instances of these web services -Describe full operation as a workflow -User will not see the workflow, but the high level operation There may be several ways to execute a workflow -Because the same sub-operations are offered by several sites -Choose the “cheapest” workflow -Cost of a workflow based on amount of data moved

21. Slide 21 GO-ESSP Paris. June 2007 Slide 21 Conclusion TIGGE Phase 1 is progressing well -Strong governance -Very good working relationship between CMA, NCAR and ECMWF BRIDGE project gives us the funding to explore how to implement TIGGE Phase 2