Slide 1 ECMWF, 5 September 2007 Slide 1 The SIMDAT project Baudouin Raoult Head of Data and Services Section ECMWF

Slide 2 ECMWF, 5 September 2007 Slide 2 SIMDAT and the VGISC project SIMDAT is a four years EU funded project SIMDAT was an opportunity to fund the development of the VGISC -DWD invited ECMWF to join and take a lead rôle in the SIMDAT project, in order to develop the VGISC. -For ECMWF it was an opportunity to develop expertise in modern technologies (GRID, Service oriented architecture,…) The partners are DWD, ECMWF, EUMETSAT, Météo France and UK Met Office The software was designed according to all WIS related documents present on the WMO web site, as well as previous VGISC documents -In particular the functional roles of GISCs, DCDPs and NCs, as well as their interactions -The virtualisation requirements of the VGISC

Slide 3 ECMWF, 5 September 2007 Slide 3 The VGISC architecture

Slide 4 ECMWF, 5 September 2007 Slide 4 Access to existing data repositories in a non- intrusive fashion

Slide 5 ECMWF, 5 September 2007 Slide 5 Connectivity to another site via secure connection layer

Slide 6 ECMWF, 5 September 2007 Slide 6 A scalable system: addition of new sites

Slide 7 ECMWF, 5 September 2007 Slide 7 Catalogue Synchronisation: each site has a copy of the global catalogue Synchronisation

Slide 8 ECMWF, 5 September 2007 Slide 8 Data from anywhere can be accessed from everywhere Retrievals

Slide 9 ECMWF, 5 September 2007 Slide 9 Finding data Google like search -Fast: people are now used get results in a few milliseconds -Scoring: people are used to find the most relevant hits in the first result page Yahoo like directory -Datasets can be categorized -A dataset can be in several categories at once

Slide 10 ECMWF, 5 September 2007 Slide 10 Portal: Searching…

Slide 11 ECMWF, 5 September 2007 Slide 11 Portal: Search results

Slide 12 ECMWF, 5 September 2007 Slide 12 Portal: Browsing by Category

Slide 13 ECMWF, 5 September 2007 Slide 13 Portal: Legible Metadata

Slide 14 ECMWF, 5 September 2007 Slide 14 Sites are organised in a mesh network Each site is connected to 2-3 peers at most Each site can reach any sites which is part of the network Provides a scalable solution

Slide 15 ECMWF, 5 September 2007 Slide 15 Data policies and Access control We want authentication -Who is this user? We want to support data policies -e.g. WMO Resolution 40 We want access control -According to his/her profile, can this user access data with this data policy? We want a decentralised solution -No central security server -User from anywhere can access data from everywhere

Slide 16 ECMWF, 5 September 2007 Slide 16 Decentralised access control This site holds some data … and defines under which policy this data is available (e.g. for research and education)

Slide 17 ECMWF, 5 September 2007 Slide 17 Decentralised access control This site registers a user … and defines its profile (e.g. a researcher)

Slide 18 ECMWF, 5 September 2007 Slide 18 Decentralised access control The sites must establish a relation of trust -The trust relation is defined by having the same understanding of the term “researcher” -More that one sites can be part of a trust relation -The trust relation is enforced by exchanging X509 certificates A scalable solution: -Each site manages its own users

Slide 19 ECMWF, 5 September 2007 Slide 19 How to publish data in SIMDAT Download the software Install the catalogue node and the portal (optional) -Edit the configuration file -Start the software Install the data repository -Edit the configuration file -Start the software Edit some metadata files to describes your datasets -Drop the files in a directory Write a shell script that translate SIMDAT requests into requests to your existing database

Slide 20 ECMWF, 5 September 2007 Slide 20 From architecture to usage

Slide 21 ECMWF, 5 September 2007 Slide 21 Collaboration With the WMO community -Experience gain in the project are fed back to WMO via expert teams. Evaluation copies of the software -Australia, Canada, China, Finland, Japan, Korea, Morocco, Norway, Russia, Sweden, Zimbabwe -The software is available under an Open Source license Other communities are interested -EUMETNET OPERA project -Global Monitoring for Environment and Security (GMES) -International Polar Year (IPY)

Slide 22 ECMWF, 5 September 2007 Slide 22 Status: 11 sites connected Satellite data ERA40 data TIGGE data Climate Time Series Aviation data (TAF, METAR) Lightning data Model output Real-time GTS data Model output Climate Time Series Model output Observation Model output Satellite data Model output Wave Observation Research datasets Oceanographic data More than 27,000 datasets discoverable

Slide 23 ECMWF, 5 September 2007 Slide 23 Data available BoMNWP outputs, ObservationsGRIB, ASCII CMANWP output, satellite data, TIGGEGRIB, BUFR DWDClimate Time Series, NWP outputsXML, HTML, GRIB ECMWFNWP outputs (ERA15, ERA40, TIGGE), Sample GTS observations (1 day) GRIB1, GRIB2, BUFR, PDF EUMETSATMETEOSAT 8 imagesJPEG JMANWP outputs, MTSAT images, Observations GRIB, BUFR, animated GIFs KMAClimate Time Series, NWP outputsXML, GRIB Météo FranceNWP outputs Real-time GTS dataGRIB, ASCII NCAR~ 6000 datasetsNetCDF, other RNODCOceanographic data (BATHY, SHIP, TESAC) ASCII UKMOAviation data (METAR, TAF)XML

Slide 24 ECMWF, 5 September 2007 Slide 24 Support of WIS requirements Routine collection of observation dataWork in progress. Automated dissemination of all observed data and products, both real ‑ time and non real-time : Push model Work in progress. Ad-hoc requests for data and products : Pull model Done. Support of different user profiles and data policies Done. Support of data and network securityDone. Based on SSL Support the integration of diverse datasetsDone. Support for Any data type, from any data repository. Reliable infrastructureDone. Technologically sustainable and appropriate to local expertise Done. Based on Standards. Run on a PC. Open Source components. Modular, flexible and scalableDone.

Slide 25 ECMWF, 5 September 2007 Slide 25 Conclusion SIMDAT is a fully decentralised system -All sites are true peers and have equal rights -Decentralised user and data policies management -Integrated catalogue and data retrieval functions (one-stop-shop) -Designed for operational use Interfacing with any existing data repositories -Without any impact on the local infrastructure or disruption of operational activities -Support for any data types (GTS bulletins, Model outputs, Satellite images, climate time-series, ….) Generates interest in meteorology and other environmental communities -A synergy has been established Data available from 11 sites worldwide -Software running continuously, catalogues are constantly updated