Distributed Marine Data System: Intercommission Coordination Group on WIS IODE/JCOMM Distributed Marine Data System: E2EDM and Ocean Data Portal Pilot Projects Nikolay Mikhailov, IODE/JCOMM ETDMP Chair, RIHMI-WDC Russian Federation Brazil, 14-17 July 2008

The IODE/JCOMM initiatives Data&information exchange is important for World Ocean studies – on regional and global levels Observation systems The IOC IODE system forms a worldwide service-oriented network consisting of NODC’s, DNA’s, WDC’s. The IODE/JCOMM initiatives for Distributed Marine Data System development IODE/JCOMM E2EDM technology prototype Pilot Project (2004-2006) IODE Ocean Data Portal (2007-…) Data and product delivering systems Delay-mode data base’s Analysis and Forecasting systems

IODE/JCOMM Ocean Data Portal/E2EDM Technology Vision “The Oceans Data Portal will deliver a standards based infrastructure that provides integration of marine data and information from a network of distributed IODE NODCs as well as resources from other participating systems.“ Ocean Data Portal supports the data access requirements of all IOC programmes areas, including GOOS, JCOMM, HAB and the Tsunami warning system. Ocean Data Portal will provide the JCOMM WIGOS operations

Data and Services The Ocean Data Portal will provide on-line access to the marine data and information resources of the participating data centres including: operational and delayed-mode data data and services from the oceanographic and marine meteorological domains from multiple data source formats and local data systems (DBMS, data files, map services, electronic documents) from multiple data providers in different geographic regions

ODP/E2E Distributed Marine Data System Principles The ODP will be interoperability with existing systems IODE data centres will need to accept and implement a set of agreed interoperable arrangements including the technical specifications and Web-services Interoperability will be achieved through the use of internationally endorsed standards (such as SOA, ISO and OGC) NODCs will not be required to change their internal (local) data management systems

ODP/E2E Technical Concept IODE Portal Portal of system 1 (e.g. SeaDataNet) Portal of system N (e.g. WIS) Network of distributed IODE data centres Portal Web-services Metadata Data Internet Dedicated data sources network- E2E

E2EDM Architecture Software Metadata Use External layer E2EUserMD E2EInterfaceMD Discovery - E2ESeachMD, Access - E2ELinkMD Integration Server layer Encoding E2EElementsMD, - E2ECodesMD Request/Responce - E2E Request, - E2EResponce, - Transport Data File (Point, Profile, Grid) – NetCDF, ASCII, XML Communication layer Config.file Mapping file Data Provider layer Local data system layer

Model of IODE/JCOMM Distributed Marine Data System E2E Integration Server Monitoring, Administrating, Discovery, Interaction services Users Ocean Data Portal Publishing services (end-user applications) E2E Data Provider DBMS, Data files (structured), Object files (documents,…) Distributed Marine Data System IODE/JCOMM Data Centres – national and specialized Local Data Systems System Coordinator Centre – global, regional, task-oriented Data Access

Model of IODE/JCOMM Distributed Marine Data System Portal Integration Server Integration Server Integration Server

IODE/JCOMM Distributed Data System operation (current status) User access Portal HTTP communication API‘s Registry Metadata management Data discovery Data Delivery Monitoring & Security E2E Integration Server Registry Connection Mapping Data Access HTTP communication Web-services Data providers community E2E Data Provider Request message Response message netCDF transport data-file Local Data system Data Centers Network

IODE/JCOMM Distributed Marine Data System coverage Data Provider (VLIZ) Data Providers JCOMM PAs RIHMI-WDC mirror Integration server Integration server operational planned finished PP IODE Project Office Data Provider (RIHMI-WDC) Data Providers IODE ODINs Data Provider (IFREMER) Data Provider (MetOffice)

E2E Integration Server operation

E2E Data Provider operation

IODE Ocean Data Portal prototype http://e2edm.vliz.be/e2edmclient/

IODE Ocean Data Portal prototype Providing some Google-like search

IODE Ocean Data Portal prototype - Result of data search

IODE Ocean Data Portal prototype - Advanced metadata search

IODE Ocean Data Portal prototype (WOD, North sea) User data view page

IODE Ocean Data Portal prototype User data view page – different type of products

E2EDM Technology Documentation Title Contents Version and updating E2EDM Overview General information about E2EDM technology and implementation of this technology Jan 2007 Updated May 2008 The Checklist of technical requirements for data providers for E2E technology implementation Describes requirements for IODE data centres to build the E2E Data providers network E2EDM technical specification v.1.0 Describes technical specifications (model, metadata structures, data exchange protocol and etc.) the E2EDM technology (TO BE UPDATED!!!) Updated 15 Jan 2007 E2EDM Concept XML Schema XML document includes ocean and marine meteo parameters which circulates under the distributed data system (current data source federation). Updated 16 November 2007 E2EDM Global XML Schema Includes XML structures of metadata classes (packages) used for E2EDM metadata records construction. Updated 14 October 2007 E2EDM User's Manual Document is considered as quick-start guide for the E2EDM technology users (general architecture of the E2EDM technology, installation and use of the E2EDM Data Provider and Integration Server software). The document is intended for data centres to built the distributed data system. E2EDM Integration Server User Guide Includes the detail rules and procedures how to install, configure and run E2EDM Integration Server software. v.1.0., updated 05 May 2007 E2EDM Data Provider User Guide - Installation and setup Describes detail rules, procedures and web-forms for the Data Provider installation, registration and maintenance. The document is intended for system administrators of the data centres and other institutions responsible for providing data to the system. v.1.1.0 September 2007 Updated May 2008 E2EDM Data Provider User Guide - Registration and maintenance of resources. Describes detail rules, procedures and web-forms for registration and maintenance of the local resources of distributed data sources system. The document is intended for operators of the data centres. E2EDM Ocean Data Portal. User Guide Describes detail rules, procedures and web-forms for use Ocean Data Portal basing on the E2EDM technology. The document is intended for end-user for enter and access to resources of the distributed data system. v1.0 Updated December 2007

Links to ODP/E2E E2EDM Web site http://e2edm.meteo.ru – documentation E2EDM Integration Server - IODE Project Office (VLIZ) http://e2edm.vliz.be/iserv/backend - RIHMI-WDC (mirror) http://data.meteo.ru:8080/iserv/backend JCOMM/IODE Data Portal -http://e2edm.vliz.be/e2edmclient JCOMM/IODE Data Portal web-site will be launched soon

Interaction with other systems (WIS case) OAI Client (GISC) OAI Server DCPC Obninsk WMO Schema «Register» WMO metadata records (all with global unique ID) «Request» «Response» E2EDM records are reformatted into WMO standard records are migrated from (A) to (B) « A » B GISC catalogue / registry Resource catalogue E2EDM Integration Server) E2EDM Data Provider VGISC protocol communication

The IODE/JCOMM DDS organization The dedicated data center should support the Integration Server operations The data center which agrees to be the E2E data provider should provide: the appropriate middleware for communications: web server and application server installation of the E2E Data Provider software to connect the local data system with E2E Integration Server registration of the data source and its resources support of the local data system. .

(IODE Officers and DMCG-3 meetings) ODP Future Plan (IODE Officers and DMCG-3 meetings) Establish the pilot project working group – in process Draft of a detailed work plan – Oct.2008 The JCOMM WIGOS basing on E2E Technology with expanding the DPs – 2008-2010 The Design Docs and take into account other similar initiatives to avoid duplication of effort – beginning 2009

Thank you !