1 NODC, Russia SeaDataNet TTG meeting Paris, May Overview and potential use of E2EDM technology for SeaDataNet Sergey Belov, Nick Mikhailov

2 Scope NODC, Russia E2EDM provides following functionality: Wrapping local data systems Mapping local codes and dictionaries Data discovery, delivery and processing (export and data production from delivered data) Integrated and secured online access to the distributed data E2EDM is integrating operational and delay-mode time scale data (oceanographic and marine meteorological data) from multiple data formats and multiple data providers in different geographic regions.

3 NODC, Russia Basis NODC, Russia Concept model Namespace Metadata/data objects specifications Data exchange protocol Software components

4 NODC, Russia E2EDM concept model NODC, Russia Providing formalization, identification, description and interpretation for main model objects: Data source Resource User (in particular cases – external application) Model is based on the following structured semantic objects: Element Class - set of elements Record - set of classes These semantic objects defines: Granularity and description for original and transferred data Metadata description Codes and dictionaries unification Data and user policy

5 NODC, Russia E2EDM namespace NODC, Russia Defines metadata and data structures in terms of XML scheme. Within E2EDM consists of two sub-namespaces: global and conceptual (thematic). Global namespace – structures for the model objects Conceptual (thematic domain) namespace – list of thematic elements dealing with grouping, e.g. some discipline and available for discovery and exchange Both designed in XML Schema. 1.Global XML Schema (ISO WMO Core Metadata + CDI) 2.Concept XML Schema (DiGIR) XML Schemas are also used for validation and development issues (XML Schema -> UML -> software design)

6 NODC, Russia E2EDM metadata objects specifications NODC, Russia More than 100 elements, 27 classes, 7 metadata records and 4 data records Metadata Information about local data source (linkage) Unique identification of metadata object Information about data set Information about references and responsible party Information about temporal, vertical extent Information about data policy Information about data quality E2EObjectIDE2ELinkage E2EDataQuality E2EDistributionInfoE2EVerticalExtentE2ETemporalExtentE2EObjectContactE2EObjectReferenceE2ERestrictions List of classes, defining Instruments and processing tools, projects, platforms, geographic coverage, data structure and granularity

7 NODC, Russia E2EDM data objects specifications NODC, Russia E2EDM data records, based on NetCDF: E2EDataPoint – for point data E2EDataProfile – for profile data E2EDataGrid – for grids E2EDataObject – object files (http-based and local) Header (DataHeader) ListElement (list of elements) DataContainer «SequenceElement» «StructureElement» Common transport NetCDF construction

8 NODC, Russia E2EDM data objects specifications (II) NODC, Russia class/elementNetCDF componentNetCDF structure E2EObjectHierarchy + objectHierarchyLevel DataHeader + DataContainer Group objectHierarchyIdDataHeaderAttribute objectHierarchyListDataContainerVariable Classes and data constructions relationship: 1 objectSystemId, objectHierarchyId, countryId, organizationId, dateTime, Latitude, Longitude, platfType, platformId, platformName platformId 2 DEPH, TEMP, PSAL dateTime, Latitude, Longitude dateTime

9 NODC, Russia E2EDM data exchange protocol NODC, Russia Data exchange protocol is responsible for data requesting and data retrieving. Request message (XML) Response message (XML) Transport data file (NetCDF,v.2.2)

10 NODC, Russia E2EDM software components 1.E2EDM Data Provider (DP) – set of services, providing interface between local data systems and distributed system. Must be plugged on the local system. Services are using common library (E2EDM API). DP supports data bases, structured and object data files. 2.E2EDM Integration Server – set of services, providing security issues, resource catalogue maintenance and communication with DP (discovery, requesting and retrieving data)

11 NODC, Russia E2EDM Architecture Integration Server layer Communication layer Data Provider layer Local data system layer External layer

12 NODC, Russia Integration Server. Features Metadata update & maintenance (automatic) Resource catalogue management Discovery services for data, metadata and common codes Request/response communication service Data sources monitoring Downloading services (for obtaining data files, metadata, resource and instance descriptions synchronization) Export tools (e.g. export in XML from E2EDM resource description into WMO Core Profile, to ASCII) HTTP GET/POST (REST) and Web service interfaces Security service Schedule mechanisms Administration tools (CRM, CMS)

13 NODC, Russia Data Provider. Features Data source/resources registry and metadata update Codes and dictionaries mapping Request handling Access to the local data source (DBMS, files) Data extraction based on search criteria Transport data file generation HTTP GET/POST (REST) and Web service interfaces Security service Schedule mechanisms Administration tools (online maintenance for resource and instances)

14 NODC, Russia General data exchange scheme User Integration Server Interface layer (HTTP, Web-services) request Data Provider Request message reply Transport data file Response message

15 NODC, Russia Current operational status of E2EDM E2EDM was originated as one of the JCOMM/ETDMP pilot projects in 2003 MetOffice, UK IFREMER, France VLIZ, Belgium NODC, Russia

16 NODC, Russia Pilot project data sources The following data sources are involved in the E2EDM prototype Historical marine meteorological data Historical ocean cruise data Real-time GTS ocean (BATHY and TESAC) data Real-time GTS ocean (TESAC/ARGO data) Real-time GTS (SHIP) data Monthly climatic fields of ocean parameters (imageries) Ocean SST satellite data Geographic area – North Atlantic (test-area)

17 NODC, Russia Pilot project data sources The following data sources are involved in the E2EDM prototype Historical marine meteorological data Historical ocean cruise data Real-time GTS ocean (BATHY and TESAC) data Real-time GTS ocean (TESAC/ARGO data) Real-time GTS (SHIP) data Monthly climatic fields of ocean parameters (imageries) Ocean SST satellite data Geographic area – North Atlantic (test-area)

18 NODC, Russia User data access interface – via Integration Server Resource catalogue Parameters Additional search options

19 NODC, Russia User Data Access interface (cont.) Search status

20 NODC, Russia E2EDM Web site E2EDM Integration Server - IODE Project Office (VLIZ) RIHMI-WDC (mirror) User data access interface (cont.) Point data grid data profile data

21 NODC, Russia E2EDM using for SeaDataNet E2EDM Integration Server E2EDM Data Provider Data providers community HTTP communication Local Database systemLocal Data file system SDN Data Centres Network Web-portal NetCDF transport data-file Object transport data-file Responce message Request message User access Registry Discovery Delivery Connection Encoding Access

22 NODC, Russia E2EDM adaptation for SeaDataNet JRA2 - Technical development of the interoperable system: E2EDM covers many of JRA2 tasks, but all actions & decisions should be discussed. E2EDM model deepness is enough for many of the SeaDataNet positions. ( + - there is solution in E2EDM) Discovery services Metadata update & maintenance + Potential decision: CDI adoption. Security services Data access level + Transferred data size level + Delivery services Downloading services + Viewing services Developed user interface, libraries, external tools (based on NetCDF) +

23 NODC, Russia Thank you! Questions?