1. 1 TECO-WIS 6 - 8 November, Seoul JCOMM E2EDM Pilot Project Nickolay Mikhailov, JCOMM/IODE ETDMP chair, Russia Sergey Belov, RNODC/RIHMI-WDC, Russia

2. 2 Contents JCOMM/IODE ETDMP Pilot Project scope Concept, Functionality, Architecture Implementation result E2EDM within WIS E2EDM future

3. 3 Expert Team for Data Management Practices recommend and implement of end-to-end data management system for JCOMM/IODE review and assess the effectiveness of data management practices, recommend best practices ETDMP was organized by IOC/WMO JCOMM (JCOMM-1, 2001) IODE GETADE have been merged with JCOMM ETDMP (2003) Terms of Reference JCOMM/IODE ETDMP

4. 4 1 - Metadata management to develop and provide practical testing of a comprehensive metadata model to build and demonstrate a prototype system for real-time data fusion from distributed sources into sample products of interest to JCOMM/IODE users to examine a scheme of assigning a unique tag to original data, and the variations in QC procedures and quality flags for a various disciplines 2 - Data Assembly, Quality Control and Quality Assurance 3 - E2EDM Prototype ETDMP Pilot Projects

5. 5 E2EDM Prototype should provides functionality for: local data systems “wrapping” by E2EDM services data extraction from distributed sources on user requests/scheduling on-line aggregation and visualization creation of new products based on the obtained data E2EDM Prototype should integrates the data: at operational and delay-mode time scale across multidisciplinary ocean and marine meteo datasets/flows from geographically distributed data sources existing in different formats (codes, dictionaries and etc.) Pilot Project scope

6. 6 RIHMI-WDC/NODC, Russia Flanders Marine Institute (VLIZ), Belgium IFREMER, France Met Office, UK Pilot Project participants

7. 7 E2EDM Concept Data model Metadata/data objects specifications Namespace Data exchange protocol Software components

8. 8 E2EDM data model Providing identification and formalization of features for main model E2E objects: Data source Resource User (external software application) Model is based on the following structured semantic objects: Element Class - set of elements Record - set of classes These semantic objects are used for: Data granularity and description for local and transferred data Metadata description Codes and dictionaries unification Data sources and users interaction

9. 9 E2EDM metadata specifications E2ESearchMD – discovery metadata E2ELinkMD – data physical location registry E2EElementsMD – managed data/metadata elements E2ECodesMD – code lists, controlled dictionaries/vocabularies E2EUserMD – end-user external application registry E2EInterfaceMD – end-user interaction and data delivery specifications (interaction type, scheduling, data formats and delivering mechanisms – push/pull)

10. 10 E2EDM data specifications E2EDM data records: E2EDataPoint – data with fixed spatial (geographical, depth/height) coordinates, point data type E2EDataProfile – data with fixed geographical and temporal coordinates, and depth (height) coordinates, profile data type E2EDataGrid – data distributed in the scope by definite geometrical model

11. 11 E2EDM data objects specifications (II) class/elementNetCDF componentNetCDF structure E2EObjectHierarchy + objectHierarchyLevel DataHeader + DataContainer Group objectHierarchyIdDataHeaderAttribute objectHierarchyListDataContainerVariable E2EDM classes for data granularity and NetCDF data constructions relationship: 1 objectSystemId, objectHierarchyId, countryId, organizationId, dateTime, Latitude, Longitude, platfType, platformId, platformName platformId 2 DEPH, TEMP, PSAL dateTime, Latitude, Longitude dateTime

12. 12 E2EDM namespace Within E2EDM consists of two sub-namespaces: global and conceptual (thematic). Global namespace – structures for the E2E data model objects Conceptual (thematic domain) namespace – list of thematic elements which are available for discovery and exchange (E2EElementsMD) Both designed in XML Schema. 1.Global XML Schema (ISO 19115 + WMO Core Metadata + extension) 2.Concept XML Schema XML Schemas are also used for validation and development issues (XML Schema -> UML -> software design)

13. 13 E2EDM data exchange protocol Data exchange protocol is responsible for data requesting and data retrieving. Request message (XML) Response message (XML) Transport data file (NetCDF,v.2.2) * REQUEST MESSAGE EXAMPLE * 0.95 2005-08-03T14:47:52+03:00 data.meteo.ru http://data.meteo.ru:80/digir/DiGIR.php search 78.54 false * RESPONSE MESSAGE EXAMPLE * 2005-09-29T15:35:17+04:00 search http://data.meteo.ru:8090/dpms/ncResults/1127993716743-result.nc 7

14. 14 E2EDM software components E2EDM Data Provider (DP) – set of services, providing interface between local data systems and distributed system. Must be plugged on the local system. Services are using common library (E2EDM API). DP supports data bases, structured and object data files E2EDM Integration Server – set of services, providing system monitoring, security issues, resource catalogue maintenance and communication with DP (discovery, requesting and retrieving data)

15. 15 Integration Server. Functions Metadata update & maintenance Resource catalogue management Discovery services for data, metadata and common codes Request/response communication service Data sources monitoring Downloading services (for obtaining data files, metadata, resource and instance descriptions synchronization) Export tools (e.g. export E2EDM resource description into WMO Core Profile, to ASCII) HTTP GET/POST (REST) and Web service interfaces Security service Schedule mechanisms Administration tools (CRM, CMS)

16. 16 Data Provider. Functions Data source/resources registry and metadata update Codes and dictionaries mapping Request handling Access to the local data source (DBMS, files) Data extraction based on search criteria Transport data file generation HTTP GET/POST (REST) and Web service interfaces Security service Schedule mechanisms Administration tools (online maintenance for resource and instances)

17. 17 E2EDM architecture E2EDM Integration Server E2EDM Data Provider Data providers community HTTP communication Web-services Local Database systemLocal Data file system Data Centers Network JCOMM/IODE Data Portal netCDF transport data-file Object transport data-file Response message Request message User access Security Registry Discovery Delivery Monitoring Connection Mapping Data Access Data

18. 18 Implementation results MetOffice, UK IFREMER, France VLIZ, Belgium NODC, Russia

19. 19 Implementation results (II) The following data sources are involved in the E2EDM prototype Historical marine meteorological data Historical ocean cruise data Real-time GTS ocean (BATHY and TESAC) data Real-time GTS ocean (TESAC/ARGO data) Real-time GTS (SHIP) data Monthly climatic fields of ocean parameters (imageries) Ocean SST satellite data Geographic area – North Atlantic

20. 20 Administrative tools

21. 21 E2EDM web-site http://e2edm.meteo.ru/ Project overview Documentation Software components (coming soon) JCOMM/IODE Data Portal link

22. 22 Current JCOMM/IODE Data Portal Resource catalogue Parameters Additional search options Requests status

23. 23 Current JCOMM/IODE Data Portal interface (II) Point dataProfile dataGrid dataObject files

24. 24 E2EDM within WIS Harvester OAI DR DCPC Obninsk WMO and E2EDM metadata records «A» «B» GISC catalogue / registry E2EDM Integration Server E2EDM Data Providers VGISC protocol communication

25. 25 ETDMP IODE/JCOMM Ocean Data Portal Scheme for regional/task-oriented approach Community end-users Global Ocean Data Portal (linked with E2E Integration Server) Regional Ocean Data Portal Data Provider Users Themathic Ocean Data Portal Data Provider Users Data sources federation Data Provider E2E future implementation IOC IODE – XIX IODE/JCOMM Ocean Data Portal ODIN Black Sea

26. 26 Thank you! Questions?