E2EDM Technology: Overview Training on the management of the End to End data Management (E2EDM) prototype system at IODE NODCs E2EDM Technology: Overview 22- 25 October, Oostende, 2007

History… From the beginning to now days… September, 2003 Draft version of E2EDM concept presented by Nick Mikhailov (head of Russian NODC) at the First Session of the JCOMM Expert Team on Data Management Practices (ETDMP) , Oostende, 15-18 September 2003) May, 2004 Review of PILOT PROJECT III presented at the Ad hoc Session of the ETDMP (Oostende, Belgium). The E2EDM concept and working plan for the prototype has been adopted September, 2004 A representative of JCOMM, Mr. David Thomas, presented the report of the ad hoc Session of the JCOMM/IODE ETDMP on Interprogramme Task Team on the Future Information System (Geneva, Switzerland) December, 2004 Release of technical specification document and alpha-version of E2EDM software components; web site on E2EDM technology was launched April, 2005 First version of the Data Provider was installed in Flanders Marine Institute (VLIZ) and NODC of Russia, providing regular GTS and cruise historical data. The E2EDM prototype was successfully demonstrated at IODE-XVII session 22- 25 October, Oostende, 2007 22- 25 October, Oostende, 2007

History… August, 2005 The Data Provider has been installed at IFREMER (France) and MetOffice (UK), providing ARGO buoy and voluntary ships data (MCSS project). September, 2005 Demonstration at IOC/WMO JCOMM - II session October, 2005 A representatives of JCOMM, Robert Keeley and Nick Mikhailov presented the report of the current status of E2EDM technology at the Interprogramme Coordination Group for WMO Information System (Boulder, USA). The decision to use E2EDM as block of WMO IS has been adopted. March, 2006 WIS GISC-DCPC Developer meeting. Collaboration with WIS on establishing DCPC based on E2EDM technology. May, 2006 The E2EDM data sources were expanded to provide functions for Marine warning system support. The appropriated external application – E2EDM Marine Warning System prototype has been developed. March, 2007 The E2EDM prototype was successfully demonstrated at IODE-XVII session. Adoption of IODE Ocean Data Portal based on E2EDM was concerned. 22- 25 October, Oostende, 2007 22- 25 October, Oostende, 2007

About the E2EDM technology BASIS The “end to end” data management (E2E) technology is considered as an “covering layer” that comprises local data systems managed by IODE data centres and provides “transparent” access to the metadata, data and products (resources) generated by these local data systems The goal of this E2EDM technology is to integrate the non-homogeneous local data systems into unified distributed marine data system, that will provide the transparent exchange between these local systems and a end-user access to numerous data flows/sets/bases in a “single stop shopping” manner 22- 25 October, Oostende, 2007 22- 25 October, Oostende, 2007

Background The E2EDM technology in it’s original idea is able to provide the real-time access to the distributed marine data: at operational and delay-mode time scale across oceanographic and marine meteorological disciplines of a various data categories – observation, climate, forecast, analysis from multiple local data formats and data storage types from multiple data providers around the world 22- 25 October, Oostende, 2007

Overall solution The basis of the E2EDM technology is the model of distributed information resources on the concept of the E2EDM technology objects (i.e. any entities producing or using data/metadata under the framework of the E2EDM system). These objects are: local data system data source information resource transport data file services end-users 22- 25 October, Oostende, 2007

Basic terms and definitions Data set (local) - set of data (or metadata) being created by a Data Center. It is characterized by a specific data model, data coding system, data storage system and format, conditions of access to data Distributed Data Source System (further, DDSS) – set of data sources in Data Centers operating under the E2EDM technology Data Source - combination of the Data Provider (software component) and local data sets (information component). One Data Provider installed in a Data Center processes all local data sets of the Center Resources - local data sets representing data sources. Resource is a local data set or part of it. Resource has a unique identifier and may consist of structural data units (resource instances, or subsets). The content and structure of resource are recorded in the resource description and represent descriptive metadata of the DDSS. Code - names of objects and their properties, code values of objects in the form of character sequence used for unambiguous understanding of data in information systems. Code is defined by a coding system (ISO, IOC, WIS, etc) 22- 25 October, Oostende, 2007

Basic terms and definitions System codes – codes of resource description elements (data accessibility, spatial presentation, storage system, etc). System codes are presented by fixed list of codes and used for description of distributed resources System dictionaries – codes and structured descriptions of objects (organizations, ships, buoys, satellites, etc.). System dictionaries are used for unification of data coding by mapping local codes into system codes Data exchange protocol – agreements regulating data exchange inside distributed system. Data exchange protocol consists of request-message, response-message and transport data file Transport data file – binary data representation based on data search results from data source. Local data elements composing local data sets in Data Centers. System data elements – data elements managed by E2EDM technology 22- 25 October, Oostende, 2007

Technical specifications Technical specifications are based on a single data model and include specifications of: Namespace metadata and data records data exchange protocol 22- 25 October, Oostende, 2007

Namespace Defines metadata and data structures. Within E2EDM consists of two sub-namespaces: global and conceptual (thematic) Global namespace – structures for the model objects Conceptual (thematic domain) namespace – list of thematic elements dealing with some discipline, e.g. meteorology, and available for discovery and exchange. Both written on XML Schema language. Global XML Schema (ISO TC211 (particularly ISO 19115 + ISO19139) + WMO Core Metadata, which is an extension of ISO 19115). Since not all fields/attributes can be found in ISO model it was extended in E2EDM implementation Concept XML Schema (DiGIR) 22- 25 October, Oostende, 2007

ISO 19115 - 19139 ISO TC211 : http://www.isotc211.org/ is working on the development of International Standards ISO 19115:2003 is the International Standard for metadata for geographical data sets and is a content model. It has been officially published in May 2003 and a Technical Correction / Update has been published in July 2006 The TC211 is now working on an extension to ISO 19115, i.e. ISO 19115-2, with extra elements which are relevant for the acquisition of imagery and gridded data. The adoption of this extension is planned for 2008 ISO 19139 is the technical implementation of the ISO 19115 model and provides the XML schema’s (XSD) for the parsing / validation of generated XML files The final ISO 19139 schema is available at an unofficial 19139 XSD repository:  http://eden.ign.fr/xsd/isotc211 22- 25 October, Oostende, 2007

Metadata records E2ESearchMD – description of a local data (or metadata) source as an information resource generated by Data Provider software and with the data granulation into instances using specified rules and scheme. Functions – data discovery and distributed data source monitoring E2ECodesMD – description of the common (system) code lists and dictionaries used for data unification by mean reformatting the local codes into system codes. Functions – supporting the resource’s descriptions for coding the E2ESearchMD elements and unification of data from a various local data sources with different code systems E2EElementsMD – description of thematic elements (data and related metadata elements) managed by the E2EDM technology and placed in Conceptual XML Schema. Functions – defining the parameter dictionary for concrete system implementation and providing the flexibility of E2E technology use for a various implementation cases – via data source federation concept 22- 25 October, Oostende, 2007

Metadata records E2ELinkMD – description of a data/metadata source to provide access to the local data/metadata system: DBMS, structured data files, object data files. Functions – data source linkage to the central entry point E2ECodesMD – description of the common (system) code lists and dictionaries used for data unification by mean reformatting the local codes into system codes. Functions – supporting the resource’s descriptions for coding the E2ESearchMD elements and unification of data from a various local data sources with different code systems E2EInterfaceMD – description of an information interface of the end-user external software applications (i.e. system Portal services) with Integration Server – descriptions of distributed system resources that external application is required. Functions – data exchange status between Integration Server and external application 22- 25 October, Oostende, 2007

Data records E2EDataPoint – data with fixed spatial (geographical, depth/height) coordinates, point data type E2EDataProfile – data with fixed geographical and temporal coordinates, and depth (height) coordinates, profile data type E2EDataGrid – data distributed in the scope by definite geometrical model E2EObjectDD – representation of object data (e.g. documents, images, GIS shapes, etc.). 22- 25 October, Oostende, 2007

Data exchange protocol The Data Exchange Protocol provides data interaction between the E2EDM Data Provider and Integration Server (or Integration Server and end-user applications) and consists of: request/response messages and transport data file. The formats and other specifications of request/response messages are based on the DiGIR Protocol. The extension to the DiGIR request/response formats has been developed providing requests/responses for local file-oriented data systems and transferring data using a transport data file. 22- 25 October, Oostende, 2007

JCOMM/IODE Data Portal E2EDM Integration Server Architecture User access JCOMM/IODE Data Portal HTTP communication Web-services Security Registry Discovery Delivery Monitoring E2EDM Integration Server HTTP communication Web-services Data providers community Request message E2EDM Data Provider Connection Mapping Data Access Response message NetCDF transport data-file Data Data Centers Network Object transport data-file Local Database system Local Data file system 22- 25 October, Oostende, 2007

Architecture. Features Using open standards and open source components Based on Service-Oriented Architecture concept (SOA) User is isolated from whole distributed system schema (common user doesn’t know how many data providers are operating and where are they located) User now is more than just a human being with web-browser. External applications, subsystems and systems are considered as users too High scalability and nested hierarchy allows to build up any number of levels inside the distributed system 22- 25 October, Oostende, 2007

Software components E2EDM Data Provider (DP) – set of services, providing the access to the local data system of participating centres. As soon as the Data Provider is installed as a “wrapper” of the local data system, this system becomes a data source for the E2E-based distributed data system E2EDM Integration Server – set of services, providing the management and use of the data sources network via interaction with Data Providers 22- 25 October, Oostende, 2007

E2EDM Integration Server. Features Automatic metadata update & maintenance Resource catalogue management Discovery services for data, metadata and common codes Request/response communication service Data sources monitoring Download services (for obtaining data files, metadata, resource and instance descriptions synchronization) Export tools (e.g. export in XML from E2EDM resource description into WMO Core Profile, ASCII, etc.) HTTP GET/POST,REST and Web service interfaces (SOA) Security services Schedule mechanisms Administration tools (CRM, CMS) 22- 25 October, Oostende, 2007

E2EDM Integration Server Monitoring of the distributed data sources 22- 25 October, Oostende, 2007

E2EDM Integration Server Federations 22- 25 October, Oostende, 2007

E2EDM Integration Server Data providers registry 22- 25 October, Oostende, 2007

E2EDM Integration Server Administrator workplace. Resources of Data Providers E2EDM resources 22- 25 October, Oostende, 2007

E2EDM Integration Server Tasks and user management 22- 25 October, Oostende, 2007