1. E2EDM Technology: Overview
Training on the management
of the End to End data Management (E2EDM)
prototype system at IODE NODCs
E2EDM Technology: Overview
October, Oostende, 2007

2. History… From the beginning to now days… September, 2003
Draft version of E2EDM concept presented by Nick Mikhailov (head of Russian NODC) at the First Session of the JCOMM Expert Team on Data Management Practices (ETDMP) , Oostende, September 2003)
May, 2004
Review of PILOT PROJECT III presented at the Ad hoc Session of the ETDMP (Oostende, Belgium). The E2EDM concept and working plan for the prototype has been adopted
September, 2004
A representative of JCOMM, Mr. David Thomas, presented the report of the ad hoc Session of the JCOMM/IODE ETDMP on Interprogramme Task Team on the Future Information System (Geneva, Switzerland)
December, 2004
Release of technical specification document and alpha-version of E2EDM software components; web site on E2EDM technology was launched
April, 2005
First version of the Data Provider was installed in Flanders Marine Institute (VLIZ) and NODC of Russia, providing regular GTS and cruise historical data. The E2EDM prototype was successfully demonstrated at IODE-XVII session
October, Oostende, 2007
October, Oostende, 2007

3. History…
August, 2005
The Data Provider has been installed at IFREMER (France) and MetOffice (UK), providing ARGO buoy and voluntary ships data (MCSS project).
September, 2005
Demonstration at IOC/WMO JCOMM - II session
October, 2005
A representatives of JCOMM, Robert Keeley and Nick Mikhailov presented the report of the current status of E2EDM technology at the Interprogramme Coordination Group for WMO Information System (Boulder, USA). The decision to use E2EDM as block of WMO IS has been adopted.
March, 2006
WIS GISC-DCPC Developer meeting. Collaboration with WIS on establishing DCPC based on E2EDM technology.
May, 2006
The E2EDM data sources were expanded to provide functions for Marine warning system support. The appropriated external application – E2EDM Marine Warning System prototype has been developed.
March, 2007
The E2EDM prototype was successfully demonstrated at IODE-XVII session. Adoption of IODE Ocean Data Portal based on E2EDM was concerned.
October, Oostende, 2007
October, Oostende, 2007

4. About the E2EDM technology
BASIS
The “end to end” data management (E2E) technology is considered as an “covering layer” that comprises local data systems managed by IODE data centres and provides “transparent” access to the metadata, data and products (resources) generated by these local data systems
The goal of this E2EDM technology is to integrate the non-homogeneous local data systems into unified distributed marine data system, that will provide the transparent exchange between these local systems and a end-user access to numerous data flows/sets/bases in a “single stop shopping” manner
October, Oostende, 2007
October, Oostende, 2007

5. Background
The E2EDM technology in it’s original idea is able to provide the real-time access to the distributed marine data:
at operational and delay-mode time scale
across oceanographic and marine meteorological disciplines
of a various data categories – observation, climate, forecast, analysis
from multiple local data formats and data storage types
from multiple data providers around the world
October, Oostende, 2007

6. Overall solution
The basis of the E2EDM technology is the model of distributed information resources on the concept of the E2EDM technology objects (i.e. any entities producing or using data/metadata under the framework of the E2EDM system). These objects are:
local data system
data source
information resource
transport data file
services
end-users
October, Oostende, 2007

7. Basic terms and definitions
Data set (local) - set of data (or metadata) being created by a Data Center. It is characterized by a specific data model, data coding system, data storage system and format, conditions of access to data
Distributed Data Source System (further, DDSS) – set of data sources in Data Centers operating under the E2EDM technology
Data Source - combination of the Data Provider (software component) and local data sets (information component). One Data Provider installed in a Data Center processes all local data sets of the Center
Resources - local data sets representing data sources. Resource is a local data set or part of it. Resource has a unique identifier and may consist of structural data units (resource instances, or subsets). The content and structure of resource are recorded in the resource description and represent descriptive metadata of the DDSS.
Code - names of objects and their properties, code values of objects in the form of character sequence used for unambiguous understanding of data in information systems. Code is defined by a coding system (ISO, IOC, WIS, etc)
October, Oostende, 2007

8. Basic terms and definitions
System codes – codes of resource description elements (data accessibility, spatial presentation, storage system, etc). System codes are presented by fixed list of codes and used for description of distributed resources
System dictionaries – codes and structured descriptions of objects (organizations, ships, buoys, satellites, etc.). System dictionaries are used for unification of data coding by mapping local codes into system codes
Data exchange protocol – agreements regulating data exchange inside distributed system. Data exchange protocol consists of request-message, response-message and transport data file
Transport data file – binary data representation based on data search results from data source.
Local data elements composing local data sets in Data Centers. System data elements – data elements managed by E2EDM technology
October, Oostende, 2007

9. Technical specifications
Technical specifications are based on a single data model and include specifications of:
Namespace
metadata and data records
data exchange protocol
October, Oostende, 2007

10. Namespace
Defines metadata and data structures. Within E2EDM consists of two sub-namespaces: global and conceptual (thematic)
Global namespace – structures for the model objects
Conceptual (thematic domain) namespace – list of thematic elements dealing with some discipline, e.g. meteorology, and available for discovery and exchange.
Both written on XML Schema language.
Global XML Schema (ISO TC211 (particularly ISO ISO19139) + WMO Core Metadata, which is an extension of ISO 19115). Since not all fields/attributes can be found in ISO model it was extended in E2EDM implementation
Concept XML Schema (DiGIR)
October, Oostende, 2007

11. ISO
ISO TC211 : is working on the development of International Standards
ISO 19115:2003 is the International Standard for metadata for geographical data sets and is a content model. It has been officially published in May 2003 and a Technical Correction / Update has been published in July 2006
The TC211 is now working on an extension to ISO 19115, i.e. ISO , with extra elements which are relevant for the acquisition of imagery and gridded data. The adoption of this extension is planned for 2008
ISO is the technical implementation of the ISO model and provides the XML schema’s (XSD) for the parsing / validation of generated XML files
The final ISO schema is available at an unofficial XSD repository:  
October, Oostende, 2007

12. Metadata records
E2ESearchMD – description of a local data (or metadata) source as an information resource generated by Data Provider software and with the data granulation into instances using specified rules and scheme.
Functions – data discovery and distributed data source monitoring
E2ECodesMD – description of the common (system) code lists and dictionaries used for data unification by mean reformatting the local codes into system codes.
Functions – supporting the resource’s descriptions for coding the E2ESearchMD elements and unification of data from a various local data sources with different code systems
E2EElementsMD – description of thematic elements (data and related metadata elements) managed by the E2EDM technology and placed in Conceptual XML Schema.
Functions – defining the parameter dictionary for concrete system implementation and providing the flexibility of E2E technology use for a various implementation cases – via data source federation concept
October, Oostende, 2007

13. Metadata records
E2ELinkMD – description of a data/metadata source to provide access to the local data/metadata system: DBMS, structured data files, object data files.
Functions – data source linkage to the central entry point
E2ECodesMD – description of the common (system) code lists and dictionaries used for data unification by mean reformatting the local codes into system codes.
Functions – supporting the resource’s descriptions for coding the E2ESearchMD elements and unification of data from a various local data sources with different code systems
E2EInterfaceMD – description of an information interface of the end-user external software applications (i.e. system Portal services) with Integration Server – descriptions of distributed system resources that external application is required.
Functions – data exchange status between Integration Server and external application
October, Oostende, 2007

14. Data records
E2EDataPoint – data with fixed spatial (geographical, depth/height) coordinates, point data type
E2EDataProfile – data with fixed geographical and temporal coordinates, and depth (height) coordinates, profile data type
E2EDataGrid – data distributed in the scope by definite geometrical model
E2EObjectDD – representation of object data (e.g. documents, images, GIS shapes, etc.).
October, Oostende, 2007

15. Data exchange protocol
The Data Exchange Protocol provides data interaction between the E2EDM Data Provider and Integration Server (or Integration Server and end-user applications) and consists of: request/response messages and transport data file.
The formats and other specifications of request/response messages are based on the DiGIR Protocol.
The extension to the DiGIR request/response formats has been developed providing requests/responses for local file-oriented data systems and transferring data using a transport data file.
October, Oostende, 2007

16. JCOMM/IODE Data Portal E2EDM Integration Server
Architecture
User access
JCOMM/IODE Data Portal
HTTP communication
Web-services
Security
Registry
Discovery
Delivery
Monitoring
E2EDM Integration Server
HTTP communication
Web-services
Data providers community
Request message
E2EDM
Data Provider
Connection
Mapping
Data Access
Response message
NetCDF transport
data-file
Data
Data Centers Network
Object transport
data-file
Local Database system
Local Data file system
October, Oostende, 2007

17. Architecture. Features
Using open standards and open source components
Based on Service-Oriented Architecture concept (SOA)
User is isolated from whole distributed system schema (common user doesn’t know how many data providers are operating and where are they located)
User now is more than just a human being with web-browser. External applications, subsystems and systems are considered as users too
High scalability and nested hierarchy allows to build up any number of levels inside the distributed system
October, Oostende, 2007

18. Software components
E2EDM Data Provider (DP) – set of services, providing the access to the local data system of participating centres. As soon as the Data Provider is installed as a “wrapper” of the local data system, this system becomes a data source for the E2E-based distributed data system
E2EDM Integration Server – set of services, providing the management and use of the data sources network via interaction with Data Providers
October, Oostende, 2007

19. E2EDM Integration Server. Features
Automatic metadata update & maintenance
Resource catalogue management
Discovery services for data, metadata and common codes
Request/response communication service
Data sources monitoring
Download services (for obtaining data files, metadata, resource and instance descriptions synchronization)
Export tools (e.g. export in XML from E2EDM resource description into WMO Core Profile, ASCII, etc.)
HTTP GET/POST,REST and Web service interfaces (SOA)
Security services
Schedule mechanisms
Administration tools (CRM, CMS)
October, Oostende, 2007

20. E2EDM Integration Server Monitoring of the distributed data sources
October, Oostende, 2007

21. E2EDM Integration Server
Federations
October, Oostende, 2007

22. E2EDM Integration Server Data providers registry
October, Oostende, 2007

23. E2EDM Integration Server
Administrator workplace. Resources of Data Providers
E2EDM
resources
October, Oostende, 2007

24. E2EDM Integration Server Tasks and user management
October, Oostende, 2007