Distributed system development, including standards and interoperability By Dick M.A. Schaap – technical coordinator Bologna, September 08

SeaDataNet Infrastructure objectives SeaDataNet aims to set up and operate an efficient Pan-European distributed infrastructure for managing marine and ocean data by connecting: 40 National Oceanographic Data Centres (NODC’s), national oceanographic focal points, and ocean satellite data centres, in Europe these Data Centres are mostly divisions of major national marine research institutes and based in 35 countries, surrounding the European seas SeaDataNet aims to ensure a comparable quality of data sets and to make data sets easily accessible on-line through a unique portal, while the data sets are stored and managed at the Data Centres.

SeaDataNet Infrastructure overall concept

SeaDataNet Infrastructure JRA activities JRA1: to define and develop common standards and protocols for SeaDataNet, that will the basis for interoperability inside the SeaDataNet network of Data Centres and outside to other systems. JRA2: to design and develop the architecture and software components for the SeaDataNet infrastructure and to coordinate its implementation at the Data Centres and central SeaDataNet portal. Close interaction between JRA1 and JRA2. The outcome of JRA1 also sets requirements towards JRA3: adapting the stand-alone ODV data analysis and viewing software package for integrated use with SeaDataNet output. Because of this close relation the activities in JRA1, JRA2 and JRA3 are coordinated by a Technical Task Team.

SeaDataNet Technical Task Team Oversees and coordinates the technical developments Includes 11 core technical partners 8 Meetings sofar: Paris, France May 2006 Hamburg, Germany 4-5 September 2006 The Hague, The Netherlands 7-8 December 2006 Trieste, Italy March 2007 The Hague, The Netherlands 2-3 July 2007 Nice, France, August 2007 Liverpool, UK, December 2007 Paris, France, May 2008

SeaDataNet Technical Task Team Methodology: Agenda with relevant topics, leading speakers, open discussion and brainstorming, possible subgroups for further explorations and analyses, resulting in technical specifications of system architecture and its modules, followed by implementation, evaluation and operation All presentations and documents are included in the SeaDataNet Extranet in the TTT section. Note: All Project Partners are encouraged to visit this regularly. All actions / decisions are minuted in a TTT Action List, which is updated regularly and also can be found in the Extranet TTT section

SeaDataNet Infrastructure versions and planning Version 0 = Continuation and maintenance of existing Sea-Search systems with minor modifications Version 1 = Harmonised and upgraded metadatabases; Transparent data access involving all Data Centres. Prototype ready by March 2008; all TTT partners by end 2008; thereafter gradually migrating all partners – operational by mid Version 2 = Adding OGC viewing services and further virtualisation of data access. Operational by 2010.

SeaDataNet Version Set-up of SeaDataNet portal ( with CMSwww.seadatanet.org Continuation of existing Sea-Search metadata systems: EDMED - European Directory of Marine Environmental Data Sets EDMERP – European Directory of Marine Research Projects CSR – Cruise Summary Reports EDIOS – European Directory of Ocean observing Systems EDMO – European Directory of Marine Organisations CDI – Common Data Index Operation of existing data access systems / procedures of all Data Centres, referred in the Common Data Index directory Set-up of SeaDataNet extranet Set-up of SeaDataNet mailing lists

SeaDataNet Portal website

SeaDataNet System Approach for V1 and V2 An approach has been adopted, which is in line with INSPIRE. The SeaDataNet infrastructure should consist of the following services: Discovery services = Metadata directories Security services = Authentication, Authorization & Accounting (AAA) Delivery services = Data access & downloading of data sets Viewing services = Visualisation of metadata, data and data products Product services = Generic and standard products Monitoring services = Statistics on usage and performance of the system Maintenance services = Updating of metadata by Data Centres A network of interconnected Data Centres and a central Portal, that will give users access to the various SeaDataNet services, and information on data management standards, tools and protocols.

SeaDataNet User’s portal architecture V1 services for metadata Data downloading services Visualization services (WMS) for V2 Ifremer Database BODC Database BSH Database... Download managers in Data centers CSREDIOS EDMED CDI Shopping basket Requests status manager datametadata CSR CDI Project info Metadata & Data catalogues Data request Status of request General requestMetadata request Organisation + data source id Data download at BODCat BSHat MARIS + EDMERP Entry point for access hits EDMO EDIOSEDMED EDMERPEDMO Software Vocabularies StandardsCross search AAA Registr. My transact. User Register User registration

SeaDataNet Interoperability Interoperability is the key to distributed data management system success. This is achieved in SeaDataNet via: Using common quality control protocols and flag scale Using common and controlled vocabularies, including international content governance Adopting the ISO metadata standard for all metadata directories Providing XML Validation Services to quality control the metadata maintenance Providing standard metadata entry tools Using harmonised Data Transport Formats (NetCDF, ODV ASCII and MedAtlas ASCII) for data sets delivery Adopting of OGC standards for mapping and viewing services Using SOAP Web Services in the SeaDataNet architecture

SeaDataNet Quality Control Guideline A guideline (V1) of recommended QC procedures, reviewing NODC schemes and other known schemes (e.g. WGMDM guidelines, World Ocean Database, GTSPP, Argo, WOCE, QARTOD, ESEAS,SIMORC, etc.). QC methods for CTD (temperature and salinity), Current meter data (including ADCP), Wave data, Sea level data A scheme of QC flags to be used in SeaDataNet.These flags are for assigning to individual data values. They are not for allocating to whole data series, or to accompanying information. Compiled in discussion with IOC, ICES and JCOM, to ensure an international acceptance and tuning. Important feedback from the joint IODE/JCOMM Forum on Oceanographic Data Management and Exchange Standards (January 2008), joined by SeaDataNet and international experts to consider on-going work on standards and to seek harmonisation, where possible. Now extending the guideline with QC methods for surface underway data, nutrients, geophysical data, and biological data. V2: April 2009.

SeaDataNet Common Vocabularies Use of common vocabularies in all metadatabases and data formats is an important prerequisite towards consistency and interoperability. Set-up and population of Common Vocabularies. The SeaDataNet Vocabulary service is based upon the NERC DataGrid (NDG) vocabulary Web service. For end-users there is a vocabulary Client Interface for searching and browsing and to export selected entries in csv format. The Web service is compliant to WS Basis Profile 1.1, which is adopted as standard for all Web services in SeaDataNet. Content governance of the vocabularies is very important and is done by a combined SeaDataNet and MarineXML Vocabulary Content Governance Group (SeaVoX), moderated by BODC, and including experts from SeaDataNet, MMI, MOTIIVE, JCOMMOPS and more international groups. SeaVox operates by mailing list server.

SeaDataNet Common Vocabularies Vocabularies User Interface

SeaDataNet Common Data Transport Formats V1: data sets are accessable by downloading services. Delivery of data sets to users requires common data transport formats, which interact with other SeaDataNet standards (Vocabularies, Quality Flag Scale) and analysis & presentation tools (ODV, DIVA) The following formats have been defined: SeaDataNet ODV ASCII for profiles, time series and trajectories SeaDataNet MedAtlas as optional extra format. NetCDF with CF compliance for gridded data sets ODV and MedAtlas have been outfitted with a SeaDataNet semantic header International cooperation is underway from SeaDataNet with the CF community for a common NetCDF format for the oceanographic and meteorological domains, including a semantic header

Authentication, Authorization and Administration Single Sign On system required for access to distributed system User’s authentication information based on personal login / password User must register in order to get one login Web form to provide necessary information online user agreement on “SeaDataNet General Licence” After processing, login/password sent by ( check) Choice in V1 for CAS system (= Centralized) SeaDataNet Data Policy Authorisation based on “Roles”

User registration and registration validation process Registration Web form SDN licence agreement + User information SeaDataNet web portal 12 Validation Web form + SeaDataNet role Validation of user registration and SeaDataNet role assignment SeaDataNet user directory User SDN User Desk 3 3 Transmission by User personal identifier (login) + password Registration request Validation NODC of the user’s country or SDN User Desk (default) User directory update

SeaDataNet AAA service Interface for central log-in

SeaDataNet infrastructure V1 Discovery services EDMED - Data Sets EDMERP – Research Projects CSR – Cruise Summary Reports EDIOS – Monitoring systems EDMO – Marine Organisations CDI – Common Data Index PLUS Cross search on top of the directories

SeaDataNet Discovery services Activities undertaken for: Reviewing and streamlining the logical formats of each of the Directories Expanding the number of Common vocabularies, further population and upgrading of Vocabularies Web services Defining XML schema’s and formats, using the ISO metadata standard as basis Defining and developing maintenance modalities for each of the Directories

SeaDataNet Discovery services Defining and developing new User Interfaces for each of the Directories Defining and implementing XML Validation Web services, that will be used to validate XML output from data centres, before import into the public Directories Developing Web services for the Directories

Vocabs EDMO … … … … CDI EDMERP CSR EDMED EDIOS SeaDataNet : Formats review Review of the format and use of common vocabularies for each of the Directories to achieve harmonisation and integration, and paving the path to data access via the CDI Data Access …

SeaDataNet XML Schema’s The ISO content model is the basis for the XML formats and exchange schema’s (XSD). Guidance document has been prepared on how to use XML for SeaDataNet, including how to declare references to Common Vocabularies, EDMO and EDMERP For each Directory has been prepared: Description of the format and XML tags XML Schema XML example file Extended XML Schema’s have been prepared, using Schematron and OCL to support the checking of mandatory fields, use of codes from the Common Vocabularies and use of organization codes from EDMO. These schema’s are used in the XML Validation Web services

SeaDataNet Maintenance modalities Maintenance: Depending on the Directory, the following maintenance modalities are provided: Online maintenance via online Content Management System XML export from local system Local XML export can be produced by partners via: own software using the MIKADO Java tool, that has been developed for entering and editing partner’s shares of the V1 Directories. MIKADO interacts with the Web services of the vocabularies, EDMO and EDMERP and produces valid XML files, that can be imported into the central V1 Directories. MIKADO also includes functionality for coupling to partner’s local database(s) for generating CDI XML files in bulk

SeaDataNet MIKADO Java tool Available under multiple environments : Microsoft : Windows 2000, XP, VISTA APPLE Unix - Solaris Linux MIKADO Java code Native Drivers MYSQL ORACLE POSTGRES SQLServer Bridge Drivers using Microsoft ODBC (ACCESS, EXCEL, SQL SERVER) DATABASE JDBC Java DataBase Connectivity EXCEL File XML files for SeaDataNet catalogues CSR EDMED EDMERP CDI [EDIOS] Manual Automati c Other Drivers Downloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory)

SeaDataNet Maintenance modalities DirectoryOnline CMS by partners XML exchange via use of MIKADO at partners XML exchange via local system at partners Online CMS by moderator XML Validation service EDMED (BODC = authority) SDN DC’s initial content conversion free text => vocabs via ‘sandbox’ Stand-alone incl. local storage + QC – loop for content Synchronisation Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP EDMERP (MARIS = authority) SDN DC’s manage their national records + ‘sandbox’ for institutes Stand-alone incl. local storage Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP CSR (BSH = authority) Online entries by chief scientists Stand-alone incl. local storage Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP

SeaDataNet Maintenance modalities DirectoryOnline CMS by partners XML exchange via use of MIKADO at partners XML exchange via local system at partners Online CMS by moderator XML Validation service CDI (MARIS = authority) Tool embedded in local system Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP EDIOS (BODC = authority) XML SCHEMA NOT YET READY SDN DC’s initial revision + new entries via ‘sandbox’ Stand-alone incl. local storage NOT YET READY Generated from in-house system NOT YET READY Validation of XML syntax and use of Vocabs, EDMO and EDMERP EDMO (MARIS = authority) SDN DC’s manage their national records Vocabularies (BODC = authority) BODC with SeaVox governance mailing list User Register + AAA service (IFREMER = authority) SDN DC’s manage their national records

SeaDataNet Content Management Systems EDMERP CSREDMED EDMO

SeaDataNet new User Interfaces EDMERP CSR EDIOS EDMO

SeaDataNet Discovery services - status Reviewing and streamlining the logical formats of each of the Directories – READY Expanding the number of Common vocabularies, further population and upgrading of Vocabularies Web services – READY Defining XML schema’s and formats, using the ISO metadata standard as basis – READY, EXCEPT FOR EDIOS Defining and developing maintenance modalities for each of the Directories: EDMO, EDMERP, EDMED and CSR – READY VIA ONLINE CMS EDMED, EDMERP, CSR and CDI – READY VIA MIKADO UPGRADE Common Vocabularies and AAA - READY VIA MASTER CMS IMPORT OF XML FOR CDI READY IMPORT OF XML FOR EDMERP, CSR, and EDMED ALMOST READY EDIOS – VIA MIKADO AND XML IMPORT NOT YET READY, WAITS FOR XML SCHEMA Defining and implementing XML Validation Web services, that will be used to validate XML output from data centres, before import into the public Directories – READY FOR CDI, EDMED, EDMERP and CSR

SeaDataNet Discovery services - status Defining and implementing new User Interfaces READY FOR EDMO, CDI, EDMERP, CSR, EDIOS, Common Vocabularies and AAA services UNDER DEVELOPMENT FOR EDMED CROSS SEARCH DEVELOPMENT STARTS WHEN ALL ARE READY Content upgrading from V0 to V1 – Activity by all NODC’s EDMO READY EDMERP and CSR WELL UNDERWAY – DEADLINE NOV 08 EDMED START VERY SOON – DEADLINE DEC 08 EDIOS – so far done for United Kingdom and underway for Black Sea countries; others waiting for MIKADO extension Implementing Web services EDMO, EDMERP, Common Vocabularies, Validator and AAA services READY Note: CDI upgrade is undertaken as part of the Data Access activities;

SeaDataNet : URLs URL for CSR V1 Retrieval: URL for CSR V1 CMS URL for EDIOS V1 Retrieval URL for EDMED V1 CMS

SeaDataNet : URLs EDMERP Retrieval EDMERP CMS EDMO V1 retrieval EDMO V1 CMS

SeaDataNet Delivery Services See Separate presentation