Common Data Index CDI V1 How to proceed By Dick M.A. Schaap – technical coordinator Madrid, March 09

Central and homogeneous metadatabase plus transparent access to the distributed data sets via a unique user interface at the SeaDataNet portal and downloading services SeaDataNet Delivery Services – CDI V1

SeaDataNet CDI V1 system components CDI V1 Metadatabase and User Interface: Searching and browsing of metadata of data sets, managed by SeaDataNet Data Centres Central User Register - Web service: Details of users, their organizations and addresses, license period, Id-Passwords, SeaDataNet Roles Shopping Basket – User Interface: Preparing a user request of multiple data sets, handling validation of users and roles via the Central User Register, and routing requests to the Request Status Manager Request Status Manager – Web service + User Interface: Processing and administration of all requests and data deliveries (downloads), for users in communication with data centres. For data centres to oversee all transactions.

SeaDataNet CDI V1 system components Data Centre Download Manager – Java Tool: Handles communication with Request Status Manager and takes care that requested files are made ready for downloading by User (if OK) in the agreed formats via a local website address at the Data Centre

SeaDataNet CDI V1 – implementation step by step Prepare CDI V1 XML test records using the new MIKADO software and the latest CDI XML schema Consider carefully the Data Access Restriction, that you want to apply to the data Send the test records to the XML Validation Services for validation If ok, then forward the test records to MARIS for inclusion in the CDI V1 test database

SeaDataNet CDI V1 – implementation step by step Install the Download Manager software; 2 possible situations You have a file management system You have a database management system In case of file management system: Convert data files to SeaDataNet standard formats Fill the ‘coupling table’ to connect the CDI V1 records (via LOCAL_CDI_ID) to the associated files (by name OR by query) Test together with MARIS the set-up and good functioning of the connection via the Test interface

SeaDataNet CDI V1 – implementation step by step If ok, then make full production of CDI V1 records, complete coupling table and make sure that the files are ready / the queries are ok. Possible interim step: full CDI V1 production in the Import system for checking the quality of the CDI V1 records Final step: MARIS moves the production CDI V1 records to the operational CDI V1 system and the data centre is active. Maintenance and updating; only modifications => LOCAL_CDI_ID is important key to recognise updated or new CDI records

CDI is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record). This is in contrast to EDMED, which is an inventory of data sets, each of which comprises a large number of measurements. CDI XML Schema is based upon the ISO standard At the SeaDataNet website you can find: CDI V1 XML Schema CDI V1 XML example CDI V1 logical description in XLS file CDI V1 XML files can be prepared with the MIKADO software SeaDataNet Prepare CDI V1 XML test records

SeaDataNet MIKADO Java tool Available under multiple environments : Microsoft : Windows 2000, XP, VISTA APPLE Unix - Solaris Linux MIKADO Java code Native Drivers MYSQL ORACLE POSTGRES SQLServer Bridge Drivers using Microsoft ODBC (ACCESS, EXCEL, SQL SERVER) DATABASE JDBC Java DataBase Connectivity EXCEL File XML files for SeaDataNet catalogues CSR EDMED EDMERP CDI [EDIOS] Manual Automati c Other Drivers Downloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory)

User must register once in order to get a personal login - password Web form to provide necessary information User agrees with “SeaDataNet User Licence” After processing, login/password sent by ( check) Licence is part of the SeaDataNet Data Policy NODCs intermediate in the registration of their national users to the SeaDataNet central user register => users will get a user role Role together with the data access restriction determines the access: User will have immediate access to the data The User request will be considered by the Data Centre and possibly the user will be contacted User will be denied access SeaDataNet – Consider the Data Access Restriction

User Roles in AAA Register, of relevance for data access SDRN02Public SDRN03Academic SDRN04Commercial SDRN05National and local government SDRN06Pan-National government See Vocabulary for definitions Multiple roles may apply

Data access restrictions in CDI V1 Collection cost charge Commercial charge Licence Moratorium No access Distribution cost Organisation Restricted Academic SeaDataNet licence Unrestricted See Vocabulary L081 for definitions Recommended to use: SeaDataNet Licence!

Business Matrix Restriction Role ABCD U V W OK TBD NO Recommended to use: SeaDataNet Licence!

SeaDataNet – Test CDI V1 records imported Send the test records to the XML Validation Services for validation If ok, then forward the test records to MARIS for inclusion in the CDI V1 test database

SeaDataNet CDI V1 – Data Access example Search Results Include in Basket Shopping list Submit + Authentication Request Confirmed Data Check Status In RSM Ready at DC x Download SDN format

SeaDataNet Download Manager V1.1 Installation Install and configure the Download Manager V1.1 software, following the installation instructions and in discussion with MARIS, who has to register the new data centre in the RSM configuration. At the SeaDataNet Extranet (BSCW) in the Technical Task Team folders you can find all the relevant documentation and tools for the upgrade: CDI V0 – V1 Steps guidance document (see Delivery Services folder) Specifications of the Download Manager (see Delivery Services folder) Technical documentation of the Download Manager (see Delivery Services folder) Installation instruction of the Download Manager (see Delivery Services folder) Software package of the Download Manager V1.1d (see Delivery Services folder) Transition document if you go from V1.1c to V1.1d (see Delivery Services folder)

SeaDataNet Download Manager V1.1 Installation Install the Download Manager software; 2 possible situations You have a file management system You have a database management system In case of file management system: You have to convert data files to SeaDataNet standard formats In case of database management system: Conversion is done ‘on the fly’ by the Download Manager software Fill the ‘coupling table’ to connect the CDI V1 records (via LOCAL_CDI_ID) to the associated files (by name OR by query) Test together with MARIS the set-up and good functioning of the connection via the Test interface

SeaDataNet Download Manager V1.1 Functions Communication with RSM Web service about process progress Processing order lists => retrieving requested data files Retrieving data sets from: file management systems database systems (e.g. Oracle, MySQL, MS-SQL, Sybase, ….) If data sets are retrieved from a database => export of data files in SDN ODV4 format Note: Data sets from a file management system must be pre- processed in SDN transport formats (ODV4 and optional MedATLAS for ASCII type data, or NetCDF (CF) for grid type data) Combining data set files in daily zip files in a user download directory Generating dynamic user download webpage, following SDN template, to give user overview and option of downloadable zip files

SeaDataNet – Converting to SDN Data Formats V1: data sets are accessable by downloading services. Delivery of data sets to users requires common data transport formats, which interact with other SeaDataNet standards (Vocabularies, Quality Flag Scale) and analysis & presentation tools (ODV, DIVA) The following formats have been defined: SeaDataNet ODV4 ASCII for profiles, time series and trajectories SeaDataNet MedAtlas as optional extra format. NetCDF with CF compliance for gridded data sets ODV4 and MedAtlas have been outfitted with a SeaDataNet semantic header NEMO tool ready for conversion of ASCII files to SeaDataNet ODV and MedATLAS Med2MedSDN V1.0 ready for conversion of MEDATLAS files without the SeaDataNet extensions to MEDATLAS files with SeaDataNet extensions (SeaDataNet MEDATLAS)

SeaDataNet CDI V1 – next steps If the DM – RSM connection works fine in the Test system, then you are ready to move to full production and operational status Make full production of CDI V1 records Complete the coupling table Make sure: associated files are ready OR all SQL queries are ok. Possible interim step: full CDI V1 production in the Import system for checking the quality of the CDI V1 records

SeaDataNet CDI V1 – final steps Final step: MARIS moves the production CDI V1 records to the operational CDI V1 system and the data centre is active. Maintenance and updating; only modifications => LOCAL_CDI_ID is important key to recognise updated or new CDI records

SeaDataNet CDI V1 – Quick Search interface Using an intuitive approach which is easy for users

SeaDataNet CDI V1 interim as temporary alternative All Data Centres must populate the CDI V1 database: Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO software V1 interim = no installation of Download Manager, but data centres will be informed by the Request Status Manager by of all requests Data centres have to process all requests manually using the Request Status Manager Delivering data sets in SeaDataNet Formats (conversion via new NEMO tool) – pre-processing required. Effect is, that users will experience a common approach to all data centres. The difference is behind the scenes. Thereafter centres should move to full V1 status by installing DM