OBSERVATIONS & PRÉVISIONS CÔTIÈRES 3 rd SeaDataNet training course – Ostende – June 2008 Tools for data management in the frame of SeaDataNet project M. Fichaut

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 2 Training Workshop – Ostende – October 2009 CDI CSR EDMERP EDMED SEADATANET PORTALSEADATANET PORTAL European portal Collection of ASCII files Format X Data in Database Metadata in Database Data Input Metadata Input Metadata In Excel files Partner system : pilot data centre SeaDatanet Vocabulary NEMO ODV Med2MedSDN Coupling table Download Manager Collection of ASCII files Format SDN XML Metadata Files Local copy of data to download Data request Data download MIKADO XML Validator

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 3 Training Workshop – Ostende – October 2009 CDI CSR EDMERP EDMED SEADATANET PORTALSEADATANET PORTAL European portal Collection of ASCII files Format X Data in Database Metadata in Database Data Input Metadata Input Metadata In Excel files Partner system : other data centre SeaDatanet Vocabulary NEMO ODV Med2MedSDN Collection of ASCII files Format SDN XML Metadata Files Data download Local copy of data to download Data request by Manual preparation of data MIKADO XML Validator

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 4 Training Workshop – Ostende – October 2009 Overview (1) MIKADO Objectives and technical characteristics of Mikado Mikados main features Manual input of metadata Use of the common SDN vocabularies Automatic XML generation Connection to the local database Query writing Mapping to the local database Batch mode Coupling table for download manager XML VALIDATION SERVICE

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 5 Training Workshop – Ostende – October 2009 Overview (2) Med2MedSDN Objectives and main features NEMO Objectives, main features and principles of NEMO Technical characteristics Description of the different steps to follow to be able to reformat ASCII files to SeaDataNet format Link between NEMO and MIKADO Link between NEMO and SDN Download Manager – coupling table

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 3 rd SeaDataNet training course – Ostende – June 2008 MIKADO Tool for the generation of XML descriptions of SeaDanaNet catalogue records

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 7 Training Workshop – Ostende – October 2009 Objective MIKADO is used to generate XML catalogue descriptions, it creates XML ISO files using SDN common vocabularies for metadata exchange of CSR - Cruise Summary Reports EDMED - Marine Environmental Data sets CDI - Common Data Index EDMERP - Marine Environmental Research Projects [EDIOS – Permanent Ocean-observing System] Is freely available on SeaDataNet Web site

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 8 Training Workshop – Ostende – October 2009 Technical characteristics Written in Java Language (Version >= 1.6) Available under multiple environments : Microsoft : Windows 2000, XP, VISTA, APPLE Unix - Solaris Linux. Use of the SeaDataNet common vocabularies web services to update lists of values needs network connection in order to have up to date lists of values. but Mikado works offline once the lists are up-to-date

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 9 Training Workshop – Ostende – October 2009 MIKADO main features (1) MIKADO can be used in 2 different ways One manual way, to input manually information for the catalogues and CDI in order to generate XML files. One automatic way, to generate XML descriptions automatically, from information catalogued in a relational database or in an Excel file. Automatic way is needed for those who have many entries referenced in a relational database Only one interface for all catalogues Same look Same principles

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 10 Training Workshop – Ostende – October 2009 MIKADO main features (2) MIKADO Java code Native Drivers MYSQL ORACLE POSTGRES SYBASE MSServer Bridge Drivers using Microsoft ODBC (ACCESS, EXCEL, SQL SERVER) DATABASE JDBC Java DataBase Connectivity EXCEL File XML files for SeaDataNet catalogues CSR EDMED EDMERP CDI [EDIOS] Manual Automati c Other Drivers Downloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory)

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 11 Training Workshop – Ostende – October 2009 MIKADO and SDN vocabulary lists (2) List code List Name List code List Name C16SeaDataNet Sea AreasL061SeaDataNet Platform Classes C77ICES ROSCOP data typesL071SeaDataNet data access mechanisms C174SeaDataNet CSR ship metadataL081SeaDataNet Data Access Restriction Policies C320ISO countriesL101SeaDataNet geographic co-ordinate reference frames C371Ten-degree Marsden SquaresL111Height and Depth Vertical Co-ordinate Reference Datum C381Ports GazetteerL181ROSCOP sample quantification units L05SeaDataNet device categoriesL231SeaDataNet metadata entities L021SeaDataNet Geospatial Feature TypesL241SeaDataNet data transport formats L031SeaDataNet Measurement Periodicity ClassesP021BODC Parameter Discovery Vocabulary EDMERPEuropean marine projectsP081SeaDataNet Parameter Disciplines EDMOEuropean marine organisation

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 12 Training Workshop – Ostende – October 2009 MIKADO and SDN vocabulary lists (2) Automatic check of the version of the vocabulary lists : once when MIKADO starts If On is clicked in the Vocabulary Update Menu MIKADO downloads locally the latest version of each list Possible to enable-disable the automatic check If Off is clicked Manual check Update once now

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 13 Training Workshop – Ostende – October 2009 MIKADO – Manual input Available for 4 catalogues : EDMED, CSR, CDI, EDMERP Each input generates an XML file that can be sent to the central catalogue For EDMERP and CSR : EDMERP CMS and CSR online can also be used, but MIKADO is useful if you have problems with the NETWORK connection if you want to keep locally an XML description of your catalogues For EDMED and CDI, there is no online input tools.

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 14 Training Workshop – Ostende – October 2009 MIKADO – Manual input

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 15 Training Workshop – Ostende – October 2009 MIKADO manual : LOCAL Identifier The LOCAL Identifier is vital because it is kept in the central catalogue and is the entry point to know if the record is new or if it is an update. This LOCAL ID exists for all the catalogues and is under the responsibility of the data centre who generates the XLM descriptions

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 16 Training Workshop – Ostende – October 2009 MIKADO manual – vocabulary lists (1) Common vocabulary BODC, list C320 (Country ISO codes)

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 17 Training Workshop – Ostende – October 2009 MIKADO manual – vocabulary lists (2)

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 18 Training Workshop – Ostende – October 2009 MIKADO – Automatic XML generation Principle Read the information about CSR, EDMED, EDMERP or CDI in a database or in an Excel file MIKADO has predefined variables which correspond to the XML tags definition for each catalogues MIKADO helps user to write the SQL orders to fulfill these variables with the information available in the database or in the Excel file

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 19 Training Workshop – Ostende – October 2009 MIKADO – Automatic XML generation 4 STEPS Connect to a database or an Excel file and test the connection Write the queries to retrieve information in the database or in the Excel file, test the queries Save the queries in a Configuration file Generate the XML files using the Configuration file

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 20 Training Workshop – Ostende – October 2009 MIKADO – Automatic XML generation

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 21 Training Workshop – Ostende – October 2009 MIKADO automatic - connection Help for the connection to the database Pre-filled information for some databases Check of the connection

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 22 Training Workshop – Ostende – October 2009 MIKADO automatic – connection OK Green message in the Check box

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 23 Training Workshop – Ostende – October 2009 MIKADO – automatic – connection KO Red message in the Check box

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 24 Training Workshop – Ostende – October 2009 Expendable trees Main query Return the LOCAL ID Single queries Return 1 row Multiple queries Return 1 to n rows Single an multiple queries related to each LOCAL ID returned by the main query. MIKADO automatic – queries

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 25 Training Workshop – Ostende – October 2009 Write the queries SQL syntax (for Oracle, Excel, MySQL, …) and SQL variables must be adapted to your own data base Check the Queries Green OK Red KO : read the error message MIKADO automatic – queries

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 26 Training Workshop – Ostende – October 2009 MIKADO automatic – single queries All the XML variables are listed in the expendable tree In bold : mandatory fields 1 to n single subquery can be written In green : fields already fulfilled Add or delete variables in a query Delete a full query Check the query

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 27 Training Workshop – Ostende – October 2009 MIKADO automatic – multiple queries All the XML variables are listed in the expendable tree Number of queries is pre-defined The list of variables for each of these multiple queries is also pre- defined In bold : mandatory field In green : fields already fulfilled

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 28 Training Workshop – Ostende – October 2009 MIKADO automatic – multiple queries In a group of variables (same XML block, same pre-defined set of variable in a query), the non mandatory variables can be left to null

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 29 Training Workshop – Ostende – October 2009 MIKADO automatic - Save the queries When all the queries are written They can be saved in an XML file to be re-used later on

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 30 Training Workshop – Ostende – October 2009 MIKADO automatic - generate the XML files 1.Select the catalogue you want to generate 2.Open the corresponding configuration file 3.Choose the output directory 4.Choose the type of export files 5.Export the XML files Progress bar Cancel allowed

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 31 Training Workshop – Ostende – October 2009 While generating the XML files for all the catalogues Each time that MIKADO does not recognized a value (entrykey or entryterm) which should come from the common vocabulary, it asks the user for mapping MIKADO manages a demand-driven continuous (incremental) extension of a local mapping : mapping of the local database to the common vocabulary Mapping tables can be modified Delete rows Modify the LOCAL value MIKADO automatic - local mapping

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 32 Training Workshop – Ostende – October 2009 MIKADO automatic - local mapping Example CSR generation Mapping of the platform type

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 33 Training Workshop – Ostende – October 2009 MIKADO automatic - local mapping Modification of the local mapping If wrong entries have been input, it is possible to: Delete one entry Delete all the entries Change the local code

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 34 Training Workshop – Ostende – October 2009 MIKADO in batch mode MIKADO can be run in batch mode using existing configuration files Several arguments can be added on the command line Java –Djava.endorsed.dirs=dist/lib –jar dist/Mikado.jar mikado-home=[path] argument2= … argumentn= Log file to register the errors

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 35 Training Workshop – Ostende – October 2009 Batch mode : mandatory arguments (1) batch-type : XmlFiles, Zipfile, Both batch-mode : CDI, EDMERP, CSR, EDMED conf-file : path and name of the configuration file with the SQL queries output-dir : path and name of the output directory where the XML files will be written continue-on-error : true or false If true : if one record with mapping missing, or one record with mandatory field(s) null in the database, warning for this record in the logfile, and MIKADO processes next records.. If false : MIKADO stops at the first error.

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 36 Training Workshop – Ostende – October 2009 Batch mode : optional arguments (2) log-file : Path and name of the Log file of MIKADO. By default, mikado.log in Mikado home directory trace : to have a summary of the time response of each SQL query (useful for tuning of the queries bad if time responses) max-file-in-zip : number maximum of XML files per zip file generated by Mikado. By default, one zip file contains 1000 XML files. zip-prefix : To personalize the names of the Zip files generated by Mikado. By default the zip file name are SeaDataNet_[catalogue]_[x].zip

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 37 Training Workshop – Ostende – October 2009 Coupling file for Download manager (1) MIKADO is able to generate this coupling file The coupling file is used by SeaDataNet download manager to make the mapping between a LOCAL_CDI_ID (one profile, one time-series or one trajectory) and the name of the file containing this LOCAL_CDI_ID (MODUS1 and 3) : if the metadata is in a data base and the data in files the SQL Query to retrieve the meta data and the data of this LOCAL_CDI_ID in the local database (MODUS 2)

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 38 Training Workshop – Ostende – October 2009 Coupling file for Download manager (2) The principle to create this coupling file is the same than to create XML files for catalogue descriptions User has to create a configuration file that will be used for the generation of the coupling file. to write the queries to retrieve the filename or the data for each LOCAl_CDI_ID

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 39 Training Workshop – Ostende – October 2009 Coupling file for Download manager (3)

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 40 Training Workshop – Ostende – October 2009 MIKADO – User manual User manual is provided : File : SDN_MIKADO_UserManual_v1_1_4.pdf Detailed explanation for MIKADO use, lots of snapshots

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 41 Training Workshop – Ostende – October 2009 ANY QUESTION?

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 42 Training Workshop – Ostende – October 2009 VALIDATION of XML files XML validation Services have been developed in the frame of SeaDataNet by the Russian NODC It is a Web validation Service available at

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 3 rd SeaDataNet training course – Ostende – June 2008 XML validation tool Web service for validation of the XML files generated by MIKADO

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 44 Training Workshop – Ostende – October 2009 Select the catalogue you want to validate

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 45 Training Workshop – Ostende – October 2009 View all CDI schema versions View statistics about detected errors Run the validation with the last version of XML schema

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 46 Training Workshop – Ostende – October 2009 Upload the file you want to validate Validation works only file per file

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 47 Training Workshop – Ostende – October 2009 The file is displayed on the screen Press Validate

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 48 Training Workshop – Ostende – October 2009 Errors and warning are displayed Warning are not obstacles for XML delivery Errors must be corrected

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 49 Training Workshop – Ostende – October 2009 ANY QUESTION?