Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP regional node and data network for SNDM-Argentina, October 2013, Buenos Aires, Argentina

2 Registration steps Design Registration Maintenance

3 Design Data source storage type DBMS – all types of data; Structured data files – forecasts, climate, small volumes of real-time data; Objective files – images, documents, shape-files, …; Web-application – WMS; Data source structure (data files, tables/views)

4 Data source storage type - DBMS Supported databases: MySQL, PostgreSQL, Oracle, MS SQL Server (native JDBC driver) Table structure requirement: one parameter – one column. Possibility to get the data from one or several tables Filter the data Benefits: Metadata is updating automatically Caching mode Fast processing of large amounts of data. Flexible adjustment of the data.

5 Data source storage type – Structured data files Supported formats: text files with separator or text files with fix column positions. Structure must be ‘plain’, non-hierarchical Options to link data files: Specify URL to HTTP/FTP server, Upload data file to Data Provider server using web interface, Copy data file to local Data Provider file system

6 Data source storage type – Structured data files Benefits: Metadata is updating automatically Caching mode In case of using HTTP/FTP data source it’s enough to update data files, Data Provider will synchronize data, update data cache (instances) and update dynamic metadata (temporal extent, geographic bounding box,…) Connect set of data files to one metadata (specifying all file names or using filename mask (e.g. “*.txt”))

7 Objective data files Supported formats: any data format – Data Provider don’t analyze the content and provide data “as is”. In case if several object data files should be linked to one metadata, they can be zipped to archive. E.g. GIS data (shp, shx, dbf). The ways to link objective data files: specify URL to HTTP/FTP server, upload data files to Data Provider server using web interface

8 Objective data files At least one object data file must be linked to discovery metadata. Linked data files call ‘instances’ and have their own discovery metadata. Metadata for instance based on resource metadata. Features: Once data file is uploaded it can’t be updated automatically: only manually. You should add new data file manually via web-interface Dynamic metadata is not updating automatically by scheduler

9 Design Data granularity issue: a) Split data to several metadata records by geographic box, time period, measured parameter;

10 Design b) Split data to data subsets (instances) by “key” parameter;

11 Data granularity – point data

12 Data granularity – point data T00:00:00;-7.778; ;V2AP6.00;0; T00:00:00;-7.778; ;V2AP6.00;68; T00:00:00;-7.778; ;V2AP6.00;92; T00:00:00;-7.778; ;V2AP6.00;100; T00:00:00; ; ;DCFH2.00;2; T00:00:00; ; ;DCFH2.00;4; T00:00:00; ; ;DCFH2.00;6; T00:00:00; ; ;DCFH2.00;8;17.9

13 Data granularity – profile data

14 Data granularity – profile data Specifying record header

15 Data granularity – profile data Specifying list of parameters in the data table

16 Data granularity – profile data T00:00:00; ; ;V2AP6.00 0; ; ; ; T00:00:00; ; ;DCFH2.00 2;17.9 4;17.9 6;17.9 8;17.9

17 System parameters analysis Make cross-mapping between local data columns and system parameters Contact global node (MINCYT) to add new parameters if required

18 Registration Use web-interface to register metadata and connect the data (Notice: Chrome browser is not supported yet) Use metadata templates Specify maximum details to fit metadata quality requirements

19 Maintenance Update data in-time Update metadata on data frequency basis – create schedulers for automatic metadata actualization Analyze daily reports from Integration Server

20 Questions?