1. NEMO – Reformating tool
Michèle Fichaut - Ifremer
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017

2. NEMO [current version 1.6.3]
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO [current version 1.6.3]
Can be downloaded from SeaDataNet Web site
Written in Java Language (Version >= 1.7)
Bilingual (French, English)
Available under multiple environments :
Microsoft : Windows
Unix - Solaris
Linux
Interactive and batch modes available
Use of SeaDataNet common vocabularies web services
to update lists of values of the SeaDataNet common vocabularies
need network connections in order to have up to date lists of values.
But NEMO works offline once the lists are up-to-date

3. EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, 18-19 May 2017
NEMO main features
Reformat ASCII text file of vertical profiles, time-series or trajectories to a SeaDataNet ASCII format (ODV, NetCDF, MEDATLAS).
The input ASCII files can be:
one file per station for vertical profiles or time series
one file for one cruise/collection of vertical profiles, time series or trajectories
Interact with Mikado, to be able to generate ISO XML descriptions of the data
Generate a coupling table that can be used with the Download manager

4. EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, 18-19 May 2017
NEMO principles
NEMO able to read almost any ASCII format to translate it to SeaDataNet formats
Users of NEMO describe the entry files format NEMO able to find information necessary for SeaDataNet formats
Mandatory pre-requirement : set of input files must be homogeneous
be located at the same position : same line in the file, same position on the line or same column if CSV format
be in the same format
For example the measured temperature is:
In the second column of the measured parameters,
from character 10 to character 14

5. Files which cannot be converted by NEMO (1)
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Files which cannot be converted by NEMO (1)
Binary format : like EXCEL, WORD …
File have to be first reformatted to text file : either CSV file, either text with pre-requirements 2
Files which do not respect NEMO pre-requirements
be located at the same position : same line in the file, same position on the line or same column if CSV format
be in the same format
Files which are not :
Vertical profiles (with depth or pressure as vertical reference)
Time series (with time as reference)
Trajectories (with latitude, longitude and time as reference)

6. Files which cannot be converted by NEMO (2)
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Files which cannot be converted by NEMO (2)
"Cruise:archive number numeric","Date and time","Latitude point","Longitude point","Observation level: measured","Temperature water","Salinity","Chlorine","Alkalinity","Nitrate","Nitrite: observed","Nitrogen ammonical","Nitrogen organic","Oxidability","Oxygen: observed","Phosphate","Phosphorus total","pH of the water column","Silicate: observed","Country: name","Organisation: name","Platform: name(eng)","Geographical object: name" "13380","1981_11_17T12:00:00"," "," ","0.00","10.040","7.790","","","86.0","5.0","0.30","","","9.10","5.0","3.9","7.22","370","USSR(Russia)","Klaipeda HM Obs.","OKEANOGRAF","Baltic Sea,North Sea" "13380","1981_11_07T12:00:00"," "," ","10.00","10.040","7.790","","","65.0","5.0","0.40","","","9.26","8.0","4.1","7.66","230","USSR (Russia)","Klaipeda HM Obs.","OKEANOGRAF","Baltic Sea,North Sea“ ……
The extra “” characters have to be deleted  NEMO can be used

7. Files which cannot be converted by NEMO (3)
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Files which cannot be converted by NEMO (3)
CDI;Aktivitet_ID;Cruise;Station;Time;Longitude;Latitude;BotDepth;Type;MinDepth;fMinDepth;MaxDepth;fMaxDepth;Sample_ID;fSample_ID;SamplingArea;fSamplingArea;SubSample_ID;fSubSample_ID;SubCoef;fSubCoef;LatinName;fLatinName;Worms_ID;fWorms_ID;Sex;fSex;Stage;fStage;Count;fCount;LNMAX;fLNMAX;DRYWT%;fDRYWT%;LIPID;fLIPID;LIPID%;fLIPID%;HG;fHG;PB;………….;BBJKF;fBBJKF;BEP;fBEP;BAA;fBAA;ANT;fANT;Modified;Active ;169639;unknown;ARH170020;1998_11_26T00:00:00;10,3455;56,0799;16;B;16;1;16;1;2587;1;;9;61;1;;9;Mytilus_edulis;1;urn:lsid:marinespecies.org:taxname:140480;1;U;1;U;1;;9;44,3;1;18,7;1;0,85;1;1,57;1;0,028;1;0,6;1;0,85;1;87;1;6,1;1;;9;1,6;1;;9;;9;12,1;1;1,7;1;0,7;1;;9;0,99;1;0,31;1;0,77;1;0,25;1;0,15;1;0,41;1;0,09;1;;9;;9;;9;;9;;9;;9;;9;;9;;9;;9;;9;;9;; ;169639;unknown;ARH170020;1998_11_26T00:00:00;10,3455;56,0799;16;B;16;1;16;1;2587;1;;9;62;1;;9;Mytilus_edulis;1;urn:lsid:marinespecies.org:taxname:140480;1;U;1;U;1;;9;40,7;1;18,4;1;0,87;1;1,53;1;0,025;1;0,8;1;0,84;1;91;1;6,6;1;;9;2,2;1;;9;;9;10,6;1;1,8;1;0,8;1;;9;1,22;1;0,42;1;0,94;1;0,31;1;0,18;1;0,41;1;0,09;1;;9;;9;;9;;9;;9;;9;;9;;9;;9;;9;;9;;9;;1
NEMO would need some adaptations to be used

8. Interaction with MIKADO
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Interaction with MIKADO
SeaDataNet Vocabulary
NVS 2
Collection of
ASCII files
SeaDataNet
files
CDI summary
CSV file
CDI
MIKADO
XML
CDI files
summary_CDI_NEMO.xml
Delivered with NEMO
Coupling table
Export
ODV
NetCDF
MEDATLAS

9. Running NEMO 1 step to configure NEMO
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Running NEMO
1 step to configure NEMO
5 steps to convert the entry files to a SeaDataNet format
Describe the type of file(s)
[Describe the cruise, if the files are related to 1 cruise]
NEMO is able to read XML CSR generated with MIKADO
Describe the station information
Describe the measured parameters
Convert the file
One more step to
Save the description of the format (Model)

10. NEMO settings – Select the type of file 2 – Browse the file 1
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO settings
– Select the type of file 2 – Browse the file 1
1 – Only for MedAtlas conversion
2 – Mandatory : EDMO_CODE of the CDI_Partner
3 – If user needs to generate a coupling file
4 – If user wants to generate a csv file used by MIKADO 2 3 4

11. NEMO - Description of input files
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Description of input files
Where are the files?
Is it a cruise?
Is it one file per cruise or one file per station ?
Is it a collection of station files ?
grouped by cruises or not?
Are the files with separators?
Tabulations? Semicolon? Comma?
Are they vertical profiles, time series or trajectories data?

12. NEMO – Open the file Multistation file 1 – Select the type of file
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO – Open the file
Multistation file
1 – Select the type of file
2 – Browse the file
3 - File to convert is displayed in NEMO window 3 2 1

13. NEMO - Describe the input file(s) - 1
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Describe the input file(s) - 1
One Multistation file
1 - describe the type of the input file
2 - select the output format 1 2

14. NEMO - Describe the input file(s) – 2 Multi-station file
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Describe the input file(s) – 2 Multi-station file
Describe how the file is organised:
File header
Station header
End of station
Data termination

15. NEMO - Describe the input file(s) -3
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Describe the input file(s) -3
File header
For multi-station files
Lines which appear once at the beginning of the file
Select
Set
Station header
If there are line(s) with no measurements
before each station data 1 2

16. NEMO - Describe the input file(s) -4
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Describe the input file(s) -4
End of station:
If there is a (or more) specific line after the data in each station
Data termination:
If no specific line, then how does NEMO knows that we change station
Select the indicator
Set 1 2

17. NEMO - Describe the input file(s) - 5
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Describe the input file(s) - 5 1
N monostation files with a constant number of lines for the station header
Select the lines if the station header
Set 2

18. NEMO - Describe the input file(s) - 6
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Describe the input file(s) - 6
N monostation files with a non constant number of lines for the station header
Select the text which identifies the end of the station header
Set 1 2

19. NEMO - Describe the input file(s) - 7
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Describe the input file(s) - 7
When finished
Validate the step
Look at the log
Green:OK, move to the next step 1 2

20. NEMO - Describe the input file(s) - 8
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
NEMO - Describe the input file(s) - 8
When finished
Validate the step
Look at the log
Green:OK, move to the next step
Red: KO, correction needed 1 2

21. Cruise/collection description
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Cruise/collection description
Can be:
1 -Manually input
2 - Imported from a SeaDataNet XML CSR
3 - Imported from a database (IFREMER only)
Mandatory fields depend on the output format 3 2 1

22. Cruise/collection description
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Cruise/collection description
Can be:
1 -Manually input
2 - Imported from a SeaDataNet XML CSR
3 - Imported from a database (IFREMER only)
Mandatory fields depend on the output format
Mandatory
1: Cruise reference
2: Local Cruise ID
3: Cruise name
4: Ship code

23. Cruise/collection description in output ODV
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Cruise/collection description in output ODV
1: Cruise reference
3: Cruise name
4 : Ship code
2: Local Cruise ID

24. Station description Mandatory information
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Station description
Mandatory information

25. Station description 1- Select the date in the file
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017 1
Station description
1- Select the date in the file
2 - Input the date format
3 - Set
4 –Test and check 2 3 4

26. Station description 1- Select the LON in the file
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017 1
Station description
1- Select the LON in the file
2 - Input the LON format
3 - Set
4 –Test and check 2 3 4

27. Data description Choose the parameter list
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Choose the parameter list

28. Data description Choose the parameter list Give the code
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Choose the parameter list
Give the code

29. Data description Choose the parameter list Give the code, the unit
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Choose the parameter list
Give the code, the unit

30. Data description Choose the parameter list
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Choose the parameter list
Give the code, the unit, the position in the line

31. Data description Choose the parameter list
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Choose the parameter list
Give the code, the unit, the position in the file, the format

32. Data description Choose the parameter list
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Choose the parameter list
Give the code, the unit, the position in the file, the format

33. Data description Choose the parameter list
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Choose the parameter list
Give the code, the unit, the position in the file, the format, the default value (1)
Then check (2) 2 1

34. Data description Add all the measured parameters Validate the step
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Add all the measured parameters
Validate the step
Error!!
Depth missing

35. Data description Add depth = 0 (surface) Validate the step Error!!
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Add depth = 0 (surface)
Validate the step
Error!!
Depth must be in the 1st position

36. Data description Add depth = 0 (surface) Validate the step Error!!
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Add depth = 0 (surface)
Validate the step
Error!!
Depth must be in the 1st position

37. Data description Save the model Using Menu – Model > Save
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Data description
Save the model
Using Menu – Model > Save

38. EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, 18-19 May 2017
File conversion

39. Non numeric parameters
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Non numeric parameters
For sediment description in Sediment Core
Compatible with ODV

40. Other additional information
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Other additional information
Instrument used for the measured parameters can be added
Quality flags already existing in the file => if not SeaDataNet flag, mapping is necessary

41. NEMO and MIKADO interaction– CDI summary file NEMO and SDN Download Manager – coupling table
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017

42. While converting … NEMO is able to Generate a SeaDataNet CDI Summary
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
While converting …
NEMO is able to
Generate a SeaDataNet CDI Summary
Text file containing the minimum mandatory information needed in the CDI ISO description
This file can be converted to an Excel file
Which can be read by MIKADO to generate the XML CDI files
Generate a coupling table that will be used by the Download manager of SeaDataNet
It is the link between the LOCAL_CDI_ID and the file

43. EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, 18-19 May 2017
CDI summary
Further information must be added in NEMO settings (menu Options of NEMO)

44. CDI SUMMARY text File (.txt)
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
CDI SUMMARY text File (.txt)

45. Interaction with MIKADO
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Interaction with MIKADO
Collection of
ASCII files
SeaDataNet
files
CDI summary
CSV text file
CDI
MIKADO
XML
CDI files
summary_CDI_NEMO.xml

46. Coupling table for Download Manager
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Coupling table for Download Manager
Further information must be added in NEMO settings (menu Options of NEMO)
Coupling table =
link between each LOCAL_CDI_ID and the file(s) which contains it

47. Coupling table If the “Generate mapping” option is clicked :
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Coupling table
If the “Generate mapping” option is clicked :
NEMO will insert a record in the coupling table for each vertical profile, time series or trajectory that it converts.
The coupling table is managed by a local database imbedded in NEMO
The coupling table can be :
Edited (for modification, insertion or deletion)
exported (to be used by the download manager)
imported (from previous version of NEMO, for example)

48. Coupling table content (1)
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Coupling table content (1)
LOCAL_CDI_ID
Modus
1 : mono-station
3 multi-station
Filename
Date of creation

49. Coupling table content (2)
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
Coupling table content (2)
One unique LOCAL_CDI_ID in one mono-station ODV file
The same LOCAL_CDI_ID in 2 different files :
One multi-station MEDATLAS file
One mono-station ODV file

50. EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, 18-19 May 2017
NEMO in batch mode
NEMO can be run in batch mode, from NEMO install directory, using existing models
Several arguments can be added on the command line
Log file in NEMO install directory

51. EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, 18-19 May 2017
NEMO – User manual
User manual is provided on line on SeaDataNet website:
Detailed explanation for NEMO installation and use, lots of snapshots
Available also on this page
zip file of examples of vertical profiles, time series and trajectories with corresponding NEMO
FAQ
This presentation of NEMO

52. EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, 18-19 May 2017
Next release of NEMO
New management of CSV files as input files of NEMO
Will manage the column number and the “PF_“ temporary files will not exist any more
Management of deprecated parameters : replacement by the new parameter defined by the ‘Replaced by’ attributes in the vocabulary
Bug corrections
Planned released date : Summer 2017

53. New management of the CSV files
EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, May 2017
New management of the CSV files

54. EMODnet Chemistry 3 Kick-off meeting, Trieste, Italy, 18-19 May 2017
Any questions?