10 th Argo data management 2009 Toulouse Argo format and CF compatibility OceanOBS09 Strengthen and enhance the international framework under GCOS, GOOS, WCRP, IGBP and supporting regional and national frameworks for sustained world ocean observing and information systems supporting the needs of society about ocean weather, climate, ecosystems, carbon and chemistry A strong message on data management : NetCDF and CF compatibility will enhance data sharing and use. The Argo data format is an implementation of NetCDF by the Argo community. Initiated in December 2001 Version 1.0 adopted in July 2002 CF-1.0 convention became available in October 2003

10 th Argo data management 2009 Toulouse CF metadata convention NetCDF Climate and Forecast (CF) Metadata Convention The conventions for climate and forecast (CF) metadata are designed to promote the processing and sharing of files created with the NetCDF API The CF conventions define metadata that provide a definitive description of what the data in each variable represents, and the spatial and temporal properties of the data. This enables users of data from different sources to decide which quantities are comparable, and facilitates building applications with powerful extraction, regridding, and display capabilities.

10 th Argo data management 2009 Toulouse Argo implementation of CF ? Can we implement CF compatibility without breaking down the data management ? Probably Yes : if we enrich the existing format, with global attributes and more parameter attributes. CF does not standardize NetCDF variable names. It standardizes some attributes of NetCDF variables.

10 th Argo data management 2009 Toulouse Status of the existing profile format CF file checker CHECKING NetCDF FILE: /tmp/D69001_010.nc ===================== Using CF Checker version Using Standard Name Table Version 12 ( T12:05:02Z) Using Area Type Table Version 1 (5 December 2008) WARNING (2.6.1): No 'Conventions' attribute present Checking variable: DATA_TYPE WARNING (3): No standard_name or long_name attribute specified Checking variable: TEMP … ERRORS detected: 0 WARNINGS given: 9

10 th Argo data management 2009 Toulouse Addition needed for a CF-1.4 compatibility Can be done in 3 steps Add a global attribute section Add a long name to all variables Add a standard name and X, Y, Z, T axis attributes to physical parameters Step 1 : add a global attribute section // global attributes: :data_type = "Argo vertical profile" ; :format_version = "2.2"; "2.3" ? :user_manual_version = "2.3" ; :Conventions = "CF-1.4" ; Step 2 : add a long name attribute to all variables char DATA_TYPE(STRING16) ; DATA_TYPE:long_name = "Data type" ;

10 th Argo data management 2009 Toulouse Addition needed for a CF-1.4 compatibility Step 3 : add standard_name and axis attributes double JULD(N_PROF) ; JULD:long_name = "Julian day (UTC) of the station relative to REFERENCE_DATE_TIME" ; JULD:standard_name = "time" ; JULD:units = "days since :00:00 UTC" ; JULD:conventions = "Relative julian days with decimal part (as parts of day)" ; JULD:_FillValue = ; JULD:axis = "T" ;

10 th Argo data management 2009 Toulouse Addition needed for a CF-1.4 compatibility Step 3 : add standard_name and axis attributes double LATITUDE(N_PROF) ; LATITUDE:long_name = "Latitude of the station, best estimate" ; LATITUDE:standard_name = "latitude" ; LATITUDE:units = "degree_north" ; LATITUDE:_FillValue = ; LATITUDE:valid_min = -90. ; LATITUDE:valid_max = 90. ; LATITUDE:axis = "Y" ; double LONGITUDE(N_PROF) ; LONGITUDE:long_name = "Longitude of the station, best estimate" ; LONGITUDE:standard_name = "longitude" ; LONGITUDE:units = "degree_east" ; LONGITUDE:_FillValue = ; LONGITUDE:valid_min = ; LONGITUDE:valid_max = 180. ; LONGITUDE:axis = "X" ;

10 th Argo data management 2009 Toulouse Addition needed for a CF-1.4 compatibility Step 3 : add standard_name and axis attributes float PRES(N_PROF, N_LEVELS) ; PRES:long_name = "SEA PRESSURE" ; PRES:standard_name = "sea_water_pressure" ; PRES:_FillValue = f ; PRES:units = "decibar" ; PRES:valid_min = 0.f ; PRES:valid_max = f ; PRES:comment = "In situ measurement, sea surface = 0" ; PRES:C_format = "%7.1f" ; PRES:FORTRAN_format = "F7.1" ; PRES:resolution = 0.1f ; PRES:axis = "Z" ; float TEMP(N_PROF, N_LEVELS) ; TEMP:long_name = "SEA TEMPERATURE IN SITU ITS-90 SCALE" ; TEMP:standard_name = "sea_water_temperature" ; TEMP:_FillValue = f ; TEMP:units = "degree_Celsius" ; TEMP:valid_min = -2.f ; TEMP:valid_max = 40.f ; TEMP:comment = "In situ measurement" ; TEMP:C_format = "%9.3f" ; TEMP:FORTRAN_format = "F9.3" ; TEMP:resolution = 0.001f ;

10 th Argo data management 2009 Toulouse We can do more with CF… as OceanSITES does We can take advantage of CF possibilities to handle rich description for complex parameters such as Oxygen and bio-geo-chemical parameters. float DOXY(N_PROF, N_LEVELS); DOXY:standard_name = “sea_water_dissolevd_oxygen”; DOXY:units = "micromole/kg"; DOXY:_FillValue = f ; DOXY:long_name = "sea water dissolved oxygen"; DOXY:valid_min = 0.f; DOXY:valid_max = 650.f; DOXY:comment = "in situ measurement"; DOXY:C_format = "%9.3f "; DOXY:FORTRAN_format = " F9.3"; DOXY:resolution = 0.01; DOXY:ancillary_variables = “DOXY_QC,DOXY_ADJUSTED,DOXY_ERROR” ; DOXY:uncertainty = 0.2; DOXY:accuracy = 0.1; DOXY:precision = 0.1; DOXY:resolution = 0.01; DOXY: cell_methods = “median”; DOXY:reference_scale = “???”

10 th Argo data management 2009 Toulouse We can do more with CF… QC flags char TEMP_QC(N_PROF, N_LEVELS) ; TEMP_QC:long_name = "quality flag" ; TEMP_QC:conventions = "Argo reference table 2" ; TEMP_QC:_FillValue = " " ; TEMP_QC: flag_values = "0,1,2,3,4,5,7,8,9" ; TEMP_QC: flag_meanings = "no qc, good, probably good, probably bad,bad,value changed,interpolated value,missing value" ;

10 th Argo data management 2009 Toulouse How to handle more parameters Add new entries in : Reference table 3: parameter code table Remark on duplicate sensors Some floats are equipped with 2 different sensors, measuring the same physical parameter. In that case, add the integer "2" at the end of the code of the duplicate parameter (e.g. DOXY2). If more sensors that measure the same physical parameter are added, then the integer will simply increase by 1 (i.e. DOXY3, DOXY4, and so on). Use “ancillary_variables” attributes to identify relations between different parameters. Example : DOXY:ancillary_variables = “DOXY_QC,DOXY_ADJUSTED,DOXY_ERROR” ;

10 th Argo data management 2009 Toulouse How to handle more parameters Vertical reference issue If parameters have different vertical reference, a proposal is to handle one profile per vertical reference. Example : 2 CTD sensors on a float Temperature from CTD dbar to surface in 70 levels Temperature from CTD2 50 dbar to surface in 100 levels We have 2 profiles, the profile from CTD2 is profile N°2 Add a cycle_profile_number equal to 2

10 th Argo data management 2009 Toulouse How to handle more parameters Bouncing files : a float can perform more than 2 profiles per cycle : Add a cycle_profile_number