CTD Data Processing Current BIO Procedure

Current Processing Software Matlab Migrating to R & Python Code Version Control SVN Migrating to GitHub

The quality control of STD data is based on the algorithms found in the UNESCO manual: UNESCO (1990) GTSPP Real-Time Quality Control Manual, Intergovernmental Oceanographic Commission, Manuals and Guides 22, SC/90/WS-74, 121 pp. The thresholds of certain GTSPP tests have been adapted to conditions found in the estuary and Gulf of St. Lawrence. Several additional tests were added to improve the overall quality control of the STD profile. The current quality control procedure validates the pressure (or depth), temperature, salinity, and sigma-T data as well as the principal metadata concerning the time–space coordinates of the CTD profile. Eventually other variables, such as dissolved oxygen, fluorescence, and light transmission, could also be included. Text taken from IML CTD QC Manual

The quality control procedure is divided into five steps: Step 1: Tests validating the important metadata such as the time and position Step 2: Tests comparing data values within a profile Step 3: Comparison of the profile to a climatology Step 4: Comparison of the profile to other profiles from the same mission Step 5: Visual inspection of the cruise track and of the profiles themselves All metadata can be modified, in particular the time–space coordinates, without making the profile unusable. ** No data are modified by the quality control procedure. During the step 2 tests, quality flags are added (see the next section) to qualify the data as good, doubtful, erroneous, or missing. If the data must be modified for some reason, these modifications are made outside the quality control procedure and the quality flags must be adjusted in consequence. ** Historically at BIO, bad data has always been removed from CTD files. This fiscal year we are implementing flagging. A future goal of this upgrade will be going back through historical data files as time allows to fix them so that removed data is reinserted with the appropriate flag assigned.

Quality Control Checks Example (IML code modified for use at BIO) >> run_mli_checks_HUD MISSION INFORMATION Mission Number: HUD Chief Scientist: DAVE HEBERT Platform: HUDSON Mission Start Date: 20-SEP :00:00.00 Mission End Date: 11-OCT :59:59.00 COMMENTS RESULTS FROM THE QUALITY CONTROL BIO Implementation of the GTSPP 2010 Real-Time Quality Control Tests Test 1.0 Time Sequence (IML) -> ok Test 1.1 Platform Identification (GTSPP) -> ok Test 1.2 Impossible Date/Time (GTSPP) -> ok Test 1.3 Impossible Location (GTSPP) -> ok Test 1.4 Position on Land (GTSPP) -> ok Test 1.5 Impossible Speed (GTSPP) -> ok Stage 5: Visual Inspection..... Cruise Track Inspection -> Please wait for plot to load...

Stage 2: Profile Tests for file: CTD_HUD _001_1_DN.ODF Tests on file: CTD_HUD _047_1_DN.ODF Test 2.1 Global Impossible Parameter Values -> ok Test 2.1 Regional Impossible Parameter Values -> ok Test 2.3 Increasing Depth -> ok Test 2.4 Profile Envelope -> ok Test 2.5 Freezing Point -> ok Test 2.7 Spike in Temperature and Salinity (one point) -> ok Test 2.8 Top and Bottom Spike in Temperature and Salinity (one point) -> ok Test 2.9 Gradient (point to point) -> ok Test 2.10 Density Inversion (point to point) -> Density inversion found

A new History Header was added to each ODF file during processing to indicate what was done to the file. HISTORY_HEADER, CREATION_DATE='03-MAR :59:05.38', PROCESS='The following edits were completed by JEFF JACKSON.', PROCESS='Edit Cruise: Field "Organization" has been changed from "BIO" to "DFO BIO".', PROCESS='Edit Cruise: Field "Chief_Scientist" has been changed from "Dave Herbert" to "DAVE HEBERT".', PROCESS='Edit Cruise: Field "Start_Date" has been changed from "17-NOV :00:00.00" to "20-SEP :00:00.00".', PROCESS='Edit Cruise: Field "End_Date" has been changed from "17-NOV :00:00.00" to "11-OCT :59:59.00".', PROCESS='Edit Cruise: Field "Platform" has been changed from "Hudson" to "HUDSON".', PROCESS='Edit Cruise: Field "Cruise_Name" has been changed from "AZMP Scotian Shelf" to "SCOTIAN SHELF AND SLOPE".', PROCESS='Edit Cruise: Field "Cruise_Description" has been changed from " " to "ATLANTIC ZONE MONITORING PROGRAM (AZMP)".', PROCESS='The following Parameter was removed from the ODF file: PSAR_01', PROCESS='The following Parameter was removed from the ODF file: PHPH_01', PROCESS='The primary conductivity calibration coefficient "Offset" was changed from its original value [0.0] to [ ].', PROCESS='The primary conductivity calibration coefficient "Slope" was changed from its original value [1.0] to [1.0014].', PROCESS='The secondary conductivity calibration coefficient "Offset" was changed from its original value [0.0] to [ ].', PROCESS='The secondary conductivity calibration coefficient "Slope" was changed from its original value [1.0] to [1.0010].', PROCESS='The primary oxygen (#3026) was replaced with sensor (#0042) after Event 19.', PROCESS='The primary oxygen (#3026) calibration coefficient "Soc" was changed from its original value of [ e-001] to [4.4504e-001].', PROCESS='The primary oxygen (#0042) calibration coefficient "Soc" was changed from its original value of [ e-001] to [4.8410e-001].', PROCESS='The secondary oxygen (#3030) calibration coefficient "Soc" was changed from its original value of [ e-001] to [4.6940e-001].', PROCESS='Please note the secondary oxygen (DOXY_02) should be used instead of the primary oxygen (DOXY_01) because its data is of better quality.', PROCESS='Edit Record: Field "Num_Calibration" has been added to the RECORD_HEADER and set to value 0.', PROCESS='Edit Record: Field "Num_Swing" has been added to the RECORD_HEADER and set to value 0.'

HISTORY_HEADER, CREATION_DATE='03-MAR :22:41.30', PROCESS='Quality control of the CTD data contents performed by Jeff Jackson', PROCESS='Nulled the ALTB_01 value "99.92" corresponding to the pressure value "132" in record "131"', PROCESS='Nulled the ALTB_01 value "99.77" corresponding to the pressure value "148" in record "147"', PROCESS='Nulled the ALTB_01 value "99.98" corresponding to the pressure value "232" in record "231"', PROCESS='Nulled the ALTB_01 value "99.94" corresponding to the pressure value "251" in record "250"', PROCESS='Nulled the ALTB_01 value "100.45" corresponding to the pressure value "279" in record "278"', PROCESS='Nulled the ALTB_01 value "100.32" corresponding to the pressure value "297" in record "296"', PROCESS='Nulled the ALTB_01 value "99.79" corresponding to the pressure value "314" in record "313"', PROCESS='Nulled the ALTB_01 value "100.14" corresponding to the pressure value "323" in record "322"', PROCESS='Nulled the ALTB_01 value "100.24" corresponding to the pressure value "352" in record "351"', PROCESS='Nulled the ALTB_01 value "100.19" corresponding to the pressure value "361" in record "360"', PROCESS='Nulled the ALTB_01 value "102.85" corresponding to the pressure value "379" in record "378"', PROCESS='Nulled the ALTB_01 value "101.61" corresponding to the pressure value "426" in record "425"', PROCESS='Nulled the ALTB_01 value "2.64" corresponding to the pressure value "2" in record "1"', …

Some of the functionality required for new Visual QC tool: Plot Profile and/or Time-Series Plot both Down & Up traces for profiles if applicable Plot adjacent stations – allowing the user to specify how many before and after or the range of events to plot. Pick points and assign flags and make the points different colors based on the flag chosen. Option to remove points from plot but not from file. What else?

A Current BIO Goal: Load all CTD ODF files into an Oracle Database Tested the functionality using the languages: Matlab, R and Python The test involved reading 10 ODF files (1.68 MB) and loading them into Oracle tables. The resulting elapsed times were: Matlab : seconds R : seconds Python : seconds