Project Status Brief to NOAA IOOS Program February 18, 2011 Implementing QA/QC Standards for In Situ Ocean Sensors Using OGC-Sensor Web Enablement a.k.a. QARTOD to OGC a.k.a. Q2O Project Status Brief to NOAA IOOS Program February 18, 2011

Project Goals: Implement semantic interoperability through use of registered terms, including references to authoritative, published methods Implement syntactic interoperability through strict compliance with SWE Common (SWE 1.0) Describe and encode QARTOD recommended tests Enable dynamic quality assessment through access to sensor history, processing history and results of QC tests

Community activities were addressing individual parts… OOSTethys/OpenIOOS

Bringing together community members as Q2O Team Janet Fredericks - WHOI, MMI, MVCO, QARTOD Mike Botts/Tony Cook - UAH, OGC SWE Julie Bosch - NOAA, MMI, IOOS DMAC, QARTOD Harvey Seim/Sara Haines - SECOORA, NCCOOS Philip Bogden/Eric Bridger - GoMOOS, IOOS DMAC, SURA, MMI, OOSTethys, OpenIOOS OIE = Ocean Interoperability Experiment

Funded by NOAA CSC/IOOS (January 2008 – December 2010) What is Q2O? Funded by NOAA CSC/IOOS (January 2008 – December 2010) Deliverables: Implement the QARTOD recommendations into the OGC Sensor Web Enablement framework providing SensorML profiles for QARTOD tests and Documenting results by providing a tutorial and Test the deliverables by implementing services at participating data centers Methodology: Bring together IT specialists with domain experts (for waves, in situ currents, CTD observations and Dissolved Oxygen) Partner with community building projects such as OOSTethys and MMI

What does that mean? What do we have (know) to start with? A sensor (wave buoy or ADCP) with certain characteristics A sensor history QA info associated with a sensor Deployment characteristics Methods to process the data QC Tests to apply to the data … What information can we provide to data users via systems (OOSTethys, OpenIOOS)? What sensors we have available as a service Description of the sensor Description of where / how / when it is deployed List of the processing methods used on the data List of the QC tests applied The criteria used in the QC tests The results of the QC tests The data … How do we convey that information in SOS? Get Capabilities Lists available data offerings Returns SensorML Describe Sensor Provides sensor and deployment characteristics and processing methods Returns SensorML Get Observation Provides the data, test results and points to file with processing/test info Returns O&M What does that mean? Julie Bosch

What had/has to be done by Q2O? Engage QARTOD/Domain experts Gather QARTOD information identify recommendations Define Processes input / output / criteria Develop/Register vocabularies Build SWE instances Test implementation Develop Guidance QARTOD - WAVES - QC RECOMMENDED TESTS Graphic - QARTOD materials on the CDIP web site, example of one of the items gathered to identify/define the recommendations for waves QC.

Most QARTOD generic tests, parameters and flags have been defined, terms have been registered (MMI/ORR) and SensorML encodings have been created to describe them into OGC/SWE instances.

The details… developing vocabularies

The details… on the Q2O Project website http://q2o.whoi.edu - materials available to public - account access (working materials)

Using MMI Vine tool to generate relationship between Q2O Boolean QC flag and IGOSS flagging convention

Original Equipment Manufacturer (OEM) file MVCO ADCP -> waves Processing has been fully described: Model Specific Files Original Equipment Manufacturer (OEM) file input (observable properties), output (observed properties), Identifiers (what is it? An RDI Workhorse 1200), classifiers (acoustic Doppler, etc) characteristics (size, accuracy, etc) Parameters (what is adjustable and what are the options) Contact (manufacturer) The Setup and Deployment File Similar to the OEM – but Contact (operator) historyEventList (sensorDeployment, sensorChange, pressurePort, cleanedFaces) these are time-stamped and include time-stamped references to Serial Numbers, firmware versions, calibrations etc relating to the sensor history

Links OEM/Setup-Deployment files with the processing SensorML files MVCO ADCP -> waves Processing has been fully described: 2. ADCP_System File -- Links OEM/Setup-Deployment files with the processing SensorML files MVCO_Observatory.xml – Provides an overview of the MVCO GO to the new Q2O MVCO page and to the MVCO to see how it all fits together. Notice …. Different offerings from same data set … From ADCP_System system file notice the reference to properties (like windWaves and how well content can be defined!) and how the files are linked together (see Components in table – and links in SensorML documents Also note PARAMS capability .. As a way to have seasonal changes to QC Tests

Other Current Works in Progress (<6/30/2011) (see deliverables page) UNC Sonde – OEM file; SD file … plus ingesting data from NetCDF file (where the QC-flags are served) Guides – in progress Standards Document – in progress Fully-describe (SensorML) RDI-NEMO processing – USF/COMPS in situ current demo (now just the describeSensor file – buoy is offine) Incorporate Q2O into OOSTethys Cookbooks to the extent possible

Lessons learned vocabulary development (use template to be thorough even if it’s not flushed out); include references, figures, equations when possible Categorize the terms: observableProperties, properties, parameters, process, platforms … registration of vocabulary/ontology so terms can be mapped across authorities SWE process chains, parameters, components,…How important it is to really define processing … you need input, output and parameters defined and well constructed descriptions. it’s HARD – need manufacturer’s help for OEM/SD files … and better tools for editing and registering SensorML documents …

Some Very Positive Outcomes -- Worked with QARTOD community with test definitions for ADCP Waves and Currents -- Developed a vocabulary and registered it. -- Worked with MMI as they developed ontology registry and repository. -- Designed and built up the structure of data flow in SensorML of system of sensors before we could begin working on implementing QC tests -- Work with Teledyne RDI established example and guidance for other manufactures to describe sensor in SensorML. -- Had to build work-around for some limitation of SensorML v1.0 -- provided valuable feedback for development of SensorML v2.0

What's next? NEXT STEPS (after 6/30/2011) Demonstrate (eg., OpenIOOS) that the services can be ingested and we can indeed perform dynamic quality assessment! Upgrade Q2O to SWE 2.0 (when it’s released) Develop more OEM/SD files for common oceanographic sensors. Continue to develop forms and tools to generate SensorML documents to fully describe sensors! SUGGESTIONS?? (and Thank you!!)