1. Towards validating observation data in WaterML 2.0 WATER FOR A HEALTHY COUNTRY You can change this image to be appropriate for your topic by inserting an image in this space or use the alternate title slide with lines. Note: only one image should be used and do not overlap the title text. Enter your Business Unit or Flagship name in the ribbon above the url. [delete instructions before use] Jonathan Yu | Software Engineer 16 th July 2012 Hydroinformatics 2012 An architecture for validating structure and content of WaterML 2.0 documents

2. Outline: 1. Overview of WaterML 2.0 2. Validation problem 3. Proposed validation approach 4. Discussion and challenges 5. Conclusion 2 | Towards validating observation data in WaterML 2.0 | Jonathan Yu

3. Need for water data standards Towards validating observation data in WaterML 2.0 | Jonathan Yu 3 | A data standard is vital

4. WaterML 2.0 International standard XML encoding for transfer of water information Result of harmonization of a number of identified exchange formats Ongoing effort by WMO/OGC Hydro DWG headed up by Peter Taylor from CSIRO Where possible, enhance reusability of standards Towards validating observation data in WaterML 2.0 | Jonathan Yu 4 |

5. What does WaterML 2.0 enable? delivery and consumption of water observations data any Sensor Observation Service (SOS) implementation integration of water observations data with data from closely related domains in environmental sciences such as geology and meteorology, where OGC-conformant systems are being deployed. applications such as –groundwater interoperability –climate monitoring Towards validating observation data in WaterML 2.0 | Jonathan Yu 5 |

6. Validation problem Current implementation target of WaterML 2.0 is XML Common practice is to use XML schema to describe grammar/structure Can we adequately validate WaterML 2.0 using XML schema? XML schema validation is inadequate Towards validating observation data in WaterML 2.0 | Jonathan Yu 6 |

7. XML Schema validation inadequate WaterML 2.0 Information Model Co-constraints XML Schema Towards validating observation data in WaterML 2.0 | Jonathan Yu 7 |

8. Excerpt of Timeseries – default timeValuePair … <wml2:quality xlink:href="http://www.opengis.net/WaterML2.0/def/quality/unchecked" xlink:title="unchecked data"/> <wml2:dataType xlink:href="http://www.opengis.net/WaterML2.0/def/timeseriesType/AveragePrec" xlink:title="Average in preceeding interval"/> <wml2:processing xlink:href="http://www.opengis.net/WaterML2.0/def/processing/raw" xlink:title="As measured data"/> Towards validating observation data in WaterML 2.0 | Jonathan Yu 8 | Content validation may be required

9. WaterML 2.0 Example – Point metadata -41.57583 147.45822 +10:00 AEST Towards validating observation data in WaterML 2.0 | Jonathan Yu 9 | Content validation may be required

10. Actual Points in Timeseries - TimeValuePair 2010-10-10T07:45:00+10:00 0.54 Towards validating observation data in WaterML 2.0 | Jonathan Yu 10 | Content validation may be required

11. How do we go about enhancing XML Schema validation? Option 1: Overload the XML Schema. E.g. Ship vocabulary definitions as static enumerations in the schema. Option 2: Create custom code to handle co-constraints to parse XML and apply constraints checking Opaque, non-standard, reporting format is also non-standard Option 3: Other standards-based constraints checking technology i.e. Schematron Towards validating observation data in WaterML 2.0 | Jonathan Yu 11 |

12. Schematron Schematron is an ISO standard ISO/IEC 19757-3:2006 Information technology -- Document Schema Definition Language (DSDL) -- Part 3: Rule-based validation -- Schematron Has a defined language for reporting: Schematron Validation Report Language (SVRL) We can apply standard transformation on SVRL outputs to further process or convert this report to human readable formats (HTML) or some other machine readable format Towards validating observation data in WaterML 2.0 | Jonathan Yu 12 |

13. Proposed validation service architecture Vocabulary Service RDF Triple Store Validation Service User interface Schematron Rules XSD Validation HTTP REST Interface SKOS/RDF Vocab Interfaces SPARQL Queries WaterML 2.0 Doc Towards validating observation data in WaterML 2.0 | Jonathan Yu 13 | Conformance Certificate Report First pass: XML Schema validation Second pass: Schematron validation - Involves vocabulary checking Report is generated and returned

14. Requirement class: measurement time series exchange Req 1Req 2 Conformance class: measurement time series exchange Conf Test(s) 1 Conf Test(s) 2 Structuring content validation rules Requirement class: measurement time series exchange Req 1 Req 2 Conformance class: measurement time series exchange Conf Test(s) 1 Conf Test(s) 2 Towards validating observation data in WaterML 2.0 | Jonathan Yu 14 | Exchanging water observation data Conformance certification report Use the OGC modular spec to define the WaterML 2.0 requirements classes and the associated conformance classes for validation rules.

15. Wider implications: decoupled architecture Decoupling of vocabulary services allows: Distributed vocabulary services Reference vocabularies to emerge Makes vocabulary services highly reusable for other purposes -Inclusion in validation of other encoding formats (e.g. WaterML 2.0 – P.2. Ratings and gauges?) -Documentation generation, user interface elements Decoupling allows validation service to be generic Adapt for other XML based markup language validation Towards validating observation data in WaterML 2.0 | Jonathan Yu 15 |

16. Potential scenario WDTF Validation WaterML 2.0 Validation SI Units VocService International Authority VocService Aust. Authority VocService BOM Authority VocService Towards validating observation data in WaterML 2.0 | Jonathan Yu 16 |

17. ‘Goldilocks’ of content rule definition Tension in determining content rules to provide out-of-the-box Too constrained: trade-off in flexibility of the format can restrict its usage and be more prescriptive of the use than is required Users not able to express what they want Not constrained enough: greater flexibility yield ‘conformant’ documents that may have problems Working on getting the balance right… Towards validating observation data in WaterML 2.0 | Jonathan Yu 17 |

18. Towards validating observation data in WaterML 2.0 | Jonathan Yu Conclusion and Future work WaterML 2.0 and the validation service the need for standards and appropriate validation mechanism proposed a validation service for schematic and semantic validation enhanced with vocabulary checking importance of the decoupling of validation and vocabularies Future work: balance of content rules - flexible but prescriptive enough develop a set of reference vocabularies for timeseries reporting output to outline the level of conformance according to the WaterML 2.0 specification finding a home for the validation service 18 |

19. Land and Water Jonathan Yu Software Engineer t+61 3 9252 6440 ejonathan.yu@csiro.au wwww.csiro.au/clw ICT Centre Peter Taylor Software Engineer t+61 3 6232 5530 epeter.taylor@csiro.au wwww.csiro.au/ict WATER FOR A HEALTHY COUNTRY Thank you

20. Information models reference controlled vocabularies Towards validating observation data in WaterML 2.0 | Jonathan Yu 20 | WaterML 2.0 Information Model Observations and Measurements Information modelsControlled vocabularies Unit of Measure Vocabs Interpolation Type Vocabs Interpolation Type Vocabs Interpolation Type Vocabs Interpolation Type Vocabs Interpolation Type Vocabs Interpolation Type Vocabs Unit of Measure Vocabs

21. Validating WaterML 2.0 Propose 2-pass method of validation Syntactic level – XML Schemas Content level – Business/Logic Rules 1.XML Schema validation can verify data-types and basic patterns 2.Validating at Content level. This involves checking Valid identifiers –e.g. verify the URI exists Co-constraints and vocabulary checking –e.g. the uom is suitable for the property (expressed as a URI) Towards validating observation data in WaterML 2.0 | Jonathan Yu 21 | <wml2:unitOfMeasure xlink:href="m"/> <wml2:quality xlink:href="http://www.opengis.net/WaterML2.0/de f/quality/unchecked" xlink:title="unchecked data"/> <wml2:dataType xlink:href="http://www.opengis.net/WaterML2.0/de f/timeseriesType/AveragePrec" xlink:title="Average in preceeding interval"/> <wml2:processing xlink:href="http://www.opengis.net/WaterML2.0/de f/processing/raw" xlink:title="As measured data"/> WaterML 2.0 XML Fragment

22. Need for water data standards Towards validating observation data in WaterML 2.0 | Jonathan Yu 22 |

23. Need for water data standards Large number of agencies collecting water information Nationally Internationally Water observation data is a key element of a water resources information system. A data standard is vital Reporting, monitoring and analysis Observational and forecast data Various delivery mechanisms Towards validating observation data in WaterML 2.0 | Jonathan Yu 23 |