1. The What, Why and How of the OGC/WMO Hydrology Domain Working Group or How Too Many Dave’s is Never Enough Dr David Lemon (CSIRO) OGC/WMO Hydrology Domain Working Group Workshop JRC Ispra March 2010

2. Helping the World to Communicate Geographically Water

3. Helping the World to Communicate Geographically Water

4. Helping the World to Communicate Geographically However... In many places we can’t asses –How much we have –Where it is –Who owns it –What it is fit for We can’t forecast –How much we will have –Where it will be We certainly can’t share information in a useful timeframe –Expect by private contract

5. Helping the World to Communicate Geographically Increased Irrigation Fire Recovery Expanding Plantations Changing Climate Increasing groundwater extraction Increasing interception Growing Urban Demand Environmental Flows Global water challenges Vertessy 2007

6. Helping the World to Communicate Geographically To address these We will struggle to respond to these unless we know what’s happening. Knowing what’s happening will be difficult without sharing and exploiting interdisciplinary data sources, models, and processes We need interoperability of knowledge and tools, not just the data –This is complicated by: the volumes of data being generated differing semantics institutional barriers

7. Helping the World to Communicate Geographically Standards For some they have a bad reputation –For Developers Take time to develop Must be useful to have impact Need to identify and travel the path to adoption –For Implementers Can be more difficult to implement than custom solutions Need to identify where they are needed Need to identity which ones to use – so many to choose from. These can be overcome and the benefits far outweigh the pain

8. Helping the World to Communicate Geographically Standards Increase Choice Reduce Cost Encourage market competition –Choose based on functionality desired – Avoid “lock in” to a proprietary architecture Decisions to share information and services become policy decisions Needed for technologies to survive the ‘Valley of Death’

9. Helping the World to Communicate Geographically The Hydrology Domain Working Group

10. Helping the World to Communicate Geographically Towards the Hydrology DWG March 2007 –CUAHSI submit WaterML as OGC Discussion paper September 2007 (Canberra, Australia) –Meeting coordinated by NR Canada and CSIRO –34 participants, 18 organisations, (Oceania, Nth America, Europe) –Looked at Current international activities Methodologies, patterns and technologies Futures –Agreed time was right to move forward: IWDIF is born.

11. Helping the World to Communicate Geographically Towards the Hydrology DWG Early 2008 –Much discussion between CSIRO, CUAHSI, OGC and WMO September 2008 (Atlanta) –First ad-hoc –Presentations from CUAHSI and CSIRO –Discussion about value of group –Agreed to develop charter (CSIRO lead). December 2008 (Valencia) –3 Presentations (CSIRO, GRDC)

12. Helping the World to Communicate Geographically Towards the Hydrology DWG Early 2009 –CUAHSI and CSIRO starting working on WaterML2.0 March 2009 (Athens) –6 presentations (EEA, INSPIRE, WMO, GRDC, CUAHSI, CSIRO –TC agrees to Charter The OGC\WMO Hydrology Domain Working Group is formed

13. Helping the World to Communicate Geographically Since Formation June 2009 (Boston) –First official meeting –Groundwater IE proposed September 2009 (Darmstadt) –Presentation of Water Data Exchange Formats Harmonisation Report December 2009 (Mountainview) –Surface Water IE proposed

14. Helping the World to Communicate Geographically Since Formation March 2010 (Ispra) –24 participants, 19 organisations, (UN, Oceania, Nth America, Europe) –Vendors, Government, Research –Looking at Methodologies, patterns and technologies Current activities Futures

15. Helping the World to Communicate Geographically The OGC A non-profit, international, voluntary consensus standards organisation developing standards for geospatial and location based services Focussed on technology standards –Eg. Web Map Service, Web Feature Service, Observations and Measurements, KML, SensorML, … 350+ members including: GIS vendors (ESRI), large software vendors (Oracle, Microsoft), government agencies (USGS, NASA, US DOD), systems integrators, … Key relationships with other standards groups (ISO, W3C, WMO) Work undertaken by interested participants through Domain and Standards Working Groups

16. Helping the World to Communicate Geographically WMO ‘... the UN system's authoritative voice on the state and behaviour of the Earth's atmosphere, its interaction with the oceans, the climate it produces and the resulting distribution of water resources’ ‘... facilitates the free and unrestricted exchange of data and information, products and services in real- or near-real time...’

17. Helping the World to Communicate Geographically Hydrology Domain Working Group A joint working group of the OGC and WMO constituted as an OGC Domain Working Group. Brings together interested parties to develop and promote the technology for greatly improving the way in which water information is described and shared. Co-chaired by representatives nominated by the OGC TC and the World Meteorological Organisation’s (WMO) Commission for Hydrology (CHy). Current Co-Chairs: Ilya Zaslavsky (SDSC), Ulrich Looser (GRDC) and David Lemon (CSIRO) > 50 Participants, > 30 Organisations

18. Helping the World to Communicate Geographically Charter To provide a venue and mechanism for seeking technical and institutional solutions to the challenge of describing and exchanging data describing the state and location of water resources, both above and below the ground surface. The path to adoption will be through OGC papers and standards, advanced to ISO where appropriate, and also through the WMO’s CHy and WIS activities. The Hydrology DWG will provide a means of developing candidate standards for adoption by CHy as appropriate.

19. Helping the World to Communicate Geographically © 2009 Open Geospatial Consortium, Inc. 19 Expected Outcomes An agreed feature model (ie. what are the features of the hydrosphere (from an information perspective) and how are they related.) An agreed observation model. Agreed vocabularies, endorsed by the community, and by WMO in particular. Agreeing on semantics is a long process, but we should be able to recommend some vocabularies Also: services carrying the above

20. Helping the World to Communicate Geographically Activities Analysis and harmonization of existing commonly used protocols and standards in the domain Interoperability Experiments (IEs) focused on selected sub-domains of water data (e.g. surface water, groundwater, water quality, water use, hydrologic forecasts, real time data) Participation in GEO Architecture Implementation Pilots (AIPs), focused on broader scenarios within the entire domain and across domains Submission of discussion papers and best practice papers (the latter following IE and AIP experiences) Collaboration with other DWGs, in particular ESS (on modeling), MetOcean (on handling gridded data), SWE (on real time data management) © 2009 Open Geospatial Consortium, Inc. 20

21. Helping the World to Communicate Geographically © 2009 Open Geospatial Consortium, Inc. 21 Timeline September 2009: Discussion paper on harmonization of water data exchange formats (1st version) ; IE1 announced December 2009: Update on the harmonization report; Proposed WaterML 2.0 schema announced; IE1 officially launched, IE 2 announced June 2010: WaterML 2.0 submitted as a discussion paper ; IE2 launched (to be reported 3/11); Ontology task force started (with GRDC, CSIRO, CUAHSI, others) ; IE1 preliminary report; December 2010: IE1 final report, a best practice paper based on the results June 2011: IE2 final report, and a paper

22. Helping the World to Communicate Geographically Current Activities

23. Helping the World to Communicate Geographically Groundwater Interoperability Experiment (Boyan Brodaric – GCS, Nate Booth – USGS)

24. Helping the World to Communicate Geographically  Groundwater is an increasingly significant source of water  Ground/water feature and observation data - distributed: many data providers (~1000 in Canada alone) - heterogeneous: structure and content (e.g. level, flow, quality) - massive instances: millions of wells (features/sensors) & time series (observ.) - massive schema: thousands of properties (e.g. for quality) - few standard services: very few OGC gr/water services No standard way to share groundwater information Context

25. Helping the World to Communicate Geographically Sensor sites and well locations

26. Helping the World to Communicate Geographically Groundwater feature and observation interoperability: 1.Advance design of WaterML 2 schema harmonize existing schemas [WaterML (USA), GroundwaterML (CAN), WTDF (AU), WQX (USA), OGC O&M, SensorML, …] 2.Advance fit of OGC services with groundwater data SOS, WFS, WMS, CSW 3.Advance groundwater data exchange between US & Canada in cross-border Great Lakes area (Lake Superior) for water wells and levels Objectives

27. Helping the World to Communicate Geographically 1.Advance design of WaterML 2 schema merge or link existing schemas (WaterML, O&M, GWML)? when to restrict: what is special about groundwater observations? when not to restrict: does it need to be extensible? 2.Advance fit of OGC services with gr/water data SOS 1: not designed for massive networks (i.e. GetCapability lists each sensor) CSW: include individual sensors? 3.Advance GW data exchange between US & CAN what is a groundwater observation? 1/n sensors, 1/n time points, 1/n time series, 1/n properties? Challenges

28. Helping the World to Communicate Geographically  View US & Canadian well/sensor locations - WMS (GetMap)  Query on one well/sensor and display data - WMS (GetFeatureInfo) - well: WFS (getFeature via BBOX) or - level: SOS (GetObservation via BBOX) - return GWML or WaterML2.0 and display as HTML  Download cross-border wells and levels - Zoom to area of interest (find BBOX) - well: WFS (GetFeature via BBOX and thematic filter) or - level: SOS (GetObservation via BBOX and time filter) - Export as GWML, WaterML2.0, Excel, or KML Use Cases

29. Helping the World to Communicate Geographically  Dec 2009: Milestone 1 - preliminary WaterML 2 schema, example instances, documents (CSIRO) - preliminary services: SOS & WFS (GSC), SOS (USGS), CSW (SDSC) - preliminary demo of use-case  June 2010: Milestone 2 - intermediate results at OGC TC  Dec 2010: Final Milestone - final results - technology demo - change requests re: OGC specs - wrap and report at OGC TC and AGU 2010 Dates, Workplan Outputs

30. Helping the World to Communicate Geographically query view Milestone 1 Results download

31. Helping the World to Communicate Geographically WaterML 2.0 (Peter Taylor - CSIRO)

32. Helping the World to Communicate Geographically Harmonised water data model “At least one of the people out there has a smarter idea about what to do with your content than you do.” - James Boyle

33. Helping the World to Communicate Geographically Harmonised water data model The need for standards Sharing of hydrological data is somewhat ad-hoc, based largely on ad-hoc formats, CSV files and FTP Leads to very inefficient (or non-existent) transfer and sharing of data Sharing requires a lot of transformation effort on the behalf of the data requestor –Parse different file formats –Understand the concepts –Metadata is often limited or non-existent, requiring follow up conversations : JAN RESERVOIR DATA 900 AM CST THU MAR 28 1985.B JAN 0328 C DH06/HP/HW/PP/QT : STN ID POOL TAILW PCPN DISCH K CFS JSNM6 295.71 / 259.51 / 0.50 / 4.300 : BARNETT DNNM6 410.1 / 330.3 /.31 / : BAY SPRINGS L&D ABEM6 190.1 / 163.6 /.17 / : ABERDEEN L&D CBUM6 163.5 / 137.1 /.39 / : COLUMBUS L&D ARKM6 210.1 / /.07 /.650 : ARKABUTLA SRDM6 242.8 / /.10 /.250 : SARDIS ENDM6 235.1 / /.37 /.100 : ENID GRNM6 197.6 / /.70 /.100 : GRENADA BMDA4 / / 0.00 / 6.393 : BLAKELY, AR DGDA4 / /.10 / 2.634 : DEGRAY, AR NARA4 / / 0.00 / 1.244 : NARROWS, AR.END

34. Helping the World to Communicate Geographically Harmonised water data model A common situation Need flow data! Hmm maybe Don can help… *RING RING* Hi Don, I need some upper Derwent flow readings for my geochemical model. Any ideas? Don Hmm, I’ve got one site. I’ll send it through… 10 minutes… To: Jack 01/02/09, 3.2, 3, 1 01/02/09, 3.1, 3, 1 10 minutes… *RING RING* Ok. Got the data. Where is the site located? Oh, it’s at laughing jack bridge. Coordinates? Ummm. (papers shuffle) 147.123 -41.588 What reference system?? I think it’s GDA94 Ok. What sensor is used? It’s calculated from the stream gauge reading using a rating curve.. Oh…how accurate is that? Umm...... DON? Hydro Jack *CLICK*

35. Helping the World to Communicate Geographically Harmonised water data model Hydrological observations Measurements of –Water level, –Water temperature, –Discharge, –Sediment, –Water quality –... Different sampling techniques (in-situ, laboratory etc.) Why do people want to share hydrological observations? –Provide data to national systems (AWRIS, CUAHSI) –Share data between agencies Hydro company to local water authority Researchers (learning hydrology, modelling) –Other domains Climatology Oceanography Meteorology

36. Helping the World to Communicate Geographically Harmonised water data model The need for harmonisation A number of initiatives exist to develop standards for water observation data: –WaterML1.0, 1.1 –WDTF, WOML –XHydro –UK Environmental Agencies time series transfer format –SANDRE –...; A lot of commonalities exist between the standards –Partial duplication of effort –Inconsistencies across the standards –Hard to re-use tools Potential for a harmonised, re-usable information model –Define common semantics of concepts –See Document 09-124 (Harmonising standards for water observation data)

37. Helping the World to Communicate Geographically Harmonised water data model The need for a common model Sharing can be easy; understanding is difficult.

38. Helping the World to Communicate Geographically Harmonised water data model A proposed approach Identify common concepts within existing hydrological data standards Identify use cases/patterns/structures of hydrological observation data Identify methodologies that can be leveraged to assist the development of a harmonised standard Re-use existing definitions of concepts where appropriate – don’t re-invent the wheel

39. Helping the World to Communicate Geographically Harmonised water data model Re-use existing standards Sensor Web Enablement –SWE Common –Sensor Observation Service –Observations & Measurements GML & ISO –Primitive types –Spatial Allows us to focus on the domain concepts rather than types that are fairly common across domains (units, spatial types etc.)

40. Helping the World to Communicate Geographically Harmonised water data model OGC Observations & Measurements Provides a definition of common concepts and relationships of the observation process A baseline from which we can develop a specific model for hydrological observations Appears to be the emerging best practice, used by: –Integrated Ocean Observing System (IOOS) –Climate Science Markup Language (CSML) –Global Runoff Data Centre’s (GRDC) meta data profile –Submitted to ISO

41. Helping the World to Communicate Geographically Harmonised water data model Model-driven approaches A model-driven approach to information modeling allows specification of model in UML and various encodings can be generated (e.g. GML XML) Also an emerging best practice in ISO211 for developing specifications for spatial data (ISO19101, ISO19109) Tools exist that facilitate the development process: –FullMoon for encoding UML into GML-compliant XML –HollowWorld to allow re-use of exisiting standards in UML –SolidGround to manage model

42. Helping the World to Communicate Geographically Harmonised water data model UK Environmental Agency – TS format Sampling features Time series

43. Helping the World to Communicate Geographically Harmonised water data model Sampling Features Result type (time series) Observed property Procedure

44. Helping the World to Communicate Geographically Harmonised water data model Benefits of a harmonised model Access to a wider range of developed tools, libraries and applications – reduced duplication of effort Adapters for existing systems can be shared Access to documented and explicit definitions of concepts Helps pave the way for increased interoperability of water information systems

45. Helping the World to Communicate Geographically Harmonised water data model Going forward - issues How well do existing site/location/station/measuring point concepts fit into the model? How do we provide common grouping mechanisms that satisfy existing and future discovery requirements? How do we describe our sampling regimes? How do we encode observation results (time series etc.)? Managing dependencies between derived time series? Define a core set of use cases to address

46. Helping the World to Communicate Geographically Harmonised water data model Development of a harmonised model will bring benefits to a large number of parties and will promote improved sharing of data and meta data, by both humans and computers.

47. Helping the World to Communicate Geographically Harmonised water data model The future? Hydro Jack Laughing Jack Bridge 147.123 -41.588 (AGD66) Upper Derwent River Observed properties: River level, River Flow, Rainfall River Flow --Measured by Sontek Acoustic Doppler Sensor (ID232) --Accuracy: +/- 0.02 --Installed: 01/04/2012 Need flow data!

48. Helping the World to Communicate Geographically Future Activities?

49. Helping the World to Communicate Geographically Surface Water IE (Peter Fitch – CSIRO)

50. Helping the World to Communicate Geographically Objectives of IE Extend and complement the work already underway in GW IE –Further develop WaterML2.0 for surface water Test compatibility of WaterML 2.0 with existing participants infrastructure. Advance exchange of surface water data between Germany and France in the cross-border area of the Rhine/Rhin river. Advance exchange of surface water data between Mexico and USA in the cross-border area of xx. The data will be served by all participants using WaterML 2.0 and OGC services (SOS, WFS, WMS). Copyright © 2008, Open Geospatial Consortium, Inc., All Rights Reserved.

51. Helping the World to Communicate Geographically Participants (so far) OrganisationIdeas for IEContributions IOW-SandreCross border quality/quantity data exchange Data and services 52° NorthService implementations Federal WaterCatchment oriented approach/ cross border data exchange Data and services CUAHSICross US/Mexico border data exchange TBA KistersCheck feasibility/suitability of service definitions Consume and provide services CSIRODevelopment of WaterML2.0 IMTA (Mexican Institute of Water Technology) Data and services

52. Helping the World to Communicate Geographically Model Integration and Workflows?

53. Helping the World to Communicate Geographically Insert presentation title Current Practices in Hydrology The team hydrologist’s initial specification

54. Helping the World to Communicate Geographically Insert presentation title A use case: current practices Input data checks

55. Helping the World to Communicate Geographically Insert presentation title A use case: current practices Create calibration tasks

56. Helping the World to Communicate Geographically Insert presentation title A use case: current practices Calibration runs

57. Helping the World to Communicate Geographically Insert presentation title A use case: current practices Output analysis

58. Helping the World to Communicate Geographically Insert presentation title Issues with current practices Heavy reliance on individuals and manual steps for the ‘process’ to sustain itself –Lack of resilience, critical sections –Poor use of valuable skills (copy-paste in Excel); increased risk of non-reproducible errors Dispersed information: emails, post-its, script logs, sharepoint, ‘wiki sites’, personal documents, brains, etc. Heterogeneity of toolsets and capabilities; islands of specialisation with manual transmission in-between –XML documents –R, Matlab, Python Scripts; Excel Idiosyncratic access arrangements Basic traceability requires “off-line” documentation.

59. Helping the World to Communicate Geographically... And Finally

60. Helping the World to Communicate Geographically Too many Dave’s? David Maidment (CUAHSI) David Valentine (CUAHSI) David Tarboton (CUAHSI) David Kinny (BOM) David Schell (OGC) David Arctur (OGC) David Thomas (WMO) David Lemon (CSIRO) Peter Taylor (CSIRO) Peter Fitch (CSIRO) Peter Gijsbers (Deltares) Pierre Lagarde (BRGM) Pierre Kerherve (WMO)

61. Helping the World to Communicate Geographically Public mailing list https://lists.opengeospatial.org/mailman/listinfo/hydro.dwg https://lists.opengeospatial.org/mailman/listinfo/hydro.dwg Twiki http://external.opengis.org/twiki_public/bin/view/HydrologyDWG http://external.opengis.org/twiki_public/bin/view/HydrologyDWG Thank you...