1. CUAHSI WaterML Ilya Zaslavsky (SDSC), David Valentine (SDSC), Tim Whiteaker (UT-Austin) /editors/ CUAHSI = Consortium of Universities for the Advancement of Hydrologic Sciences, Inc.;

2. Background CUAHSI HIS: NSF-supported collaborative project: UT Austin + SDSC + Drexel + Duke + Utah State (www.cuahsi.org/his/). PI: David R. Maidment (UT-Austin)www.cuahsi.org/his/ A cyberinfrastructure project Current focus: providing uniform access to heterogeneous observations data, from different agencies Plus an easy way to publish hydrologic observations data And to assemble comprehensive observations databases for your watershed, catchment, etc. From ad hoc SOAP wrappers for agency data to a consistent schema, agency buy-in, and support for multiple clients => WaterML

3. WaterOneFlow Web Services Data accessthrough web services Data storage through web services Downloads Uploads Observatory servers Workgroup HIS SDSC HIS servers 3 rd party servers e.g. USGS, NCDC GIS Matlab IDL Splus, R D2K, I2K Programming (Fortran, C, VB) Web services interface Web portal Interface (HDAS) Information input, display, query and output services Preliminary data exploration and discovery. See what is available and perform exploratory analyses HTML -XML WSDL - SOAP Hydrologic Information System Service Oriented Architecture

4. Super computer Centers: NCSA, TACC Domain Sciences: Unidata, NCAR LTER, GEON Government: USGS, EPA, NCDC, USDA Industry: ESRI, Kisters, OpenMI HIS Team WATERS Testbed WATERS Network Information System CUAHSI HIS The CUAHSI Community, HIS and WATERS CUAHSI: 116 Universities (Nov. 2006) HIS Team: Texas, SDSC, Utah, Drexel, Duke

5. CUAHSI HIS as a mediator across multiple agency and PI data Keeps identifiers for sites, variables, etc. across observation networks Manages and publishes controlled vocabularies (in the Annex to the WaterML paper), and provides vocabulary/ontology management and update tools Provides common structural definitions for data interchange Provides a sample protocol implementation Governance framework: a consortium of universities, MOUs with federal agencies, collaboration with key commercial partners, led by renowned hydrologists, and NSF support for core development and test beds

6. WaterML design principles Driven largely by hydrologists; the goal is to capture semantics of hydrologic observations discovery and retrieval Relies to a large extent on the information model as in ODM (Observations Data Model), and terms are aligned as much as possible Several community reviews since 2005 Driven by data served by USGS NWIS, EPA STORET, multiple individual PI-collected observations Is no more than an exchange schema for CUAHSI web services The least barrier for adoption by hydrologists A fairly simple and rigid schema tuned to the current implementation Conformance with OGC specs not in the initial scope

7. Point Observations Information Model A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water An observation series is an array of observations at a given site, for a given variable, with start time and end time A value is an observation of a variable at a particular time A qualifier is a symbol that provides additional information about the value Data Source Network Sites Observation Series Values {Value, Time, Qualifier} USGS Streamflow gages Neuse River near Clayton, NC Discharge, stage, start, end (Daily or instantaneous) 206 cfs, 13 August 2006 Return network information, and variable information within the network Return site information, including a series catalog of variables measured at a site with their periods of record Return time series of values

8. Observations Data Model Schema (version 4.0) Data Source and Network SitesVariablesValuesMetadata Depth of snow pack Streamflow Landuse, Vegetation Windspeed, Precipitation Controlled Vocabulary Tables e.g. mg/kg, cfs e.g. depth e.g. Non-detect,Estimated, A site is a point location where one or more variables are measured A data source operates an observation network A network is a set of observation sites Metadata provide information about the context of the observation. A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time From Ernest To, David Maidment, CRWR

9. Challenges… (1/2) Sites STORET has stations, and measurement points, at various offsets… Site metadata lacking and inconsistent (e.g. 2/3 no HUC info, 1/3 no state/county info); agency site files need to be upgraded to ODM… A groundwater site is different than a stream gauge… Censored values Values have qualifiers, such as “less than”, “censored”, etc. – per value. Sometimes mixed data types.. Units There are multiple renditions of the same units, even within one repository There may be several units for the same parameter code (STORET) If no value recorded – there are no units?? Unit multipliers E.g. NCDC ASOS keeps measurements as integers, and provides a multiplier for each variable Sources STORET requires organization IDs (which collected data for STORET) in addition to site IDs Time stamps: ISO 8601 A service to determine UTC offsets given lat/lon and date??

10. Challenges… (2/2) Values retrieval USGS: by site, variable, time range EPA: by organization-site, variable, medium, units, time range NCDC: fewer variables, period of record applies to site, not to seriesCatalog Variable semantics Variable names and measurement methods don’t match E.g. NWIS parameter # 625 is labeled ‘ammonia + organic nitrogen‘, Kjeldahl method is used for determination but not mentioned in parameter description. In STORET this parameter is referred to as Kjeldahl Nitrogen. One-to-one mapping not always possible E.g. NWIS: ‘bed sediment’ and ‘suspended sediment’ medium types vs. STORET’s ‘sediment’.  Ontology tagging, semantic mediation  Contolled vocabularies are in Annex A

11. - From different database structures, data collection procedures, quality control, access mechanisms  to uniform signatures … Water Markup Language - Tested in different environments - Standards-based - Can support advanced interfaces via harvested catalogs - Accessible to community - Templates for development of new services - Optimized, error handling, memory management, versioning, run from fast servers - Working with agencies on setting up services and updating site files NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, ODM

12. WaterML key elements Response Types –SiteInfo –Variables –TimeSeries Key Elements –site –sourceInfo –seriesCatalog –variable –timeSeries values –queryInfo GetValues GetVariableInfo GetSiteInfo

13. variables variablesResponse variable 1 many timeSeriesqueryInfo criteria timeSeriesResponse variable sourceInfo queryURL values site queryInfo criteria sitesResponse seriesCatalog siteInfo queryURL variable series variableTimeInterval 1 many Structure of responses

14. DatasetInfoTypeSiteInfoType Elements Defining Spatial Location for observation sitesfor continuous surfaces (other site information) SourceInfoType GeogLocationType (other dataset information) LatLonPointType child elements GeogLocationType LatLonBoxType LatLonPointType

15. SiteInfoResponseType Namespaces queryInfo site Network Sites Variables

16. queryInfo example user parameters query URL Parameters sent to service URLs called (if external resource)

17. siteInfo Name Site Code Location

18. geoLocation geogLocation – geographic coordinates –LatLon point –LatLon box localSiteXY – projected coordinates

19. series variable – what is measured valueCount – how many measurements variableTimeInterval – when is it measured TimePeriodType

20. variable variableCode – global identifier variableName units Sites Variables Values TimePeriodType

21. variableTimeInterval TimePeriodType – date range (including “last n days” TimeInstantType – single measurement Compare with…

22. SiteInfo response queryInfo site name code location seriesCatalog variables what how many when TimePeriodType

23. VariablesResponseType variable – same as in series element Code, name, units Sites Variables Values

24. TimeSeriesResponseType queryInfo timeSeries –sourceInfo – “where” –variable – “what” –values Sites Variables Values

25. sourceInfo SiteInfoType –Same as siteInfo element –code, name, location DataSetInfoType –For data continuous in space –LatLonPointType –LatLonBoxType

26. values Each time series value recorded in value element Timestamp, plus metadata for the value, recorded in element’s attributes ISO Timevalue qualifier Compare with…

27. value metadata examples qualifiers censorCode (lt, gt, nc) qualityControlLevel (Raw, QC’d, etc.) methodID offset –offsetValue –offsetUnitsAbbreviation –offsetDescription –offsetUnitsCode

28. TimeSeries response queryInfo location variable values

29. Clients Tested with.Net and Java Desktop clients: Excel, Matlab, ArcGIS, VB.NET, more being written Web client: DASH (Data Access System for Hydrology): http://river.sdsc.edu/DASH (beta) http://river.sdsc.edu/DASH

30. DASH AGS Server IIS Windows 2003 Server 4 GB Ram 1 TB Disk Quad Core CPU SQL Server VS 2005 WaterOneFlow Web Services ArcGIS 9.2 GIS Data Mxd Service ODM LoaderODM tools ODM Current Deployment Architecture Direct DB connection

31. SQL Server ODMs and catalogs. All instances exposed as ODM (i.e. have standard ODM tables or views: Sites, Variables, SeriesCatalog, etc.) NWIS-IID NWIS-DV ASOS STORET TCEQ BearRiver... Spatial store Geodatabase or collection of shapefiles or both NWIS-IID points NWIS-DV points ASOS points STORET points TCEQ points BearRiver points... My new ODM My new points More databases More synced layers DASH Web Application Background layers (can be in the same or separate spatial store) WOF services Web services from a common template NWIS-IID WS NWIS-DV WS ASOS WS STORET WS TCEQ WS BearRiver WS... My new WS More WS from ODM-WS template USGS NCDC EPA TCEQ Web Configuration file Stores information about registered networks MXD Stores information about layers WSDLs, web service URLs Connection strings Layer info, symbology, etc. ODM DataLoader 2 6 5 3 1 4 WORKGROUP HIS SERVER ORGANIZATION STEPS FOR REGISTERING OBSERVATION DATA

32. New network registration steps Using the ODM DataLoader, load your data into a blank ODM instance (this will create all ODM tables, including Sites, Variables and SeriesCatalog that HIS application relies on) Copy Web Services template to a new folder, edit the template web.config file to point to the new ODM, test to make sure the new service works as expected Create a point layer (a feature class in GDB, or a shapefile) from the new ODM’s Sites table or from GetSites web service (using GetSitesTool – this will also test the service) Add the point layer to the MXD document, specify symbology, scale-dependent rendering, etc. Add information about the new ODM, the associated web service, and the associated point layer, to HIS configuration file (see the first slide for the exact content) Restart the HIS service 6 5 4 3 2 1

33. Near future Need further reviewed, based on initial implementation Within and beyond OGC membership Further engage with hydrologic observation groups at agencies Internationalization (with CSIRO WRON, European WISE, others?) CUAHSI O&M profile? Carry CUAHSI WaterML messages over O&M - need to talk with Simon about it… Test it within an initiative (e.g. Oceans IE, or Water Data Interoperability Testbed, OWS-5? ) Have a list of suggestions for versions after 1.0 (including linear referencing, GetCapabilities, WFS integration, handling multiple siteCodes and variableCodes in a single call) – more suggestions are welcome Divorcing from implementation? Looking for golden middle in standard complexity…

34. OGC Harmonization Best Practices WaterML text includes steps for harmonizing with GML/O&M Align spatial feature descriptions (e.g. using gml:Point, gml:Envelope) Align service signatures (getCapabilities) Align terminology with O&M Provides guidance to other communities on harmonization steps

35. Motion EO/NRE WG recommends to the TC that the CUAHSI WaterML (document 07-041) be released as an OGC Discussion paper Pending minor editorial changes Proposed Second: Ben Domenico Unanimous Roadmap: Reporting initial implementation experience CUAHSI WaterML 1.1, aligning with O&M Testing within an OGC initiative