1. CUAHSI Hydrologic Information System: web services and related technologies HIS PIs: David Maidment Ilya Zaslavsky David Tarboton Michael Piasecki Jon Goodall and Rick Hooper (CUAHSI) WaterML & Web Services team: David Valentine Tom Whitenack Tim Whiteaker Matt Rodriguez www.cuahsi.org/his/

2. Test bed HIS Servers Central HIS servers ArcGIS Matlab IDL MapWindow Excel Programming (Fortran, C, VB) Desktop clients Customizable web interface (DASH) HTML - XML WSDL - SOAP Hydrologic Information System Service Oriented Architecture Modeling (OpenMI) Global search (Hydroseek) WaterOneFlow Web Services, WaterML Controlled vocabularies Metadata catalogs Ontology ETL services HIS Lite Servers External data providers Deployment to test beds Other popular online clients ODM DataLoader Streaming Data Loading Ontology tagging (Hydrotagger) WSDL and ODM registration Data publishing ODMTools Server config tools

3. CUAHSI Member Institutions 112 Universities as of June 2007

4. HIS Goals Hydrologic Data Access System – better access to a large volume of high quality hydrologic data Support for Observatories – synthesizing hydrologic data for a region Advancement of Hydrologic Science – data modeling and advanced analysis Hydrologic Education – better data in the classroom, basin-focused teaching

5. Project co-PI in Phase 2 Collaborator in Phase I CUAHSI HIS Partner Institutions

6. 10 WATERS Testbed Sites

7. Windows 2003 Server 4 GB Ram 500 GB Disk Quad Core CPU SQLServer 2005 IIS (Internet Information Server) DASH ArcGIS Server Visual Studio 2005 WaterOneFlow Web Services ArcGIS 9.2 GIS Data Mxd Services ODDataLoader ODM tools COTS Software HIS Applications Data NWIS DV NWIS IID Your ODM

9. HIS Team WATERS Testbed WATERS Network Information System

10. Super computer Centers: NCSA, TACC Domain Sciences: Unidata, NCAR LTER, GEON Government: USGS, EPA, NCDC, USDA Industry: ESRI, Kisters, OpenMI HIS Team WATERS Testbed WATERS Network Information System CUAHSI HIS International Partners CSIRO Land and Water Resources Water Resources Observations Network (WRON) European Commission Water database design and model integration (HarmonIT and OpenMI)

11. Digital Watershed How can hydrologists integrate observed and modeled data from various sources into a single description of the environment?

12. Rainfall & Snow Water quantity and quality Remote sensing Water Data Modeling Meteorology Soil water

13. Water Data Web Sites

14. NWISWeb site output # agency_cd Agency Code # site_no USGS station number # dv_dt date of daily mean streamflow # dv_va daily mean streamflow value, in cubic-feet per-second # dv_cd daily mean streamflow value qualification code # # Sites in this file include: # USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC # agency_cdsite_nodv_dtdv_vadv_cd USGS020875002003-09-011190 USGS020875002003-09-02649 USGS020875002003-09-03525 USGS020875002003-09-04486 USGS020875002003-09-05733 USGS020875002003-09-06585 USGS020875002003-09-07485 USGS020875002003-09-08463 USGS020875002003-09-09673 USGS020875002003-09-10517 USGS020875002003-09-11454 Time series of streamflow at a gaging station USGS has committed to supporting CUAHSI’s GetValues function

15. Observation Stations Ameriflux Towers (NASA & DOE)NOAA Automated Surface Observing System USGS National Water Information SystemNOAA Climate Reference Network Map for the US

16. Water Quality Measurement Sites in EPA Storet Substantial variation in data availability from states Data from Bora Beran, Drexel University

17. Water Quality Measurement Sites from Texas Commission for Environmental Quality (TCEQ)

18. Geographic Integration of Storet and TCEQ Data in HIS

19. Point Observations Information Model A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water An observation series is an array of observations at a given site, for a given variable, with start time and end time A value is an observation of a variable at a particular time A qualifier is a symbol that provides additional information about the value Data Source Network Sites Observation Series Values {Value, Time, Qualifier} USGS Streamflow gages Neuse River near Clayton, NC Discharge, stage, start, end (Daily or instantaneous) 206 cfs, 13 August 2006 Return network information, and variable information within the network Return site information, including a series catalog of variables measured at a site with their periods of record Return time series of values

20. Observations Catalog Specifies what variables are measured at each site, over what time interval, and how many observations of each variable are available

21. Observations Data Model Schema (version 4.0) Data Source and Network SitesVariablesValuesMetadata Depth of snow pack Streamflow Landuse, Vegetation Windspeed, Precipitation Controlled Vocabulary Tables e.g. mg/kg, cfs e.g. depth e.g. Non-detect,Estimated, A site is a point location where one or more variables are measured A data source operates an observation network A network is a set of observation sites Metadata provide information about the context of the observation. A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time From Ernest To, David Maidment, CRWR

22. WaterOneFlow Set of query functions Returns data in WaterML NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)

23. WaterML key elements Response Types –SiteInfo –Variables –TimeSeries Key Elements –site –sourceInfo –seriesCatalog –variable –timeSeries values –queryInfo GetValues GetVariableInfo GetSiteInfo

24. variables variablesResponse variable 1 many timeSeriesqueryInfo criteria timeSeriesResponse variable sourceInfo queryURL values site queryInfo criteria sitesResponse seriesCatalog siteInfo queryURL variable series variableTimeInterval 1 many Structure of responses

25. More Information about WaterML… next 20 slides… /we may skip them/ Or check the specification online at http://www.opengeospatial.org/standards/dp

26. DatasetInfoTypeSiteInfoType Elements Defining Spatial Location for observation sitesfor continuous surfaces (other site information) SourceInfoType GeogLocationType (other dataset information) LatLonPointType child elements GeogLocationType LatLonBoxType LatLonPointType

27. SiteInfoResponseType Namespaces queryInfo site Network Sites Variables

28. queryInfo example user parameters query URL Parameters sent to service URLs called (if external resource)

29. siteInfo Name Site Code Location

30. geoLocation geogLocation – geographic coordinates –LatLon point –LatLon box localSiteXY – projected coordinates

31. series variable – what is measured valueCount – how many measurements variableTimeInterval – when is it measured TimePeriodType

32. variable variableCode – global identifier variableName units Sites Variables Values TimePeriodType

33. variableTimeInterval TimePeriodType – date range (including “last n days” TimeInstantType – single measurement Compare with…

34. SiteInfo response queryInfo site name code location seriesCatalog variables Series how many when TimePeriodType

35. VariablesResponseType variable – same as in series element Code, name, units Sites Variables Values

36. TimeSeriesResponseType queryInfo timeSeries –sourceInfo – “where” –variable – “what” –values Sites Variables Values

37. sourceInfo SiteInfoType –Same as siteInfo element –code, name, location DataSetInfoType –For data continuous in space –LatLonPointType –LatLonBoxType

38. values Each time series value recorded in value element Timestamp, plus metadata for the value, recorded in element’s attributes ISO Timevalue qualifier

39. value metadata examples qualifiers censorCode (lt, gt, nc) qualityControlLevel (Raw, QC’d, etc.) methodID offset –offsetValue –offsetUnitsAbbreviation –offsetDescription –offsetUnitsCode

40. TimeSeries response queryInfo location variable values

41. OGC Harmonization Best Practices WaterML text includes steps for harmonizing with GML/O&M Align spatial feature descriptions (e.g. using gml:Point, gml:Envelope) Align service signatures (getCapabilities) Align terminology with O&M

42. More on OGC Interactions WaterML is published as OGC Discussion paper We are working with OGC O&M (Observations & Measurements) authors to reconcile WaterML with OGC specs: –WOML proposal (Water Observations ML) Think of it as WFS + O&M type service (with GetSiteInfo proxied as OGC’s GetFeatureInfo request) –The plan is to provide WOML-compliant services alongside WaterML 1.1 –There is an international group that Dave Valentine is coordinating, focused on developing a standard water information exchange schema (started at WaterML workshop in Canberra in September’07) As we are going with OGC through this harmonization, we shall be able to assist WQX with similar mappings (more below on mapping between WaterML and WQX)

43. Hydrologic Information Server Microsoft SQLServer Relational Database Observations Data & Catalogs Geospatial Data GetSites GetSiteInfo GetVariables GetVariableInfo GetValues DASH – data access system for hydrologyWaterOneFlow services ArcGIS Server

44. SQL Server ODMs and catalogs. All instances exposed as ODM (i.e. have standard ODM tables or views: Sites, Variables, SeriesCatalog, etc.) NWIS-IID NWIS-DV ASOS STORET TCEQ BearRiver... Spatial store Geodatabase or collection of shapefiles or both NWIS-IID points NWIS-DV points ASOS points STORET points TCEQ points BearRiver points... My new ODM My new points More databases More synced layers DASH Web Application Background layers (can be in the same or separate spatial store) WOF services Web services from a common template NWIS-IID WS NWIS-DV WS ASOS WS STORET WS TCEQ WS BearRiver WS... My new WS More WS from ODM-WS template USGS NCDC EPA TCEQ Web Configuration file Stores information about registered networks MXD Stores information about layers WSDLs, web service URLs Connection strings Layer info, symbology, etc. ODM DataLoader 2 6 5 3 1 4 WORKGROUP HIS SERVER ORGANIZATION STEPS FOR REGISTERING OBSERVATION DATA

45. New network registration steps Using the ODM DataLoader or other, load your data into a blank ODM instance (this will create all ODM tables that HIS relies on) Copy Web Services template to a new folder, edit the template web.config file to point to the new ODM, test to make sure the new service works as expected Create a point layer (a feature class in GDB, or a shapefile) from the new ODM’s Sites table using the GetSitesTool. Add the point layer to the MXD document, specify symbology, scale-dependent rendering, etc. Add information about the new ODM, the associated web service, and the associated point layer, to HIS configuration file (see the first slide for the exact content) Restart the HIS service Register and test the new service at the HIS Central: http://water.sdsc.edu/centralhis/ 6 5 4 3 2 1 7

46. Central HIS Data Services Catalog

49. Beyond Syntactic Uniformity: Semantic Mediator What we are doing now ….. NWIS NAWQA NARR generic request GetValues GetValues ODM Michael Piasecki Drexel University

50. Hydroseek http://www.hydroseek.net Supports search by location and type of data across multiple observation networks including NWIS, Storet, and university data

51. Semantic Tagging of Harvested Variables

52. CUAHSI HIS as a mediator across multiple agency and PI data –Keeps identifiers for sites, variables, etc. across observation networks –Manages and publishes controlled vocabularies, and provides vocabulary/ontology management and update tools –Provides common structural definitions for data interchange –Provides a sample protocol implementation –Governance framework: a consortium of universities, MOUs with federal agencies, collaboration with key commercial partners, led by renowned hydrologists, and NSF support for core development and test beds

53. US Map of USGS Observations Antarctica Puerto Rico Hawaii Alaska

54. Different types of nutrients by decade: Available Data Total

55. Some physical properties by decade: Available Data Total

56. Measurement Counts by Decade

57. Measurement counts by Decade in the Western States

58. STORET Datacube The EPA STORET datacube contains 273K sites and 2.7M series. Florida is the source of about 25% of the total records. Number of years of record by start decade About 60% of the water quality records are short term measurements (one year or less in duration). The starting decade of the longer series are to the right. 93% of the series are water quality data. Demo