1. CUAHSI HIS Service Oriented Architecture Support EAR 0622374 CUAHSI HIS Sharing hydrologic data http://his.cuahsi.org/ Ilya Zaslavsky, David R. Maidment, David G. Tarboton, Michael Piasecki, Jon Goodall, David Valentine, Thomas Whitenack, Jeffery S. Horsburgh, Tim Whiteaker and the entire CUAHSI HIS Team

2. Data Discovery and Integration Data Publication Data Analysis and Synthesis HydroCatalog HydroDesktopHydroServer ODMGeo Data CUAHSI Hydrologic Information System Services-Oriented Architecture Data Services Metadata Services Search Services WaterML, Other OGC Standards Information Model and Community Support Infrastructure

3. What is a “service oriented architecture”? A design strategy for information systems that enables loose coupling among components Essential relationships and dependencies shall be preserved, non-essential can be discarded Service == unit of work, performed based on a contract between service provider and service consumer – Hides the internal workings of service – Implementation/platform-independent – Presents a relatively simple interface – Can be published, discovered and invoked using this interface Everything is a service: data, models, visualization, …… "Things should be made as simple as possible, but no simpler."

4. What makes an open community- driven hydrologic information system Agreeing on standards for information models and services: WaterML, WaterOneFlow services, OGC specs Making the services easily discoverable, sharing and indexing a lot of quality data: HISCentral Reliable core services: monitoring; logging/reporting; user support; high availability Sharing code: Codeplex, etc. 4

5. WaterML as a Web Language Discharge of the San Marcos River at Luling, June 28 - July 18, 2002 Streamflow data in WaterML language First presented as an OGC Discussion Paper in 2007 Adopted by USGS, NCDC, multiple academic groups, internationally

6. OGC ® 6 M-WRIIMs System Implementation WRIIMs WaterML (Water Markup Language) Site n On site sensor query interface and results water quality, real time and historic data requesting responding Feng-Chia University, Taiwan – Presented 6/16/2011, HydroDWG

7. HIS Central Catalog GetSites GetSiteInfo GetVariableInfo GetValues WaterOneFlow Web Service WaterML Discovery and Access Hydro Desktop Water Metadata Catalog Harvester Service RegistryHydrotagger Search Services http://hiscentral.cuahsi.org Integrates data services from multiple sources Supports concept based data discovery CUAHSI Data Server 3 rd Party Server e.g. USGS

8. Map integrating NWIS, STORET, & Climatic Sites 69 public services 18,000+ variables 1.96+ million sites 23.3 million series Referencing 5.2 billion data values HIS Central Content Available via HISCentral discovery services Available via GetValues requests

9. Growth in GetValues calls for all services reporting to HIS Central May-June 2011

10. Federal Agency Water Data Services at HISCentral Network NameSite Count Value Count (thousands) Earliest ObservationNotes NWISDV 31,800304,000 10/18/1847 WaterML-compliant GetValues service from NWIS, catalog ingested EPA 236,00078,000 01/11/1900 SOAP wrapper over WQX services, catalog ingested NWISUV 11,800169,000 120 DAYS WaterML-compliant GetValues Service, catalog ingested NCDC ISH11,6003,000*1/1/2005 WaterML-compliant GetValues service from NCDC NCDC ISD24,80018,2001/1/1892 WaterML-compliant GetValues service from NCDC NWISIID 376,00086,500 9/1/1867 SOAP wrapper over NWIS web site, catalog ingested NWISGW 834,000 8,4901/1/1800 SOAP wrapper over NWIS web site, catalog ingested RIVERGAGES 1,300264,000 1/1/2000 WaterML compliant REST services from the Army Corps of Engineers * Estimated

11. Hydrologic Ontology http://hiscentral.cuahsi.org/startree.aspx Semantic heterogeneity: water data source use their own vocabularies, which makes it difficult to discover and interpret data Solutions: controlled vocabularies community vocabulary of hydrologic parameters, semantic tagging, and semantic query rewriting acre feetacre-feet micrograms per kilogram micrograms per kilgram FTUNTU mhoSiemens ppmmg/kg Dissloved oxygen

12. HydroTagger Each Variable is connected to a corresponding Concept http://water.sdsc.edu/hiscentral/startree.aspx

13. HISCentral Hosting Facility Redundant Continuously monitored (R-U-On) Synchronized databases Fail over management Monitoring of external servers Usage reporting

14. Service Monitors 14

15. Local CZO DB CZO Data Publication System Spatial, hydrologic, geophysical, geochemical, imagery, spectral… Local CZO DB Web site Standard CZO Services Shared vocabularies CZO Metadata Ontology Archive Harvester Standard CZO data display formats CZO Desktop Matlab R Excel ArcGIS Modeling CZO Desktop Applications CZO Data Products CZO Web-based Data Discovery System External cross- project registries DataNet CZO Data Repository and Indexing (CZO Central)

16. International Standardization of WaterML Hydrology Domain Working Group - working on WaterML 2.0 - organizing Interoperability Experiments focused on different sub-domains of water - towards an agreed upon feature model, observation model, semantics and service stack http://external.opengis.org/twiki_public/bin/view/HydrologyDWG/WebHome Iterative Development Timeline Groundwater IE – GSC+USGS – Dec 09 – Dec 10 Surface Water IE – CSIRO+many – Jun 10 – Sep 11 Forecasting IE – NWS+Deltares? – Sep 11 – Sep 12? Water Quality IE Water Use IE WaterML 2 SWG (Mar 2011) June’11

17. New requirements, and the path forward Transition to OGC model – for better interoperability, including international: what are new service interfaces; how we transition an operational system? Federation of catalogs – since many data providers stand up catalogs, also better scalability: what is the suggested combination of catalog technologies and interfaces? Recognition that we don’t need to search over all services: what are the better search patterns (e.g. 3-step data access: identify services, then extract time series metadata, and then request data content for the time series)? Recognition that we can (and need to) rely on common implementations of mature, modular standard specifications: what is an appropriate operational governance model for distribution of roles and responsibilities within such a modular system?

18. The Migration Path (1) Step 1: Prototyping a new infrastructure and assimilating results of international validation of new OGC specifications : A client developed at UT-Austin that implements the Who (data service providers) – What (variables) - Where (locations) search pattern using OGC CSW and WFS services. The CSW interface provides federation of catalog services, while WFS is used to relay time series catalogs A Kisters WISKI-based client demonstrating access to WFS (for locations of sampling features) and SOS (for observational data encoded in WaterML 2.0), developed as part of Hydrology DWG’s Surface Water IE The Groundwater (2009-2010) and Surface Water (2010-2011) Interoperability Experiments of the OGC/WMO Hydrology Domain Working Group have demonstrated serving water data encoded in WaterML2 using SOS1 and SOS2 services. Goal: smooth transition of the operational HIS

19. The Migration Path (2) Step 2: Settle on a time series catalog information model that can be relayed via common WFS implementations Step 3: Create WFS interfaces over observation networks in the HIS Central catalog, integrated with HIS Central administration interface http://hiscentral.cuahsi.org/wfs/52/cuahsi.wfs?request=getCapabilities http://hiscentral.cuahsi.org/pub_network.aspx?n=52 An additional WFS endpoint for this network An observation network page in HISCentral administration interface for network #52 (Little Bear River)

20. The Migration Path (3) Step 4: Make the networks registry in HISCentral CSW compatible Step 5: Establish a distributed system of federated hydrologic catalogs, using the CSW standard

21. The Migration Path (4) Step 6: Create WaterML2/SOS endpoints, initially for networks already registered in the HIS Central Metadata Catalog at SDSC:

22. The Migration Path (5) Step 7: Integrate the WaterML2/SOS2 endpoints in HydroServer software stack Step 8: Integrate WFS-based series catalog in HydroServer software stack Step 9: Update HISCentral harvesting routines to rely on WFS services Step 10: Update HydroDesktop client to interact with CSW and WFS services –These are to be completed

23. Conclusions HISCentral maintains a large collection of hydrologic time series from distributed data sources, both academic and government – Supports data discovery queries and vocabulary queries – Monitors and validates services – Regular harvesting of registered services – Supports variety of clients – High-availability setup Water data exchange standards are the backbone of HIS SOA: – The specifications have seen wide adoption One of the benefits of SOA: smooth migration to a new set of standards (OGC) Building a community hydrologic information system: – Sharing data and code; reliable core services; access to large volumes of quality data Catalog ServerDesktop Metadata Services Catalog Services Data Services