CUAHSI Hydrologic Information System an introduction Ilya Zaslavsky Director, Spatial Information Systems Lab San Diego Supercomputer Center University of California San Diego Presentation at DID Data Management, Kuala Lumpur, Malaysia, July 24, 2009

San Diego Supercomputer Center Founded in 1985, as one of the five original supercomputer centers, funded by the National Science Foundation 400 employees Advanced research in high- performance computing and networking R&D and cyberinfrastructure projects: in neuroscience, geology, astronomy, environmental sciences, molecular biology, hydrology SDSC building on UCSD campus

Consortium of Universities for the Advancement of Hydrologic Science, Inc. An organization representing more than one hundred United States universities, receives support from the National Science Foundation to develop infrastructure and services for the advancement of hydrologic science and education in the U.S US Universities

CUAHSI HIS: NSF support through 2012 (GEO) Partners: Academic: 11 NSF hydrologic observatories, CEO:P projects, LTER… Government: USGS, EPA, NCDC, NWS, state and local Commercial: Microsoft, ESRI, Kisters International: Australia, UK Standardization: OGC, WMO (Hydrology Domain WG, CHy); adopted by USGS, NCDC An online distributed system to support the sharing of hydrologic data from multiple repositories and databases via standard water data service protocols; software for data publication, discovery, access and integration. What is CUAHSI HIS?

Observation Stations Ameriflux Towers (NASA & DOE)NOAA Automated Surface Observing System USGS National Water Information SystemNOAA Climate Reference Network Map for the US Build a common window on water data using web services

Rainfall & Snow Water quantity and quality Remote sensing Water Data Modeling Meteorology Soil water

Sources of Observations Data

Point Water Observations Time Series A point location in spaceA series of values in time

Getting Water Data (the old way) Different Query PagesDifferent Query Responses

Web Pages and Web Services Uses Hypertext Markup Language (HTML) Uses WaterML (a Markup Language for water data)

HTML as a Web Language Text and Pictures in Web Browser Texas Water Development Board HyperText Markup Language

WaterML as a Web Language Discharge of the San Marcos River at Luling, June 28 - July 18, 2002 Streamflow data in WaterML language

Point Observations Information Model Data Source Network Sites Variables Values {Value, Time, Metadata} Utah State Univ Little Bear River Little Bear River at Mendon Rd Dissolved Oxygen 9.78 mg/L, 1 October 2007, 5PM A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time Metadata provide additional information about the value GetSites GetSiteInfo GetVariableInfo GetValues WaterOneFlow Service

Site Codes Variable Codes Date Ranges WaterML and WaterOneFlow GetSites GetSiteInfo GetVariableInfo GetValues WaterOneFlow Web Service Client DEC UVM USGS Data Repositories Data EXTRACT TRANSFORM LOAD WaterML WaterML is an XML language for communicating water data WaterOneFlow is a set of web services based on WaterML

Standard Water Data Services Set of query functions Returns data in WaterML NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites) Next Step: OGC-WMO Hydrology Domain Working Group: WaterML Contact: Ilya Zaslavsky, co-chair

Test bed HIS Servers Central HIS servers ArcGIS Matlab IDL, R MapWindow Excel Programming (C#, VB..) Desktop clients Customizable web interface (DASH) HTML - XML WSDL - SOAP Modeling (OpenMI) Global search (Hydroseek) Water Data Web Services, WaterML Controlled vocabularies Metadata catalogs Ontology ETL services HIS Lite Servers External data providers Deployment to test beds Other popular online clients ODM DataLoader Streaming Data Loading Ontology tagging (Hydrotagger) WSDL and ODM registration Data publishing ODMTools Server config tools HIS Central Registry & Harvester Hydrologic Information System Service Oriented Architecture HIS Desktop

Central HIS Data Services Catalog

Semantic Tagging of Harvested Variables

Hydroseek Supports search by location and type of data across multiple observation networks including NWIS, Storet, and academic data

Against the NIH Syndrome 2006: ► CUAHSI HIS web services are discussed on the BASINS mailing list as a new way to access hydrologic data. The list is mostly used by hydrologists and developers outside academia; ► NCDC develops ASOS web services following WaterML 2007: ► MOU with USGS; USGS is developing WaterML-compliant GetValues service; ► GLEON uses an early version of ODM to develop their own schema (VEGA); ► Phoenix LTER is developing ODM (in MySQL) and WaterML services (in Java); ► A Google Earth-based client for CUAHSI web services is developed at CSIRO, Australia; ► Deployment to 11 hydrologic observatory test beds, + CBEO (CEOP project) : ► KISTERS develops WaterML-compliant web services over their database; ► Workshops at state agencies ► MapWindow open source GIS develops WaterOneFlow parsers; ► Florida, Texas and Idaho use ODM and WaterOneFlow web services to provide access to state data repositories; New Jersey is considering the same; ► Another CEOP project, at UC-Davis, is implementing ODM (in Postgres) and web services (in Java); ► Stroud Water Research Center; WRON; CZO; … many that we don’t know… ► Now SBRP: data from UCSD, UA, more? ► Integration with streaming data middleware (Open Source Data Turbine)

The International Workshop on Hydrologic Data Management and Modeling in South East Asia July University of Malaya Learning how the system works Publishing hydrologic data Setting up a server for SEA Already published: sample data from JPS (Malaysia) and from Indonesia

Data published as web service: These are results of GetValues for JPS: , Streamflow data In HydroExcel

Charts of the same data In HydroExcel

Area of interest In HydroSeek

Finding JPS stations In HydroSeek

More information about JPS stations In HydroSeek, and data download

JPS data downloaded from HydroSeek

Zooming in on Indonesia

Looking for COD measurements In HydroSeek

Zooming in to stations

Summary CUAHSI HIS = Cyberinfrastructure for managing and publishing observational data –Supports many types of point observational data –Overcomes syntactic and semantic heterogeneity using a standard data model and controlled vocabularies –Supports a national network of observatory test beds –Maintains national registry of services (1.75 million stations – the largest in the world) WaterML is a standard language for consistently communicating water observations data from academic and government sources using web services; already adopted by several federal agencies. Joint WMO and OGC activity to enhance it. The system is already deployed at multiple locations It is free and open source