Lecture 8 Database Implementation

Slides:

Advertisements

Similar presentations

HydroServer A Platform for Publishing Space- Time Hydrologic Datasets Support EAR CUAHSI HIS Sharing hydrologic data Jeffery.

Advertisements

Sharing Hydrologic Data with the CUAHSI Hydrologic Information System Support EAR CUAHSI HIS Sharing hydrologic data David.

This work is funded by the Inland Northwest Research Alliance INRA Constellation of Experimental Watersheds: Cyberinfrastructure to Support Publication.

ICEWATER: INRA Constellation of Experimental Watersheds Cyberinfrastructure to Support Publication of Water Resources Data Jeffery S. Horsburgh, Utah State.

SAN DIEGO SUPERCOMPUTER CENTER HYDROLOGIC METADATA CATALOG AND SEMANTIC SEARCH SERVICES IN CUAHSI HIS CUAHSI HIS Sharing hydrologic.

Managing Sensor Infrastructure: A Sensor Extension for ODM2 Amber Spackman Jones, Jeffery S. Horsburgh, Juan Caraballo, Maurier Ramírez Utah Water Research.

C van Ingen, D Agarwal, M Goode, J Gupchup, J Hunt, R Leonardson, M Rodriguez, N Li Berkeley Water Center John Hopkins University Lawrence Berkeley Laboratory.

Linking HIS and GIS How to support the objective, transparent and robust calculation and publication of SWSI? Jeffery S. Horsburgh CUAHSI HIS Sharing hydrologic.

This work is funded by National Science Foundation Grant EAR Accessing and Sharing Data Using the CUAHSI Hydrologic Information System CUAHSI HIS.

SENSORS, CYBERINFRASTRUCTURE, AND EXAMINATION OF HYDROLOGIC AND HYDROCHEMICAL RESPONSE IN THE LITTLE BEAR RIVER OBSERVATORY TEST BED Jeffery S. Horsburgh.

Components of an Integrated Environmental Observatory Information System Cyberinfrastructure to Support Publication of Water Resources Data Jeffery S.

This work was funded by the U.S. National Science Foundation under grant EAR Any opinions, findings and conclusions or recommendations expressed.

HydroServer A Platform for Publishing Space- Time Hydrologic Datasets Support EAR CUAHSI HIS Sharing hydrologic data Jeffery.

SENSORS, CYBERINFRASTRUCTURE, AND WATER QUALITY IN THE LITTLE BEAR RIVER Jeffery S. Horsburgh David K. Stevens, Amber Spackman Jones, David G. Tarboton,

Time Series Analyst An Internet Based Application for Viewing and Analyzing Environmental Time Series Jeffery S. Horsburgh Utah State University David.

Development of a Community Hydrologic Information System Jeffery S. Horsburgh Utah State University David G. Tarboton Utah State University.

Using GIS in Creating an End-to- End System for Publishing Environmental Observations Data Jeffery S. Horsburgh David G. Tarboton, David R. Maidment, Ilya.

Integrating Historical and Realtime Monitoring Data into an Internet Based Watershed Information System for the Bear River Basin Jeff Horsburgh David Stevens,

Introducing the CUAHSI Hydrologic Information System Desktop Application (HydroDesktop) and Open Development Community Jiří Kadlec, Daniel Ames, Teva Velupillai.

Deployment and Evaluation of an Observations Data Model Jeffery S Horsburgh David G Tarboton Ilya Zaslavsky David R. Maidment David Valentine

An End-to-End System for Publishing Environmental Observations Data Jeffery S. Horsburgh David K. Stevens, David G. Tarboton, Nancy O. Mesner, Amber Spackman.

Tools for Publishing Environmental Observations on the Internet Justin Berger, Undergraduate Researcher Jeff Horsburgh, Faculty Mentor David Tarboton,

Using HydroServer Organize, Manage, and Publish Your Data Support EAR CUAHSI HIS Sharing hydrologic data Jeffery S. Horsburgh.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Database Management System Lecture 2 Introduction to Database management.

About CUAHSI The Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) is an organization representing 120+ universities.

Ocean Sciences What is CUAHSI? CUAHSI – Consortium of Universities for the Advancement of Hydrologic Science, Inc Formed in 2001 as a legal entity Program.

Information Requirements for Integrating Spatially Discrete, Feature- Based Earth Observations Jeffery S. Horsburgh Anthony Aufdenkampe, Kerstin Lehnert,

About CUAHSI The Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) is an organization representing 120+ universities.

I Copyright © 2004, Oracle. All rights reserved. Introduction.

Course Introduction Introduction to Databases Instructor: Joe Bockhorst University of Wisconsin - Milwaukee.

Exercises: Organizing, Loading, and Managing Point Observations Using HydroServer Support EAR CUAHSI HIS Sharing hydrologic data

What’s New? – BCS 2013 Brett Lonsdale.  Co-founder of Lightning Tools  One of the hosts on the SharePoint Pod Show  Co-organizer of SharePoint Saturday.

1 INTRODUCTION TO DATABASE MANAGEMENT SYSTEM L E C T U R E

Data Interoperability in the Hydrologic Sciences The CUAHSI Hydrologic Information System David Tarboton, David Maidment, Ilya Zaslavsky, Dan Ames, Jon.

CIS 103 — Applied Computer Technology Last Edited: September 17, 2010 by C.Herbert Using Database Management Systems.

Advancing an Information Model for Environmental Observations Jeffery S. Horsburgh Anthony Aufdenkampe, Richard P. Hooper, Kerstin Lehnert, Kim Schreuders,

Workshop on QC in Derived Data Products, Las Cruces, NM, 31 January 2007 ClimDB/HydroDB Objectives Don Henshaw Improve access to long-term collections.

Water and Catchment Data Services David R. Maidment Center for Research in Water Resources University of Texas at Austin River Science Symposium Swansea,

CERN - IT Department CH-1211 Genève 23 Switzerland t DB Development Tools Benthic SQL Developer Application Express WLCG Service Reliability.

The CUAHSI Hydrologic Information System Presented by Dr. Tim Whiteaker The University of Texas at Austin 22 February, 2011.

The CUAHSI Community Hydrologic Information System Jeffery S. Horsburgh Utah Water Research Laboratory Utah State University CUAHSI HIS Sharing hydrologic.

The CUAHSI Observations Data Model Jeff Horsburgh David Maidment, David Tarboton, Ilya Zaslavsky, Michael Piasecki, Jon Goodall, David Valentine,

Data Model / Database Implementation (continued) Jeffery S. Horsburgh Hydroinformatics Fall 2014 This work was funded by National Science Foundation Grants.

National Center for Supercomputing Applications University of Illinois at Urbana–Champaign Data Management for the Mississippi River Luigi Marini September.

From Prototype to Service: A CUAHSI Datacenter for Hydroinformatics Richard Hooper Consortium of Universities for the Advancement of Hydrologic Science,

Lecture 9 Using Structured Query Language (SQL) Jeffery S. Horsburgh Hydroinformatics Fall 2012 This work was funded by National Science Foundation Grant.

Lecture 5 Data Model Design Jeffery S. Horsburgh Hydroinformatics Fall 2012 This work was funded by National Science Foundation Grant EPS

Hydroinformatics Lecture 15: HydroServer and HydroServer Lite The CUAHSI HIS is Supported by NSF Grant# EAR CUAHSI HIS Sharing hydrologic data.

UNIVERSITY OF UTAH GREEN INFRASTRUCTURE MONITORING DATABASE CVEEN 7970 Hydroinformatics Semester Project Zachary Magdol, Jai Kanth Panthail, Pratibha Sapkota,

Using Python to Retrieve Data from the CUAHSI HIS Web Services Jeffery S. Horsburgh Hydroinformatics Fall 2015 This work was funded by National Science.

Using Structured Query Language (SQL) Jeffery S. Horsburgh Hydroinformatics Fall 2014 This work was funded by National Science Foundation Grants EPS

1 of 42 Lecture 5 Data Model Design Jeffery S. Horsburgh Hydroinformatics Fall 2013 This work was funded by National Science Foundation Grants EPS

Using Python to Retrieve Data from the CUAHSI HIS Web Services Jon Goodall Hydroinformatics Fall 2014 This work was funded by National Science Foundation.

Developing a community hydrologic information system David G Tarboton David R. Maidment (PI) Ilya Zaslavsky Michael Piasecki Jon Goodall

Fundamental of Database Systems

Jeffery S. Horsburgh Hydroinformatics Fall 2014

The CUAHSI Hydrologic Information System Spatial Data Publication Platform David Tarboton, Jeff Horsburgh, David Maidment, Dan Ames, Jon Goodall, Richard.

Hydroinformatics Lecture: HydroServer .NET/PHP

Using GIS in Creating an End-to-End System for Publishing Environmental Observations Data Jeffery S. Horsburgh David G. Tarboton, David R. Maidment, Ilya.

Sharing Hydrologic Data with the CUAHSI* Hydrologic Information System

The CUAHSI Hydrologic Information System and NHD Plus A Services Oriented Architecture for Water Resources Data David G Tarboton David R. Maidment (PI)

Database System Concepts and Architecture

Jeffery S. Horsburgh Utah State University

Using Structured Query Language (SQL) (continued)

CUAHSI HIS Sharing hydrologic data

Hydroinformatics Lecture 15: HydroServer (and HydroServer Lite)

Data Acquisition, Management and Manipulation

HydroDesktop: A Key Component of the CUAHSI/CZO HIS for Hydrologic Data Discovery, Visualization, and Analysis Daniel P. Ames, Ph.D. P.E. Idaho State University.

Presentation transcript:

Lecture 8 Database Implementation Jeffery S. Horsburgh Hydroinformatics Fall 2012 This work was funded by National Science Foundation Grant EPS 1135482

Objectives Examine the features of a Relational Database Management System (RDBMS) Create a physical implementation of an Observations Data Model (ODM) database within a RDBMS Load observational data into an ODM database

What We have Talked About Data management and the data life cycle Metadata Information needed to describe observational data Data models and data model design How do you structure the data and metadata for storage and retrieval Google Fusion Tables and Excel Tools

Where We are Going Database implementation – TODAY!! How do you physically implement a data model and load data Using Structured Query Language (SQL) Using code to slice and dice your data Accessing databases from client software Getting data from a database into R

Relational Database Management Systems Robust software for managing data Data stored in tables that may be related Based on Structured Query Language (SQL) Adopted by the American National Standards Institute (ANSI) and the International Standards Organization (ISO) as the standard data access language

File Versus Server-Based RDBMS File-based Everything contained within a single file Generally good for single user, desktop applications Examples: SQLite, Microsoft Access Server-based A database “server” manages databases and all transactions Good for multiple, simultaneous connections and transactions Examples: Oracle, Microsoft SQL Server, MySQL, PostgreSQL

The Beauty of Server-Based RDBMS Yay! Database Server With RDBMS Authentication and Access Control I want some data! Do I recognize your IP address? Do I know you? Are you authorized? Database Data Firewall Simultaneous Data Users

Server-Based RDBMS – “Granules” Database Server Databases Tables Records DB1 DB2 DB3 DBn . . . Site Variable Date Value 1 Temperature 8/2/2007 14:00 12.4 8/2/2007 14:30 12.7 8/2/2007 15:00 13.1 1 Temperature 8/2/2007 14:00 12.4

Physical Implementation of a Data Model within a RDBMS Create a new database Create tables for entities Define attributes and data types Create relationships and define their properties Define constraints All of this can be scripted/automated using SQL

DEMO: Microsoft SQL Server and SQL Server Management Studio

CUAHSI HIS – Sharing Hydrologic Data The CUAHSI Hydrologic Information System (HIS) is an internet based system to support the sharing of hydrologic data. It is comprised of hydrologic databases and servers connected through web services as well as software for data publication, discovery and access. Data Discovery and Integration platform Data Publication platform Data Synthesis and Research platform Data Services Metadata Services Metadata Search HIS Central HydroDesktop HydroServer Service registration Catalog harvesting Service and data theme metadata Data carts Water Data Services Spatial Data Services Like search portals Google, Yahoo, Bing Like browsers Like web servers Like HTML

HydroServer Software Stack Observations Data Model (ODM) Relational database schema for hydrologic observations (store data) ODM Software Utilities ODM Data Loader (load table-based data) ODM Streaming Data Loader (load streaming data) ODM Tools (visualize and manage data) WaterOneFlow web services (publish data)

ILO-4 Database Implementation and Loading Data GOAL: Organize and load observational data for a continuous water quality monitoring site in the Little Bear River to an ODM database on the class server.

The Data You Will be Using 7 water quality and streamflow monitoring sites Temperature Dissolved Oxygen pH Specific Conductance Turbidity Water level/discharge 4 weather stations Relative Humidity Solar radiation Precipitation Barometric Pressure Wind speed and direction Soil moisture Spread spectrum radio telemetry network

Authentication and Access Control The Class HydroServer hydroserver.uwrl.usu.edu Authentication and Access Control Firewall ODM

DEMO: Create a blank ODM database in Microsoft SQL Server Download the ODM blank SQL Server schema Extract the SQL Server blank schema database to the SQL Server data directory Attach database to the SQL Server database engine

Authentication in SQL Server Windows Authentication Uses a Windows account to authenticate a user Users must have an account on the server SQL Server Authentication Specific to SQL Server Enables you to create accounts for SQL Server without giving access to the rest of the server

Authorization in SQL Server Assigning specific permissions to specific users for specific database objects

DEMO: Creating Database Users and Assigning Database Permissions in SQL Server

Load Data into the Database Load metadata using the ODM Data Loader Load observations using the ODM Streaming Data Loader Advantages of using the ODM Data Loaders Designed specifically for ODM Validate data against the business logic of ODM Protect the consistency of data and avoid errors

ODM Data Loading Strategy

ODM Data Loader Loading data to any table in ODM – e.g., Sites, Variables, Methods, etc. Loading Data Values in serial format Multiple sites and variables in one file All data values in a single column Site Variable Date Value 1 Temperature 8/2/2007 14:00 12.4 8/2/2007 14:30 12.7 8/2/2007 15:00 13.1 … Dissolved Oxygen 9.78 9.70 9.56 2 Specific Conductance 351.7 350 351.2

ODM Streaming Data Loader Focused on loading data values Crosstab data Single site, multiple variables Date in one column, each variable in one column Datalogger files from field sensors Date Variable_1 Variable_2 … Variable_n 8/2/2007 14:00 13.01298 1.155137 9.391667 8/2/2007 14:30 13.222 1.15333 9.586666 8/2/2007 15:00 13.19934 1.153291 9.75 8/2/2007 15:30 13.19094 1.153561 9.78 8/2/2007 16:00 13.17751 1.16075 9.703334

Visualize Data Using ODM Tools Connect to the database using a client software application Visualize and manage data using the software tools

DEMO: ODM Tools

What You Will Do for the ILO In your personal ODM database on the server Load Site, Variable, Method, and Source metadata using the ODM Data Loader Load sensor observations using the ODM Streaming Data Loader Visualize and explore the data using ODM Tools Evaluate ODM as a data management tool for sensor data from the Little Bear River experimental watershed