1. Lecture 8 Database Implementation
Jeffery S. Horsburgh
Hydroinformatics
Fall 2012
This work was funded by National Science Foundation Grant EPS

2. Objectives
Examine the features of a Relational Database Management System (RDBMS)
Create a physical implementation of an Observations Data Model (ODM) database within a RDBMS
Load observational data into an ODM database

3. What We have Talked About
Data management and the data life cycle
Metadata
Information needed to describe observational data
Data models and data model design
How do you structure the data and metadata for storage and retrieval
Google Fusion Tables and Excel Tools

4. Where We are Going Database implementation – TODAY!!
How do you physically implement a data model and load data
Using Structured Query Language (SQL)
Using code to slice and dice your data
Accessing databases from client software
Getting data from a database into R

5. Relational Database Management Systems
Robust software for managing data
Data stored in tables that may be related
Based on Structured Query Language (SQL)
Adopted by the American National Standards Institute (ANSI) and the International Standards Organization (ISO) as the standard data access language

6. File Versus Server-Based RDBMS
File-based
Everything contained within a single file
Generally good for single user, desktop applications
Examples: SQLite, Microsoft Access
Server-based
A database “server” manages databases and all transactions
Good for multiple, simultaneous connections and transactions
Examples: Oracle, Microsoft SQL Server, MySQL, PostgreSQL

7. The Beauty of Server-Based RDBMS
Yay!
Database Server
With RDBMS
Authentication and Access Control
I want some data!
Do I recognize your IP address?
Do I know you?
Are you authorized?
Database
Data
Firewall
Simultaneous Data Users

8. Server-Based RDBMS – “Granules”
Database Server
Databases
Tables
Records
DB1
DB2
DB3
DBn
. . .
Site
Variable
Date
Value 1
Temperature
8/2/ :00
12.4
8/2/ :30
12.7
8/2/ :00
13.1 1
Temperature
8/2/ :00
12.4

9. Physical Implementation of a Data Model within a RDBMS
Create a new database
Create tables for entities
Define attributes and data types
Create relationships and define their properties
Define constraints
All of this can be scripted/automated using SQL

10. DEMO: Microsoft SQL Server and SQL Server Management Studio

11. CUAHSI HIS – Sharing Hydrologic Data
The CUAHSI Hydrologic Information System (HIS) is an internet based system to support the sharing of hydrologic data. It is comprised of hydrologic databases and servers connected through web services as well as software for data publication, discovery and access.
Data Discovery and Integration platform
Data Publication platform
Data Synthesis and Research platform
Data Services
Metadata Services
Metadata Search
HIS Central
HydroDesktop
HydroServer
Service registration
Catalog harvesting
Service and data theme metadata
Data carts
Water Data Services
Spatial Data Services
Like search portals Google, Yahoo, Bing
Like browsers
Like web servers
Like HTML

12. HydroServer Software Stack
Observations Data Model (ODM)
Relational database schema for hydrologic observations (store data)
ODM Software Utilities
ODM Data Loader (load table-based data)
ODM Streaming Data Loader (load streaming data)
ODM Tools (visualize and manage data)
WaterOneFlow web services (publish data)

13. ILO-4 Database Implementation and Loading Data
GOAL: Organize and load observational data for a continuous water quality monitoring site in the Little Bear River to an ODM database on the class server.

14. The Data You Will be Using
7 water quality and streamflow monitoring sites
Temperature
Dissolved Oxygen pH
Specific Conductance
Turbidity
Water level/discharge
4 weather stations
Relative Humidity
Solar radiation
Precipitation
Barometric Pressure
Wind speed and direction
Soil moisture
Spread spectrum radio telemetry network

15. Authentication and Access Control
The Class HydroServer
hydroserver.uwrl.usu.edu
Authentication and Access Control
Firewall
ODM

16. DEMO: Create a blank ODM database in Microsoft SQL Server
Download the ODM blank SQL Server schema
Extract the SQL Server blank schema database to the SQL Server data directory
Attach database to the SQL Server database engine

17. Authentication in SQL Server
Windows Authentication
Uses a Windows account to authenticate a user
Users must have an account on the server
SQL Server Authentication
Specific to SQL Server
Enables you to create accounts for SQL Server without giving access to the rest of the server

18. Authorization in SQL Server
Assigning specific permissions to specific users for specific database objects

19. DEMO: Creating Database Users and Assigning Database Permissions in SQL Server

20. Load Data into the Database
Load metadata using the ODM Data Loader
Load observations using the ODM Streaming Data Loader
Advantages of using the ODM Data Loaders
Designed specifically for ODM
Validate data against the business logic of ODM
Protect the consistency of data and avoid errors

21. ODM Data Loading Strategy

22. ODM Data Loader
Loading data to any table in ODM – e.g., Sites, Variables, Methods, etc.
Loading Data Values in serial format
Multiple sites and variables in one file
All data values in a single column
Site
Variable
Date
Value 1
Temperature
8/2/ :00
12.4
8/2/ :30
12.7
8/2/ :00
13.1 …
Dissolved Oxygen
9.78
9.70
9.56 2
Specific Conductance
351.7
350
351.2

23. ODM Streaming Data Loader
Focused on loading data values
Crosstab data
Single site, multiple variables
Date in one column, each variable in one column
Datalogger files from field sensors
Date
Variable_1
Variable_2 …
Variable_n
8/2/ :00
8/2/ :30
13.222
8/2/ :00
9.75
8/2/ :30
9.78
8/2/ :00

24. Visualize Data Using ODM Tools
Connect to the database using a client software application
Visualize and manage data using the software tools

25. DEMO: ODM Tools

26. What You Will Do for the ILO
In your personal ODM database on the server
Load Site, Variable, Method, and Source metadata using the ODM Data Loader
Load sensor observations using the ODM Streaming Data Loader
Visualize and explore the data using ODM Tools
Evaluate ODM as a data management tool for sensor data from the Little Bear River experimental watershed