1. Data Warehousing Concepts

2. Introduction to Data Warehousing Data Warehouse Architecture
Contents
Introduction to Data Warehousing
Data Warehouse Architecture
Dimensional Modeling
Data warehouse techniqued
Data mining
OLAP

3. 1. Introduction to Data Warehousing
Data Warehousing Definition
Online Transaction Processing (OLTP) System
Data Warehousing System
Difference between OLTP and DW System
Reasons for Building a Data Warehouse
Benefits of Data Warehousing

4. 1.1 What is a Data Warehouse?
Data Warehouse is
- primarily a centralized repository of an organization’s data.
- holds large amount of data including historical info.
- designed to support efficient data analysis and reporting.

5. 1.2 OLTP Systems
Focus:
Designed to get data in quickly and to analyze the current events. Transaction Oriented.
Organized around business processes such as Order Entry, Purchasing, Campaign Management, Trading etc.
Avoidance of data duplication, maintainability etc.
Characteristics:
Process Oriented.
Normalized Data.
Current Data.
Volatile Data.
Real Time Updates.

6. 1.3 Data Warehousing Systems
Focus:
Designed to get data out and quickly analyze.
Concerned with customer, product etc. rather than order entry, campaign management.
Focus on easy data access .
Contains slices of data across different periods of time.
Historical data supports trending, forecasting and time based performance reporting.
Characteristics:
Subject oriented rather than process oriented.
Integrated across subjects and entire enterprise.
De-Normalized Data.
Time-Variant.
Historical Data.
Non Volatile
Atomic and Summary Data.

7. 1.4 OLTP Vs Data Warehouse OLTP Systems Data Warehouse Normalized Data
De-Normalized Data
Used to run the business
Used to analyze the business
Real-Time data update
Updated on a predefined schedule
Volatile Data
Non-Volatile Data
Current Data
Historical Data
Wider Audience. Transaction throughput
Limited Audience. Fast Query Response
Small to large database
Large to Very Large Database

8. 1.5 Why Build a Data Warehouse?
No Single Version of Truth.
Lack of standardized data across the enterprise for easy understanding and further decision-making.
Absence of historical data for the purpose of analysis and decision making.

9. 1.6 Benefits of Data Warehousing
Rapid Access to data.
Integrated data.
Reliable Reporting.
Better Decision making.

10. 2. Data Warehouse Architecture
Logical Architecture
Elements of Data Warehouse

11. Data Warehouse - Logical Architecture
BI Tools
ETL
Staging Area DQ
Query Tools
Datamarts
OLAP Tools
Data Mining
Data
Visualization

12. Elements of A Data Warehouse1 2 3
ETL Tool or
Process
SAP
CRM
Inventory
Manufacturing
Staging
70% of Effort in a Data Warehousing solution is in developing a successful ETL strategy
Operational Data Storage
ETL & Staging
ETL tool will interface with all the sources in the enterprise and extract data in a batch cycle or in real time
Data
Warehouse
Quality
Accounts
Inventory
Data Storage
Enterprise Information is stored in the warehouse structure
ETL Tool
BI Tools, Portals
Quality
Finance
Marktng
Secured
Access
BI Tools interface with the databases to generate reports
Reporting Layer
METADATA
Extracting
The extract step is the first step involved in getting data into the data ware house environment.
Extracting means reading and understanding the source data, and copying the parts that are needed to the data staging area for further work
Extracting data needs to be done carefully so as not to effect production environments

13. Staging Area Transforming
Once the data is extracted into the data staging area, there many possible transformation steps, including:
Cleaning the data by correcting misspellings, resolving domain conflicts (such as a city name that is incompatible with a postal code), dealing with missing data elements, and parsing into standard formats
Staging Area
A storage area and set of processes that clean, transform, combine, duplicate, household, archive, and prepare source data for use in the data warehouse
The data staging area is everything in between the source system and the presentation server
The data staging area is not part of the physical data warehouse
The staging area is dominated by the simple activities of sorting and sequential processing

14. Loading Data
At the end of the transformation process, the data is in a position to be loaded across to the target warehouse
First time bulk load to get the historical data into the Data Warehouse
Periodic Incremental loads to bring in modified data
Loading in the data warehouse environment usually takes the form of inserting data into dimension tables and fact table. These are the tables that are typically queried on by the users/tools while executing reports
Bulk loading is a very important capability that is to be contrasted with record-at-a-time loading, which is far slower and can cause load times to be in the 10 hours+ range
It may be required to drop and recreate indexes on the target warehouse structure each time data loading occurs

15. Data warehouse techniques
 Data Mining
 OLAP
Data MINING
Data mining access of a database differs from this
traditional accesses in several ways:
Query: The query might not be well formed or precisely stated. The data miner might not even be exactly sure of what he wants to see.
Data: The data accessed is usually a different version from that of the original operational database. The data have been cleaned and modified to better support the mining process.
Output: The output of the data mining query probably is not a subset of the database. Instead it is the output of some analysis of the contents of the database.

16. Data mining algorithms can be characterized as consisting of three parts:
Model – The purpose of the algorithm is to fit a model to the data.
Preference – Some criteria must be used to fit one model over another.
Search – All algorithms require some technique to search the data.

17. 4. OLAP

18. A B C Time D 1 2 3 4 5 Product SALES CUBE Q2 Q1 Dimensions Sales
CUSTOMER
Time
Sales
The general activity of querying and presenting text and number data from data warehouses in a dimensional format is known as OLAP
The OLAP vendors’ technology is non relational and is almost always based on an explicit multidimensional cube of data
OLAP databases are also known as multidimensional databases, or MDDBs.
OLAP installations would be classified as small, individual data marts when viewed against the full range of data warehouse application
SALES CUBE
CUSTOMER A 11 43 12 49 71 B 33 15 65 94 45 C 59 77 37 78 12
Time Q2 D 09 53 20 73 32 Q1
Product

19. Thank You