1. CISB594 – Business Intelligence
Data Warehousing
Part II

2. Reference
Materials used in this presentation are extracted mainly from the following texts, unless stated otherwise.

3. Objectives At the end of this lecture, you should be able to:
Describe the processes used in developing and managing data warehouses
Explain data integration and the extraction, transformation, and load (ETL) processes
Describe real-time (active) data warehousing
Understand data warehouse administration and security issues
CISB594 – Business Intelligence

4. Data Integration: The Extraction, Transformation, and Load (ETL) Process
Data integration is a term that covers three processes which combine to move data from multiple sources into a data warehouse:

5. Data Integration : The Extraction, Transformation, and Load (ETL) Process
Extraction, transformation, and load (ETL) technologies
Fundamentally, a DW could not exist without ETL
The ETL process consists of
extraction (reading data from one or more databases),
transformation (converting the extracted data from its previous form into the form in which it needs to be so that it can be placed into a data warehouse)
load (putting the data into the data warehouse)
The ETL process also contributes to the quality of the data in a DW, its purpose is to load the data warehouse with integrated and cleansed data

6. Data Transformation Tools – To purchase or to build?
The process can either be done through
purchasing data transformation tools and setup ETL, or
purchase the tool together with the expertise to setup ETL, or
developing the tools using programming languages.
Programmers can set up ETL processes using almost any programming language, but building such processes from scratch is very complex. (we are trying to talk to different sources here!)

7. Data Transformation Tools – To purchase or to build?
Increasingly, companies are buying ETL tools to help in the creation of ETL process (Wikipedia)
Examples of ETL tools
IBM InfoSphere DataStage
Microsoft SQL Server Integration Services (SSIS)
Oracle Data Integrator (ODI)

8. Data Transformation Tools – To purchase or to build? A Consideration
Should organization purchase data transformation tools or build the transformation process itself ?
Data transformation tools are expensive
Data transformation tools may have a long learning curve
It is difficult to measure how the IT organization is doing until it has learned to use the data transformation tools
Not an easy decision !
However many believe purchasing the tools should be able to kick start the project faster and simplify the maintenance of the data warehouse .

9. Data Warehouse Development
Choosing the vendors
Six guidelines to consider a vendor
Financial strength
Qualified consultants
Market share
Industry experience
Established partnerships
These indicate that a vendor is likely to be in business for the long term, to have the support capabilities its customers need, and to provide products that interoperate with other products the potential user has or may obtain.

10. Data Warehouse Development
A data warehousing project is a major undertaking
MORE COMPLICATED AS IT COMPRISES AND INFLUENCES MANY DEPARTMENTS, INPUT OUTPUT INTERFACES AND CAN BE PART OF BUSINESS STRATEGY
Data warehouse development approaches
Inmon Model: EDW approach
Kimball Model: Data mart approach
Which model is best?
There is no one-size-fits-all strategy to data warehousing
It depends on the need and the capacity of the organization
For many organizations, data mart approach is a convenient first step in implementing DW

11. Data Warehouse Development
Kimball Model
Inmon Model
Pros:
Easy to build organizationally
Easy to build technologically
Cons:
Enterprise wide view unavailable
Redundant data costs
High ETL costs
High DBA costs
Business Enterprise View
Design consistency
Data reusability
Require corporate leadership and vision

12. Eventually it can be this …

13. Data Warehouse Development
Describe the major similarities and differences between the Inmon and Kimball data warehouse development approaches.
Similarities: Both methods can produce an enterprise data warehouse and subset data marts.
Differences:
Inmon’s approach starts with an enterprise data warehouse, creating data marts as subsets of that EDW if appropriate. The focus is on proven, traditional methods and technologies.
Kimball’s starts with data marts, consolidating them into an EDW later if appropriate. It focuses in creating a useful end-user capability quickly.

14. Data Warehouse Development
Effort
Data Mart (Kimball Model)
EDW (Inmon Model)
Scope
One subject area
Several subject areas
Development Time
Months
Years
Development Cost
$10K - $100 K ++ $
Development Difficulty
Low to medium
High
Sources
Only some operational and external systems
Many operational and external systems
Size
Megabytes to several gigabytes
Gigabytes to petabytes
Hardware
Workstations and departmental servers
Enterprise servers and mainframe OS
Windows and Linux
Unix, S/390
Number of simultaneous users
10s
100s to 1000s

15. Real-Time Data Warehousing
Traditionally, a data warehouse are not business critical, data are commonly updated on a weekly basis – not allowing for responding to transactions in near real time
Today, organizations are facing the need for real-time data warehousing, as decision support has become operational.
The emergence of real-time data warehousing (RDW) or active data warehousing (ADW)
The process of loading and providing data via a data warehouse as they become available

16. Real-Time Data Warehousing
The need for real-time data
A business often cannot afford to wait a whole day for its operational data to load into the data warehouse for analysis
Real-time data collection can reduce or eliminate the nightly batch processes

17. Mistakes
Starting with wrong sponsorship chain – You need the right champion, somebody influential over the sources and directions
Setting unrealistic expectations – remember, BI initiative relies on management’s good decision making as much as it relies on the technology used.
Engaging in political naïve behaviour – Don’t imply that managers have been making bad decisions without BI
Loading DW with information just because it is available
Believing that DW design is the same as transactional DB design.
Choosing a DW admin who is only technology oriented.

18. Data Warehouse Administration and Security Issues
Due to its huge size and complicated nature, DW requires strong monitoring and administrating
Needs more than a DBA
Needs data warehouse administrator (DWA)
A person responsible for the administration and management of a data warehouse

19. Data Warehouse Administration and Security Issues
What skills should a DWA possess? Why?
Familiarity with high-performance hardware, software and networking technologies, since a data warehouse is based on those.
Solid business insight, to understand the purpose of the DW and its business justification, familiarity with business decision, making processes to understand how the DW for business strategic purpose will be used easy.
Excellent communication skills, to communicate with the rest of the organization

20. Data Warehouse Administration and Security Issues
Effective security in a data warehouse should focus on
four main areas:
1. Establishing effective corporate and security policies and procedures. An effective security policy should start at the top and be communicated to everyone in the organization.
2. Implementing logical security procedures and techniques to restrict access. This includes user authentication, access controls, and encryption.
3. Limiting physical access to the data center environment.
4. Establishing an effective internal control review process for security and privacy

21. Assignment question
Discuss the architecture and data warehouse model approach – include diagram and justify the choice
Discuss whether your team will recommend the organization to purchase tools from vendors or build the system itself. Justify your suggestion and include cost estimation.

22. Now ask if .. You are able to:
Describe the processes used in developing and managing data warehouses
Explain data integration and the extraction, transformation, and load (ETL) processes
Describe real-time (active) data warehousing
Understand data warehouse administration and security issues
CISB594 – Business Intelligence