4.1 Opening Vignette: Data Warehousing and DSS at Group Health Cooperative 2-3 million data records are processed monthly How to use for decision support? How to hold down costs? How to improve customer service? How to utilize resource effectively? How to improve service quality? Answers Develop a comprehensive database (data warehouse) and DSS approach Very effective

Data Warehousing, Access, Analysis and Visualization What to do with all the data that organizations collect, store and use? Information overload! Solution Data warehousing Data access Data mining Online analytical processing (OLAP) Data visualization Data sources Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ

4.3 The Nature and Sources of Data Data: Raw Information: Data organized to convey meaning Knowledge: Data items organized and processed to convey understanding, experience, accumulated learning, and expertise DSS Data Items –Documents –Pictures –Maps –Sound –Animation –Video Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ

Data Sources Internal External Personal Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ

The Internet and Commercial Database Services For External Data The Internet: Major supplier of external data Commercial Data “Banks”: Sell access to specialized databases Can add external data to the MSS in a timely manner and at a reasonable cost

Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ

The Internet/Web and Corporate Databases and Systems Use Web Browsers to Access vital information by employees and customers Implement executive information systems Implement group support systems (GSS) Database management systems provide data in HTML Web-browsers as DBMS front-ends

Database Management Systems in DSS DBMS: Software program for entering (or adding) information into a database; updating, deleting, manipulating, storing, and retrieving information A DBMS combined with a modeling language is a typical system development pair, used in constructing DSS or MSS DBMS are designed to handle large amounts of information Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ

Database Organization and Structure Relational Databases Hierarchical Databases Network Databases Object-oriented Databases Multimedia-based Databases Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ

Data and Applications OS Application 1 Application 2 Application 3 Application 1 Application 2 Application 3 OSDBMSOS O-O DBMS Application 1 Application 2 Application 3

Traditional File Systems n Advantages –simple data design to support single or small group of applications –fast data access –inexpensive n Disadvantages –lack of data relation –redundancy –lack of standards –low application development productivity

DBMS n Advantages –integration, sharing of data –increased data accessibility –minimized redundancy –easier application development and maintenance –improved data security –logical/physical data independence n Disadvantage –complex data design –slow access –expensive

Conceptual View Physical View Physical storage structure of data Internal view Logical, integrated view External view Users view of data 3 level DB Architecture Data Definition Language Data Manipulation Language Query Language

Customer Invoice Item Line Item Relational Database Invoice#, Inv.Date Customer#, Cname, Caddress Item, Item-Type, Item-Color, Item-Price, Quantity

Relational Database n Primary Key –duplicate rows not allowed –cannot have missing (NULL) value(s) for PK n Foreign Key –defines relationship between tables –FK values either reference existing PK values or they are NULL (referential integrity) CustomerInvoice FK

Relational Database n Relational Operators –Select: subset of rows –Project: subset of columns –Join: creates new table by linking on common attributes Select Items with Price > $100 Show Item#, Type, and Price All Invoices for Customer Sid B. Customer Invoice JOIN on Cust#

Data Warehousing Physical separation of operational and decision support environments Purpose: to establish a data repository making operational data accessible Uses TPS data needed for decision support Data are transformed and integrated into a consistent structure Data warehousing (or information warehousing): a solution to the data access problem End users perform ad hoc query, reporting analysis and visualization

Data Warehouse: A Decision Support Focus n DW technology A set of methods, techniques and tools that may be leveraged together to produce a vehicle that delivers data to end users on an integrated platform A framework to support the merging of operational data, informational data, external data, and personal data Issue is one of applying the technology to solve a business problem

What is a data warehouse? Databases that support decision making and that are subject oriented time-variant integrated non volatile –organized around the essential business entities (customer, product, policy, claim, order, etc.) –contains data that has been cleansed, transformed, integrated –data organized by various time periods; often summarized on time; data is time-stamped –not updated in real time; not updated by users

Figure 6. Transformation of the operational state information Operational state information is not carried to the data warehouse Data is transferred to the data warehouse after all state changes Or, data is transferred with period snapshots Order Processing System Data Warehouse Daily closed orders Order Up Inventory Down Weekly inventory snapshot Inventory snapshot 1 Inventory snapshot 2 Orders (Closed)

What is a data warehouse? Kinds of information in data warehouse old detail data current detail data lightly summarized data highly summarized data meta-data

Meta Data n “data on data” source, history, and many other aspects of data. n Business meta data definitions,descriptions and rules used for reporting. n Technical meta data structures and mapping rules for the data extraction and staging process. n Allows information stored in warehouse to be used effectively for reporting and analysis, and ensure that all users have “one version of the truth”.

Data Warehousing Benefits Increase in knowledge worker productivity Supports all decision makers’ data requirements Provide ready access to critical data Insulates operation databases from ad hoc processing Provides high-level summary information Provides drill down capabilities Yields –Improved business knowledge –Competitive advantage –Enhances customer service and satisfaction –Facilitates decision making –Help streamline business processes

DW Suitability For organizations where Data are in different systems Information-based approach to management in use Large, diverse customer base Same data have different representations in different systems Highly technical, messy data formats Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ

Data Loader Data Converter Data Scrubber Data Transformer Data Warehouse OLAP Interface OLAP Server PC Files LAN Servers Mainframe OLTP Databases External Sources

Data Marts n Data warehouse designed to meet the needs of a specific group of users n Should (but may not) be designed with corporate standards and accessibility in mind –incorporate standards for hardware, software, networking, DBMS, naming conventions, etc. –vendor’s attempt to bypass IT and sell directly to end-users?

Operational Data Store n Used for operational processing, may be used to feed the DW n An architectural construct that is subject-oriented integrated volatile current valued comprised of only corporate detailed data n Multiple applications may use the data, with updating in one place n Effective in organizations trying to move legacy systems to integrated environment

OLAP: Data Access and Mining, Querying and Analysis Online Analytical processing (OLAP) –DSS and EIS computing done by end-users in online systems –Versus online transaction processing (OLTP) OLAP Activities –Generating queries –Requesting ad hoc reports –Conducting statistical analyses

OLAP (On-Line Analytical Processing) n To gain insight into data through fast, interactive access to a wide variety of possible views of information that has been transformed from raw data n view and analyze data across multiple dimensions n allows flexible and easy “slicing and dicing” of data, drill down capabilities move from a general view to one which is more detailed (known as "drill-down"), or from a very detailed level to one which is more aggregated (“roll-up”). view data from a different perspective by introducing a completely different analysis criterion ("dicing" or changing view).

OLAP n Multidimensional OLAP vs. Relational OLAP –MOLAP: data stored in multi-dimensional arrays; use of sparse matrix techniques –ROLAP: data stored in relational DBMS; use of star-schema design

ROLAP: Star Schema Design Dimension Key 1 Dimension Key 2 Dimension Key 3 ……. Fact 1 Fact 2 Fact 3 ……. Fact Table Dimension Key 1 Description 1 Aggregation Lvl 1.1 Aggregation Lvl 1.2 Aggregation Lvl 1.3 Dimension Key 2 Description 2 Aggregation Lvl 2.1 Aggregation Lvl 2.2 Aggregation Lvl 2.3 Dimension Table 1 Dimension Table 2 Dimension Key 3 Description 3 Aggregation Lvl 3.1 Aggregation Lvl 3.2 Dimension Table 3

Star-Schema example ZIP Code City State/Province Country Dimension Tables ZIP Code City State/Province Country Dimension Tables Sales Rep ID Sales Rep Name Store ID Store Name Store Location Distribution Channel Product Code Product Name Category Product Type Customer Type Cust Type Desc Cust Category Cust Category Desc Dimension Tables Sales Rep ID Product Code Cust Zip Code Customer Type Sales Period Date Total Qty Total $ Quota Qty Returned Qty Promotion Qty Fact Table Sales Rep ID Sales Rep Name Store ID Store Name Store Location Distribution Channel Product Code Product Name Category Product Type Customer Type Cust Type Desc Cust Category Cust Category Desc Dimension Tables Multi-dimensional data measures

OLTP vs OLAP schema Company PO PO-itemItem Ship-from Ship-to Efficient create, update and processing of orders Query: list purchases by companies, cost of items, source and destination Item Company Purchases Date Ship-from Ship-to

Dimensional hierarchies Item Company Purchases Date #Units $-value Name City StateRegion Name Item-Type Product Category Day Week Year Qtr Month Query examples: Purchases by Item Purchases by Item and Date Purchases by Item, Date and Company Purchases by Item by Week Purchases by Item-Type by Qtr Purchases by Item-Type by State by Year

Further OLAP queries n Compare average purchase-$ in 1998 to that in 1997 n Compare average monthly purchase-$ by Region n What are the total 1st-quarter purchase-units by company over the past 5 years n What is the deviation in weekly purchase-$ by company over the past year n Model weekly purchase-units by Item-Category over the past 5 years n Analytical queries - requirements beyond traditional SQL-type querying constructs

OLAP uses the data warehouse and a set of tools, usually with multidimensional capabilities Query tools Spreadsheets Data mining tools Data visualization tools Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ

Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson Copyright 1998, Prentice Hall, Upper Saddle River, NJ