1 Categories of data Operational and very short-term decision making data Current, short-term decision making, related to financial transactions, detailed data are stored, not structured for decision making. Historical and long-term decision making data Saved for a pre-determined period of time, usually related to long- term decision making, structured for decision making. Contains data that will support decisions of strategic importance. Referred to as a “data warehouse”. Archival data Saved for a pre-determined period of time, used to track transactions for audit, not structured for decision making.

2 Webflix data storage requirements Operational needs. What are examples of questions management needs to be able to answer to handle daily operations effectively? Decision support needs. What are examples of questions management needs to be able to answer to manage the organization effectively on a short and long-term basis? Governmental, legal or auditing needs. What types of questions might be relevant for this type of organization?

3 Operational data Includes: Master data (also called reference data): Customer, employee, video, distribution center, critic, keyword. Transaction data: Queue, Copy, Customer Contract. Must store both master and transaction data. Must store changes to both master and transaction data.

4 Problems with operational data May not be integrated. May not be of good quality: Incomplete. Not accurate. Inconsistent. The meaning of the data is not fully defined and/or understood by all stakeholders.

5 Archival data Examples of archived data: Emergency dispatch calls. Credit card transactions. Accounts payable transactions. Tax-related data. Does not usually have to be accessed quickly. Must have procedures for extracting, transforming and loading (ETL) data as necessary. Archive database design is usually a copy of the transaction database design.

6 Topics about Data Warehouses What is a data warehouse? How does a data warehouse differ from a transaction processing database? What are the characteristics of a data warehouse? What are the components of a data warehousing system? How is a data warehouse created? How is a data warehouse accessed?

Compare and Contrast TPS and DSS

We use data to answer management questions TPS Questions How many customers currently have “Skyfall” in the queue? How many copies of “Skyfall” are in inventory in Sacramento? How many customers do we have in Nevada City? When is “Cloud Atlas” going to be released? Data Warehouse Questions How long does a customer usually keep a video? Which customers return videos within 2 days of receiving them? Which city has the most customers who return videos within 2 days of receiving them? What is the most popular genre for customers in Reno? 8

Operational vs. Data Warehouse databases

So, can one database support both transaction processing and decision support applications? Yes?? No??

11

12 Historical Data

A Business Intelligence “System” A business intelligence system encompasses all processes, hardware and software necessary to extract data, transform it, integrate it, store it, and provide information. The information is then made effective and accessible to users to support decision making. Sounds like just another information system So what makes it different?

14 Big Data!

15

16 Components of a business intelligence/data warehousing system Data store. Extraction/transformation/loading processes. Analysis tools – both end-user and IT professional. Visualization tools – primarily end-user.

What is a data warehouse (data store)? A data warehouse is a database designed to support a decision support system. A data warehouse is: Integrated: It is a centralized, consolidated database integrating data from an entire organization. Subject-oriented: Data warehouse data are organized around key subjects. The data are usually arranged by topic, such as customers, products, suppliers, etc. Time-variant: Data in the warehouse contain a time dimension so that they may be used as a historical aggregation. Non-volatile: Once data enter, they seldom leave. Data are appended rather than overwritten. Data are updated in batches.

18 Issues in creating a data warehouse How to get accurate and complete data? How to consolidate data? Differing data meanings. Differing storage mechanisms. Differing data formats.

Data mart extraction data warehouse 20

Two-tier data warehouse architecture

Three-tier data warehouse architecture

23 Issues in designing a data warehouse Must have a predefined subject focus. Has the potential to be very large – must define the “grain” or granularity level of storage. Will always have a dimension of time. May contain derived data. May be a summary of data, rather than each detailed transaction. Does not always adhere to standard normalization rules.

Analysis tools Standard old queries Online Analytical Processing Data Mining 24

25 Online analytical processing Provides multi-dimensional data analysis techniques. Works primarily with data aggregation. Provides advanced statistical analysis. Supports access to very large databases. Provides enhanced query optimization algorithms. Lots of acronyms: OLAP, ROLAP, MOLAP, HOLAP. Can be add-ons to existing products, example is Excel. Can have their own user interfaces.

OLAP vs. Data Mining questions

27 Data mining Data mining tools: analyze the data; uncover patterns hidden in the data; form computer models based on the findings; and use the models to predict business behavior. Proactive tools. Based on artificial intelligence software such as decision trees, neural networks, fuzzy logic systems, inductive nets and classification networking.

28 Visualization tools Graphical. Spreadsheet format - usually Excel look-and-feel. Beyond the spreadsheet using discovery tools. Example: Dashboard. Examples: examples/ examples/ Web-based.