1 Categories of data Operational and very short-term decision making data Current, short-term decision making, related to financial transactions, detailed data are stored, not structured for decision making. Historical and long-term decision making data Saved for a pre-determined period of time, usually related to long- term decision making, structured for decision making. Contains data that will support decisions of strategic importance. Referred to as a “data warehouse”. Archival data Saved for a pre-determined period of time, used to track transactions for audit, not structured for decision making.

2 CutGlass data storage requirements Operational needs. What are examples of questions management needs to be able to answer to handle daily operations effectively? Decision support needs. What are examples of questions management needs to be able to answer to manage the organization effectively on a short and long-term basis? Governmental or auditing regulation needs. What types of questions might be relevant for this type of organization?

3 Operational data Includes: Master data (also called reference data): Customer, employee, material, item, supplier/vendor, etc. Transaction data: Order, orderline, purchase order, invoice, shipment, etc. Must store both master and transaction data. Must store changes to both master and transaction data.

4

5 Problems with operational data Not integrated. “Dirty”: Incomplete. Not accurate. Inconsistent. The meaning of the data is not fully defined and/or understood by all stakeholders.

6 Archival data Examples of archived data: Emergency dispatch calls. Credit card transactions. Accounts payable transactions. Tax-related data. Does not usually have to be accessed quickly. Can store data on tape or other cheaper, less accessible media. Must have procedures for extracting, transforming and loading (ETL) data as necessary. Archive database design is usually a copy of the transaction database design.

What about decision support data? How does data used for decision support differ from transaction or archival data? Does decision support data need to be stored separately from transaction data? Does decision support data require a different database structure? 7

We use data to answer management questions TPS Questions What proposals have no decision as of today? Which work orders are overdue as of today? How much material was purchased for work order 10010? What are the labor costs currently for work order 10010? Data Warehouse Questions What are the characteristics of the proposals that weren’t accepted by clients? Which work orders were over cost estimates for the last 3 years? Are our proposal estimates improving (closer to actual) over the last 3 years? How about the last year? 8

Compare and Contrast TPS and DSS

Operational vs. Data Warehouse databases

11

So, can one database support both transaction processing and decision support applications? Should we just add a few tables to the transaction processing database, as shown in the prior slide?? YesNo

A Business Intelligence “System” A business intelligence system encompasses all processes, hardware and software necessary to extract data, transform it, integrate it, store it, and provide information. The information is then made effective and accessible to users to support decision making. Sounds like just another information system So what makes it different?

14 Big Data!

The “V’s” of Big Data Volume: scale of data Velocity: frequency of change Variety: Different forms and sources of data Veracity: Uncertainty of the accuracy of data 15

16

17 Components of a business intelligence/data warehousing system Data store. Extraction/transformation/loading processes. End user query tools. End user visualization tools.

What is a data warehouse? A data warehouse is a database designed to support a decision support system. A data warehouse is: Integrated: It is a centralized, consolidated database integrating data from an entire organization. Subject-oriented: Data warehouse data are organized around key subjects. The data are usually arranged by topic, such as customers, products, suppliers, etc. Time-variant: Data in the warehouse contain a time dimension so that they may be used as a historical aggregation. Non-volatile: Once data enter, they seldom leave. Data are appended rather than overwritten. Data are updated in batches.

19 Issues in creating a data warehouse How to get accurate and complete data? How to consolidate data? Differing data meanings. Differing storage mechanisms. Differing data formats.

Data mart extraction data warehouse 21

Two-tier data warehouse architecture

Three-tier data warehouse architecture

24 Issues in designing a data warehouse Must have a predefined subject focus. Has the potential to be very large – must define the “grain” or granularity level of storage. Will always have a dimension of time. May contain derived data. May be a summary of data, rather than each detailed transaction. Does not always adhere to standard normalization rules.

Potential Data Mart for Consulting Company 25

26 Accessing a data warehouse Visualization tools. Graphical. Spreadsheet format - usually Excel look-and-feel. Beyond the spreadsheet using discovery tools. Example: Dashboard. Examples: Query tools. OLAP: Online analytical processing. Data mining: Artificial intelligence based query methods.

27 Online analytical processing Provides multi-dimensional data analysis techniques. Works primarily with data aggregation. Provides advanced statistical analysis. Supports access to very large databases. Provides enhanced query optimization algorithms. Lots of acronyms: OLAP, ROLAP, MOLAP, HOLAP. Can be add-ons to existing products, example is Excel. Can have their own user interfaces.

OLAP vs. Data Mining questions

29 Data mining Data mining tools: analyze the data; uncover patterns hidden in the data; form computer models based on the findings; and use the models to predict business behavior. Proactive tools. Based on artificial intelligence software such as decision trees, neural networks, fuzzy logic systems, inductive nets and classification networking.