CS346: Advanced Databases Graham Cormode G.Cormode@warwick.ac.uk Data Warehousing and OLAP

Outline Chapter: “Overview of Data Warehousing and OLAP” in Elmasri and Navathe What is a data warehouse and what is it for? The multidimensional data model and common schema designs Special indexes: bitmap and join indexes Why? Another model of data to study and contrast with RDBMS A different perspective on using data for insight A relatively recent development (1990s): still developing CS346 Advanced Databases

CS910 Foundations of Data Analytics Data Warehouses Data Warehouses were introduced to handle large data stores Typically, historical data for business analytic purposes Separate from the organization’s “live” operational database “A data warehouse is a subject-oriented, integrated, time-variant, non-volatile collection of data” W. Inmon, “father of data warehouse” Subject-oriented: focused on one topic (e.g. all sales records) Integrated: data brought together from many sources, cleaned Time-variant: covers a long history of data (e.g. last decade) Nonvolatile: only periodically updated, not “live” data Data warehouse products from Oracle, IBM, Microsoft, Teradata CS910 Foundations of Data Analytics

OLAP, OLTP, DSS OLAP: Online Analytical Processing Analysis of complex data stored in a data warehouse Often using distributed storage and processing (Hive…) In contrast to Online Transaction Processing (OLTP) Insertions, updates, deletions and queries Decision Support Systems or Enterprise Information Systems Allow organization’s leader to make complex strategic decisions Support data mining / machine learning for knowledge discovery “Business Intelligence” CS346 Advanced Databases

Data Warehouse Characteristics Data Warehouses adopt a different data model to RDBMS Typically a multidimensional data model Warehouses often store integrated data from many sources Contrast to DBMS which encourages multiple disjoint DBs Warehouses typically support time-series, trend analysis Need more historical data, not just the current values Warehouses typically nonvolatile Data is added to only periodically. No need for transactions! Warehouses typically handle very large amounts of data Often two orders of magnitude (100x) larger than “live” databases May be terabytes-petabytes in size CS346 Advanced Databases

ETL: Extract, Transform, Load Putting data into a data warehouse is a complex process Denoted ETL: Extract, Transform, Load Extract: pull data out of whatever system it is stored in Via appropriate interchange format: XML, flat files, etc. Transform: put data into a usable format Pick which attributes, harmonize formats, sort and join as needed Format for consistency: names of entities should agree Clean the data: identify errors and fill in missing values Return cleaned data to update original source: backflushing Fit the data to the model of the warehouse: ensure it fits schema CS346 Advanced Databases

ETL: Extract, Transform, Load Load: store in an appropriate format Many warehouses use simple structures, e.g. sorted flat files Refresh policy: How up to date is the data? Can it be offline? How long does it take to load into the warehouse? Store metadata on the data as well: metadata repository Technical metadata: how data was processed, stored, updated Business metadata: relevant business rules and organization details CS346 Advanced Databases

Characteristics of Data Warehouses A few key properties of data warehouses (DW): Multidimensional: allow many levels of aggregation Support multiple users via client-server architecture Should be intuitive and responsive to use Many variations of the central concept: Enterprise-wide DW: corral everything about an organization Virtual DWs: provide a materialized view of an operational DB Data marts: DWs restricted to a subset of an organization Two common architectures for warehouses: Distributed: must handle replication, partitioning, consistency Federated: collection of autonomous warehouses (data marts) CS346 Advanced Databases

CS910 Foundations of Data Analytics OLAP and Data Cubes Warehouses often support Online Analytical Processing (OLAP) A multidimensional view of data Represents data as a data cube Explored by aggregating or refining dimensions in the data CS910 Foundations of Data Analytics

Aggregating Multidimensional Data E.g. Sales volume as a function of product, month, and region Dimensions: Product, Location, Time Hierarchical summarization paths Region Industry Region Year Category Country Quarter Product City Month Week Office Day Product Month

A Sample Data Cube * (all) Total annual sales of TVs in U.S.A. 1Qtr Date Product Country sum TV DVD PC 1Qtr 2Qtr 3Qtr 4Qtr U.S.A Canada Mexico * (all)

CS910 Foundations of Data Analytics OLAP Operations Roll up (drill-up): summarize data by climbing up hierarchy or by dimension reduction Drill down (roll down): inverse of roll-up from higher level summary to lower level summary or detailed data, or introducing new dimensions Slice and dice: project and select Zoom in on particular value, or drop some attributes Apply aggregation: on a given dimension Count, Sum, Min, Max, Average, Variance, Median, Mode CS910 Foundations of Data Analytics

Multidimensional Storage Model The DW multidimensional storage model has two table types: Dimension tables and fact tables Fact table: many tuples, 1 per stored fact, pointing to dimensions E.g. sale of an item: which product, which store, which customer Dimension table: tuples of attributes of the dimension E.g. details of the product, of the store, of the customer CS346 Advanced Databases

Data Warehouse Schemas Star schema: fact table with a single table for each dimension Snowflake schema: variation of a star schema Fact tables are arranged hierarchically after normalization CS346 Advanced Databases

Fact constellations Fact constellation: a set of fact tables that share some dimension tables CS346 Advanced Databases

Bitmap Indexes Bitmap indexes used to support high-performance access One of various techniques used in the database Takes the form of a bit vector for each value in a table Set to 1 if a particular value occurs, 0 if it does not Can be quite compact if the domain size is small E.g. 1M rows and domain size of 4: bitmap index size 0.5MB Efficient to check conjunctive conditions: intersect (AND) bitmaps CS346 Advanced Databases

Join indexing A join index connections dimension data to tuples in a fact table Assuming a star schema A join index is a traditional index linking primary and foreign keys Lists all the keys that meet the (equi)join condition e.g. consider a sales fact table that has city as one dimension Join index on city: list of sales tuple ids for each different city Can make a join index as a bitmap index CS346 Advanced Databases

Data Warehouse versus Views Recall views: result of a (stored) query on a database Could achieve warehouse functionality via (materialized) views Data warehouses are more than just views: Warehouses are stored, not materialized on demand Different data model: multidimensional, not relational Data warehouses can be indexed (views cannot) Warehouses support various analysis tasks (mining, time series) Warehouses typically contain more (historic) data than one DB CS346 Advanced Databases

Data Warehouses: Pros and Cons Data warehouses have many strengths for data analysis: Support fast exploration and aggregation of data Designed to handle very large data sets (TBs / billions of records) Software supports analytics (data mining/machine learning) on top Clustering, Regression, Classification, Rule mining However, they have their limitations: A big undertaking: bringing together all an organization’s data Need a thorough understanding of the organizational structure Can be costly to maintain (time-consuming to clean and load data) As underlying data organization changes, so must the warehouse CS910 Foundations of Data Analytics

Summary What is a data warehouse and what is it for? Storing and querying all the data of a large organization The multidimensional data model and common schema designs Roll up, drill down, slice & dice; star and snowflake schemas Special indexes: bitmap and join indexes Chapter: “Overview of Data Warehousing and OLAP” in Elmasri and Navathe CS346 Advanced Databases