Chapter 33 OLAP Transparencies © Pearson Education Limited 1995, 2005

Chapter 33 - Objectives The purpose of Online Analytical Processing (OLAP). The relationship between OLAP and data warehousing. The key features of OLAP applications. The potential benefits associated with successful OLAP applications. © Pearson Education Limited 1995, 2005

Chapter 33 - Objectives How to represent multi-dimensional data. The rules for OLAP tools. The main categories of OLAP tools. OLAP extensions to the SQL standard. How Oracle supports OLAP. © Pearson Education Limited 1995, 2005

Business Intelligence Technologies Accompanying the growth in data warehousing is an ever-increasing demand by users for more powerful access tools that provide advanced analytical capabilities. There are two main types of access tools available to meet this demand, namely Online Analytical Processing (OLAP) and data mining. © Pearson Education Limited 1995, 2005

Business Intelligence Technologies OLAP and Data Mining differ in what they offer the user and because of this they are complementary technologies. An environment that includes a data warehouse (or more commonly one or more data marts) together with tools such as OLAP and /or data mining are collectively referred to as Business Intelligence (BI) technologies. © Pearson Education Limited 1995, 2005

Online Analytical Processing (OLAP) The dynamic synthesis, analysis, and consolidation of large volumes of multi-dimensional data, Codd (1993). Describes a technology that uses a multi-dimensional view of aggregate data to provide quick access to strategic information for the purposes of advanced analysis. © Pearson Education Limited 1995, 2005

Online Analytical Processing (OLAP) Enables users to gain a deeper understanding and knowledge about various aspects of their corporate data through fast, consistent, interactive access to a wide variety of possible views of the data. Allows users to view corporate data in such a way that it is a better model of the true dimensionality of the enterprise. © Pearson Education Limited 1995, 2005

Online Analytical Processing (OLAP) Can easily answer ‘who?’ and ‘what?’ questions, however, ability to answer ‘what if?’ and ‘why?’ type questions distinguishes OLAP from general-purpose query tools. Types of analysis ranges from basic navigation and browsing (slicing and dicing) to calculations, to more complex analyses such as time series and complex modeling. © Pearson Education Limited 1995, 2005

OLAP Benchmarks OLAP Council published an analytical processing benchmark referred to as the APB-1 (OLAP Council, 1998). Aim is to measure a server’s overall OLAP performance rather than the performance of individual tasks. © Pearson Education Limited 1995, 2005

OLAP Benchmarks APB-1 assesses the most common business operations including: bulk loading of data from internal or external data sources incremental loading of data from operational systems; aggregation of input level data along hierarchies; © Pearson Education Limited 1995, 2005

OLAP Benchmarks APB-1 assesses the most common business operations including (continued): calculation of new data based on business models; time series analysis; queries with a high degree of complexity; drill-down through hierarchies; ad hoc queries; multiple online sessions. © Pearson Education Limited 1995, 2005

OLAP Benchmarks OLAP applications are judged on their ability to provide just-in-time (JIT) information, a core requirement of supporting effective decision-making. This requirement is more than measuring processing performance but includes its abilities to model complex business relationships and to respond to changing business requirements. © Pearson Education Limited 1995, 2005

OLAP Benchmarks APB-1 uses a standard benchmark metric called AQM (Analytical Queries per Minute). AQM represents the number of analytical queries processed per minute including data loading and computation time. Thus, the AQM incorporates data loading performance, calculation performance, and query performance into a singe metric. © Pearson Education Limited 1995, 2005

OLAP Benchmarks Publication of APB-1 benchmark results must include both the database schema and all code required for executing the benchmark. An essential requirement of all OLAP applications is the ability to provide users with JIT information, which is necessary to make effective decisions about an organization's strategic directions. © Pearson Education Limited 1995, 2005

OLAP Applications JIT information is computed data that usually reflects complex relationships and is often calculated on the fly. Also as data relationships may not be known in advance, the data model must be flexible. © Pearson Education Limited 1995, 2005

Examples of OLAP applications in various functional areas © Pearson Education Limited 1995, 2005

OLAP Applications Although OLAP applications are found in widely divergent functional areas, they all have the following key features: multi-dimensional views of data support for complex calculations time intelligence © Pearson Education Limited 1995, 2005

OLAP Applications - multi-dimensional views of data Core requirement of building a ‘realistic’ business model. Provides basis for analytical processing through flexible access to corporate data. The underlying database design that provides the multi-dimensional view of data should treat all dimensions equally. © Pearson Education Limited 1995, 2005

OLAP Applications - support for complex calculations Must provide a range of powerful computational methods such as that required by sales forecasting, which uses trend algorithms such as moving averages and percentage growth. Mechanisms for implementing computational methods should be clear and non-procedural. © Pearson Education Limited 1995, 2005

OLAP Applications – time intelligence Key feature of almost any analytical application as performance is almost always judged over time. Time hierarchy is not always used in the same manner as other hierarchies. Concepts such as year-to-date and period-over-period comparisons should be easily defined. © Pearson Education Limited 1995, 2005

OLAP Benefits Increased productivity of end-users. Reduced backlog of applications development for IT staff. Retention of organizational control over the integrity of corporate data. Reduced query drag and network traffic on OLTP systems or on the data warehouse. Improved potential revenue and profitability. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Example of two-dimensional query. What is the total revenue generated by property sales in each city, in each quarter of 2004?’ Choice of representation is based on types of queries end-user may ask. Compare representation - three-field relational table versus two-dimensional matrix. © Pearson Education Limited 1995, 2005

Multi-dimensional Data as Three-field table versus Two-dimensional Matrix © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Example of three-dimensional query. ‘What is the total revenue generated by property sales for each type of property (Flat or House) in each city, in each quarter of 2004?’ Compare representation - four-field relational table versus three-dimensional cube. © Pearson Education Limited 1995, 2005

Multi-dimensional Data as Four-field Table versus Three-dimensional Cube © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Cube represents data as cells in an array. Relational table only represents multi-dimensional data in two dimensions. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Use multi-dimensional structures to store data and relationships between data. Multi-dimensional structures are best visualized as cubes of data, and cubes within cubes of data. Each side of a cube is a dimension. A cube can be expanded to include other dimensions. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data A cube supports matrix arithmetic. Multi-dimensional query response time depends on how many cells have to be added ‘on the fly’. As number of dimensions increases, number of the cube’s cells increases exponentially. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data However, majority of multi-dimensional queries use summarized, high-level data. Solution is to pre-aggregate (consolidate) all logical subtotals and totals along all dimensions. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Pre-aggregation is valuable, as typical dimensions are hierarchical in nature. (e.g. Time dimension hierarchy - years, quarters, months, weeks, and days) Predefined hierarchy allows logical pre-aggregation and, conversely, allows for a logical ‘drill-down’. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Supports common analytical operations Consolidation Drill-down Slicing and dicing © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Consolidation - aggregation of data such as simple ‘roll-ups’ or complex expressions involving inter-related data. Drill-Down - is the reverse of consolidation and involves displaying the detailed data that comprises the consolidated data. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Slicing and Dicing - (also called pivoting) refers to the ability to look at the data from different viewpoints. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Can store data in a compressed form by dynamically selecting physical storage organizations and compression techniques that maximize space utilization. Dense data (that is, data that exists for a high percentage of cells) can be stored separately from sparse data (that is, a significant percentage of cells are empty). © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data Ability to omit empty or repetitive cells can greatly reduce the size of the cube and the amount of processing. Allows analysis of exceptionally large amounts of data. © Pearson Education Limited 1995, 2005

Representation of Multi-dimensional Data In summary, pre-aggregation, dimensional hierarchy, and sparse data management can significantly reduce the size of the cube and the need to calculate values ‘on-the-fly’. Removes need for multi-table joins and provides quick and direct access to arrays of data, thus significantly speeding up execution of multi-dimensional queries. © Pearson Education Limited 1995, 2005

OLAP Tools There are many varieties of OLAP tools available in the marketplace. This choice has resulted in some confusion with much debate regarding what OLAP actually means to a potential buyer and in particular what are the available architectures for OLAP tools. © Pearson Education Limited 1995, 2005

Codd’s Rules for OLAP Systems In 1993, E.F. Codd formulated twelve rules as the basis for selecting OLAP tools. © Pearson Education Limited 1995, 2005

Codd’s Rules for OLAP Systems Multi-dimensional conceptual view Transparency Accessibility Consistent reporting performance Client-server architecture Generic dimensionality © Pearson Education Limited 1995, 2005

Codd’s rules for OLAP Dynamic sparse matrix handling Multi-user support Unrestricted cross-dimensional operations Intuitive data manipulation Flexible reporting Unlimited dimensions and aggregation levels © Pearson Education Limited 1995, 2005

Codd’s Rules for OLAP Systems There are proposals to re-defined or extended the rules. For example to also include Comprehensive database management tools Ability to drill down to detail (source record) level Incremental database refresh SQL interface to the existing enterprise environment © Pearson Education Limited 1995, 2005

Categories of OLAP Tools OLAP tools are categorized according to the architecture used to store and process multi-dimensional data. There are four main categories: Multi-dimensional OLAP (MOLAP) Relational OLAP (ROLAP) Hybrid OLAP (HOLAP) Desktop OLAP (DOLAP) © Pearson Education Limited 1995, 2005

Multi-dimensional OLAP (MOLAP) Use specialized data structures and multi-dimensional Database Management Systems (MDDBMSs) to organize, navigate, and analyze data. Data is typically aggregated and stored according to predicted usage to enhance query performance. © Pearson Education Limited 1995, 2005

Multi-dimensional OLAP (MOLAP) Use array technology and efficient storage techniques that minimize the disk space requirements through sparse data management. Provides excellent performance when data is used as designed, and the focus is on data for a specific decision-support application. © Pearson Education Limited 1995, 2005

Multi-dimensional OLAP (MOLAP) Traditionally, require a tight coupling with the application layer and presentation layer. Recent trends segregate the OLAP from the data structures through the use of published application programming interfaces (APIs). © Pearson Education Limited 1995, 2005

Typical Architecture for MOLAP Tools © Pearson Education Limited 1995, 2005

MOLAP Tools - Development Issues Underlying data structures are limited in their ability to support multiple subject areas and to provide access to detailed data. Navigation and analysis of data is limited because the data is designed according to previously determined requirements. © Pearson Education Limited 1995, 2005

MOLAP Tools - Development Issues MOLAP products require a different set of skills and tools to build and maintain the database, thus increasing the cost and complexity of support. © Pearson Education Limited 1995, 2005

Relational OLAP (ROLAP) Fastest-growing style of OLAP technology due to requirements to analyze ever-increasing amounts of data and the realization that users cannot store all the data they require in MOLAP databases. © Pearson Education Limited 1995, 2005

Relational OLAP (ROLAP) Supports RDBMS products using a metadata layer - avoids need to create a static multi-dimensional data structure - facilitates the creation of multiple multi-dimensional views of the two-dimensional relation. © Pearson Education Limited 1995, 2005

Relational OLAP (ROLAP) To improve performance, some products use SQL engines to support the complexity of multi-dimensional analysis, while others recommend, or require, the use of highly denormalized database designs such as the star schema. © Pearson Education Limited 1995, 2005

Typical Architecture for ROLAP Tools © Pearson Education Limited 1995, 2005

ROLAP Tools - Development Issues Performance problems associated with the processing of complex queries that require multiple passes through the relational data. Middleware to facilitate the development of multi-dimensional applications. (Software that converts the two-dimensional relation into a multi-dimensional structure). © Pearson Education Limited 1995, 2005

ROLAP Tools - Development Issues Development of an option to create persistent, multi-dimensional structures with facilities to assist in the administration of these structures. © Pearson Education Limited 1995, 2005

Hybrid OLAP (HOLAP) Provide limited analysis capability, either directly against RDBMS products, or by using an intermediate MOLAP server. Deliver selected data directly from the DBMS or via a MOLAP server to the desktop (or local server) in the form of a datacube, where it is stored, analyzed, and maintained locally. © Pearson Education Limited 1995, 2005

Hybrid OLAP (HOLAP) Promoted as being relatively simple to install and administer with reduced cost and maintenance. © Pearson Education Limited 1995, 2005

Typical Architecture for HOLAP Tools © Pearson Education Limited 1995, 2005

HOLAP Tools - Development Issues Architecture results in significant data redundancy and may cause problems for networks that support many users. Ability of each user to build a custom datacube may cause a lack of data consistency among users. Only a limited amount of data can be efficiently maintained. © Pearson Education Limited 1995, 2005

Desktop OLAP (DOLAP) Store the OLAP data in client-based files and support multi-dimensional processing using a client multi-dimensional engine. Requires that relatively small extracts of data are held on client machines. They may be distributed in advance, or created on demand (possibly through the Web). © Pearson Education Limited 1995, 2005

Desktop OLAP (DOLAP) As with multi-dimensional databases on the server, OLAP data may be held on disk or in RAM, however, some DOLAP products allow only read access. Most vendors of DOLAP exploit the power of desktop PC to perform some, if not most, multi-dimensional calculations. © Pearson Education Limited 1995, 2005

Desktop OLAP (DOLAP) The administration of a DOLAP database is typically performed by a central server or processing routine that prepares data cubes or sets of data for each user. Once the basic processing is done, each user can then access their portion of the data. © Pearson Education Limited 1995, 2005

Typical Architecture for DOLAP Tools © Pearson Education Limited 1995, 2005

DOLAP Tools - Development Issues Provision of appropriate security controls to support all parts of the DOLAP environment. Since the data is physically extracted from the system, security is generally implemented by limiting the information compiled into each cube. Once each cube is uploaded to the user's desktop, all additional meta data becomes the property of the local user. © Pearson Education Limited 1995, 2005

DOLAP Tools - Development Issues Reduction in the effort involved in deploying and maintaining the DOLAP tools. Some DOLAP vendors now provide a range of alternative ways of deploying OLAP data such as through e-mail, the Web or using traditional client/server architecture. Current trends are towards thin client machines. © Pearson Education Limited 1995, 2005

OLAP Extensions to SQL Advantages of SQL include that it is easy to learn, non-procedural, free-format, DBMS-independent, and that it is a recognized international standard. However, major limitation of SQL is the inability to answer routinely asked business queries such as computing the percentage change in values between this month and a year ago or to compute moving averages, cumulative sums, and other statistical functions. © Pearson Education Limited 1995, 2005

OLAP Extensions to SQL Answer is ANSI adopted a set of OLAP functions as an extension to SQL to enable these calculations as well as many others that used to be impossible or even impractical within SQL. IBM and Oracle jointly proposed these extensions early in 1999 and they now form part of the current SQL standard, namely SQL: 2003. © Pearson Education Limited 1995, 2005

OLAP Extensions to SQL - RISQL The extensions are collectively referred to as the ‘OLAP package’ and are described as follows: Feature T431, ‘Extended Grouping capabilities’ Feature T611, ‘Extended OLAP operators’ © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities Aggregation is a fundamental part of OLAP. To improve aggregation capabilities the SQL standard provides extensions to the GROUP BY clause such as the ROLLUP and CUBE functions. © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities ROLLUP supports calculations using aggregations such as SUM, COUNT, MAX, MIN, and AVG at increasing levels of aggregation, from the most detailed up to a grand total. CUBE is similar to ROLLUP, enabling a single statement to calculate all possible combinations of aggregations. CUBE can generate the information needed in cross-tabulation reports with a single query. © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities ROLLUP and CUBE extensions specify exactly the groupings of interest in the GROUP BY clause and produces a single result set that is equivalent to a UNION ALL of differently grouped rows. © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities ROLLUP Extension to GROUP BY enables a SELECT statement to calculate multiple levels of subtotals across a specified group of dimensions. ROLLUP appears in the GROUP BY clause in a SELECT statement using the following format: SELECT ... GROUP BY ROLLUP(columnList) © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities ROLLUP creates subtotals that roll up from the most detailed level to a grand total, following a column list specified in the ROLLUP clause. ROLLUP first calculates the standard aggregate values specified in the GROUP BY clause and then creates progressively higher level subtotals, moving from right to left through the column list until finally completing with a grand total. © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities ROLLUP creates subtotals at n + 1 levels, where n is the number of grouping columns. For instance, if a query specifies ROLLUP on grouping columns of propertyType, yearMonth, and city (n = 3), the result set will include rows at 4 aggregation levels. © Pearson Education Limited 1995, 2005

Example - Using the ROLLUP Group Function Show the totals for sales of flats or houses by branch offices located in Aberdeen, Edinburgh, or Glasgow for the months of September and October of 2004. © Pearson Education Limited 1995, 2005

Example - Using the ROLLUP Group Function SELECT propertyType, yearMonth, city, SUM(saleAmount) AS sales FROM Branch, PropertyFor Sale, PropertySale WHERE Branch.branchNo = PropertySale.branchNo AND PropertyForSale.propertyNo = PropertySale.propertyNo AND PropertySale.yearMonth IN ('2004-08', '2004-09') AND Branch.city IN (‘Aberdeen’, ‘Edinburgh’, ‘Glasgow’) GROUP BY ROLLUP(propertyType, yearMonth, city); © Pearson Education Limited 1995, 2005

Example - Using the ROLLUP Group Function © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities CUBE Extension to GROUP BY CUBE takes a specified set of grouping columns and creates subtotals for all of the possible combinations. CUBE appears in the GROUP BY clause in a SELECT statement using the following format: SELECT ... GROUP BY CUBE(columnList) © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities CUBE generates all the subtotals that could be calculated for a data cube with the specified dimensions. CUBE can be used in any situation requiring cross-tabular reports. The data needed for cross-tabular reports can be generated with a single SELECT using CUBE. Like ROLLUP, CUBE can be helpful in generating summary tables. © Pearson Education Limited 1995, 2005

Extended Grouping Capabilities CUBE is typically most suitable in queries that use columns from multiple dimensions rather than columns representing different levels of a single dimension. © Pearson Education Limited 1995, 2005

Example - Using the CUBE Group Function Show all possible subtotals for sales of properties by branches offices in Aberdeen, Edinburgh, and Glasgow for the months of September and October of 2004. © Pearson Education Limited 1995, 2005

Example - Using the CUBE Group Function SELECT propertyType, yearMonth, city, SUM(saleAmount) AS sales FROM Branch, PropertyFor Sale, PropertySale WHERE Branch.branchNo = PropertySale.branchNo AND PropertyForSale.propertyNo = PropertySale.propertyNo AND PropertySale.yearMonth IN ('2004-08', '2004-09') AND Branch.city IN (‘Aberdeen’, ‘Edinburgh’, ‘Glasgow’) GROUP BY CUBE(propertyType, yearMonth, city); © Pearson Education Limited 1995, 2005

Example - Using the CUBE Group Function © Pearson Education Limited 1995, 2005

Elementary OLAP Operators Supports a variety of operations such as rankings and window calculations. Ranking functions include cumulative distributions, percent rank, and N-tiles. Windowing allows the calculation of cumulative and moving aggregations using functions such as SUM, AVG, MIN, and COUNT. © Pearson Education Limited 1995, 2005

Elementary OLAP Operators Ranking Functions Computes the rank of a record compared to other records in the dataset based on the values of a set of measures. There are various types of ranking functions, including RANK and DENSE_RANK. The syntax for each ranking function is: RANK( ) OVER (ORDER BY columnList) DENSE_RANK( ) OVER (ORDER BY columnList) © Pearson Education Limited 1995, 2005

Elementary OLAP Operators The difference between RANK and DENSE_RANK is that DENSE_RANK leaves no gaps in the sequential ranking sequence when there are ties for a ranking. © Pearson Education Limited 1995, 2005

Example - Using the RANK and DENSE_RANK Functions Rank the total sales of properties for branch offices in Edinburgh. SELECT branchNo, SUM(saleAmount) AS sales, RANK() OVER (ORDER BY SUM(saleAmount)) DESC AS ranking, DENSE_RANK() OVER (ORDER BY SUM(saleAmount)) DESC AS dense_ranking FROM Branch, PropertySale WHERE Branch.branchNo = PropertySale.branchNo AND Branch.city = ‘Edinburgh’ GROUP BY(branchNo); © Pearson Education Limited 1995, 2005

Example - Using the RANK and DENSE_RANK Functions © Pearson Education Limited 1995, 2005

Elementary OLAP Operators Supports a variety of operations such as rankings and window calculations. Ranking functions include cumulative distributions, percent rank, and N-tiles. Windowing allows the calculation of cumulative and moving aggregations using functions such as SUM, AVG, MIN, and COUNT. © Pearson Education Limited 1995, 2005

Elementary OLAP Operators Windowing Calculations Can be used to compute cumulative, moving, and centered aggregates. They return a value for each row in the table, which depends on other rows in the corresponding window. © Pearson Education Limited 1995, 2005

Elementary OLAP Operators Windowing Calculations Can be used to compute cumulative, moving, and centered aggregates. They return a value for each row in the table, which depends on other rows in the corresponding window. These aggregate functions provide access to more than one row of a table without a self-join and can be used only in the SELECT and ORDER BY clauses of the query. © Pearson Education Limited 1995, 2005

Example - Using Windowing Calculations Show the monthly figures and three-month moving averages and sums for property sales at branch office B003 for the first six months of 2004. © Pearson Education Limited 1995, 2005

Example - Using Windowing Calculations SELECT yearMonth, SUM(saleAmount) AS monthlySales, AVG(SUM(saleAmount)) OVER (ORDER BY yearMonth, ROWS 2 PRECEDING) AS 3-month moving avg, SUM(SUM(salesAmount)) OVER (ORDER BY yearMonth ROWS 2 PRECEDING) AS 3-month moving sum FROM PropertySale WHERE branchNo = ‘B003’ AND yearMonth BETWEEN ('2004-01' AND '2004-06’) GROUP BY yearMonth ORDER BY yearMonth; © Pearson Education Limited 1995, 2005

Example - Using Windowing Calculations © Pearson Education Limited 1995, 2005