Presentation is loading. Please wait.

Presentation is loading. Please wait.

Dimensional modelling - Star-join schemas

Similar presentations


Presentation on theme: "Dimensional modelling - Star-join schemas"— Presentation transcript:

1 Dimensional modelling - Star-join schemas
Service Dimension Time Dimension Sales Dimension Customer Dimension Fact table - Transactions Sum Number of calls C210 S1 F11 991011 25:00 3 S3 05:00 1 C212 S2 F13 89:00 C213 12:00 C214 S4 991012 08:00 När man modellerara fokuserar man på de viktiga affärshändelserna -transactions, i verkamheten. Det kan till exempel vara säljhändelser. Att man säljer varor eller tjänster. Jag kommer återkomma till andra typer av händelser - men centarlt när man modellerar på det här sättet är att identifiera händelserna. Man samla dessa händelser och fakta, dvs värdet, om händelserna i en entitet som vi kallar försäljningsfakta. Försäljningsfaktaentiteten innehåller alltså det här värdet vi såg förut i kuben. Sedan kan man studera dessa händelser ur olika aspekter, dimensioner. Dimensionerna motsvarar dimensionerna vi såg förut. När man modellerar gäller det alltså att välja dimensionerna, till exempel vilka tjänster eller tjänstegrupper som säljs, vilka kunder eller kundkategorier som köper tjänster. Vid vilka tidpunkter eller tidsperioder som tjänsterna säljs. Vilka försäljare eller försäljningskontor som sålt tjänsterna. Sedan väljer på attributen för dessa dimensioner - för tjänst kanske vi lägger på tjänstenamn och tjänstegrupp. För kund väljer vi kundnamn, postadress, region, inkomstgrupp. Fakta i mitten, och runt om får vi dimensionser Gör om till databasschema med tabeller. Entiteterna blir tabeller. Vi får faktatabelloch dimensionstabeller. Då hamnar attributen som kolumner. Bofolkar tabellerna med instanser, rader, tupler. Mellan dimensionstabellerna och faktatabellen får vi ett till många förhållande. Dimensionerna innehåller få rader. Ej normaliserat. Dubbellagra information Faktatabellen innehåller massor med rader. Den innehåller den överväldigasnde delen av information. Den vill manska ha så få och små kolumner som mjligt. I faktatabellen finns främmande nycklar till de olika dimensionerna. Plockar bort rader i tabellerna. Går in och joinar mellan dimensioner och faktatabell för att ta bort rader i faktatabell. Tar bort alla rader som inte är S1 Enkelt att förstå en sådan här struktur för verksamhetschefer än en relationsdatabasstruktur. Normalisera av två orsaker– för att ha en så efffektiv lagring, spara utrymmer, som möjligt +redundans/inkonsistens. Men insonstistesen hanteras av transformationlagret. Och spara utrymme är sekundärt. OLAP-verktyget kan använda den här datastrukturen effektivt. Normaliserar vissa attribut – vinne kortare rader, men innebär att man måste joina.

2 Partitioning strategy
for performace-related and manageablity reasons usually for handling the fact table Horisontal partitioning - speed up queries by minimising the data to be scanned (without using an index) - partition by time most common Vertical partitioning - data is split vertically - two forms: normalisation and row splitting - consider row splitting if some columns are access infrequently Dim Fact Dim Dim

3 A family of stars

4 A family of stars A dimensional model of a data warehouse for a large data warehouse consists of between 10 and 25 similar-looking star-join schemas. Each star join will have 5 to 15 dimensional tables. Conformed (shared) dimensions for drill-across. A Conformed dimension is a dimension that means the same thing with every possible fact table to which it can be joined.

5 Value chains as families of star-join schemas
There are two sides to the value chain the demand side - the steps needed to satisfy the customers’ demand for the product the supply side - the steps needed to manufacture the products from original ingredients or parts The chain consists of a sequence of inventory and flow star-join schemata joining the different star-join schemata is only possible when two sequential schemata have a common, identical dimension Sometimes the represented chain can be extended beyond the bounds of the business itself

6 Value chains as families of star-join schemas
Supply Chain Row material production Ingredient purchasing Ingredient delivery Ingredient inventory Bill of materials Manufacturing process control Manufacturing costs Packaging Trans-shipping to warehouse Finished goods inventory Demand Chain Finished goods inventory Manufacturing shipments Distributor inventory Distributor shipments Retail inventory Retail sales

7 The Data Warehouse Bus Dimensions Orders Production
Time Sales Rep Customer Promotion Product Plant Distr. Center Allows the parallell dvlpmt of business process data marts with ability to integrate

8 Problems of Data Warehousing
Complexity of integration Hidden problems with source systems Data homogenisation Underestimation of resources for data loading Required data not captured High maintenance Long duration projects Why not integrating the legacy applications (OLTP systems) instead? Svårt att få ut data. Svårt att göra data homogen – så att den betyder samma sak. Den data man behöver finns inte. Förvaltningen av dw

9 The stove pipe problem Business process IT- system1 system2 system3
Market Purchase Production Shipment Service Business process IT- system1 system2 system3 system4 IT system5

10 The stove pipe problem Business process IT systems IT- system1 system2
Market Purchase Production Shipment Service Business process IT- system1 system2 system3 system4 IT system5 IT systems

11 Integrating the Enterprise - ERP/ES & DW
Reporting and DW Managers Customers Suppliers Sales force Customer service Back office Services Sales & delivery Finan- cials Manufac- turing Inventory Central database Any organisation has to interface to their customers by means of front office functions such as sales and services, as well as to their suppliers by means of corresponding back office units. The organisational functions that need to be supported are hence such as sales and delivery, services, financials, Manufacturing Inventory But also the human resource has to be managed And managers need to be supported by information and system functionality that help them make wise tactical and strategic decisions. In an enterprise system all of this would be supported by an integrated set of databases with no unnecessary redundancy. Human resource Employees

12 case Integration of Telecom systems - point-to-point

13 Integration of Telecom systems - Message Brokers & DW

14 Dimensional modelling vs. ER-modelling
Entity-relationship modelling - a logical design technique to eliminate data redundancy to keep consistency and storage efficiency - makes transaction simple and deterministic - ER models for enterprise are usually complex, e.g. they often have hundreds, or even thousands, of entities/tables Dimensional modelling - a logical design technique that present data in a intuitive way and that allow high-performance access - aims at model decision support data - easier to navigate for the user and high performance

15 Why dimensional modelling?
the logical model is easy understand a predictable standard framework for end user applications the logical design can be done nearly independent of expected query pattern handle changes easy - at least adding new dimensional attributes high performance “browsing” across the attributes, eliminating joins and make use bit vector indexes strategy to handling aggregates, e.g. summery records that are logical redundant with base table to enhance query performance the database engine can make strong assumption how to optimise strategies for handling slowly changing dimensions, heterogenous products, event-handling (“factless fact tables”)

16 Steps in the Design Process
1 Choose a business process to model A business process is a major operational process in an organi- sation, that is supported by some kind of a legacy system(s) from which data can be collected, e.g., orders, invoices, shipments, inventory. 2 Choose the grain of the business process The grains is the level of detail at which the data is represented in the DW. Typical grains are individual transactions, individual daily (monthly) snapshots. 3 Choose the dimensions that will apply to each fact table record Typical dimensions are time, product, customer, store, etc. 4 Choose the measured facts that will populate fact table E.g., quantity sold, dollars sold

17 Consider the following questions
How much total business did my newly remodelled stores do compared with the chain average? How did leather goods items costing less than $5 do with my most frequent shoppers? What was the ratio of nonholiday weekend days total revenue to holiday weekend days? A major point of the analysis done so far is to show that the foundation layers of all these dimensional data warehouses have huge number of records. This large number of detailed records is needed, not because an analyst is going to need a single record at this level, but because the analyst must be able to make very precise cuts between records. If we don’t tract the individual accounts, we can’t group accounts into the most interesting clumps. If we don’t track the lowest-level product subdivision, then we can’t pull out all the groups of products we might want to track. Even though many queries will opportunistically need to retrieve clumps of base-level data, most queries will relax the constraints on ton or more of the dimensions. If all we have is base-level data, then relaxing constrains on one or more dimensions means that we let a vast number of records into our query. Consider the queries … In all three cases, we need very detailed data from one of the major dimensions, but we are either summarising or omitting detail from some of the other dimensions. In the 1st q., we need specific stores but we are summarising over all products. In the 2nd q. We need specific products but we are summarising across all stores. In the 3rd q, we need specific says on the calendar but we are summarising across both product and store. All of these questions will be expensive to process if we have only base-level data. What is needed here is a set of several different precomputed aggregates that will accelerate each of these queries. The effect on performance is not small. It is typical to expect anywhere from a 10-fold to a 1000-fold improvements in runtime by having the right aggregates available.

18 Aggregation Aggregations can be created on-the-fly or by the process of pre-aggregation An aggregate is a fact table record representing a summarisation of base-level fact table records Category-level product aggregates by store by day District-level store aggregates by product by day Monthly sales aggregates by product by store Category-level product aggregates by store district by day Category-level product aggregates by store district by month The use of prestored summaries (aggregates) is the single most effective tool the data warehouse designer has to control performance. An aggregate is a fact table record representing a summarisation of base-level fact table record. An aggregate fact table record is always associated with one or more aggregate dimension table records. For example, in a base level fact table dimensioned by UPC (Unified product code) level products, individual stores, and individual days, we can imagine the following aggregate records: Not only do we have these new fact table records, but we must have aggregate dimension table entries describing category level products, store districts, and months.

19 How to store aggregates
as new Level fields in an already existing Fact table as new fact tables

20 Example of a Dimensional Model

21 New Level field for Aggregates
One approach to storing aggregates is by using Level fields in the affected dimension tables, and thereby allowing the aggregate fact records to reside in the original fact table. If we look at the grocery store aggregate example with category totals would result in the shown model. The Level field describe the aggregation level of every record in the dimension table. All of the original base-level records are encoded with Level = Base. The new category total records are encoded with Level=Category. These category product aggregate records need to have keys that are compatible with and do not conflict with the original base-level keys in the product dimension table. However, all the rest of the fields have no valid meaning, and have to take on null or ‘NA’ or ‘Total’ values. The most serious application problem with this solution is that it is possible to double count. This happens if the requesting application fails to constrain on a single value for a Level field. For instance, if the only product table constraint is Category = Paper, then both the base-level records and the category-level records will be retrieved. Tissue NB! Constraint the queries to avoid double counting of the Level fields

22 An example ? $sold napkin/day ? $sold tissue/day ? $sold paper/day

23 New Level field for Aggregates - an example
? $sold napkin/day ? $sold tissue/day ? $sold paper/day Well, is this a solution you would chose?

24 New Tables

25 New Fact Tables for Aggregates
The creation of aggregate fact table requires the creation of: a derivative dimension an artificial key for each new derivative dimension

26 How to store aggregates
as new Level fields in an already existing Fact table problems with double count visible for the users as new fact tables + no problems with double count + invisible for the users + are easily introduced and/or reduced at different points in time + simpler metadata + simpler choice of key + the size of the field for the summarised data does not increase the size of the field for the basic data

27 Sparsity Failure The planning of aggregate table sizes can
be tricky because of the phenomenon called sparsity failure This phenomenon appears when we build aggregates on sparse tables. For example: In the grocery store item movement database, only about 10% of the products in the store are actually sold in a given store on a given day. Even disregarding the promotion dimension, the database is only occupied 10% in the primary keys of product, store, and time. However when we build aggregates, the occupancy rate shoots up dramatically.

28 Aggregation Navigator
Query Tool Client or Application Server base level SQL aggregated results Aggregation Metadata Aggregate Navigator aggregate-aware SQL aggregated results DBMS data + aggregations

29 An example of SQL query SELECT category_description, sum(sales_dollars) FROM base_sales_fact, product, store, time WHERE base_sales_fact.product_key = product.product_key AND base_sales_fact.store_key = store.store_key AND base_sales_fact.time_key = time.time_key AND store.city = ‘Cincinnati’ AND time.day = ‘January 1, 1996’ GROUP BY category_description category_sales_fact category_product

30 An example of SQL query, cont
SELECT category_description, sum(sales_dollars) FROM category_sales_fact, category_product, store, time WHERE category_sales_fact.product_key = category_product.product_key AND category_sales_fact.store_key = store.store_key AND category_sales_fact.time_key = time.time_key AND store.city = ‘Cincinnati’ AND time.day = ‘January 1, 1996’ GROUP BY category_description

31 Aggregation Navigator
Insulates end user applications from the changing portfolio of aggregates Allows the DBA to dynamically and seamlessly for the end user adjust the aggregates without having to roll over the applications base

32 Aggregations - summary
Pre-aggregation demands more storage space but provides better query performance Lowest level of aggregation is determined by the granularity of the fact table Aggregation is easier when facts are all additive

33 Bitmap indexing An effective indexing technique for attributes with low-cardinality domains There is a distinct bit vector BV for each value V of the domain Example: the attribute sex has value M and F. A table of 100 million people needs 2 lists of 100 million bits.

34 Bitmap Index Base Table Region Index Rating Index SELECT Customers
Text from Ramakrishnan&Gehrke, sid 691 (men exempeln är det samma som jag hade förra året) Consider a table that describes customers: CUST(Custid:Int, Region:String, Rating:String, …) where Domain(Region) = {N,W,S,E} and Domain(Rating) = {H,M,L} Columns with few possible values are called sparse [like, Region and Rating]. We can exploit sparsity to construct a new kind of index that greatly speeds up queries on these columns. The idea is to record values for sparse columns as a sequence of bits, one for each possible value.” For example, a rating value is 100, 010, or 001, a 1 in first position denotes ‘high’, a 1 in second position denotes ‘medium’, and a 1 in third position denotes ‘low’. Similarly, for region, 1000 denotes ‘North’, 0100 denotes ‘Sought’, 0010 denotes ‘East’, and 0001 denotes ‘West’. If we consider the rating values for all rows in the Customer table, we can treat this as a collection of three bit vectors, one of which has the associated value ‘high’, the other one the associated value ‘medium’ and the third one has the associated value ‘low’. Each bit vector has one bit per row in the Customers table, indication whether the value in that row is the value associated with the bit vector. The collection of bit vectors for a column is called a bitmap index for that column. SELECT Customers FROM Base Table WHERE Region = W AND Rating = L

35 Bitmap Index Base Table Region Index Rating Index Region = W AND
Bitmap indexes offer two important advantages over conventional hash and tree indexes. First, they allow the use of efficient bit operations to answer queries. For example, consider the query “How many customer from the West region are rated as low”. We can take the fourth bit vector for Region and do a bit-wise and AND with the third bit vector for Rating to obtain a bit vector that has 1 for every West-living customer with rating Low. We can then count the number of 1s in this bit vector to answer the query. Second, bitmap indexes can be much more compact than a traditional B+ tree index and are very amenable to the use of compression techniques. Region = W AND Rating = L

36 Bitmap Index Base Table Region Index Rating Index Region = W AND
Rating = L

37 Mullet-dimensional OLAP (MOLAP)
Relational DB server and/or legacy systems End-user access tools MOLAP server data request load result set Database & application logic layer Presentation layer

38 Relational OLAP (ROLAP)
db server ROLAP server End-user access tools SQL data request result set result set Database layer Application logic layer Presentation layer

39 DB2’s Integration Server Architecture
Desktop OLAP Model OLAP Metaoutline Integration Server desktop TCP/IP DB2 OLAP server TCP/IP Server ODBC Relational data source ODBC TCP/IP OLAP Metadata Catalog OLAP Command Interface DV2 OLAP database


Download ppt "Dimensional modelling - Star-join schemas"

Similar presentations


Ads by Google