1. Indexing Techniques CS 543 – Data Warehousing

2. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS2 Indexing Goal: Increase efficiency of data access by reducing the number of I/Os required to find desired record(s). Library analogy: Indexed access is analogous to using the card catalog in a library rather than searching through every shelf in the library until the desired book is found (e.g., avoids full table scan).

3. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS3 DW Indexing Issues Indexes and loading Indexing for large tables Index-only reads Selecting columns for indexing A staged approach

4. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS4 B-Tree Index

5. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS5 Bitmapped Index

6. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS6 Bitmapped Index

7. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS7 Indexing the Fact Table If the DBMS does not create an index for the primary key, create one using B-tree indexing In the concatenated primary key, place the primary keys of frequently accessed dimension tables in the top order Create indexes for combinations of dimension table primary keys based query performance Do not overlook indexing metric columns Bitmapped indexing does not apply to fact tables; there is hardly any low-selectivity columns

8. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS8 Indexing the Dimension Tables Create a unique B-tree index on the single-column primary key Index any column that is used frequently to constrain queries Create index for combination of columns that are used frequently together in queries Index every column likely to be used in a join operation

9. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS9 Hash Indexing In contrast to B-tree indexing, hash based indexes do not (typically) keep index values in sorted order. Index entry is located by hashing index value. Index entries keep in hash organized tables rather than B-tree structures. Index entry contains ROWID values for each row corresponding to the index value. ROWIDs kept in sorted order to facilitate maximum I/O performance.

10. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS10 Primary Indexing Primary index for a table in Teradata is a specification of its partitioning column(s). Primary index may be defined as unique (UPI) or non-unique (NUPI).  Automatic enforcement of uniqueness when UPI is specified. Primary index provides an implicit access path to any row just by knowing its value. Only one primary index per table.

11. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS11 Primary Indexing Primary index selection criteria: Common join and retrieval key. Distributes rows evenly across database partitions. Less than ten thousand rows per PI value when non-unique.

12. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS12 Primary Indexing Trick question: What should be the primary index of the transaction table for a large financial services firm? create table tx (tx_id decimal (15,0) NOT NULL,account_id decimal (10,0) NOT NULL,tx_amt decimal (15,2) NOT NULL,tx_dt date NOT NULL,tx_cd char (2) NOT NULL.... ) primary index (???); Answer: It depends.

13. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS13 Primary Indexing Almost all joins and retrievals will come in through the account _id foreign key.  Want account_id as NUPI. If data is “lumpy” when distributed on account_id or if accounts have very large numbers of transactions (e.g., an institutional account could easily have 10,000+ transactions).  Want tx_id as UPI for good data distribution.

14. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS14 Primary Indexing Joins and access via primary index are very efficient due to Teradata’s sophisticated row hashing algorithms that allow going directly to the data block containing the desired row. Single I/O operation for accessing a data row via UPI. Single I/O operation for accessing a data row via NUPI whenever all rows with the same PI value fit into a single block. Single VAMP operation for indexed retrieval. No spool space required.

15. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS15 Primary Indexing Primary index is free! No storage cost. No index build required. This is a direct result of the underlying hash- based file system implementation. OLTP databases use a page-based file system and therefore do not deliver this performance advantage.

16. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS16 Secondary Indexing Secondary index structures are implemented using the same underlying structure as base tables (often referred to as subtables). Secondary index may be defined as unique (USI) or non-unique (NUSI).  Automatic enforcement of uniqueness when USI is specified. Up to thirty-two secondary indexes per table in Teradata. Unlike a primary index, secondary indexes are not “free” in terms of storage.

17. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS17 Secondary Index: NUSI A non-unique secondary index (NUSI) is partitioned so that each index entry is co-located on the same Vamp (Virtual Access Module Processor) with its corresponding row in the base table. Each row access via a NUSI is a single Vamp operation (for that row) because the NUSI entry and data row are co-located. NUSI access is always performed in parallel across all Vamp whenever it is appropriate to do so.

18. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS18 Secondary Indexing: NUSI Compressed ROWID index structure: Hash on index value to get block location (ROWID for subtable). Store index value just once followed by all ROWIDs in base table corresponding to the index value. Sorted by ROWID to facilitate maximum efficiency when accessing base table, performing updates and deletes, etc. Additional blocks allocated when NUSI is non- selective and compressed ROWID structure for the index value exceeds 64K.

19. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS19 Secondary Indexing: NUSI

20. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS20 When to Build a NUSI? Building a NUSI helps when the selectivity of the indexed column is very high. Cost-based optimizer will determine when to access via NUSI: Number of rows selected by NUSI must be less than number of blocks in the table to justify access via NUSI (assumes even distribution of rows with NUSI value within table). Must also consider cost for reading the NUSI subtable and building ROWID spool file. Note that the extreme efficiency of table scanning in Teradata reduces the need for secondary indexing as compared to other databases.

21. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS21 Secondary Indexing: USI A unique secondary index (USI) is partitioned by the unique column upon which the index is built. Row access via a USI is a two Vamp operation.  First I/O is initiated on the Vamp with the USI entry.  Second I/O is initiated on the Vamp with the data row entry.

22. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS22 Secondary Indexing: USI

23. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS23 When to Build a USI? To allow data access without all VAMP operations.  Increased efficiency for (very) high selectivity retrievals. Obtain co-location of index with frequently joined tables.

24. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS24 When to Build a USI? Example: create table order_header (order_id decimal(12, 0) NOT NULL,customer_id decimal(9, 0) NOT NULL,order_dt date NOT NULL... ) primary index( customer_id ); create unique index oh_order_idx (order_id) on order_header; create table order_detail (order_id decimal(12, 0) NOT NULL,product_id integer NOT NULL,extended_price_amt decimal(15,2) NOT NULL,item_cnt integer NOT NULL... ) primary index( order_id );

25. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS25 When to Build a USI? Example: How many customers ordered green socks in the last month? Assume that green socks is quite selective. select count(distinct order_header.customer_id) from order_header,order_detail,product where order_header.order_id = order_detail.order_id and order_header.order_dt > add_months(date, -1) and order_detail.product_id = product.product_id and product.product_subcategory_cd = 'SOCKS' and product.color_cd = 'GREEN' ; The order_id USI on order_header table obviates the need for all Vamp duplication of spool result from order detail to product join when joining to the order header table.

26. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS26 A Simple Query Example: What is the average age (in years) of customers who live in California or Massachusetts, completed a graduate degree, are consultants, and have a hobby of volleyball or chess? select avg( (days(date) - days(customer.birth_dt)) / 365.25 ) from customer where customer.state_cd in (‘CA’, MA’) and customer.education_cd = ‘G’ and customer.occupation_cd = ‘CONSULTANT’ and customer.hobby_cd in (‘VOLLEYBALL’,‘CHESS’) ;

27. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS27 Sample Table Structure Assume: 20M customers. 128 byte rows. 64K data block size. Results in approximately 512 rows per block and a total of 39,063 blocks in the customer table. Note: We are ignoring block overhead for purposes of simplicity in calculations.

28. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS28 Data Demographics Assume: 8% of customers live in California. 4% of customers live in Massachusetts. 4% of customers have completed a graduate degree. 6% of customers are consultants. 2% of customers have a primary hobby of chess. 3% of customers have a primary hobby of volleyball.

29. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS29 Full Table Scan Performance Must read every block in the table. Apply where clause predicates to determine which customers to include in average. Adjust numerator and denominator of average as appropriate. Total I/O count = 39,063 Note: Data demographics have no (minimal) impact on query performance when using a full table scan operation.

30. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS30 Single Index Structure B-tree or hash organization of column values: Index entries store row IDs (RIDs), lists of RIDs, or pointers to lists of RIDs. Originally designed for columns with many unique values (OLTP legacy). Assuming an eight byte RID, we will get 8096 RIDs per 64K block.

31. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS31 Single Index Access ¶Optimizer chooses index with best selectivity based on values specified in query. ·Access next (first) index entry corresponding to specified column value(s). ¸Use RID from index entry to locate row with specified column value. ¹Validate remaining predicates to qualify row. ºAdjust average as appropriate. »Go to 2 until no more matching index values.

32. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS32 Single Index Access What are my indexing choices? state_cd (8% + 4% = 12% selectivity) education_cd (4% selectivity) occupation_cd (6% selectivity) hobby_cd (2% + 3% = 5% selectivity) Choose education_cd because it has best selectivity.

33. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS33 Single Index Performance Access via index on education_cd: 800,000 RIDs (4% of 20M) 99 blocks of RIDs to read But...4% selectivity with 512 rows per block in the base table means that 800,000 selected RIDs will cause access to every block in the base table! Total I/O count = 39,063 + 99 = 39,162 Worse than full table scan!

34. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS34 Single Index Performance Accessing via an index helps only when the selectivity of the indexed column is very high. Rule-of-thumb: Number of rows selected by an index should not be more than the number of blocks in the table to justify indexed access (assumes rows with selected value(s) have an “even” distribution within table). Must also consider cost for reading the index and sorting RIDS (if not already sorted) prior to accessing base table rows (to avoid hitting same block multiple times).

35. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS35 Single Index Performance What is the break even index selectivity (S) versus full table scan performance? Selectivity= S Row Count= 20M rows Block Size= 64K Row Width= 128 bytes RID Width= 8 bytes RIDs per Block= floor(Block Size/RID Width)= 8k Rows per Block= floor(Block Size/Row Width)= 512 Total Blocks= ceiling((Row Count) / (Rows per Block)) = 39,063

36. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS36 Single Index Performance RID I/Os = (S * Row Count) / (RIDs per Block) Indexed Base Table I/Os = (Total Blocks) * (1 - ((1 - S) ** Rows per Block)) Full Table Scan I/Os= Total Blocks Break even formula: RID I/Os + Indexed Base Table I/Os = Full Table Scan I/Os Break Even S is less than 2%.

37. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS37 Single Index Performance Larger row widths and/or smaller block sizes will generally make indexes more desirable because there is a higher probability that a given block will not contain a selected row when fewer rows fit in a block. Example: With a row width of 256 bytes (instead of 128 bytes) the break even selectivity for indexing becomes approximately 2.7%. Of course, larger row widths and/or smaller block sizes means that we are actually getting fewer rows per I/O and thus the amount of work we will do to satisfy a query will generally be higher.

38. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS38 Single Index Performance Traditional index structures work well in OLTP because selectivity is extremely high (1 customer out of 20M or a few accounts out of 50M). Selectivity is 0.000005% and thus is significantly better than the one or two percent required for break even. Bottom line: Traditional indexing is good for OLTP style queries, but is not so great for traditional DSS queries.

39. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS39 Combining Multiple Indexes Observation: Indexed access on a single column is rarely useful in a traditional data warehouse environment. Idea: Combine multiple indexes to get the selectivity required for efficient indexed access.

40. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS40 Combining Multiple Indexes While none of the index choices (state, education, occupation, hobby) are selective enough on their own to be useful...when combined we have sufficient selectivity to make indexed access efficient. Example: Start with 20M customers... Incremental Selected Column Index Selectivity Customers Selectivity by state 12% 2,400,000 Followed with selectivity by education 4% 96,000 Followed with selectivity by occupation 6% 5,760 Followed with selectivity by hobby 5% 288 Combined selectivity is 0.00144% Note: These selectivity figures assume that column values are independent (...and they usually are not).

41. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS41 Combining Multiple Indexes Must consider I/O cost of accessing four different indexes plus cost of accessing selected blocks from base table: State= 297 Education= 99 Occupation= 149 Hobby= 124 Base table= 288 ===== Total= 957 More efficient than a full table scan!

42. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS42 Combining Multiple Indexes Notice that there is a significant performance benefit if we can satisfy a query directly out of indexes without accessing base table. Example: How many customers live in California or Massachusetts, completed a graduate degree, are consultants, and have a hobby of volleyball or chess? select count(*) from customer where customer.state_cd in ('CA','MA') and customer.education_cd = 'G' and customer.occupation_cd = 'CONSULTANT' and customer.hobby_cd in ('VOLLEYBALL','CHESS') ;

43. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS43 Combining Multiple Indexes Now we only need to consider the I/O cost of accessing the four indexes (sans the cost of accessing base table): State=297 Education= 99 Occupation=149 Hobby=124 ===== Total= 669 Much more efficient than a full table scan!

44. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS44 Combining Multiple Indexes Traditional combining of multiple indexes: Requires RID list ANDing (and sometimes ORing) to combine the multiple indexes. May incur overhead of RID list sorting in order to facilitate ANDing operation (depends on RDBMS indexing implementation). This technique is useful when no one index is selective enough to produce an efficient access path, but multiple indexes taken together can provide the needed selectivity.

45. CS 543 - Data Warehousing (Sp 2007-2008) - Asim Karim @ LUMS45 Bottom Line Optimizer sophistication is critical in effectively exploiting indexes. Selectivity of indices are critical in determining their usefulness. Indexed access paths are not nearly as useful in data warehousing as compared to OLTP workloads.