Presentation is loading. Please wait.

Presentation is loading. Please wait.

CSCI5570 Large Scale Data Processing Systems

Similar presentations


Presentation on theme: "CSCI5570 Large Scale Data Processing Systems"— Presentation transcript:

1 CSCI5570 Large Scale Data Processing Systems
Distributed Data Storage Systems James Cheng CSE, CUHK Slide Ack.: modified based on the slides from Stavros Harizopoulos and Jianlin Feng

2 C-Store: A Column-oriented DBMS
Mike Stonebraker, Daniel Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Sam Madden, Elizabeth O'Neil, Pat O'Neil, Alex Rasin, Nga Tran and Stan Zdonik VLDB 2005

3 What is a column-store? row-store column-store
Date Store Product Customer Price Date Store Product Customer Price + easy to add/modify a record - might read in unnecessary data + only need to read in relevant data - tuple writes require multiple accesses => suitable for read-mostly, read-intensive, large data repositories

4 Row Store: Why? OLTP (On-Line Transaction Processing)
ATM, POS in supermarkets Characteristics of OLTP applications Transactions that involve small numbers of records (or tuples) Frequent updates (including queries) Many users Short response time OLTP needs Write-Optimized Row Store Insert or delete a record in one physical write

5 Row Store: Columns Stored Together
Data Rid = (i,N) Page i Rid = (i,2) Rid = (i,1) 20 16 24 N Pointer to start of free space N Slot Array # slots SLOT DIRECTORY Record id = <page id, slot #>

6 Current DBMS Gold Standard
Store columns in one record contiguously on disk Use B-tree indexing Use small (e.g. 4K) disk blocks Align fields on byte or word boundaries Conventional (row-oriented) query optimizer and executor (technology from 1979)

7 From OLTP to OLAP and Data Warehouse
OLAP (On-Line Analytical Processing, Codd, 1993) Flexible Reporting for Business Intelligence Characteristics of OLAP applications Transactions that involve large numbers of records Frequent ad-hoc queries and infrequent updates A few decision making users High throughput but longer response time Data warehouses are designed to facilitate reporting and analysis Read-mostly

8 A Data Warehousing example
Typical DW installation Real-world example dimension tables account fact table or RAM usage source toll star schema “One Size Fits All? - Part 2: Benchmarking Results” Stonebraker et al. CIDR 2007 Column-store Row-store Query 1 2.06 300 Query 2 2.20 Query 3 0.09 Query 4 5.24 Query 5 2.88 QUERY 2 SELECT account.account_number, sum (usage.toll_airtime), sum (usage.toll_price) FROM usage, toll, source, account WHERE usage.toll_id = toll.toll_id AND usage.source_id = source.source_id AND usage.account_id = account.account_id AND toll.type_ind in (‘AE’. ‘AA’) AND usage.toll_price > 0 AND source.type != ‘CIBER’ AND toll.rating_method = ‘IS’ AND usage.invoice_date = GROUP BY account.account_number Why? Three main factors (next slides)

9 example explained: read efficiency
row-store column-store read pages containing entire rows one row = 212 columns! is this typical? (it depends) read only columns needed in this example: 7 columns caveats: “select * ” not any faster clever disk prefetching clever tuple reconstruction What about vertical partitioning? (it does not work with ad-hoc queries)

10 example explained: compression efficiency
Columns compress better than rows Typical row-store compression ratio 1 : 3 Column-store 1 : 10 Why? Rows contain values from different domains => more entropy, difficult to dense-pack Columns exhibit significantly less entropy Examples: Caveat: CPU cost (use lightweight compression) Male, Female, Female, Female, Male 1998, 1998, 1999, 1999, 1999, 2000

11 example explained: sorting & indexing efficiency
Compression and dense-packing free up space Use multiple overlapping column collections Sorted columns compress better Range queries are faster Use sparse clustered indexes What about heavily-indexed row-stores? (works well for single column access, cross-column joins become increasingly expensive)

12 Column-Specific Optimizations
Late materialization Construct tuples as late as possible Column-specific compression Run-Length Encoding Invisible join on star schema Like semi-join in distributed DBMS

13 Effect on C-Store performance
SSBM = Star Schema BenchMark enable late materialization enable compression & operate on compressed column-oriented join algorithm “Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Hachem, and Madden. SIGMOD 2008.

14 Which Column-Specific Optimization is most significant?
Late materialization Improves performance by a factor of 3 Column-specific compression Improves performance by a factor of 2 on average Improves performance by a factor of 10 on queries that access sorted data Invisible join on star schema Improves performance by 50% - 70%

15 Compression “Super-Scalar RAM-CPU Cache Compression”
Zukowski, Heman, Nes, Boncz, ICDE’06 “Integrating Compression and Execution in Column Oriented Database Systems” Abadi, Madden, and Ferreira, SIGMOD ’06 “Query optimization in compressed database systems” Chen, Gehrke, Korn, SIGMOD’01

16 Compression Trades I/O for CPU Increased column-store opportunities:
Higher data value locality in column stores Techniques such as run length encoding far more useful Can use extra space to store multiple copies of data in different sort orders

17 Run-length Encoding … … … … … … … … Quarter Product ID Quarter
(value, start_pos, run_length) Product ID (value, start_pos, run_length) Quarter Product ID Price Price Q1 1 2 5 7 2 9 6 8 (Q1, 1, 300) (Q2, 301, 350) (Q3, 651, 500) (Q4, 1151, 600) (1, 1, 5) (2, 6, 2) 5 7 2 9 6 8 (1, 301, 3) (2, 304, 1) Q2 1 2 3 8 1 4 3 8 1 4

18 Bit-vector Encoding … … … … … … … … … … …
Product ID ID: 1 ID: 2 ID: 3 1 2 1 1 For each unique value, v, in column c, create bit-vector b b[i] = 1 if c[i] = v Good for columns with few unique values Each bit-vector can be further compressed if sparse 1 2 3 1 1 1

19 Dictionary Encoding OR … … …
Quarter Quarter Quarter For each unique value create dictionary entry Dictionary can be per-block or per-column Column-stores have the advantage that dictionary entries may encode multiple values at once Q1 Q2 Q4 Q3 1 3 2 24 128 122 + Dictionary 24: Q1, Q2, Q4, Q1 OR 122: Q2, Q4, Q3, Q3 128: Q3, Q1, Q1, Q1 + Dictionary 0: Q1 1: Q2 2: Q3 3: Q4

20 Frame Of Reference Encoding
Price Price Encodes values as b bit offset from chosen frame of reference Special escape code (e.g. all bits set to 1) indicates a difference larger than can be stored in b bits After escape code, original (uncompressed) value is written Frame: 50 45 54 48 55 51 53 40 50 49 62 52 -5 4 -2 5 1 3 4 bits per value 40 -1 Exceptions 62 “Compressing Relations and Indexes ” Goldstein, Ramakrishnan, Shaft, ICDE’98 2

21 Differential Encoding
Time Time Encodes values as b bit offset from previous value Special escape code (just like frame of reference encoding) indicates a difference larger than can be stored in b bits After escape code, original (uncompressed) value is written Performs well on columns containing increasing/decreasing sequences inverted lists Timestamps object IDs sorted / clustered columns 5:00 5:02 5:03 5:04 5:06 5:07 5:08 5:10 5:15 5:16 5:00 2 1 2 bits per value Exceptions 5:15 1 “Improved Word-Aligned Binary Compression for Text Indexing” Ahn, Moffat, TKDE’06

22 What Compression Scheme To Use?
Does column appear in the sort key? no yes Are number of unique values < ~50000 Is the average run-length > 2 yes yes no RLE Differential Encoding Does this column appear frequently in selection predicates? Is the data numerical and exhibit good locality? yes no no yes Bit-vector Compression Dictionary Compression Leave Data Uncompressed Frame of Reference Encoding OR Heavy weight Compression

23 Heavy-Weight Compression Schemes
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06 Modern disk arrays can achieve > 1GB/s Lightweight compression schemes are better Even better: operate directly on compressed data

24 Operating Directly on Compressed Data
I/O - CPU tradeoff is no longer a tradeoff Reduces memory–CPU bandwidth requirements Opens up possibility of operating on multiple records at once “Integrating Compression and Execution in Column Oriented Database Systems” Abadi et. al, SIGMOD ’06

25 Operating Directly on Compressed Data
“Integrating Compression and Execution in Column Oriented Database Systems” Abadi et. al, SIGMOD ’06 Quarter Product ID 1 2 3 ProductID, COUNT(*)) (1, 3) (2, 1) (3, 2) SELECT ProductID, Count(*) FROM table WHERE (Quarter = Q2) GROUP BY ProductID

26 Tuple Materialization and Column-Oriented Join Algorithms
“Materialization Strategies in a ColumnOriented DBMS” Abadi, Myers, DeWitt, and Madden. ICDE 2007. “Self-organizing tuple reconstruction in column-stores“, Idreos, Manegold, Kersten, SIGMOD’09 “Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008. “Query Processing Techniques for Solid State Drives”Tsirogiannis, Harizopoulos Shah, Wiener, and Graefe. SIGMOD 2009. “Cache-Conscious Radix-Decluster Projections”, Manegold, Boncz, Nes, VLDB’04

27 Column Projection Column projection involves reading needed columns from storage and extracting values for a listed set of tuples This process is called “materialization” Early materialization: project columns at beginning of query plan Straightforward since there is a one-to-one mapping across columns Late materialization: wait as long as possible for projecting columns More complicated since selection and join operators on one column obfuscates mapping to other columns from same table

28 Early Materialization
Select + Aggregate 4 2 7 QUERY: SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID 4 1 3 13 4 3 42 4 1 3 80 Construct Caveats Need to construct ALL tuples Need to decompress data Poor memory bandwidth utilization (4, 1, 4) 2 1 3 2 3 7 13 42 80 prodID storeID custID price

29 Late Materialization 1 4 2 3 QUERY: SELECT custID,SUM(price)
2 3 storeID prodID Data Source QUERY: SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID 4 2 1 3 2 3 7 13 42 80 prodID storeID custID price

30 Late Materialization 1 AND QUERY: SELECT custID,SUM(price) FROM table
AND QUERY: SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID 4 2 1 3 2 3 7 13 42 80 prodID storeID custID price

31 Late Materialization 3 1 2 QUERY: SELECT custID,SUM(price) FROM table
1 Data Source 2 custID 13 80 QUERY: SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID 7 13 42 80 price 4 2 1 3 2 3 7 13 42 80 prodID storeID custID price

32 Late Materialization QUERY: SELECT custID,SUM(price) FROM table
WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID 3 93 1 3 AGG 13 80 4 2 1 3 2 3 7 13 42 80 prodID storeID custID price

33 For plans without joins, late materialization is a win
1 2 3 4 5 6 7 8 9 10 Low selectivity Medium selectivity High selectivity Time (seconds) Early Materialization Late Materialization Ran on 2 compressed columns from TPC-H scale 10 data QUERY: SELECT C 1 , SUM(C 2 ) FROM table WHERE (C CONST) AND ( C CONST GROUP BY “Materialization Strategies in a Column-Oriented DBMS” Abadi, Myers, DeWitt, and Madden. ICDE 2007.

34 Even on uncompressed data, late materialization is still a win
Materializing late still works best QUERY: SELECT C 1 , SUM(C 2 ) FROM table WHERE (C CONST) AND ( C CONST GROUP BY 4 6 8 10 12 14 Low selectivity Medium selectivity High selectivity Time (seconds) Early Materialization Late Materialization

35 What about for plans with joins?
B, C, E, H, F Select R1.B, R1.C, R2.E, R2.H, R3.F From R1, R2, R3 Where R1.A = R2.D AND R2.G = R3.K Project Join G = K B, C Early Materialization Late Materialization F B, C, E, H, F Join 2 G = K G K B, C, E, G, H Project Join A = D Scan Join 1 A = D F, K Scan G E, H A, B, C D, E, G, H A D Scan Scan Scan Scan

36 Early Materialization Example
2 7 QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName 3 13 1 Green 3 42 2 White 3 80 3 Brown Construct Construct (4, 1, 4) 12 1 11 4 6 1 2 2 3 7 13 42 80 1 2 3 Green White Bronw custID lastName prodID storeID quantity custID price Facts Customers

37 Early Materialization Example
QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName 7 White 13 Brown 42 Brown 80 Brown Join 2 7 3 13 1 Green 3 42 2 White 3 80 3 Brown

38 Late Materialization Example
Positions of items in “price” obtained according to join result 1 2 3 4 2 3 QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName Positions of items in “lastName” obtained according to join result Join Late materialized join causes out of order probing of projected columns from the inner relation (4, 1, 4) 12 1 11 4 6 1 2 2 3 7 13 42 80 1 2 3 Green White Bronw custID lastName prodID storeID quantity custID price Facts Customers

39 Late Materialized Join Performance
Naïve LM join about 2X slower than EM join on typical queries (due to random I/O) This number is very dependent on Amount of memory available Number of projected attributes Join cardinality But we can do better Invisible Join Jive/Flash Join Radix cluster/decluster join

40 Invisible Join “Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008. Designed for typical joins when data is modeled using a star schema One (“fact”) table is joined with multiple dimension tables Example of star schema:

41 Invisible Join “Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008. Designed for typical joins when data is modeled using a star schema One (“fact”) table is joined with multiple dimension tables Typical query: select c_nation, s_nation, d_year, sum(lo_revenue) as revenue from customer, lineorder, supplier, date where lo_custkey = c_custkey and lo_suppkey = s_suppkey and lo_orderdate = d_datekey and c_region = 'ASIA‘ and s_region = 'ASIA‘ and d_year >= 1992 and d_year <= 1997 group by c_nation, s_nation, d_year order by d_year asc, revenue desc;

42 Invisible Join Apply “region = ‘Asia’” On Customer Table
custkey region nation 1 ASIA CHINA 2 INDIA 3 4 EUROPE FRANCE Hash Table (or bit-map) Containing Keys 1, 2 and 3 Apply “region = ‘Asia’” On Supplier Table suppkey region nation 1 ASIA RUSSIA 2 EUROPE SPAIN 3 JAPAN Hash Table (or bit-map) Containing Keys 1, 3 Apply “year in [1992,1997]” On Date Table Hash Table Containing Keys , , and dateid year 1997

43 Invisible Join & & = + + + Original Fact Table Hash Table Containing
orderkey custkey suppkey orderdate revenue 1 3 43256 2 33333 4 12121 23233 5 45456 6 43251 7 34235 1 1 1 1 & & = Hash Table Containing Keys , , and Hash Table Containing Keys 1, 2 and 3 Hash Table Containing Keys 1 and 3 + + + custkey 3 4 1 suppkey 1 2 3 1 orderdate 1 1

44 + + = + = + + = JOIN custkey 3 4 1 1 CHINA INDIA FRANCE 3 1 INDIA
CHINA INDIA FRANCE + + = 3 1 INDIA CHINA suppkey 1 2 3 1 + RUSSIA SPAIN JAPAN = RUSSIA + 1 orderdate 1 + 1997 = JOIN 1997

45 (between-predicate rewriting)
Invisible Join Apply “region = ‘Asia’” On Customer Table custkey region nation 1 ASIA CHINA 2 INDIA 3 4 EUROPE FRANCE Hash Table (or bit-map) Containing Keys 1, 2 and 3 Range [1-3] (between-predicate rewriting) Apply “region = ‘Asia’” On Supplier Table suppkey region nation 1 ASIA RUSSIA 2 EUROPE SPAIN 3 JAPAN Hash Table (or bit-map) Containing Keys 1, 3 Apply “year in [1992,1997]” On Date Table Hash Table Containing Keys , , and dateid year 1997

46 Invisible Join Bottom Line
Many data warehouses model data using star/snowflake schemes Joins of one (fact) table with many dimension tables is common Invisible join takes advantage of this by making sure that the table that can be accessed in position order is the fact table for each join Position lists from the fact table are then intersected (in position order) This reduces the amount of data that must be accessed out of order from the dimension tables

47 + + = + = + + = JOIN Still accessing table out of order custkey 3 4 1
CHINA INDIA FRANCE + + = 3 1 INDIA CHINA suppkey 1 2 3 1 Still accessing table out of order + RUSSIA SPAIN JAPAN = RUSSIA + 1 orderdate 1 + 1997 = JOIN 1997

48 Jive/Flash Join + = 3 1 CHINA INDIA FRANCE INDIA CHINA
“Fast Joins using Join Indices”. Li and Ross, VLDBJ 8:1-24, 1999. Still accessing table out of order “Query Processing Techniques for Solid State Drives”. Tsirogiannis, Harizopoulos et. al. SIGMOD 2009.

49 Jive/Flash Join Add column with dense ascending integers from 1 Sort new position list by second column Probe projected column in order using new sorted position list, keeping first column from position list around Sort new result by first column 3 1 + CHINA INDIA FRANCE = INDIA CHINA 1 3 2 2 1 3 + CHINA INDIA FRANCE = 2 CHINA 1 INDIA 1 INDIA 2 CHINA

50 Jive/Flash Join Bottom Line
Instead of probing projected columns from inner table out of order: Sort join index Probe projected columns in order Sort result using an added column LM vs EM tradeoffs: LM has the extra sorts (EM accesses all columns in order) LM only has to fit join columns into memory (EM needs join columns and all projected columns) Results in big memory and CPU savings LM only has to materialize relevant columns In many cases LM advantages outweigh disadvantages LM would be a clear winner if not for those pesky sorts … can we do better?

51 Radix Cluster/Decluster
The full sort from the Jive join is actually overkill We just want to access the storage blocks in order (we don’t mind random access within a block) So do a radix sort and stop early By stopping early, data within each block is accessed out of order, but in the order specified in the original join index Use this pseudo-order to accelerate the post-probe sort as well “Database Architecture Optimized for the New Bottleneck: Memory Access” VLDB’99 “Generic Database Cost Models for Hierarchical Memory Systems”, VLDB’02 (all Manegold, Boncz, Kersten) “Cache-Conscious Radix-Decluster Projections”, Manegold, Boncz, Nes, VLDB’04

52 Radix Cluster/Decluster
Bottom line Both sorts from the Jive join can be significantly reduced in overhead Only been tested when there is sufficient memory for the entire join index to be stored three times Technique is likely applicable to larger join indexes, but utility will go down a little Only works if random access within a storage block Don’t want to use radix cluster/decluster if you have variable width column values or compression schemes that can only be decompressed starting from the beginning of the block

53 LM vs EM joins Invisible, Jive, Flash, Cluster, Decluster techniques contain a bag of tricks to improve LM joins Research papers show that LM joins become 2X faster than EM joins (instead of 2X slower) for a wide array of query types

54 Tuple Construction Heuristics
For queries with selective predicates, aggregations, or compressed data, use late materialization For joins: Research papers: Always use late materialization Commercial systems: Inner table to a join often materialized before join (reduces system complexity): Some systems will use LM only if columns from inner table can fit entirely in memory

55 How Different are Column-Stores and Row-Stores?

56 Limitation of Current Comparison
Assuming a row store does not use any column-oriented physical design Columns of the same logical tuple is traditionally stored together In fact it is possible to simulate a column store in a row store

57 Simulating a Column Store in a Row Store
Vertical Partitioning Indexing Every Column Materialized Views C-Tables Vertical Partitioning + Run-Length Encoding

58 Simulation 1: Vertical Partitioning
Each column makes its own table N columns result in N tables, each table stores (tuple-id, attr) pairs Need to do tuple reconstruction In this “simulation” scenario, an integer “tuple-id” column is associated to each column Use hash join on “tuple-id” column to reconstruct tuples

59 Two Problems of Vertical Partitioning
It requires the “tuple-id” column to be explicitly stored in each column Waste space and disk bandwidth Most row-stores store a relatively large header on every tuple further waste space

60 Tuple Header A tuple header provides metadata about the tuple
For example, in Postgres, each tuple contains a 27 bytes header including information such as Insert transaction timestamp Number of attributes in the tuple NULL flags Length of tuple header

61 Tuple Header in a Column Store
A column-store puts tuple header in separate columns Some information in the tuple header can be removed For example, in MonetDB, each logical column stores (key, attr) pairs Hence, the number of attributes is always 2 And there is no need for NULL flags, say we simply don’t store the tuple with NULL value

62 Simulation 2: Indexing Every Column
Base relations are stored using a standard, row-oriented physical design And an additional secondary B-Tree is built on every column of every table Join columns on tuple-id This approach answers queries by reading values directly from the indexes

63 One Problems of Indexing Every Column
If a column has no predicate on it, this approach requires its index to be scanned to extract the needed values One optimization is to create indexes with composite keys SELECT AVG(salary) FROM emp WHERE age > 40; If we have a composite index with an (age, salary) key, we can answer this query directly from the index

64 Simulation 3: Materialized Views
For every query in the benchmark workload, create an optimal set of materialized views Involve only the columns needed to answer Problem of materialized views It requires query workload to be known in advance

65 Comparison Results by Abadi et al. (SIGMOD’2008)
Using a simplified version of TPC-H called the Star Schema Benchmark (SSBM) Major Results: None of the three attempts to simulate a column store in a row store are particularly effective The Materialized View approach is the best The Vertical Partitioning approach is the moderate The Indexing Every Column approach is the worst

66 Relevant Factors for Successful Simulation
For successfully simulating a column store in a row store, we may have to Store tuple headers separately Use virtual tuple-id to do tuple reconstruction (i.e., joins) Maintain heap files (for base relations) in guaranteed position order

67 Simulation 4: the C-Table Approach (By Bruno, CIDR’2009)
The main idea is to extend the Vertical Partitioning approach to explicitly the Run-Length Encoding (RLE) of tuple values. A C-Table is a sequence of (f, v, c) triples: f is the starting tuple-id or position. v is the data value. c is the count (length). First sort by a, then b, and finally c.

68 Comparison Results by Bruno. (CIDR’2009)
Using TPC-H Results: The C-Table approach can compete with column stores Vision: Row-stores should be able to incorporate column specific optimizations


Download ppt "CSCI5570 Large Scale Data Processing Systems"

Similar presentations


Ads by Google