Download presentation

Presentation is loading. Please wait.

Published byMegan Rone Modified over 2 years ago

1
Towards Estimating the Number of Distinct Value Combinations for a Set of Attributes Xiaohui Yu 1, Calisto Zuzarte 2, Ken Sevcik 1 1 University of Toronto 2 IBM Toronto Lab xhyu@cs.toronto.edu

2
November 3, 2005CIKM 20052 Distinct value combinations CountryCityHotel Name GermanyBremenHilton GermanyBremenBest Western GermanyFrankfurtInterCity CanadaTorontoFour Seasons CanadaTorontoIntercontinental 3 distinct value combinations 1 2 3 COLSCARD (COlumn Set CARDinality) = 3 The problem: estimating COLSCARD for a given set of attributes

3
November 3, 2005CIKM 20053 Motivation Cardinality estimation for query optimization, e.g., Estimating the size of Estimating the size of the aggregation Approximate query answering, e.g., COUNT queries SELECT sales_date, sales_person, SUM(sales_quantity) AS unit_sold FROM sales GROUP BY sales_date, sales_person

4
November 3, 2005CIKM 20054 Roadmap Related work Estimation with known marginal distributions Upper/lower bounds An estimator Estimation with histograms Experimental results Conclusions

5
November 3, 2005CIKM 20055 Related work Previous work has focused on the case of single attribute. [H Ö T88],[H Ö T89],[HNSS ’ 95],[HS ’ 98],[CCMN ’ 00] Sampling approach is used. Estimation through sampling is difficult [CCMN ’ 00] No existing statistical information is exploited.

6
November 3, 2005CIKM 20056 Our solution Considering multiple-attributes Utilizing existing statistics on individual attributes Readily available in most database systems Does not require access to the data Granularity of statistics Exact marginal frequency distributions Approximate distributions: histograms etc.

7
November 3, 2005CIKM 20057 Estimation with known marginals Number of distinct values in attribute Ai, frequency vector CountryCityHotel Name GermanyBremenHilton GermanyBremenBest Western GermanyFrankfurtInterCity CanadaTorontoFour Seasons CanadaTorontoIntercontinental

8
November 3, 2005CIKM 20058 The na ï ve estimator COLSCARD = Number of possible value combinations d i : the number of distinct values in attribute A i Sanity bound: COLSCARD cannot be greater than the table size The problem: Some value combinations with low occurrence probabilities may not appear in the table!

9
November 3, 2005CIKM 20059 Upper/Lower bounds Trivial bounds Upper bound: (the na ï ve estimator) Lower bound: Tighter bounds? In the case of two attributes, tighter bounds are available.

10
November 3, 2005CIKM 200510 Tighter bounds N = 10 4 4 2 d e f 1 1 8 a b c A2A2 A1A1 Naïve bounds: 3, 9Lower bound = 2+1+1 = 4 1 1 value freq valuefreq [2, 3] Upper bound = 3+1+1 = 5

11
November 3, 2005CIKM 200511 Expected number of combinations Assumptions 1.The data distributions of individual columns are independent 2.The occurrence of each combination in the table is independent Each element of f represents the frequency of a specific value combination. An estimate of the probability of occurrence

12
November 3, 2005CIKM 200512 Estimator The probability of the i-th combination not appearing in a particular tuple is The probability of the i-th combination not appearing in the table (of size N) is The expected number of value combinations is

13
November 3, 2005CIKM 200513 Example revisited Estimate the COLSCARD for attribute set (A 1, A 2, A 3 ), given New estimate: 5.94 Na ï ve estimate: 3*2*2 = 12

14
November 3, 2005CIKM 200514 Roadmap Related work Estimation with known marginal distributions Upper/lower bounds An estimator Estimation with histograms Experimental results Conclusions

15
November 3, 2005CIKM 200515 Estimation with histograms Histograms exist on individual attributes Two classes of histograms Partition-based End-biased Marginals can be (approximately) reconstructed from histograms Optimal histograms in each class?

16
November 3, 2005CIKM 200516 Optimal histograms Minimizing the error incurred by histograms ERR = |EST hist – EST exact | Partition-based histograms A dynamic programming algorithm similar to that for V-optimal histogram construction [Jagadish et al. 98] can be used.

17
November 3, 2005CIKM 200517 Optimal end-biased histograms An end-biased histogram with B buckets stores The exact frequencies of B-1 attribute values The average of the remaining values Which B-1 values to store exactly? Most widely used end-biased histograms store the frequencies of the most frequent values Not always optimal for COLSCARD estimation!!

18
November 3, 2005CIKM 200518 Example Attributes (A1, A2) Choose 1 frequency to store exactly Index of the frequency stored 1234 1.682.012.170.15 0.011.101.091.02 Error table N=10

19
November 3, 2005CIKM 200519 Optimal end-biased histograms Exhaustive search takes time proportional to We prove that the optimal choices can be one of the following Most frequent values Least frequent values A combination of most frequent and least frequent values Only need to search both ends Cost is linear in B, independent of d j !

20
November 3, 2005CIKM 200520 Roadmap Related work Estimation with known marginal distributions Upper/lower bounds An estimator Estimation with histograms Experimental results Conclusions

21
November 3, 2005CIKM 200521 Experiments – Data sets Synthetic data Skew: Zipfian parameter z=0 (uniform) to 4 (highly skewed) Number of tuples: 10K to 1M Real data Cover Type: 581,012 tuples, 10 attributes Census Income: 32,561 tuples, 14 attributes Error measure: ratio error ERR = max{true/est-1, est/true-1}

22
November 3, 2005CIKM 200522 Effect of data skew N=100K di=1k

23
November 3, 2005CIKM 200523 Effect of number of tuples

24
November 3, 2005CIKM 200524 Results on real data 45 pairs91 pairs

25
November 3, 2005CIKM 200525 Accuracy of end-biased histograms Results on the “ capital-gain ” attribute of Census Income data set

26
November 3, 2005CIKM 200526 Conclusions Utilizing existing knowledge maintained in database systems Proposed upper/lower bounds as well as an estimator Considered two cases exact marginal frequencies Histograms: optimal histograms Experimental results show the effectiveness of the proposed method

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google