1. Towards Estimating the Number of Distinct Value Combinations for a Set of Attributes Xiaohui Yu 1, Calisto Zuzarte 2, Ken Sevcik 1 1 University of Toronto 2 IBM Toronto Lab xhyu@cs.toronto.edu

2. November 3, 2005CIKM 20052 Distinct value combinations CountryCityHotel Name GermanyBremenHilton GermanyBremenBest Western GermanyFrankfurtInterCity CanadaTorontoFour Seasons CanadaTorontoIntercontinental 3 distinct value combinations 1 2 3 COLSCARD (COlumn Set CARDinality) = 3 The problem: estimating COLSCARD for a given set of attributes

3. November 3, 2005CIKM 20053 Motivation  Cardinality estimation for query optimization, e.g., Estimating the size of Estimating the size of the aggregation  Approximate query answering, e.g., COUNT queries SELECT sales_date, sales_person, SUM(sales_quantity) AS unit_sold FROM sales GROUP BY sales_date, sales_person

4. November 3, 2005CIKM 20054 Roadmap  Related work  Estimation with known marginal distributions Upper/lower bounds An estimator  Estimation with histograms  Experimental results  Conclusions

5. November 3, 2005CIKM 20055 Related work  Previous work has focused on the case of single attribute. [H Ö T88],[H Ö T89],[HNSS ’ 95],[HS ’ 98],[CCMN ’ 00]  Sampling approach is used. Estimation through sampling is difficult [CCMN ’ 00]  No existing statistical information is exploited.

6. November 3, 2005CIKM 20056 Our solution  Considering multiple-attributes  Utilizing existing statistics on individual attributes Readily available in most database systems Does not require access to the data  Granularity of statistics Exact marginal frequency distributions Approximate distributions: histograms etc.

7. November 3, 2005CIKM 20057 Estimation with known marginals  Number of distinct values in attribute Ai,  frequency vector CountryCityHotel Name GermanyBremenHilton GermanyBremenBest Western GermanyFrankfurtInterCity CanadaTorontoFour Seasons CanadaTorontoIntercontinental

8. November 3, 2005CIKM 20058 The na ï ve estimator COLSCARD = Number of possible value combinations d i : the number of distinct values in attribute A i Sanity bound: COLSCARD cannot be greater than the table size The problem: Some value combinations with low occurrence probabilities may not appear in the table!

9. November 3, 2005CIKM 20059 Upper/Lower bounds  Trivial bounds Upper bound: (the na ï ve estimator) Lower bound:  Tighter bounds? In the case of two attributes, tighter bounds are available.

10. November 3, 2005CIKM 200510 Tighter bounds N = 10 4 4 2 d e f 1 1 8 a b c A2A2 A1A1 Naïve bounds: 3, 9Lower bound = 2+1+1 = 4 1 1 value freq valuefreq [2, 3] Upper bound = 3+1+1 = 5

11. November 3, 2005CIKM 200511 Expected number of combinations  Assumptions 1.The data distributions of individual columns are independent 2.The occurrence of each combination in the table is independent   Each element of f represents the frequency of a specific value combination. An estimate of the probability of occurrence

12. November 3, 2005CIKM 200512 Estimator The probability of the i-th combination not appearing in a particular tuple is The probability of the i-th combination not appearing in the table (of size N) is The expected number of value combinations is

13. November 3, 2005CIKM 200513 Example revisited  Estimate the COLSCARD for attribute set (A 1, A 2, A 3 ), given New estimate: 5.94 Na ï ve estimate: 3*2*2 = 12

14. November 3, 2005CIKM 200514 Roadmap  Related work  Estimation with known marginal distributions Upper/lower bounds An estimator  Estimation with histograms  Experimental results  Conclusions

15. November 3, 2005CIKM 200515 Estimation with histograms  Histograms exist on individual attributes  Two classes of histograms Partition-based End-biased  Marginals can be (approximately) reconstructed from histograms  Optimal histograms in each class?

16. November 3, 2005CIKM 200516 Optimal histograms  Minimizing the error incurred by histograms ERR = |EST hist – EST exact |  Partition-based histograms A dynamic programming algorithm similar to that for V-optimal histogram construction [Jagadish et al. 98] can be used.

17. November 3, 2005CIKM 200517 Optimal end-biased histograms  An end-biased histogram with B buckets stores The exact frequencies of B-1 attribute values The average of the remaining values  Which B-1 values to store exactly?  Most widely used end-biased histograms store the frequencies of the most frequent values Not always optimal for COLSCARD estimation!!

18. November 3, 2005CIKM 200518 Example Attributes (A1, A2) Choose 1 frequency to store exactly Index of the frequency stored 1234 1.682.012.170.15 0.011.101.091.02 Error table N=10

19. November 3, 2005CIKM 200519 Optimal end-biased histograms  Exhaustive search takes time proportional to  We prove that the optimal choices can be one of the following Most frequent values Least frequent values A combination of most frequent and least frequent values  Only need to search both ends Cost is linear in B, independent of d j !

20. November 3, 2005CIKM 200520 Roadmap  Related work  Estimation with known marginal distributions Upper/lower bounds An estimator  Estimation with histograms  Experimental results  Conclusions

21. November 3, 2005CIKM 200521 Experiments – Data sets  Synthetic data Skew: Zipfian parameter z=0 (uniform) to 4 (highly skewed) Number of tuples: 10K to 1M  Real data Cover Type: 581,012 tuples, 10 attributes Census Income: 32,561 tuples, 14 attributes  Error measure: ratio error ERR = max{true/est-1, est/true-1}

22. November 3, 2005CIKM 200522 Effect of data skew N=100K di=1k

23. November 3, 2005CIKM 200523 Effect of number of tuples

24. November 3, 2005CIKM 200524 Results on real data 45 pairs91 pairs

25. November 3, 2005CIKM 200525 Accuracy of end-biased histograms Results on the “ capital-gain ” attribute of Census Income data set

26. November 3, 2005CIKM 200526 Conclusions  Utilizing existing knowledge maintained in database systems  Proposed upper/lower bounds as well as an estimator  Considered two cases exact marginal frequencies Histograms: optimal histograms  Experimental results show the effectiveness of the proposed method