Presentation on theme: "University of Stuttgart A Statistics Propagation Approach to Enable Cost-Based Optimization of Statement Sequences Tobias Kraft, Holger Schwarz, Bernhard."— Presentation transcript:
University of Stuttgart A Statistics Propagation Approach to Enable Cost-Based Optimization of Statement Sequences Tobias Kraft, Holger Schwarz, Bernhard Mitschang Institute of Parallel and Distributed Systems University of Stuttgart
2 Overview Motivation Cost Estimation Approach Histogram Propagation Related Work Experiments Conclusion & Future Work
University of Stuttgart 3 Motivation The Case for Optimization Many of today’s applications embed query generators. Some of these generators not only produce a single query but a sequence of SQL statements (e.g. MicroStrategy DSS tools). Rewriting these sequences may lead to significant performance improvements! Development of an optimizer based on rewrite rules and a heuristic priority-based control strategy Coarse-Grained Optimization (CGO) [VLDB03]
University of Stuttgart 4 Motivation Statement Sequences We focus on statement sequences that: compute the final result of a request in a set of subsequent steps, allow to share intermediate results by multiple subsequent steps, temporarily store intermediate results in tables that are being created and dropped within the sequence, store the final result in a table that is not being dropped within the sequence. CREATE TABLE q1 (custkey INTEGER, turnover1990 FLOAT); CREATE TABLE q2 (custkey INTEGER, turnover1991 FLOAT); CREATE TABLE q3 (custkey INTEGER, name VARCHAR(25)); INSERT INTO q1 SELECTo.custkey, SUM(o.totalprice) FROMorders o WHEREo.orderyear = 1990 GROUP BY o.custkey; INSERT INTO q2 SELECTo.custkey, SUM(o.totalprice) FROMorders o WHEREo.orderyear = 1991 GROUP BY o.custkey; INSERT INTO q3 SELECTc.custkey, c.name FROMq1, q2, customer c WHEREq1.custkey = c.custkey ANDq1.custkey = q2.custkey ANDq2.turnover1991 > q1.turnover1990; DROP TABLE q1; DROP TABLE q2; q1q1 q2q2 q3q3
University of Stuttgart 5 Motivation Problems of the Heuristic Control Strategy In some scenarios a rule application may lead to deterioration of performance. Different behavior on different platforms and database management systems. Alternative sequences of rule applications lead to different results. Need for a cost-based approach.
University of Stuttgart 6 Cost Estimation Approach Problems & Solutions Cost estimates depend on the physical layout of the database and the capabilities and strategies of the DBMS‘s query optimizer. A cost model on top of the DBMS is no feasible solution. Make use of the cost estimates provided by the DBMS‘s query optimizer. DBMSs only provide cost estimates for statements on existing tables. Execute CREATE TABLES statements before cost estimation. Missing statistics for the created tables causes the DBMS‘s query optimizer to use default values for cardinality and selectivity. Propagate statistics through the INSERT statements. Make these propagated statistics available to the DBMS‘s optimizer.
University of Stuttgart 7 Cost Estimation Approach Algorithm of Cost Estimation Input:A statement sequence S. Output:A cost estimate for S. totalcosts = 0 foreach CREATE TABLE statement c in S Execute c on the underlying database system. foreach INSERT statement i in S (in the order given by S ) Retrieve a cost estimate for i from the optimizer of the underlying database system. Add this cost estimate to totalcosts. Translate i into an algebraic tree. Retrieve histograms for the base tables from the underlying database system and propagate them through the algebraic tree to retrieve histograms for the target table of i. Store the resulting histograms in the catalog of the underlying database system. foreach CREATE TABLE statement c in S Drop the table that has been created by c. return totalcosts.
University of Stuttgart 8 Cost Estimation Approach Architectural Overview
University of Stuttgart 9 Histogram Propagation Overview We have adopted techniques from approximate query answering and added some extensions: interval arithmetic for arithmetic terms, heuristics for grouping and aggregation, unification of comparison operators, normalization of histograms, common subexpressions SQL statement sequence algebraic operator trees. Everything is a bucket / histogram: NULL values are represented by a special bucket. Constant values (used in arithmetic terms) are represented by a histogram with a single bucket. Sets of constants (used in IN-predicates) are represented by a histogram. X T1T2 T3 T4
University of Stuttgart 10 Histogram Propagation Histograms Buckets are 4-tuples: (low, high, card, dv ) The NULL value: (null, null, card, 1) A constant value c : (c, c, card, 1)
University of Stuttgart 11 Histogram Propagation Algebra and Propagation Operators: projection, selection, cartesian product, union, difference and grouping (including aggregation). A join can be represented by a cartesian product followed by a selection that contains the join condition. Arithmetic terms (projection / selection) and predicates (selection) are also represented by operator trees. Propagation is done by recursively traversing the algebraic operator tree and the arithmetic trees in a post order manner.
University of Stuttgart 12 Histogram Propagation Algebra and Propagation Arithmetic Operators: histograms as input a result histogram as output iterate over all bucket combinations compute a result bucket for each bucket combination Comparison Operators: histograms as input selectivity as output some operators also provide modified histograms iterate over all bucket combinations compute a selectivity for each bucket combination (and adapt the buckets) A 1 A 2 A1A1 A2A2 A3A3 A1A1 A2A2 A1‘A1‘A2‘A2‘ selectivity
University of Stuttgart 13 Histogram Propagation Interval Arithmetic for Arithmetic Terms Example of using interval arithmetic for adding two buckets: B O.low = B I1.low + B I2.low B O.high = B I1.high + B I2.high + = Serialization B I B I BOBO =
University of Stuttgart histogram of attribute A cardinality value Histogram Propagation Heuristics for Grouping and Aggregation Determine amount of groups and average group sizes. Use these values together with the histogram of the attribute that should be aggregated to compute the aggregate buckets. lower bound of SUM(A) = = 950 upper bound of SUM(A) = = groups with group size tuple SUM(A) ? histogram of SUM(A) 100 cardinality value
University of Stuttgart 15 Comparison operators compare histograms. A single operator implementation can be used for different purposes. No separate join implementation is necessary. E.g., the comparison operator ‘=‘ can be used in: a predicate comparing two attributes, a predicate comparing an attribute and a constant, a join condition of an equi-join cartesian product followed by a selection that contains the join-condition as predicate, an IN predicate comparing an attribute with a set of values join with a table (represented by a histogram) that contains the values of the value set. Histogram Propagation Unification of Comparison Operators C C1C1 … CnCn
University of Stuttgart 16 Related Work on Histogram Propagation Papers on Approximate Query Answering: Yannis E. Ioannidis, Viswanath Poosala: Histogram-Based Approximation of Set-Valued Query-Answers. VLDB 1999 Viswanath Poosala, Venkatesh Ganti, Yannis E. Ioannidis: Approximate Query Answering using Histograms. IEEE Data Eng. Bull Transformation of SQL queries on tables into SQL queries on histograms. Join is done by creating two tables under the uniform spread assumption such that each table represents a value distribution which fits to the respective input histogram. Only equi-joins, no support for predicates that compare two attributes, no grouping and no support of arithmetic terms.
University of Stuttgart 17 Experiments Experimental Setup Database: TPC-H benchmark database on IBM DB2 V9. Sample sequences: Sequence S1 generated by the MicroStrategy DSS tool suite. A set of semantically equivalent sequences that result from the application of the CGO rewrite rules. Variants of those sequences resulting from the use of different values in the filter predicates of the sequence.
University of Stuttgart 18 Experiments Results for Alternative Sequences
University of Stuttgart 19 Experiments Results for Different Selectivities
University of Stuttgart 20 Conclusion & Future Work Cost estimates for statement sequences are necessary to avoid rule applications that lead to performance deterioration. Making use of the cost estimates of the underlying DBMS is a feasible solution. Histogram propagation is necessary to get useable cost estimates for statements that access intermediate-result tables and to avoid bad plans. Future Work: A cost-based control strategy. Extensive measurements.
University of Stuttgart 21 Thank you for your attention!
University of Stuttgart 22
23 Cost Estimation Approach Statistics API is an interface that offers uniform DBMS-independent access to DBMS statistics, meta data and optimizer estimates provides a flexible histogram format that abstracts from proprietary data structures used in different DBMSs implementations exist for IBM DB2, Oracle and MS SQL Server DB2 database Statistics API DB2 implementation Oracle implementation SQL Server implementation Oracle database SQL Server database JDBC
University of Stuttgart 24 Histogram Propagation Common Subexpressions Identify arithmetic terms that appear multiple times in the different clauses of an SQL query. Otherwise, the arithmetic term will be recomputed for each appearance and modifications of histograms may get lost. INSERT INTO temptable SELECTyear(o.o_orderdate), count(*) FROM orders o WHERE year(o.o_orderdate) > 1992 AND o.o_orderpriority = ‘1-URGENT‘ GROUP BY year(o.o_orderdate)
University of Stuttgart 25 Histogram Propagation Normalization of Histograms Worst case: The number of buckets in the output histogram is the product of the number of buckets in the input histograms. Serializing a histogram may double the number of buckets. Normalization: prior to the enumeration phase and / or after an output histogram has been produced, reduces the number of buckets by merging adjacent buckets, trade-off between complexity / performance and quality buckets 4 buckets Merge