Query Processing and Optimization, and Database Tuning From: Fundamentals of Database Systems Sixth Edition, Chapter 19 Elmasri & Navathe Prepared by: David Marshburn IST-734, Fall 2010 CSU, Dr. Matos

Query Processing SQL – most common high-level language Scan – identifies tokens: SQL keywords, attribute name, relation names Parse – valid syntax Validate – attributes and relations are valid and semantically meaningful Query in high-level language Scanning, parsing, and validating Immediate form of query Query optimizer Execution plan Query code generator Code to execute the query Runtime database processor

Immediate form of query Optimization Really “reasonably efficient” Too time consuming to completely optimize Based on what is known in the database catalog Used in relational databases Query describes the results, not how to get the results Not used in navigational databases Used in OO-databases (but not discussed here) Immediate form of query Query optimizer Execution plan

Optimization (cont’d) Reduces the amount of data being processed Amount of data Size of data Uses relational algebra Represented by query trees Sequence from leaf nodes to root node Can use heuristic rules Immediate form of query Root Node Internal Node Leaf Node Calculus and query graphs are not typically used for optimization as they do not indicate sequence

Heuristic Optimization of Query Tree Immediate form of query based on relational algebra depiction of higher-level language (SQL) query Very inefficient If employee record count is 500, works_on record count is 150, and project record count is 100, then there will be 3,750,000 records returned to the select operation Multiply by the size of each record SELECT lname FROM employee, works_on, project WHERE pname = ‘Aquarius’ AND pnumber = pno AND essn = ssn AND bdate > ‘1957-12-31’ πlname σpname = ‘Aquarius’ AND pnumber = pno and essn = ssn AND bdate > ‘1957-12-31’ x x Project Employee Works_on

Heuristic Optimization of Query Tree (cont’d) Moving SELECT operations down the query tree Only need projects where pname = ‘Aquarius’ Only need employees whose bdate > ‘1957-12-31’ Reduced the number of records returned πlname σpnumber = pno x σessn = ssn σpname = ‘Aquarius’ x Project σbdate > ‘1957-12-31’ Works_on Employee

Heuristic Optimization of Query Tree (cont’d) Apply more restrictive SELECT operations first Switch positions of employee and project to retrieve less records (Database catalog shows that pname is a key for project; therefore a select on pname will return only 1 record) πlname σessn = ssn x σpnumber = pno σbdate > ‘1957-12-31’ x Employee σpname = ‘Aquarius’ Works_on Project

Heuristic Optimization of Query Tree (cont’d) Replace CARTESIAN PRODUCTs and SELECTs with JOINs πlname ⨝essn = ssn ⨝pnumber = pno σbdate > ‘1957-12-31’ σpname = ‘Aquarius’ Works_on Employee Project

Heuristic Optimization of Query Tree (cont’d) Move PROJECT down tree PROJECT reduces the number of columns returned This makes each record smaller πlname ⨝essn = ssn πessn σbdate > ‘1957-12-31’ ⨝pnumber = pno Employee πpnumber π essn, pno σpname = ‘Aquarius’ Works_on Project

Heuristic Optimization of Query Tree (cont’d) Transformation of query tree to more efficient query tree a step-by-step process Must ensure that each step retains the equivalency of the query Use Transformation Rules for Relational Algebra to develop a Heuristic Algebraic Optimization Algorithm

Transformation Rules for Relational Algebra Cascade of SELECT – a conjunctive selection condition can be broken up into a cascade (sequence) of individual select operations σc1 AND c2 AND . . . AND cn  (R) ≡ σc1 (σc2 (σcn(R)). . .)) Commutativity of SELECT – the SELECT operation is commutative σc1 (σc2 (R)) ≡ σc2 (σc1 (R)) Cascade of PROJECT – in a cascade (sequence) of project operations, all but the last one can be ignored πList1 (πList2 (. . . (πListn (R)) . . . )) ≡ πList1 (R)

Transformation Rules for Relational Algebra (cont’d) Commuting SELECT with PROJECT – if the selection condition c involves only those attributes A1 . . . An in the project list, the two operations can be commuted πA1, A1, . . . An (σc (R)) ≡ σc (πA1, A1, . . . An (R)) c1 AND c2 AND . . . AND cn  (R) ≡ σc1 (σc2 (σcn(R)). . .)) Commutativity of JOINs – the join operation is commutative, as is the CARTESIAN PRODUCT R ⨝ S ≡ S ⨝ R R x S ≡ S x R

Transformation Rules for Relational Algebra (cont’d) Commuting σ with ⨝ (or x) – if all the attributes in the selection condition c involve only the attributes of one of the relations being joined – say, R – the two operations can be commuted as follows: σc (R ⨝ S) ≡ (σc (R)) ⨝ S Alternatively, if the selection condition c can be written as (c1 AND c2), where condition c1 involves only the attributes of R and condition c2 involves only the attributes of S, the operations commute as follows: σc (R ⨝ S) ≡ (σc (R)) ⨝ (σc ( S)) The same rules apply if the ⨝ is replaced by a x operation

Transformation Rules for Relational Algebra (cont’d) Commuting π with ⨝ (or x) – suppose that the projection list is L = {A1, . . ., An, B1, . . ., Bm}, where A1, . . ., An are attributes of R and B1, . . ., Bm are attributes of S. If the join condition c involves only attributes in L, the two operations can be commuted as follows: π L (R ⨝c S) ≡ (πA1, . . ., AN (R)) ⨝c (πB1, . . ., BM (S))

Transformation Rules for Relational Algebra (cont’d) (cont’d) If the join condition c contains additional attributes not in L, these must be added to the projection list, and a final π operation is needed. For example, if attributes An+1, . . ., An+k of R and Bm+1, . . ., Bm+k of S are involved in the join condition c but are not int the projection list L, the operations commute as follows: π L (R ⨝c S) ≡ (πA1, . . ., AN, AN+1, . . ., AN+k (R)) ⨝c (πB1, . . ., BM, BN+1,. . ., BM+P (S)) For x, there is no condition c, so the first transformation rule always applies by replacing ⨝c with x

Transformation Rules for Relational Algebra (cont’d) Commutativity of set operations – the set operations ∪ and ∩ are commutative but – is not Associativity of ⨝, x, ∪ and ∩ - these four operations are individually associated; this is if θ stands for any one of these four operations (throughout the expression), we have: (R θ S) θ T ≡ R θ (S θ T) Commuting σ with set operations – the σ operation commutes with ∪, ∩, and -. If θ stands for any one of these three operations (throughout the expression), we have: σc (R θ S) ≡ (σc (R)) θ (σc (S))

Transformation Rules for Relational Algebra (cont’d) The operation π commutes with ∪ πL (R ∪ S) ≡ (πL (R)) ∪ (πL (S)) Converting a (σ, x) sequence into ⨝ - if the condition of c of a σ that follows a x corresponds to a join condition, convert the (σ, x) sequence into a ⨝ as follows: σc (R x S) ≡ (R ⨝L S)

Outline of a Heuristic Algebraic Optimization Algorithm Using Rule 1, break up an SELECT operations with conjunctive conditions into a cascade of SELECT operations. This permits a greater degree of freedom in moving SELECT operations down different branches of the tree

Outline of a Heuristic Algebraic Optimization Algorithm (cont’d) Using Rules 2, 4, 6, and 10 concerning commutativity of SELECT with other operations, move each SELECT operation as far down the query tree as is permitted by the attributes involved in the select condition. If the condition involves attributes from only one table, which means that it represents a selection condition, the operation is moved all the way to the leaf node that represents this table. If the condition involves attributes from two tables, which means that it represents a join condition, the condition is moved to a location down the tree after the two tables are combined.

Outline of a Heuristic Algebraic Optimization Algorithm (cont’d) Using Rules 5 and 9 concerning commutativity and associativity of binary operations, rearrange the leaf nodes of the tree using the following criteria: Position the leaf node relations with the most restrictive SELECT operations so they are executed first Make sure that the ordering of leaf nodes does not cause CARTESIAN PRODUCT operations Using Rule 12, combine a CARTESIAN PRODUCT operation with a subsequent SELECT operation in the tree into a JOIN operation

Outline of a Heuristic Algebraic Optimization Algorithm (cont’d) Using Rules 3, 4, 7, and 11 concerning the cascading of PROJECT and the commuting of PROJECT with other operations, break down and move lists of projection attributes down the tree as far as possible by creating new PROJECT operations as needed Identify subtrees that represent groups of operations that can be executed by a single algorithm

Cost-Based Query Optimization Multiple execution strategies and algorithms are generated and compared based on costs Access cost to secondary storage – transferring data blocks between secondary storage and main memory Disk storage costs – storing on disk and intermediate files for execution strategy Computation costs – in-memory operations during query execution Memory usage costs – memory buffers needed during query Communication costs – shipping query and result from database to terminal

Database Catalog Information Size of each file For a file whose records are all the same type Number of records Average record size Number of file blocks Blocking factor Primary file organization Ordered or unordered on an attribute with a primary or clustering index Hashed on a key Number of levels (for multilevel index) Number of first-level index blocks