1. 1 Overview of Query Evaluation Chapter 12

2. 2 Outline  Query Optimization Overview  Algorithm for Relational Operations

3. 3 Overview of Query Evaluation  DBMS keeps descriptive data in system catalogs.  SQL queries are translated into an extended form of relational algebra:  Query Plan Reasoning : Tree of operators with choice of one among several algorithms for each operator

4. Query Plan Evaluation Reserves Sailors sid=sid bid=100 rating > 5 sname  Query Plan Execution: Each operator typically implemented using a `pull’ interface when an operator is `pulled’ for next output tuples, it `pulls’ on its inputs and computes them.

5. 5 Overview of Query Evaluation  Query Plan Optimization :  Ideally: Want to find best plan. Practically: Avoid worst plans!  Two main issues in query optimization:  For a given query, what plans are considered? Algorithm to search plan space for cheapest (estimated) plan.  How is the cost of a plan estimated? Cost models based on I/O estimates

6. 6 System Catalogs  For each index:  structure (e.g., B+ tree) and search key fields  For each relation:  name, file name, file structure (e.g., Heap file)  attribute name and type, for each attribute  index name, for each index  integrity constraints  For each view:  view name and definition  Plus statistics, authorization, buffer pool size, etc. * Catalogs are themselves stored as relations !

7. 7 Statistics and Catalogs  Need : Information about relations and indexes.  Catalogs typically contain at least:  # tuples (NTuples) # pages (NPages) for each relation.  # distinct key values (NKeys) and NPages for each index.  Index height, low/high key values (Low/High) for each tree index.  Catalogs updated periodically.  Updating whenever many data changes occurred;  Lots of approximation anyway, so slight inconsistency ok.

8. 8 How Catalogs are stored Attr_Cat(attr_name, rel_name, type, position) System Catalog is itself a collection of tables. Catalog tables describe all tables in database, including catalog tables themselves. The code retrieves the catalog table must be handled specially.

9. 9 Query Processing  Common Techniques for Query Processing Algorithms:  Indexing: Can use WHERE conditions to retrieve small set of tuples from large relation  Iteration: Examine all tuples in an input table, one after the other (like in sorting algorithm).  Partitioning: By using sorting or hashing, we partition input tuples and replace expensive operation by similar operations on smaller inputs. Watch for these techniques as we discuss query evaluation!

10. 10 Access Paths  An access path  A method of retrieving tuples from a table  Method:  File scan,  Index that matches a selection (in the query)  Note :  Contributes significantly to cost of relational operator.

11. 11 Matching an Access Path  A tree index matches (a conjunction of) terms that involve only attributes in a prefix of search key.  Example : Given tree index on  selection a=5 AND b=3 ?  selection a=5 AND b>6 ?  selection b=3 ?

12. 12 Matching an Access Path  A hash index matches (a conjunction of) terms that has a term attribute = value for every attribute in search key of index.  Example : Given hash index on  selection a=5 AND b=3 and c =6 ?  selection c = 5 AND b = 6 ?  selection a = 5 ?  selection a > 5 AND b=3 and c =5 ?

13. 13 Access Path  Most selective access path: An index or file scan that we estimate will require fewest page I/Os.

14. 14 Query Evaluation of Selection

15. 15 Selection  Example :  R.attr OP value (R)  Case 1: No Index, NOT sorted on R.attr  Must scan the entire relation. Most selective access path = file scan Cost: M

16. 16 Selection  Example :  R.attr OP value (R)  Case 2: No Index, Sorted Data on R.attr  Binary search for first tuple. Scan R for all satisfied tuples. Cost: O(log 2 M)

17. 17 Selection Using B+ tree index  Example :  R.attr OP value (R)  Case 3: B+ tree Index on R.attr  Costs :  cost I (finding qualifying data entries), and  cost II (retrieving records)  Cost I: 2-3 I/Os. (depth of B+ tree)  Cost II:  clustered index 1 I/O and few tuples that match  unclustered index up to one I/O per qualifying tuple.

18. 18 Example : Using B+ Index for Selections  Example :  Assume uniform distribution of names,  Then say about ~ 10% of tuples qualify  Assume R is on 100 pages and has 10,000 tuples.  Clustered index: 2 IOs + little more than 10 I/Os  Unclustered index : 2 IOs + up to 1,000 I/Os SELECT * FROM Reserves R WHERE R.rname < ‘D%’

19. 19 Selection --- B+ Index  Refinement for unclustered indexes : 1. Find qualifying data entries. 2. Sort rids of data records to be retrieved. 3. Fetch rids in order. Avoid retrieving the same page multiple times. However, # of such pages likely to be still higher than with clustering.  Use of unclustered index for a range selection could be expensive.  Often better (and simpler) to instead scan data file.

20. 20 Selection – Hash Index  Hash index is good for equality selection.  Costs:  Cost I (retrieve index bucket page)  + Cost II (retrieving qualifying tuples from R)  Cost I is one I/O  Cost II could up to one I/O per satisfying tuple.

21. 21 General Condition : Conjunction  A condition with several predicates combined by conjunction (AND):  Example : day<8/9/94 AND bid=5 AND sid=3.

22. 22 General Selections (Conjunction) First approach: (utilizing single index)  Find the most selective access path, retrieve tuples using it.  To reduce the number of tuples retrieved  Apply any remaining terms that don’t match the index:  To discard some retrieved tuples  This does not affect number of tuples/pages fetched.  Example : Consider day<8/9/94 AND bid=5 AND sid=3.  A B+ tree index on day can be used; then bid=5 and sid=3 must be checked for each retrieved tuple.  Hash index on could be used day<8/9/94 must then be checked on fly.

23. 23 General Selections Second approach (utilizing multiple indices)  Assuming 2 or more matching indexes that use Alternatives (2) or (3) for data entries.  Get sets of rids of data records using each matching index.  Then intersect these sets of rids  Retrieve records and apply any remaining terms.  Example : Consider day<8/9/94 AND bid=5 AND sid=3.  A B+ tree index I on day and an index II on sid, both Alternative (2).  - Retrieve rids of records satisfying day<8/9/94 using index I,  - Retrieve rids of recs satisfying sid=3 using Index II  - Intersect rids  - Retrieve records and check bid=5.

24. 24 General Condition : Disjunction  Disjunction condition: one or more terms (R.attr op value) connected by OR (  ).  Example : ( day<8/9/94) OR (bid=5 AND sid=3)

25. 25 General Selection (Disjunction)  Case 1: Index is not available for one of terms.  Then :  Need a file scan anyways.  Thus check other conditions in this file scan.  E.g., Consider day<8/9/94 OR rname ='Joe'  No index on day. Need a File scan.  Even index is available in rname does not help.

26. 26 General Selection (Disjunction)  Case 2: Every term has a matching index.  Retrieve candidate tuples using respective index.  Then Union the results  Example : consider day<8/9/94 OR rname ='Joe'  Assume two B+ tree indexes on day and rname.  Retrieve tuples satisfying day < 8/9/94  Retrieve tuples satisfying rname = 'Joe'  Union the retrieved tuples.

27. 27 Query Evaluation of Projection

28. 28 Algorithms for Projection SELECT R.sid, R.bid FROM Reserves R

29. 29 Algorithms for Projection  The expensive part is removing duplicates.  SQL systems don’t remove duplicates unless keyword DISTINCT is specified in query.  Sorting Approach:  Sort on and remove duplicates. (Can optimize this by dropping unwanted information while sorting.)  Hashing Approach:  Hash on to create partitions. Load partitions into memory one at a time, build in-memory hash structure, and eliminate duplicates.  Indexing Approach :  If there is an index with both R.sid and R.bid in the search key, may be cheaper than sorting data entries SELECT DISTINCT R.sid, R.bid FROM Reserves R

30. 30 Summary  There are several alternative evaluation algorithms for each relational operator.

31. 31 Conclusion Not one method wins. Optimizer must assess situation to select best possible candidate