1. CPS216: Advanced Database Systems Notes 03:Query Processing (Overview, contd.) Shivnath Babu

2. parse Query rewriting Physical plan generation execute result SQL query parse tree logical query planstatistics physical query plan Query Optimization Query Execution Overview of Query Processing

3. parse Query rewriting Physical plan generation execute result SQL query parse tree logical query plan statistics physical query plan Initial logical plan “Best” logical plan Logical plan Rewrite rules

4. Query Rewriting  B,D  R.A = “c” Λ R.C = S.C X RS  B,D  R.A = “c” X RS  R.C = S.C We will revisit it towards the end of this lecture

5. parse Query rewriting Physical plan generation execute result SQL query parse tree Best logical query plan statistics Best physical query plan

6. Physical Plan Generation  B,D  R.A = “c” R S Natural join Best logical plan RS Index scan Table scan Hash join Project

7. parse Query rewriting Physical plan generation execute result SQL query parse tree Best logical query plan statistics Best physical query plan Enumerate possible physical plans Find the cost of each plan Pick plan with minimum cost

8. Physical Plan Generation Logical Query Plan P1 P2 …. Pn C1 C2 …. Cn Pick minimum cost one Physical plans Costs

9. Plans for Query Execution Roadmap –Path of a SQL query –Operator trees –Physical Vs Logical plans –Plumbing: Materialization Vs pipelining

10. Modern DBMS Architecture Disk(s) Applications OS Parser Query Optimizer Query Executor Storage Manager Logical query plan Physical query plan Access method API calls SQL File system API calls Storage system API calls DBMS

11. Logical Plans Vs. Physical Plans  B,D  R.A = “c” R S Natural join Best logical plan RS Index scan Table scan Hash join Project

12.  B,D  R.A = “c” R S Operator Plumbing Materialization: output of one operator written to disk, next operator reads from the disk Pipelining: output of one operator directly fed to next operator

13.  B,D  R.A = “c” R S Materialization Materialized here

14.  B,D  R.A = “c” R S Iterators: Pipelining  Each operator supports: Open() GetNext() Close()

15. Iterator for Table Scan (R) Open() { /** initialize variables */ b = first block of R; t = first tuple in block b; } GetNext() { IF (t is past last tuple in block b) { set b to next block; IF (there is no next block) /** no more tuples */ RETURN EOT; ELSE t = first tuple in b; } /** return current tuple */ oldt = t; set t to next tuple in block b; RETURN oldt; } Close() { /** nothing to be done */ }

16. Iterator for Select Open() { /** initialize child */ Child.Open(); } GetNext() { LOOP: t = Child.GetNext(); IF (t == EOT) { /** no more tuples */ RETURN EOT; } ELSE IF (t.A == “c”) RETURN t; ENDLOOP: } Close() { /** inform child */ Child.Close(); }  R.A = “c”

17. Iterator for Sort Open() { /** Bulk of the work is here */ Child.Open(); Read all tuples from Child and sort them } GetNext() { IF (more tuples) RETURN next tuple in order; ELSE RETURN EOT; } Close() { /** inform child */ Child.Close(); }  R.A

18. TNLJ (conceptually) for each r  Lexp do for each s  Rexp do if Lexp.C = Rexp.C, output r,s Iterator for Tuple Nested Loop Join LexpRexp

19. Example 1: Left-Deep Plan R1(A,B) TableScan R2(B,C) TableScan R3(C,D) TableScan TNLJ Question: What is the sequence of getNext() calls?

20. Example 2: Right-Deep Plan R3(C,D) TableScan TNLJ R1(A,B) TableScan R2(B,C) TableScan TNLJ Question: What is the sequence of getNext() calls?

21. Example Worked on blackboard

22. Cost Measure for a Physical Plan There are many cost measures –Time to completion –Number of I/Os (we will see a lot of this) –Number of getNext() calls Tradeoff: Simplicity of estimation Vs. Accurate estimation of performance as seen by user

23. Textbook outline Chapter 15 15.1Physical operators - Scan, Sort (Ch. 11.4), Indexes (Ch. 13) 15.2-15.6 Implementing operators + estimating their cost 15.8 Buffer Management 15.9 Parallel Processing

24. Chapter 16 16.1Parsing 16.2Algebraic laws 16.3Parse tree  logical query plan 16.4Estimating result sizes 16.5-16.7 Cost based optimization Textbook outline (contd.)

25. Chapter 5 Relational Algebra Chapter 6 SQL Background Material

26. parse Query rewriting Physical plan generation execute result SQL query parse tree logical query planstatistics physical query plan Query Processing - In class order 2; 16.1 3; 16.2,16.3 1; 13, 15 4; 16.4—16.7 First: A quick look at this

27. Why do we need Query Rewriting? Pruning the HUGE space of physical plans –Eliminating redundant conditions/operators –Rules that will improve performance with very high probability Preprocessing –Getting queries into a form that we know how to handle best  Reduces optimization time drastically without noticeably affecting quality

28. Some Query Rewrite Rules Transform one logical plan into another –Do not use statistics Equivalences in relational algebra Push-down predicates Do projects early Avoid cross-products if possible

29. Equivalences in Relational Algebra R S =SR Commutativity (R S) T = R(S T) Associativity Also holds for: Cross Products, Union, Intersection R x S = S x R (R x S) x T = R x (S x T) R U S = S U R R U (S U T) = (R U S) U T

30. Apply Rewrite Rule (1)  B,D [  R.C=S.C [  R.A=“c” (R X S)]]  B,D  R.A = “c” Λ R.C = S.C X RS  B,D  R.A = “c” X RS  R.C = S.C

31. Rules: Project Let: X = set of attributes Y = set of attributes XY = X U Y  xy (R) =  x [  y (R)]

32. Let p = predicate with only R attribs q = predicate with only S attribs m = predicate with only R,S attribs  p (R S) =  q (R S) = Rules:  combined [  p (R)] S R [  q (S)]

33. Apply Rewrite Rule (2)  B,D [  R.C=S.C [  R.A=“c” (R)] X S]  B,D  R.A = “c” X R S  R.C = S.C  B,D  R.A = “c” X RS  R.C = S.C

34. Apply Rewrite Rule (3)  B,D [[  R.A=“c” (R)] S]  B,D  R.A = “c” R S  B,D  R.A = “c” X R S  R.C = S.C Natural join

35. Rules:  combined (continued)  p  q (R S) = [  p (R)] [  q (S)]  p  q  m (R S) =  m [ (  p R) (  q S) ]  pvq (R S) = [ (  p R) S ] U [ R (  q S) ]

36.  p1  p2 (R)   p1 [  p2 (R)]  p (R S)  [  p (R)] S R S  S R  x [  p (R)]   x {  p [  xz (R)] } Which are “good” transformations?

37. Conventional wisdom: do projects early Example: R(A,B,C,D,E) P: (A=3)  (B=“cat”)  E {  p (R)} vs.  E {  p {  ABE (R)} }

38. But: What if we have A, B indexes? B = “cat” A=3 Intersect pointers to get pointers to matching tuples

39. Bottom line: No transformation is always good Some are usually good: –Push selections down –Avoid cross-products if possible –Subqueries  Joins

40. Avoid Cross Products (if possible) Which join trees avoid cross-products? If you can't avoid cross products, perform them as late as possible Select B,D From R,S,T,U Where R.A = S.B  R.C=T.C  R.D = U.D

41. More Query Rewrite Rules Transform one logical plan into another –Do not use statistics Equivalences in relational algebra Push-down predicates Do projects early Avoid cross-products if possible Use left-deep trees Subqueries  Joins Use of constraints, e.g., uniqueness

42. Announcements First homework will be posted next week –I will send an email announcement when it is posted We will start discussing past projects and project ideas from next week –Your first goal is to have a concretely defined project by the last week of September (the earlier the better!) –Small class  single student projects –October and November should be used to work on the projects –Final “demo’s” will be in the first week of December