Storage and Query Processing.

Storage and Query Processing

Files and Disk Drive Students
Tuples are accessed by their record ID (i.e., disk address) . Tuple is 50 bytes 80 tuples/page 500 pages Students 3

Data on External Storage
Storage Manager Record ID APPLICATION Record ID ??? 10010… APPLICATION search key Phone: Storage Manager Record ID File/Index record Storage Manager Buffer Manager Record ID File/Index Search key record

APPLICATION search key Storage Manager Record ID File/Index record This is our topic

File organization: Method of arranging a file of records on external storage. Record id (rid) is sufficient to physically locate record Indexes are data structures that allow us to find the record ids of records with given values in index search key fields Architecture: Buffer manager stages pages from external storage to main memory buffer pool. File and index layers make calls to the buffer manager. Record ID File/Index key record Buffer Manager Storage Manager

Alternative File Organizations
Many alternatives exist, each ideal for some situations, and not so good in others: Heap (random order) files: Suitable when typical access is a file scan retrieving all records. Sorted Files: Best if records must be retrieved in some order, or only a `range’ of records is needed. Indexes: Data structures to organize records via trees or hashing. Like sorted files, they speed up searches for a subset of records, based on values in certain fields, e.g., $80,000 ≤ Salary ≤ $150,000 Updates are much faster than in sorted files. 2

Indexes – Search Key An index on a file speeds up selections on the search key fields for the index. Any subset of the fields of a relation can be the search key for an index on the relation. Search key is not the same as key (minimal set of fields that uniquely identify a record in a relation, e.g., student ID). Search Key field A B C D E 1 2 3 4 5 6 7 Index on AB Index file Relation (data file) 7

An index contains a collection of data entries, and supports efficient retrieval of all data entries k* with a given key value k. To locate (one or more) data records with search key value k Search the index using k to find the desired data entry k* (e.g., A=3 and B=7) The data entry k* contains information to locate (one or more) data records with search key value k Data Entries Search Key A B RID 1 2 3 7 5 6 A B C D E 1 7 3 2 5 6 A data entry k* A data record Search key Search mechanism k e.g., A=3 ꓥ B=7 Index file Relation (data file) 7

Each tree node generally occupies one disk page
B+ Tree Indexes SEARCH: Follow the pointers to descend the tree to find the matching data entries in a leaf page P0 K1 P1 K2 Km Pm Index entry A non-leaf page Each tree node generally occupies one disk page Follow this pointer if K1≤ value <K2 Non-leaf Pages Leaf Pages Contains data entries Leaf pages contain data entries, and are chained (prev & next) Non-leaf pages contain index entries and direct searches: 4

B+ Tree Indexes SEARCH: Follow the pointers to descend the tree to find the matching data entries in a leaf page Index contains auxiliary information that directs searches to the desired data entries Search Key Non-leaf Pages Leaf Pages Record ID 4

B+ Tree Example Find 29* ? All ≥ 16* and < 30* ?
2* 3* Root 17 30 14* 16* 33* 34* 38* 39* 13 5 7* 5* 8* 22* 24* 27 27* 29* Entries <= 17 Entries > 17 Find * ? All ≥ 16* and < 30* ? Insert/delete: Find data entry in leaf, then change it. Need to adjust parent sometimes. And change sometimes bubbles up the tree 15

Hash-based Index Record IDs pointing to data records in the relation 3
Sequential search a bucket to find matching key 3 2 Overflow page 1 2 key h 1 h(key) = ID of hash bucket e.g., h(key) = key mod N The mod function computes the remainder of the division “key ÷N” N-1 Primary bucket pages

Hash-based Index Example
Relation Overflow page 1 key 178 Tuple 2 178 178 h . N-1 Primary bucket pages Search Key Good for equality search, Phone =

Hash-Based Indexes Good for equality selections.
Index is a collection of buckets. Bucket = primary page plus zero or more overflow pages. Hashing function h: h is applied to the search key fields of r. h(r) = ID of bucket in which record r belongs. Hash on the key fields to determine the bucket(s) Scan the data entries in these buckets to find the matching <key, rid> (i.e., alternative 2) Use rid to locate the record r 2

Data Entries There are three alternatives for data entries
B RID 1 2 3 7 5 6 A B C D E 1 7 3 2 5 6 A data record Search key Search mechanism k Index file Relation (data file) There are three alternatives for data entries This is Alternative 2 – Each data record has its own data entry 7

Alternatives for Data Entry k* in Index
Actual data record (with search key value k) B-tree Data records (instead of data entries) stored in tree node 8

Actual data record (with search key value k) <k, rid> pair, where rid is the record id of data record with search key value k Indexing technique Key COP4710 Data entries COP4710 COP4710 COP4710 Data Records Each matching data record has its own data entry 8

Actual data record (with search key value k) <k, rid> pair, where rid is the record id of data record with search key value k <k, rid-list> pair, where rid-list is a list of rids of data records with search key k Indexing technique Key COP4710 Data entries COP4710 Data Records COP4710 Matching records share a data entry 8

Data Entry formats: Pros & Cons
Alternative 1: If this is used, index structure (e.g., tree structure) is a file organization for data records (instead of a Heap file or sorted file). At most one index on a given collection of data records can use Alternative 1. (Otherwise, data records are duplicated, leading to redundant storage and potential inconsistency.) If data records are very large, # of pages containing data records is high. Implies size of auxiliary information in the index is also large, typically. Key DR DR Auxiliary information DR DR DR DR DR DR Data record, there is no separate data entry 9

Data Entry Format: Pros & Cons
Alternatives 2: Data entries <k, rid>, typically much smaller than data records. So, better than Alternative 1 with large data records, especially if search keys are small. (Portion of index structure used to direct search, which depends on size of data entries, is much smaller than with Alternative 1.) Smaller than Alternative 1 Key Data entries Variable sized data entries Data Records Alternative 3: Data entries <k, list-rid>, more compact than Alternative 2, but leads to variable sized data entries even if search keys are of fixed length. 10

Hash-based Index 1 2 key h A data entry in a hash bucket. It may be:
1 2 key h A data entry in a hash bucket. It may be: a record (Alternative 1), a <k, rid> (Alternative 2), or a <k, list_rid> (Alternative 3). N-1 Hash Buckets

No two tuples of a relation have the same value for the primary key
Index Classification No two tuples of a relation have the same value for the primary key Primary vs. secondary: If search key contains primary key, then called primary index (alternative 1 is usually used to avoid one more I/O to bring in the matching data record). Unique index: Search key contains a candidate key. Index on EMPNo & B Primary index Index on SSN & E Unique index Employee EMPNo B C SSN E F Candidate key Primary key Primary key 11

Index Classification Primary vs. secondary: If search key contains primary key, then called primary index (alternative 1 is usually used to avoid one more I/O to bring in the matching data record). Unique index: Search key contains a candidate key. Clustered vs. unclustered: If order of data records is the same as, or `close to’, order of data entries, then called clustered index. 11

Clustered Index Suppose that Alternative (2) is used for data entries, and that the data records are stored in a Heap file. To build clustered index, first sort the Heap file (with some free space on each page for future inserts). Overflow pages may be needed for inserts. (Thus, order of data records is `close to’, but not identical to, the sort order.) Data entries are always sorted Index entries direct search for data entries CLUSTERED Data entries Need to sort the heap file (Index File) (Data file) Data Records in consecutive pages An overflow page 12

Only One Clustered Index
Data records sorted according to SSN Data entries sorted according to phone# Unclustered Data entries sorted according to SSN StudentID Age Income Phone <StudentID, rid> Clustered A file can have only one clustered index & alternative 1 often used Cost of retrieving data record through index varies greatly based on whether index is clustered or not

Cost Model for Our Analysis
Relation It takes D time units to read/write a disk page D For our analysis, D is “1 disk I/O” or just “1 I/O”. R Each page holds R records B The relation is stored in B data pages A disk page is the unit for reading and writing a disk 3

An analogy Search for a paragraph Search cost = cost 1 + cost 2
Use the index to find the pages that contain the index term (cost 1) Read the paragraphs with the index term in the matching pages (cost 2) Search cost = cost 1 + cost 2

Retrieval Cost Cost depends on #qualifying tuples, and clustering. eid
Cost = Cost(finding qualifying data entries) + Cost(retrieving records) Typically small Could be large w/o clustering Emp eid sal dno age phone Finding qualifying data entries Retrieving matching data records 12

Cost Computation Fanout is F
1 F F2 B Data entries Height is approximately logFB Fanout is F B pages The I/O cost for finding a particular range of 10 matching records: Clustered Index: D(logFB + 1) /* 10 records fit in one page Number of index pages retrieved D is time to read or write a disk page 1 more I/O to read the 10 matching data records 4

Cost Computation The I/O cost for finding a particular range of 10 matching records: Clustered Index: D(logFB + 1) /* 10 records fit in one page Unclustered Index: D(logFB + 10) /* 10 records scattered over different pages Fetch 10 data pages Cost of retrieving data records through index varies greatly based on whether index is clustered or not! 11

MS SQL Server Clustered index is automatically created for PRIMARY KEY
You do not have to just accept the default. For many cases, a heap-based table would actually be better CREATE TABLE table_name { Attribute INT PRIMARY KEY NONCLUSTERED other attributes … } You can have a clustered index that is different from the primary key Attribute INT PRIMARY KEY, Attribute INT NOT NULL CLUSTERED };

Create Index Earlier versions of SQL had commands for creating indexes, but they were removed from the language because they were not at the conceptual schema level Many SQL systems still have the CREATE INDEX commands (check the syntax for your DBMS) CREATE INDEX index_name ON table_name (column1, column2, …); UNIQUE: Duplicate values are not allowed CREATE UNIQUE INDEX index_name ON table_name (column1, column2, …);

Update a table also needs to update its indexes
Trade Off Before creating an index, must also consider the impact on updates in the workload! Trade-off: Indexes can make queries go faster, updates slower. Require disk space, too. Update a table also needs to update its indexes 13

Index Selection Attributes in WHERE clause are candidates for index keys. Exact match condition suggests hash index. SELECT E.dno FROM Employees E WHERE E.num = Range query suggests tree index. Clustering is especially useful for range queries; WHERE E.age > 40 Employees older than 40 14

Is Index always helpful ?
B+ tree index on E.age can be used to get qualifying tuples SELECT E.dno FROM Emp E WHERE E.age>30 What is the selectivity of the condition ? If most employees are older than 30, a sequential scan of the relation would do almost as well Employees older than 30 18

Is Index always helpful ?
B+ tree index on E.age can be used to get qualifying tuples SELECT E.dno FROM Emp E WHERE E.age>30 What is the selectivity of the condition ? If most employees are older than 30, a sequential scan of the relation would do almost as well Index on age What if only 10% are older than 30 ? Unclustered index: Performing one I/O per qualifying tuple could be more expensive than a sequential scan Clustered index: This is a good option older than 30 18

Clustered vs Unclustered
1 If many employees collect stamps, a clustered index on E.hobby is helpful If this query is important, we should consider this index SELECT E.dno FROM Emp E WHERE E.hobby=Stamps An unclustered index on E.eid is good enough for the second query since no two employees have the same E.eid. 2 SELECT E.dno FROM Emp E WHERE E.eid= Primary key 18

Unclustered Index Does not Help
Using unclustered index on E.age may be costly Retrieving tuples with E.age > 25 is expensive SELECT E.dno, COUNT (*) FROM Emp E WHERE E.age>25 GROUP BY E.dno On age Data entries for age>25 Essentially all these data pages have matching data records 18

This Clustered Index Helps
On DNO Employees of dept. 123 SELECT E.dno, COUNT (*) FROM Emp E WHERE E.age>25 GROUP BY E.dno Clustered on dno better For each dno, count the tuples with E.age > 25 18

Indexes with Composite Search Keys
Composite Search Keys: Search on a combination of fields. Equality query: age=11 and sal =80 Range query: age=12 and sal > 10 Data entries in index sorted by <sal,age> Data entries sorted by <sal> <sal, age> <age, sal> <age> <sal> 12,20 12,10 11,80 13,75 20,12 10,12 75,13 80,11 11 12 13 10 20 75 80 Examples of composite key indexes Having multiple Data entries sue 13 75 bob cal joe 12 10 20 80 11 name age sal Data records sorted by name Data entries in index sorted by search key to support range queries 13

Order is Important: Example
Order of attributes is important for range queries key<age,sal> ≠ key<sal,age> sue 13 75 bob cal joe 12 10 20 80 11 name age sal <age, sal> 12,20 12,10 11,80 13,75 Data records sorted by name Data entries Data entries are sorted by “age” Data entries with the same age are sorted according to “sal” Data entries are sorted by “sal” Data entries with the same salary are sorted according to “age” If 20<age<30 is more “compact” than 3000<sal<5000, then select the “20<age<30” index. 20

key<age,sal> ≠ key<sal,age>
Order is Important Order of attributes is important for range queries key<age,sal> ≠ key<sal,age> To retrieve Emp records: If condition is: age=30 AND 50,000<sal<80,000: Clustered <age,sal> index much better than <sal,age> index! Reason: The qualifying data entries are grouped together in <age,sal> index, but not in <sal,age> index sue 13 75 bob cal joe 12 10 20 80 11 name age sal <age, sal> 12,20 12,10 11,80 13,75 Data records sorted by name Data entries Employee All 30-year-old employee 50,000 < sal < 80,000 If 20<age<30 is more “compact” than 3000<sal<5000, then select the “20<age<30” index. Note: Composite indexes have multiple data entries, and are updated more often 20

Composite Index: Example
Multi-attribute search keys should be considered when a WHERE clause contains several conditions SELECT E.name FROM Employees E WHERE E.age = 20 AND E.sal >= 50000 1 Age = 20 & Sal > 50K 2 An index on <age,sal> would be better than an index on age or an index on sal Reason: The qualifying data entries are grouped together ⟹ No need to search the index multiple times 14

Index-Only Evaluation
Indexes can sometimes enable index-only evaluation for important queries. Example: use leaf pages of index on age and sal to compute the average salary for each age group These data entries can be used as the two columns of the table Emp Emp eid sal dno age phone age sal Computing the averages using table is more expensive Index on age and sal 14

Index-Only Evaluation
Indexes can sometimes enable index-only evaluation for important queries. Example: use leaf pages of index on age and sal to compute the average salary for each age group The advantage is twofold: Scanning the data entries is less expensive The data entries are sorted according to age 14

Database Example We use this database for the following query examples
Emp Dept eid sal dno age phone dno mgr budget Foreign key 20

Dept 3 currently does not have employees
JOIN Operation Emp Dept eid salary dno age phone 10 100,000 2 22 20 120,000 1 40 30 32 50 110,000 45 dno mgr budget 1 20 1,000,000 2 40 2,000,000 3 null 200,000 Dept 3 currently does not have employees Emp ⋈ 𝑑𝑛𝑜 Dept eid salary dno age phone mgr budget 10 100,000 2 22 40 2,000,000 20 120,000 1 1,000,000 30 32 50 110,000 45

Without Index Need to join the two tables: Dept ⋈ 𝑑𝑛𝑜 Emp Expensive !!
SELECT D.mgr FROM Dept D, Emp E WHERE D.dno=E.dno Find mangers of departments with at least one employee Need to join the two tables: Dept ⋈ 𝑑𝑛𝑜 Emp Expensive !! Emp eid sal dno age phone Find matches 21

With Index Emp eid sal dno age phone
A number of queries can be answered without retrieving any tuples from one or more of the relations involved if a suitable index is available. SELECT D.mgr FROM Dept D, Emp E WHERE D.dno=E.dno Find mangers of departments with at least one employee If <E.dno> index is available, no need to retrieve Employees tuples, i.e., scan the E.dno data entries and pick up the corresponding Dept tuples No need to perform the join operation On dno Emp dno entries eid sal dno age phone 21

Index-Only Plans (2) Emp eid sal dno age phone SELECT E.dno, COUNT(*)
On dno Index on dno Emp eid sal dno age phone Each employee has his/her own data entry SELECT E.dno, COUNT(*) FROM Emp E GROUP BY E.dno If <E.dno> index is available, need only scan the data entries and count employees for each dno DBMS uses the data entries Good to know DBMS internal 21

Index-Only Plans (3) dno sal Create one ! eid sal dno age phone
SELECT E.dno, MIN(E.sal) FROM Emp E GROUP BY E.dno This would have been a better table Compute minimal salary for each department 70000 Dept. 123 Emp Create one ! eid sal dno age phone 21

Index-Only Plans (3) dno sal eid sal dno age phone Emp
SELECT E.dno, MIN(E.sal) FROM Emp E GROUP BY E.dno Data entries Compute minimal salary for each department 70000 Dept. 123 Index on dno & sal Emp eid sal dno age phone Need only scan the data entries to determine MIN(E.sal) for each dno 21

Index-Only Plans (4) sal age eid sal dno age phone Emp Data Entries
If <E.age,E.sal> or <E.sal,E.age> tree index is available, the average salary can be computed using only data entries in the index Compute average salary of some young employee SELECT AVG(E.sal) FROM Emp E WHERE E.age=25 AND E.sal BETWEEN AND 21

Do not scan all data entries
Index-Only Plans (5) SELECT E.dno, COUNT (*) FROM Emp E WHERE E.age=30 GROUP BY E.dno Using <dno,age> index Scan all data entries For each dno, count number of tuples with age=30 How many 30-year-olds in each department ? Using <age,dno> tree index Use index find first data entry /w age = 30 Scan data entries with age = 30, and count number of tuples for each dno (the departments are arranged continuously for age=30) Do not scan all data entries → Better !

Summary Many alternative file organizations exist, each appropriate in some situation. If selection queries are frequent, sorting the file or building an index is important. Hash-based indexes only good for equality search. Sorted files and tree-based indexes best for range search Index is: a collection of data entries (<key, rid> pairs or <key, rid-list> pairs), plus a way to quickly find entries (using hashing or a search tree) with given key values. 14

Summary (Contd.) Data entries can be (1) actual data records, (2) <key, rid> pairs, or (3) <key, rid-list> pairs. Choice orthogonal to indexing technique used to locate data entries with a given key value. Can have several indexes on a given file of data records, each with a different search key. Indexes can be classified as clustered vs. unclustered. Differences have important consequences for utility/performance. 15

Summary (Contd.) Understanding the nature of the workload for the application, and the performance goals, is essential to developing a good design. What are the important queries and updates? What attributes/relations are involved? Indexes must be chosen to speed up important queries (and perhaps some updates!). Index maintenance overhead on updates to key fields. Choose indexes that can help many queries, if possible. Build indexes to support index-only strategies. Clustering is an important decision; only one index on a given relation can be clustered! Order of fields in composite index key can be important.

Overview of Query Optimization
The slides for this text are organized into chapters. This lecture covers Chapter 8. Chapter 1: Introduction to Database Systems Chapter 2: The Entity-Relationship Model Chapter 3: The Relational Model Chapter 4 (Part A): Relational Algebra Chapter 4 (Part B): Relational Calculus Chapter 5: SQL: Queries, Programming, Triggers Chapter 6: Query-by-Example (QBE) Chapter 7: Storing Data: Disks and Files Chapter 8: File Organizations and Indexing Chapter 9: Tree-Structured Indexing Chapter 10: Hash-Based Indexing Chapter 11: External Sorting Chapter 12 (Part A): Evaluation of Relational Operators Chapter 12 (Part B): Evaluation of Relational Operators: Other Techniques Chapter 13: Introduction to Query Optimization Chapter 14: A Typical Relational Optimizer Chapter 15: Schema Refinement and Normal Forms Chapter 16 (Part A): Physical Database Design Chapter 16 (Part B): Database Tuning Chapter 17: Security Chapter 18: Transaction Management Overview Chapter 19: Concurrency Control Chapter 20: Crash Recovery Chapter 21: Parallel and Distributed Databases Chapter 22: Internet Databases Chapter 23: Decision Support Chapter 24: Data Mining Chapter 25: Object-Database Systems Chapter 26: Spatial Data Management Chapter 27: Deductive Databases Chapter 28: Additional Topics 1

Simplify Algebraic Expression
A computation proceddure Optimization 5 2+𝑥 +3 5𝑥+4 − 𝑥 2 2 =10+5𝑥+15𝑥+12 − 𝑥 2 2 =10+5𝑥+15𝑥+12 − 𝑥 4 =10+20𝑥+12 − 𝑥 4 =22+20𝑥 − 𝑥 4 A simpler procedure that compute the same thing SQL queries are processed by computing a relational algebraic expression

Project S2 7

Select S2 8

Cross-Product S1 R1 R1 × S1 Each row of R1 is paired with each row of S1 10

Joins R1 × S1 Not in join results 11

Selected combinations
Joins All combinations Selected combinations Condition Join: Example: Selection condition 11

Internal Query Representation
Query results Horizontal “filters” Vertical “filter” π ⨝ Query Plan: Tree of R.A. operations, with choice of algorithm for each operation. Each operator typically implemented using a `pull’ interface: when an operator is `pulled’ for the next output tuples, it `pulls’ on its inputs and computes them. Which algorithm for computing this JOIN T1 T2 T3 T4 𝜋((𝜎(𝑇1)⋈𝑇2) ⋈(𝑇3 ⋈ 𝜎 𝑇4 )) 2

Overview of Query Evaluation
Two main issues in query optimization: For a given query, what plans are considered? Search plan space for cheapest (estimated) plan. How is the cost of a plan estimated? Ideally: Want to find best plan. Practically: Avoid worst plans! 2

Statistics and Catalogs
Need information about the relations and indexes involved. Catalogs typically contain at least: # tuples (NTuples) and # pages (NPages) for each relation. # distinct key values (NKeys) and NPages for each index. Index height, low/high key values (Low/High) for each tree index. More detailed information (e.g., histograms of the values in some field) are sometimes stored. Avoid using unclustered index in this range 8

Statistics and Catalogs
Need information about the relations and indexes involved. Catalogs typically contain at least: # tuples (NTuples) and # pages (NPages) for each relation. # distinct key values (NKeys) and NPages for each index. Index height, low/high key values (Low/High) for each tree index. More detailed information (e.g., histograms of the values in some field) are sometimes stored. Catalogs updated periodically. Updating whenever data changes is too expensive; lots of approximation anyway, so slight inconsistency ok. 8

This step does not incur I/Os
Access Path Database Access path Matching Other terms An access path is a method of retrieving candidate tuples: Examples: File scan, or index that matches a selection (in the query) Apply any remaining terms that do not match the index to disqualify some of the candidate tuples This step does not incur I/Os 14

Access Path Example Database
1 Database B+ tree on “day” Check “bid” and “sid” Query: day < 8/9/94 AND bid=5 AND sid=3. 2 3 The predicate has 3 terms A B+-tree index on day can be used Then, bid=5 and sid=3 are checked for each retrieved tuple Similarly, (1) a hash index on <bid, sid> could be used; (2) day < 8/9/94 must then be checked. 14

Access Path: Goal Goal: Find the most selective access path to retrieve candidate tuples i.e., Find an index or file scan that require the fewest page I/Os. Database Access path Matching other terms 14

Access Path Selection A tree index matches search condition that involves only attributes in a prefix of the search key. Example: Tree index on <a, b, c> matches the selections a=5 AND b=3, and a=5 AND b>6, but not b=3. A hash index matches search condition that has a term attribute = value for every attribute in the search key of the index. Example: Hash index on <a, b, c> matches a=5 AND b=3 AND c=5; but it does not match b=3, or a=5 AND b=3, or a>5 AND b=3 AND c=5. “b” is not a prefix of “abc”. The index cannot be used The hash function requires three values as input 13

Duplicates Elimination Using Sorting
Expensive !! Remove duplicates only if “DISTINCT” is specified SELECT DISTINCT R.sid, R.bid FROM Reserves R Sorting Approach: Sort on <sid, bid> and remove the adjacent duplicates. (Can optimize this by dropping unwanted attributes while sorting.) Sorting Data Entries: If there is an index with both R.sid and R.bid in the search key, may be cheaper to sort data entries! 16

Sorting Large File is expensive
Easy Challenging - need to examine each card many times A lot of I/O’s Can we deduplicate a relation using only four passes over the file ?

Duplicates Elimination Using Hashing
SELECT DISTINCT R.sid, R.bid FROM Reserves R Memory B1 H1 1st hashing B1 B2 Reserve Disk 16

SELECT DISTINCT R.sid, R.bid FROM Reserves R Memory B1 H2 B1.1 B1.2 B1.3 B1.4 Eliminate duplicates B1 B2 Disk 16

SELECT DISTINCT R.sid, R.bid FROM Reserves R Memory B2 H2 B1.1 B1.2 B1.3 B1.4 Eliminate duplicates Process next bucket B1 B2 Disk 16

SELECT DISTINCT R.sid, R.bid FROM Reserves R Memory H2 B1.1 B1.2 B1.3 B1.4 Eliminate duplicates Process next bucket Repeat for each remaining hash bucket B1 B2 Disk 16

SELECT DISTINCT R.sid, R.bid FROM Reserves R Memory H2 B1.1 B1.2 B1.3 B1.4 Eliminate duplicates Reserve Process next bucket H1 B1 B2 I/O cost: Two passes of I/O’s per hashing phase: Disk Passes 1 & 2 due to H1, Passes 3 & 4 due to H2 16

R ⋈ S Outer relation Inner relation foreach tuple r in R do
foreach tuple s in S where ri == sj do add <r, s> to result

Simple Nested-Loops Join
R S 1 1 1 . Scan S to look for matches I/O 2 2 . . 1 page M N R S JOIN the two pages Memory One page at a time Need to scan S, one page at a time Cost = M + M·N Expensive to scan table S M times

Block Nested-Loops Join
R S JOIN the two pages Memory Need to scan S M time S JOIN Memory Bring in R 3 pages at a time Need to scan S only |𝑹|/𝟑 times R 1 2 3 4 If we have 5 pages available Simple Nested-Loop Join Block Nested-Loop Join

Join: Index Nested Loops (1)
R is the outer relation for each tuple r in R do for each tuple s in S where ri == sj do add <r, s> to result S is the inner relation If there is an index on the join column of one relation (say S), can make it the inner and exploit the index. Cost: M + ( (M*pR)  cost of finding matching S tuples) R Number of tuples in R S 1 1 1 Index Fetch into memory 2 2 . . 1 page pr tuples/page → R has (M∙pR) tuples M N

For each R tuple, cost of probing S index (i.e., finding data entry) is about 1.2 for hash index, 2-4 for B+ tree. Probing Data entries Data Records Key Typically 2~4 levels 1.2 I/O Data entries

For each R tuple, cost of probing S index (i.e., finding data entry) is about 1.2 for hash index, 2-4 for B+ tree. Cost of then finding S tuples (assuming Alt. (2) or (3) for data entries) depends on clustering. Clustered index: 1 I/O (typical), Unclustered: up to 1 I/O per matching S tuple. 1 2 M . R Index N S pr tuples/page Fetch into memory 1 page

Join: Sort-Merge (R S) i=j Sort R and S on the join column, then scan them to do a ``merge’’ (on join col.), and output result tuples. Advance scan of R until current R-tuple >= current S tuple, then advance scan of S until current S-tuple >= current R tuple; Repeat the above steps until current R tuple = current S tuple. At this point, all R tuples with same value in Ri (current R group) and all S tuples with same value in Sj (current S group) match; output <r, s> for all pairs of such tuples. Then resume scanning R and S. RelationR RelationS Often fits in one page SCAN An R group COST: R is scanned once; each S group is scanned once per matching R tuple. 11

Cost of Sort-Merge Join
Relation R has M pages Relation S has N pages Relation R Relation S Often fits in one page SCAN Merge Cost: M+N The cost of scanning could be MN (very unlikely) 12

In-memory local sorting, 2 passes of I/O – read, sort, write Merging two sorted lists using 2 input pages and 1 output page Merger Sort 1 B1 B2 B3 Sorted relation B1 B2 B3 Original relation 2 3 4 1 2 3 5 6 7 8 Memory limitation: Sorting one chunk at a time Each merging round requires 2 passes of I/Os (i.e., read two files and write the merged file) Merge Cost: Log28 or 3 rounds of merging ⇒ 6 read/write passes of I/Os

Query Processor Query Parser Catalog Manager Query Plan Evaluator
SQL Query Query Parser Catalog Manager Plan Generator Plan Cost Estimator Evaluation plan Query Plan Evaluator

Cost Estimation For each plan considered, must estimate cost:
Must estimate cost of each operation in plan tree. Depends on input cardinalities. We’ve already discussed how to estimate the cost of operations (sequential scan, index scan, joins, etc.) Must also estimate size of result for each operation in tree! Use information about the input relations For selections and joins, assume independence of predicates (next page) 8

Size Estimation and Reduction Factors
Consider a query block: A reduction factor SELECT attribute list FROM relation list WHERE term1 AND ... AND termk 1 2 Maximum # tuples in result is the product of the cardinalities of relations in the FROM clause Max # tuples = |R1| x |R2| x |R3| x ∙∙∙ Reduction factor (RF) associated with each term reflects the impact of the term in reducing result size Result cardinality = Max # tuples  product of all RF’s  Implicit assumption that terms are independent! 1 1 2 9

Result cardinality = Max # tuples  product of all RF’s
Reduction Factors Result cardinality = Max # tuples  product of all RF’s SELECT * FROM TABLE1, TABLE 2 WHERE predicate Query result Predicate (reduction) TABLE1 x TABLE2 (Max # tuples) Predicate is a logical combination of terms (e.g., age = 18 AND salary > 50,000) Each term has a reduction factor (RF) Reduction factor of the query is the product of all the RFs 𝑅𝑒𝑠𝑢𝑙𝑡 𝑠𝑖𝑧𝑒= Π𝑅𝐹⨯( 𝑇1 ⨯ 𝑇2 ) 9

Result cardinality = Max # tuples  product of all RF’s
Reduction Factors Result cardinality = Max # tuples  product of all RF’s Term col=value has RF 1/NKeys(I), given index I on col EXAMPLE: If the number of distinct key values is 100, size(result) = 1%  size(operand relation) Term col1=col2 has RF 1/MAX(NKeys(I1), NKeys(I2)) OBSERVATION: The smaller the numbers of distinct key values, the bigger the size of the join result due to more matches between the two operand relations. If Nkeys(I1)=Nkeys(I2)=1, RF=1 and size(R R2) = size(R1R2) Term col>value has RF (High(I)-value)/(High(I)-Low(I)) RF is the percentage of the tuples that satisfy the term 1/3 chance High Low value col > value Relation 9

Schema for Examples Sailors (sid: integer, sname: string, rating: integer, age: real) Reserves (sid: integer, bid: integer, day: dates, rname: string) . Tuple is 50 bytes 80 tuples/page 500 pages Sailors . Tuple is 40 bytes 100 tuples/page 1000 pages Reserves 3

Plan Generation – Relational Algebra
SELECT S.sname FROM Reserves R, Sailors S WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5 FROM clause WHERE clause SELECT clause sname( sid=sid  bid=100  rating>5 (RS)) 4

Plan Generation – Relational Algebra
SELECT S.sname FROM Reserves R, Sailors S WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5 FROM clause WHERE clause SELECT clause sname( sid=sid  bid=100  rating>5 (RS)) = sname( bid=100  rating>5 ( sid=sid (RS))) = sname( bid=100  rating>5 (R ⋈ 𝑠𝑖𝑑=𝑠𝑖𝑑 S)) 4

On-the-fly Processing
sname( bid=100  rating>5 (R ⋈ S)) Reserves Sailors sid=sid bid=100 rating > 5 sname ⇒More I/O Temp Read intermediate result Reserves Sailors sid=sid bid=100 rating > 5 sname (On-the-fly) ⇒𝑆𝑎𝑣𝑒 𝐼/𝑂 Save intermediate result 4

Plan Generation - Execution Plan
Reserves Sailors sid=sid bid=100 rating > 5 sname RA Tree sname( bid=100  rating>5 (R ⋈ S)) SELECT S.sname FROM Reserves R, Sailors S WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5 Reserves Sailors sid=sid bid=100 rating > 5 sname (Simple Nested Loops) (On-the-fly) Plan ⇒𝑆𝑎𝑣𝑒 𝐼/𝑂 4

Motivating Example SELECT S.sname FROM Reserves R, Sailors S
sid=sid bid=100 rating > 5 sname RA Tree SELECT S.sname FROM Reserves R, Sailors S WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5 Cost: 1000 I/Os By no means the worst plan! Reserves Sailors sid=sid bid=100 rating > 5 sname (Simple Nested Loops) (On-the-fly) Plan Outer relation (1,000 pages) (500 pages) 4

Motivating Example SELECT S.sname FROM Reserves R, Sailors S
sid=sid bid=100 rating > 5 sname RA Tree SELECT S.sname FROM Reserves R, Sailors S WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5 Cost: 1000 I/Os By no means the worst plan! Misses several opportunities: selections could have been `pushed’ earlier, no use is made of any available indexes, etc. Goal of optimization: To find more efficient plans that compute the same answer. Reserves Sailors sid=sid bid=100 rating > 5 sname (Simple Nested Loops) (On-the-fly) Plan Outer relation (1,000 pages) (500 pages) 4

Alternative Plan 1 (No Indexes)
Reserves Sailors sid=sid bid=100 sname (On-the-fly) rating > 5 (Scan; write to temp T1) temp T2) (Sort-Merge Join) Push selects before join to reduce the join cost (i.e., distribute select operator over join operator) Main difference: push selects Cost of plan: Scan Reserves (1000) + write temp T1 (10 pages, if we have boats, uniform distribution) Scan Sailors (500) + write temp T2 (250 pages, if we have 10 ratings) With 5 buffer pages, can do 4-way merges Sort T1: (2+2)10, sort T2: (2+6)250, merge: (10+250) (1,000 pages) (500 pages) Local sorts 2-way merge Need 2∙ log4(250/5) = 6 passes Two passes of I/Os for local sort Scan T1 & T2 Merge 4-way merge Only 5 pages available 5

Reserves Sailors sid=sid bid=100 sname (On-the-fly) rating > 5 (Scan; write to temp T1) temp T2) (Sort-Merge Join) Push selects before join to reduce the join cost (i.e., distribute select operator over join operator) (1,000 pages) (500 pages) Main difference: push selects Cost of plan: Scan Reserves (1000) + write temp T1 (10 pages, if we have 100 boats, uniform distribution) Scan Sailors (500) + write temp T2 (250 pages, if we have 10 ratings) With 5 buffer pages, can do 4-way merges Sort T1: (2+2)10, sort T2: (2+6)250, merge: (10+250) Total cost: 4,060 page I/Os 5

(On-the-fly) Alternative Plan 1 (No Indexes) sname (Block Nested Loop Join) sid=sid (Scan; (Scan; T1 has 10 pages write to write to bid=100 rating > 5 temp T1) temp T2) Reserves T2 has 250 pages Sailors (1,000 pages) (500 pages) If we use Block Nested Loop join, join cost = 10+(4250), total cost = 2770. T2 JOIN Memory Bring in T1 3 pages at a time Need to scan T2 𝟏𝟎/𝟑 , or 4 times T1 1 2 3 4 Only 5 pages available T1 has 10 pages Scan T2 for every 3 pages of T1 → Since T1 has 10 pages, need to scan T2 𝟏𝟎 𝟑 or 4 times 5

Push Projections Reserves Sailors sid=sid bid=100 sname (On-the-fly) rating > 5 (Scan; write to temp T1) temp T2) (Block Nested Loop Join) Reserves Sailors sid=sid bid=100 sname (On-the-fly) rating > 5 (Scan; write to temp T1) temp T2) (Block Nested Loop Join) sid Sid, sname If we `push’ projections, T1 has only sid, T2 only sid and sname: 5

JOIN Memory T1 fits in 3 pages Need to scan the smaller T2 only once Reserves Sailors sid=sid bid=100 sname (On-the-fly) rating > 5 (Scan; write to temp T1) temp T2) (Block Nested Loop Join) sid Sid, sname T1 If we `push’ projections, T1 has only sid, T2 only sid and sname: T1 fits in 3 pages, need to scan T2 only once cost of BNL drops to under 250 pages, total cost < 2000. 5

Alternative Plan 2 (with Indexes)
Reserves has 100,000 tuples in pages (100 tuples per page). There are 100 boats, equally popular. With clustered index on bid of Reserves, we get 1000 tuples (= 100,000/100) in 10 pages (= 1,000/100). INL with pipelining (outer is not materialized). (On-the-fly) sname 1000 x 1.2 I/Os (On-the-fly) rating > 5 1000 tuples in 10 pages (Index Nested Loops, with pipelining ) sid=sid Use clustered hash index; do not write result to disk (unclustered hash Index on sid OK) bid=100 Sailors Reserves Since 100 boats are equally popular, no hash bucket overflows. As a result, retrieving the Reserve tuples takes 10 I/Os, not (10x1.2) I/Os. (1,000 pages or 100,000 tuples) Join column sid is a key for Sailors. At most one matching tuple, unclustered hash index on sid OK. Cost: Selection of Reserves tuples ( I/Os); for each of 1,000 Reserve tuples, must get matching Sailors tuple (10001.2); total 1200 I/Os (plus 1,000 more I/Os if Sailors stored as alternative 2) 6

Alternative Plan 2 (with Indexes)
Reserves has 100,000 tuples in pages (100 tuples per page). There are 100 boats, equally popular. With clustered index on bid of Reserves, we get 1000 tuples (= 100,000/100) in 10 pages (= 1,000/100). INL with pipelining (outer is not materialized). (On-the-fly) sname 1000 x 1.2 I/Os (On-the-fly) rating > 5 1000 tuples in 10 pages (Index Nested Loops, with pipelining ) sid=sid Use clustered hash index; do not write result to disk (unclustered hash Index on sid OK) bid=100 Sailors Reserves Since 100 boats are equally popular, no hash bucket overflows. As a result, retrieving the Reserve tuples takes 10 I/Os, not (10x1.2) I/Os. (1,000 pages or 100,000 tuples) Join column sid is a key for Sailors. At most one matching tuple, unclustered hash index on sid OK. Cost: Total = ( ) ,000 ≅ 2,211 IO’s 6

Alternative Plan 2 (push projection ?)
(On-the-fly) sname From last slide (On-the-fly) rating > 5 (On-the-fly) Cannot take advantage of the index. Needs to scan sname (Index Nested Loops, with pipelining ) (On-the-fly) sid=sid rating > 5 Requires only 11 IO’s, better! (Use hash index; do not write result to temp) (unclustered Index on sid OK) bid=100 Sailors (Index Nested Loops, with pipelining ) sid=sid Projecting out unnecessary fields from outer relation (i.e., Reserves) doesn’t help because we want to use the hash index Use clustered hash index; do not write result to disk bid, sid (unclustered hash Index on sid OK) If we push “rating >5” before the join, the join operation would not be able to leverage the unclustered index on sid. Besides, it would be very expensive to perform “rating>5” using the unclustered index. bid=100 Sailors Reserves Reserves (1,000 pages or 100,000 tuples) 6

Alternative Plan 2 (push projection ?)
Decision not to push “rating > 5” before the join is based on availability of sid index on Sailors Reserves Sailors sid=sid bid=100 sname (On-the-fly) rating > 5 (Index Nested Loops, with pipelining ) (unclustered Index on sid OK) bid, sid Projecting out unnecessary fields here doesn’t help because we want to prob Sailors without writing intermedia result to temp If we push “rating >5” before the join, the join operation would not be able to leverage the unclustered index on sid. Besides, it would be very expensive to perform “rating>5” using the unclustered index. 6

Summary There are several alternative evaluation algorithms for each relational operator. A query is evaluated by converting it to a tree of operators and evaluating the operators in the tree. Must understand query optimization in order to fully understand the performance impact of a given database design (relations, indexes) on a workload (set of queries). Two parts to optimizing a query: Consider a set of alternative plans. Must prune search space; typically, left-deep plans only. Must estimate cost of each plan that is considered. Must estimate size of result and cost for each plan node. Key issues: Statistics, indexes, operator implementations. ⋈ Left-deep plan 19

Storage and Query Processing.

Similar presentations

Presentation on theme: "Storage and Query Processing."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Storage and Query Processing.

Similar presentations

Presentation on theme: "Storage and Query Processing."— Presentation transcript:

Similar presentations

About project

Feedback