Storage and Indexes Chapter 8 & 9 The slides for this text are organized into chapters. This lecture covers Chapter 7. Chapter 1: Introduction to Database Systems Chapter 2: The Entity-Relationship Model Chapter 3: The Relational Model Chapter 4 (Part A): Relational Algebra Chapter 4 (Part B): Relational Calculus Chapter 5: SQL: Queries, Programming, Triggers Chapter 6: Query-by-Example (QBE) Chapter 7: Storing Data: Disks and Files Chapter 8: File Organizations and Indexing Chapter 9: Tree-Structured Indexing Chapter 10: Hash-Based Indexing Chapter 11: External Sorting Chapter 12 (Part A): Evaluation of Relational Operators Chapter 12 (Part B): Evaluation of Relational Operators: Other Techniques Chapter 13: Introduction to Query Optimization Chapter 14: A Typical Relational Optimizer Chapter 15: Schema Refinement and Normal Forms Chapter 16 (Part A): Physical Database Design Chapter 16 (Part B): Database Tuning Chapter 17: Security Chapter 18: Transaction Management Overview Chapter 19: Concurrency Control Chapter 20: Crash Recovery Chapter 21: Parallel and Distributed Databases Chapter 22: Internet Databases Chapter 23: Decision Support Chapter 24: Data Mining Chapter 25: Object-Database Systems Chapter 26: Spatial Data Management Chapter 27: Deductive Databases Chapter 28: Additional Topics Chapter 8 & 9 1

DBMS Structure

Disks and Files DBMS stores information on (“hard”) disks. A disk is a sequence of bytes, each has a disk address. READ: transfer data from disk to main memory (RAM). WRITE: transfer data from RAM to disk. Both are high-cost, relative to in-memory operations. Data is stored and retrieved in units called disk blocks or pages. Each page has a fixed size, say 512 bytes. It contains a sequence of records. In many (not always) cases, records in a page have same size, say 100 bytes. In many (not always) cases, records implement relational tuples.

Disks and Files (cont.) How to identify a record? By record id. Record id = <page id, record position in page> In many cases, record id can be viewed as a binary number, High significant bits for page id, and low significant bits for record position in page. record position in page is the record address relative to the beginning of the page. For example, assume the disk can hold only 4 pages, each containing 8 records. For record id = 10101 we have page id = 10, and record position in page is 101. That is, it is the 5th record in the 2nd page.

Disks and Files (cont.) How to retrieve a record Step 1: Fetch the page that contains the record into the main memory Step 2: Get the record from the page The cost of step 1 dominates. So the cost of retrieving a record is measured by the number of pages that must be fetched If the rid is known, then the cost is one. If something other than rid is known, then more pages may need to be fetched to lead the DBMS to the page that contains the record. Thus how to minimize the number of pages fetched is vital to the performance (detail later).

FILE: A collection of pages, must support: insert/delete/modify record Files of Records FILE: A collection of pages, must support: insert/delete/modify record read a particular record (specified using record id) scan all records (possibly with some conditions on the records to be retrieved) 13

Alternative File Organizations Many alternatives exist, each ideal for some situation , and not so good in others: Heap files: Records in file have no particular order Suitable when typical access is a file scan retrieving all records. Sorted Files: Records are sorted on some fields. Best if records must be retrieved in some order, or only a `range’ of records is needed. Hashed Files: (later) 2

Indexes An index on a file speeds up selections on the search key fields for the index. Any subset of the fields of a relation can be the search key for an index on the relation. Search key is not the same as key (minimal set of fields that uniquely identify a record in a relation). An index contains a collection of data entries A data entry is denoted as k*, where k is a search key value and * tells where to find the record containing k Index must support efficient retrieval of all data entries k* with a given key value k Structure of data entry in more detail 7

Alternatives for Data Entry k* in Index Two alternatives: <k, rid of data record with search key value k> (often used) <k, list of rids of data records with search key value k> Examples, assuming field ‘name’ is the search key <“John Smith”, 10101> where 10101 is the rid of a record that contains “John Smith” <“John Smith”, 10101, 10111, 11010> where 10101, 10111, 11010 are records which all contain “John Smith”. 8

How to use an index We want to find a data record that contains the search key value k Step 1: find the data entry k* in the index file Step 2: follow the pointers in the data entry k* to retrieve all the data records that contain k Key point: the index file is built in such a way that the data entry k* can be found much faster than the data records can.

Index Classification Primary vs. secondary: If search key contains primary key, then called primary index, otherwise secondary. Unique index: Search key contains a candidate key. Clustered vs. unclustered: If both the data entries and the data records are sorted on the search key values A file can be clustered on at most one search key. Cost of retrieving data records through index varies greatly based on whether index is clustered or not! 11

Examples of Indexes (name is primary key in data file) name age salary 0101 0111 1101 1010 1011 1001 ,3000 ,5004 ,4003 ,2007 Ashby Cass Smith Ashby,25,3000 Basu, 33, 4003 Bristow,29,2007 Smith, 44, 3000 Jones, 40, 6003 Tracy, 44, 5004 Ashby, 25, 3000 Basu, 33, 4003 Bristow,29,2007 ,6003 0110 Cass, 50, 5004 Daniels,22,6003 Jones, 40, 6003 1110 Cass,50, 5004 Daniels,22,6003 Smith, 44, 3000 Tracy, 44, 5004 Index on search key ‘salary’ Class: unclustered, secondary Index on search Key ‘name’ Class: clustered primary Heap file Sorted file on name field This index uses alternative 2 This index uses alternative 1

Examples of Indexes (name is the primary key in data file) name age salary name age salary Ashby Basu Bristow Cass 22 33 44 Daniels,22,6003 Ashby,25,3000 Bristow,29,2007 Smith, 44, 3000 Jones, 40, 6003 Tracy, 44, 5004 Basu, 33, 4003 Jones, 40, 6003 Smith, 44, 3000 Ashby, 25, 3000 Basu, 33, 4003 Bristow,29,2007 Daniels Jones Smith Tracy Cass,50, 5004 Daniels,22,6003 Tracy, 44, 5004 Cass, 50, 5004 Index on search key ‘name’ Class: unclustered, primary Index on search Key ‘age’ Class: clustered secondary Heap file Sorted file on age field This index uses alternative 1 This index uses alternative 1

Index Classification (Contd.) Index 2 on search key ‘age’ Dense vs. Sparse: If there is at least one data entry per search key value (in some data record), then dense. Every sparse index is clustered! Sparse indexes are smaller; however, some useful optimizations are based on dense indexes. Index 1 on Search key ‘name’ name age salary Ashby, 25, 3000 Smith, 44, 3000 Data File sorted on name Bristow, 30, 2007 Basu, 33, 4003 Cass, 50, 5004 Tracy, 44, 5004 Daniels, 22, 6003 Jones, 40, 6003 Ashby Cass Smith 40 44 50 22 25 30 33 Sparse Index Dense Index 13

Index Classification (Contd.) Composite Search Keys: Search on a combination of fields. Equality query: Every field value is equal to a constant value. E.g. wrt <sal,age> index: age=20 and sal =75 Range query: Some field value is not a constant. E.g.: age =20; or age=20 and sal > 10 Data entries in index sorted by search key to support range queries. Examples of composite key indexes using lexicographic order. 11,80 11 12,10 12 name age sal 12,20 12 13,75 bob 12 10 13 <age, sal> cal 11 80 <age> joe 12 20 10,12 sue 13 75 10 20,12 Data records sorted by name 20 75,13 75 80,11 80 <sal, age> <sal> Data entries in index sorted by <sal,age> Data entries sorted by <sal> 13

Using index to speed up query processing Consider the basic SQL structure: SELECT S.name FROM Sailors S WHERE S.rating = 7 If there is no index on rating field, then we must scan all the tuples in Sailors relation If there is an index file with search key rating, then Perform binary search to find all the data entries with rating = 7 in the index file For each data entry found, follow the coupling rid(s) to retrieve all the data record containing rating = 7 For each data record retrieved, output the sailor’s name This is must faster than if there is no index on rating field