1. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Lecture 4 Practical Database Design and Tuning

2. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 2 Chapter Outline 1. Physical Database Design in Relational Databases 2. An Overview of Database Tuning in Relational Systems

3. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 3 1. Physical Database Design in Relational Databases(1) Factors that Influence Physical Database Design: A. Analyzing the database queries and transactions For each query, the following information is needed. 1.The files that will be accessed by the query; 2.The attributes on which any selection conditions for the query are specified; 3.The attributes on which any join conditions or conditions to link multiple tables or objects for the query are specified; 4.The attributes whose values will be retrieved by the query. Note: the attributes listed in items 2 and 3 above are candidates for definition of access structures.

4. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 4 Physical Database Design in Relational Databases(2) Factors that Influence Physical Database Design (contd.): A. Analyzing the database queries and transactions (contd.) For each update transaction or operation, the following information is needed. 1.The files that will be updated; 2.The type of operation on each file (insert, update or delete); 3.The attributes on which selection conditions for a delete or update operation are specified; 4.The attributes whose values will be changed by an update operation. Note: the attributes listed in items 3 above are candidates for definition of access structures. However, the attributes listed in item 4 are candidates for avoiding an access structure.

5. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 5 Physical Database Design in Relational Databases(3) Factors that Influence Physical Database Design (contd.): B. Analyzing the expected frequency of invocation of queries and transactions The expected frequency information, along with the attribute information collected on each query and transaction, is used to compile a cumulative list of expected frequency of use for all the queries and transactions. It is expressed as the expected frequency of using each attribute in each file as a selection attribute or join attribute, over all the queries and transactions. 80-20 rule 20% of the data is accessed 80% of the time

6. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 6 Physical Database Design in Relational Databases(4) Factors that Influence Physical Database Design (contd.) C. Analyzing the time constraints of queries and transactions Performance constraints place further priorities on the attributes that are candidates for access paths. The selection attributes used by queries and transactions with time constraints become higher- priority candidates for primary access structure.

7. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 7 Physical Database Design in Relational Databases(4) Factors that Influence Physical Database Design (contd.) D. Analyzing the expected frequencies of update operations A minimum number of access paths should be specified for a file that is updated frequently.

8. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 8 Physical Database Design in Relational Databases(4) Factors that Influence Physical Database Design (contd.) E. Analyzing the uniqueness constraints on attributes Access paths should be specified on all candidate key attributes — or set of attributes — that are either the primary key or constrained to be unique.

9. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 9 Physical Database Design in Relational Databases(5) Physical Database Design Decisions Design decisions about indexing Whether to index an attribute? What attribute or attributes to index on? Whether to set up a clustered index? Whether to use a hash index over a tree index? Whether to use dynamic hashing for the file?

10. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 10 Physical Database Design in Relational Databases(6) Physical Database Design Decisions (contd.) Denormalization as a design decision for speeding up queries The goal of normalization is to separate the logically related attributes into tables to minimize redundancy and thereby avoid the update anomalies that cause an extra processing overheard to maintain consistency of the database. The goal of denormalization is to improve the performance of frequently occurring queries and transactions. (Typically the designer adds to a table attributes that are needed for answering queries or producing reports so that a join with another table is avoided.) Trade off between update and query performance

11. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 11 2. An Overview of Database Tuning in Relational Systems (1) Tuning: The process of continuing to revise/adjust the physical database design by monitoring resource utilization as well as internal DBMS processing to reveal bottlenecks such as contention for the same data or devices. Goal: To make application run faster To lower the response time of queries/transactions To improve the overall throughput of transactions

12. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 12 An Overview of Database Tuning in Relational Systems (2) Statistics internally collected in DBMSs: Size of individual tables Number of distinct values in a column The number of times a particular query or transaction is submitted/executed in an interval of time The times required for different phases of query and transaction processing Statistics obtained from monitoring: Storage statistics I/O and device performance statistics Query/transaction processing statistics Locking/logging related statistics Index statistic

13. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 13 An Overview of Database Tuning in Relational Systems (3) Problems to be considered in tuning: How to avoid excessive lock contention? How to minimize overhead of logging and unnecessary dumping of data? How to optimize buffer size and scheduling of processes? How to allocate resources such as disks, RAM and processes for most efficient utilization?

14. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 14 An Overview of Database Tuning in Relational Systems (4) Tuning Indexes Reasons to tuning indexes Certain queries may take too long to run for lack of an index; Certain indexes may not get utilized at all; Certain indexes may be causing excessive overhead because the index is on an attribute that undergoes frequent changes Options to tuning indexes Drop or/and build new indexes Change a non-clustered index to a clustered index (and vice versa) Rebuilding the index

15. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 15 An Overview of Database Tuning in Relational Systems (5) Tuning the Database Design Dynamically changed processing requirements need to be addressed by making changes to the conceptual schema if necessary and to reflect those changes into the logical schema and physical design.

16. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 16 An Overview of Database Tuning in Relational Systems (6) Tuning the Database Design (contd.) Possible changes to the database design Existing tables may be joined (denormalized) because certain attributes from two or more tables are frequently needed together. For the given set of tables, there may be alternative design choices, all of which achieve 3NF or BCNF. One may be replaced by the other. A relation of the form R(K, A, B, C, D, …) that is in BCNF can be stored into multiple tables that are also in BCNF by replicating the key K in each table. Attribute(s) from one table may be repeated in another even though this creates redundancy and potential anomalies. Apply horizontal partitioning as well as vertical partitioning if necessary.

17. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 17 An Overview of Database Tuning in Relational Systems (7) Tuning Queries Indications for tuning queries A query issues too many disk accesses The query plan shows that relevant indexes are not being used.

18. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 18 An Overview of Database Tuning in Relational Systems (8) Tuning Queries (contd.): Typical instances for query tuning In some situations involving using of correlated queries, temporaries are useful. If multiple options for join condition are possible, choose one that uses a clustering index and avoid those that contain string comparisons. The order of tables in the FROM clause may affect the join processing. Some query optimizers perform worse on nested queries compared to their equivalent un-nested counterparts. Many applications are based on views that define the data of interest to those applications. Sometimes these views become an overkill.

19. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 19 An Overview of Database Tuning in Relational Systems (8) Tuning Queries (contd.): Typical instances for query tuning In some situations involving using of correlated queries, temporaries are useful. If multiple options for join condition are possible, choose one that uses a clustering index and avoid those that contain string comparisons. The order of tables in the FROM clause may affect the join processing. Some query optimizers perform worse on nested queries compared to their equivalent un-nested counterparts. Many applications are based on views that define the data of interest to those applications. Sometimes these views become an overkill.

20. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 20 An Overview of Database Tuning in Relational Systems (10) Additional Query Tuning Guidelines A query with multiple selection conditions that are connected via OR may not be prompting the query optimizer to use any index. Such a query may be split up and expressed as a union of queries, each with a condition on an attribute that causes an index to be used. Apply the following transformations NOT condition may be transformed into a positive expression. Embedded SELECT blocks may be replaced by joins. If an equality join is set up between two tables, the range predicate on the joining attribute set up in one table may be repeated for the other table WHERE conditions may be rewritten to utilize the indexes on multiple columns.

21. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Database Recovery Techniques

22. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 22 Outline Databases Recovery 1. Purpose of Database Recovery 2. Types of Failure 3. Transaction Log 4. Data Updates 5. Data Caching 6. Transaction Roll-back (Undo) and Roll-Forward 7. Checkpointing 8. Recovery schemes 9. ARIES Recovery Scheme 10. Recovery in Multidatabase System

23. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 23 Database Recovery 1 Purpose of Database Recovery To bring the database into the last consistent state, which existed prior to the failure. To preserve transaction properties (Atomicity, Consistency, Isolation and Durability). Example: If the system crashes before a fund transfer transaction completes its execution, then either one or both accounts may have incorrect value. Thus, the database must be restored to the state before the transaction modified any of the accounts.

24. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 24 Database Recovery 2 Types of Failure The database may become unavailable for use due to Transaction failure: Transactions may fail because of incorrect input, deadlock, incorrect synchronization. System failure: System may fail because of addressing error, application error, operating system fault, RAM failure, etc. Media failure: Disk head crash, power disruption, etc.

25. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 25 Database Recovery 3 Transaction Log For recovery from any type of failure data values prior to modification (BFIM - BeFore Image) and the new value after modification (AFIM – AFter Image) are required. These values and other information is stored in a sequential file called Transaction log. A sample log is given below. Back P and Next P point to the previous and next log records of the same transaction.

26. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 26 Database Recovery 4 Data Update Immediate Update: As soon as a data item is modified in cache, the disk copy is updated. Deferred Update: All modified data items in the cache is written either after a transaction ends its execution or after a fixed number of transactions have completed their execution. Shadow update: The modified version of a data item does not overwrite its disk copy but is written at a separate disk location. In-place update: The disk version of the data item is overwritten by the cache version.

27. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 27 Database Recovery 5 Data Caching Data items to be modified are first stored into database cache by the Cache Manager (CM) and after modification they are flushed (written) to the disk. The flushing is controlled by Modified and Pin- Unpin bits. Pin-Unpin: Instructs the operating system not to flush the data item. Modified: Indicates the AFIM of the data item.

28. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 28 Database Recovery 6 Transaction Roll-back (Undo) and Roll- Forward (Redo) To maintain atomicity, a transaction’s operations are redone or undone. Undo: Restore all BFIMs on to disk (Remove all AFIMs). Redo: Restore all AFIMs on to disk. Database recovery is achieved either by performing only Undos or only Redos or by a combination of the two. These operations are recorded in the log as they happen.

29. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 29 Database Recovery

30. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 30 Database Recovery

31. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 31 Database Recovery Roll-back: One execution of T1, T2 and T3 as recorded in the log.

32. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 32 Database Recovery Write-Ahead Logging When in-place update (immediate or deferred) is used then log is necessary for recovery and it must be available to recovery manager. This is achieved by Write-Ahead Logging (WAL) protocol. WAL states that For Undo: Before a data item’s AFIM is flushed to the database disk (overwriting the BFIM) its BFIM must be written to the log and the log must be saved on a stable store (log disk). For Redo: Before a transaction executes its commit operation, all its AFIMs must be written to the log and the log must be saved on a stable store.

33. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 33 Database Recovery 7 Checkpointing Time to time (randomly or under some criteria) the database flushes its buffer to database disk to minimize the task of recovery. The following steps defines a checkpoint operation: 1.Suspend execution of transactions temporarily. 2.Force write modified buffer data to disk. 3.Write a [checkpoint] record to the log, save the log to disk. 4.Resume normal transaction execution. During recovery redo or undo is required to transactions appearing after [checkpoint] record.

34. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 34 Database Recovery Steal/No-Steal and Force/No-Force Possible ways for flushing database cache to database disk: 1.Steal: Cache can be flushed before transaction commits. 2.No-Steal: Cache cannot be flushed before transaction commit. 3.Force: Cache is immediately flushed (forced) to disk. 4.No-Force: Cache is deferred until transaction commits These give rise to four different ways for handling recovery: Steal/No-Force (Undo/Redo) Steal/Force (Undo/No-redo) No-Steal/No-Force (Redo/No-undo) No-Steal/Force (No-undo/No-redo)

35. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 35 Database Recovery 8 Recovery Scheme Deferred Update (No Undo/Redo) The data update goes as follows: A set of transactions records their updates in the log. At commit point under WAL scheme these updates are saved on database disk. After reboot from a failure the log is used to redo all the transactions affected by this failure. No undo is required because no AFIM is flushed to the disk before a transaction commits.

36. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 36 Database Recovery Deferred Update in a single-user system There is no concurrent data sharing in a single user system. The data update goes as follows: A set of transactions records their updates in the log. At commit point under WAL scheme these updates are saved on database disk. After reboot from a failure the log is used to redo all the transactions affected by this failure. No undo is required because no AFIM is flushed to the disk before a transaction commits.

37. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 37 Database Recovery

38. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 38 Database Recovery Deferred Update with concurrent users This environment requires some concurrency control mechanism to guarantee isolation property of transactions. In a system recovery transactions which were recorded in the log after the last checkpoint were redone. The recovery manager may scan some of the transactions recorded before the checkpoint to get the AFIMs.

39. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 39 Database Recovery

40. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 40 Database Recovery Deferred Update with concurrent users Two tables are required for implementing this protocol: Active table: All active transactions are entered in this table. Commit table: Transactions to be committed are entered in this table. During recovery, all transactions of the commit table are redone and all transactions of active tables are ignored since none of their AFIMs reached the database. It is possible that a commit table transaction may be redone twice but this does not create any inconsistency because of a redone is “idempotent”, that is, one redone for an AFIM is equivalent to multiple redone for the same AFIM.

41. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 41 Database Recovery Recovery Techniques Based on Immediate Update Undo/No-redo Algorithm In this algorithm AFIMs of a transaction are flushed to the database disk under WAL before it commits. For this reason the recovery manager undoes all transactions during recovery. No transaction is redone. It is possible that a transaction might have completed execution and ready to commit but this transaction is also undone.

42. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 42 Database Recovery Recovery Techniques Based on Immediate Update Undo/Redo Algorithm (Single-user environment) Recovery schemes of this category apply undo and also redo for recovery. In a single-user environment no concurrency control is required but a log is maintained under WAL. Note that at any time there will be one transaction in the system and it will be either in the commit table or in the active table. The recovery manager performs: Undo of a transaction if it is in the active table. Redo of a transaction if it is in the commit table.

43. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 43 Database Recovery Recovery Techniques Based on Immediate Update Undo/Redo Algorithm (Concurrent execution) Recovery schemes of this category applies undo and also redo to recover the database from failure. In concurrent execution environment a concurrency control is required and log is maintained under WAL. Commit table records transactions to be committed and active table records active transactions. To minimize the work of the recovery manager checkpointing is used. The recovery performs: Undo of a transaction if it is in the active table. Redo of a transaction if it is in the commit table.

44. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 44 Database Recovery Shadow Paging The AFIM does not overwrite its BFIM but recorded at another place on the disk. Thus, at any time a data item has AFIM and BFIM (Shadow copy of the data item) at two different places on the disk. X and Y: Shadow copies of data items X' and Y': Current copies of data items

45. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 45 Database Recovery Shadow Paging To manage access of data items by concurrent transactions two directories (current and shadow) are used. The directory arrangement is illustrated below. Here a page is a data item.

46. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 46 Database Recovery The ARIES Recovery Algorithm The ARIES Recovery Algorithm is based on: WAL (Write Ahead Logging) Repeating history during redo: ARIES will retrace all actions of the database system prior to the crash to reconstruct the database state when the crash occurred. Logging changes during undo: It will prevent ARIES from repeating the completed undo operations if a failure occurs during recovery, which causes a restart of the recovery process.

47. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 47 Database Recovery The ARIES Recovery Algorithm (contd.) The ARIES recovery algorithm consists of three steps: 1.Analysis: step identifies the dirty (updated) pages in the buffer and the set of transactions active at the time of crash. The appropriate point in the log where redo is to start is also determined. 2.Redo: necessary redo operations are applied. 3.Undo: log is scanned backwards and the operations of transactions active at the time of crash are undone in reverse order.

48. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 48 Database Recovery The ARIES Recovery Algorithm (contd.) The Log and Log Sequence Number (LSN) A log record is written for: (a) data update (b) transaction commit (c) transaction abort (d) undo (e) transaction end In the case of undo a compensating log record is written.

49. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 49 Database Recovery The ARIES Recovery Algorithm (contd.) The Log and Log Sequence Number (LSN) (contd.) A unique LSN is associated with every log record. LSN increases monotonically and indicates the disk address of the log record it is associated with. In addition, each data page stores the LSN of the latest log record corresponding to a change for that page. A log record stores (a) the previous LSN of that transaction (b) the transaction ID (c) the type of log record.

50. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 50 Database Recovery The ARIES Recovery Algorithm (contd.) The Log and Log Sequence Number (LSN) (contd.) A log record stores: 1.Previous LSN of that transaction: It links the log record of each transaction. It is like a back pointer points to the previous record of the same transaction 2.Transaction ID 3.Type of log record For a write operation the following additional information is logged: 1.Page ID for the page that includes the item 2.Length of the updated item 3.Its offset from the beginning of the page 4.BFIM of the item 5.AFIM of the item

51. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 51 Database Recovery The ARIES Recovery Algorithm (contd.) The Transaction table and the Dirty Page table For efficient recovery following tables are also stored in the log during checkpointing: Transaction table: Contains an entry for each active transaction, with information such as transaction ID, transaction status and the LSN of the most recent log record for the transaction. Dirty Page table: Contains an entry for each dirty page in the buffer, which includes the page ID and the LSN corresponding to the earliest update to that page.

52. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 52 Database Recovery The ARIES Recovery Algorithm (contd.) Checkpointing A checkpointing does the following: Writes a begin_checkpoint record in the log Writes an end_checkpoint record in the log. With this record the contents of transaction table and dirty page table are appended to the end of the log. Writes the LSN of the begin_checkpoint record to a special file. This special file is accessed during recovery to locate the last checkpoint information. To reduce the cost of checkpointing and allow the system to continue to execute transactions, ARIES uses “fuzzy checkpointing”.

53. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 53 Database Recovery The ARIES Recovery Algorithm (contd.) The following steps are performed for recovery Analysis phase: Start at the begin_checkpoint record and proceed to the end_checkpoint record. Access transaction table and dirty page table are appended to the end of the log. Note that during this phase some other log records may be written to the log and transaction table may be modified. The analysis phase compiles the set of redo and undo to be performed and ends. Redo phase: Starts from the point in the log up to where all dirty pages have been flushed, and move forward to the end of the log. Any change that appears in the dirty page table is redone. Undo phase: Starts from the end of the log and proceeds backward while performing appropriate undo. For each undo it writes a compensating record in the log. The recovery completes at the end of undo phase.

54. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 54 Database Recovery

55. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 55 Database Recovery 10 Recovery in multidatabase system A multidatabase system is a special distributed database system where one node may be running relational database system under UNIX, another may be running object-oriented system under Windows and so on. A transaction may run in a distributed fashion at multiple nodes. In this execution scenario the transaction commits only when all these multiple nodes agree to commit individually the part of the transaction they were executing. This commit scheme is referred to as “two-phase commit” (2PC). If any one of these nodes fails or cannot commit the part of the transaction, then the transaction is aborted. Each node recovers the transaction under its own recovery protocol.

56. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 16- 56 Summary Physical Database Design in Relational Databases Database Tuning in Relational Systems

57. Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide 19- 57 Summary Databases Recovery Types of Failure Transaction Log Data Updates Data Caching Transaction Roll-back (Undo) and Roll-Forward Checkpointing Recovery schemes ARIES Recovery Scheme Recovery in Multidatabase System