1. 1 Transactions Controlling Concurrent Behavior

2. 2 Why Transactions?  Database systems are normally being accessed by many users or processes at the same time.  Both queries and modifications.

3. 3 If a CPU instruction= 1 second, a disk read takes a month! 1-5 Clocks 0.5-5 Clocks 50-100 clocks 105clocks

4. 4 Why Transactions? Concurrent execution of user programs is essential for good DBMS performance. – Disk accesses may be frequent and slow, – Idea: keep CPUs humming by working on several user programs concurrently. Optimize for throughput (# of TXs) Trade for latency (time for any one TX)

5. 5 Why Transactions?  Unlike operating systems, which support interaction of processes, a DMBS needs to keep processes from troublesome interactions.

6. 6 Example: Bad Interaction  You and your domestic partner each take $100 from different ATM’s at about the same time.  The DBMS better make sure one account deduction doesn’t get lost.  Compare: An OS allows two people to edit a document at the same time. If both write, one’s changes get lost.

7. 7 7 Example: single statements Client 1: UPDATE Product SET Price = Price – 1.99 WHERE pname = ‘Gizmo’ Client 2: UPDATE Product SET Price = Price*0.5 WHERE pname=‘Gizmo’ Two managers attempt to discount products. What could go wrong?

8. 8 8 Example: multiple statements Client 1: INSERT INTO SmallProduct(name, price) SELECT pname, price FROM Product WHERE price <= 0.99 DELETE Product WHERE price <=0.99 Client 2:SELECT count(*) FROM Product SELECT count(*) FROM SmallProduct What’s wrong ?

9. 9 9 Example: crashes Client 1: INSERT INTO SmallProduct(name, price) SELECT pname, price FROM Product WHERE price <= 0.99 (CRASH!) DELETE Product WHERE price <=0.99 What’s wrong ?

10. 10 Transactions  Transaction = process involving database queries and/or modification.  Normally with some strong properties regarding concurrency.  Formed in SQL from single statements or explicit programmer control.

11. 11 Revised Code Client 1: START TRANSACTION UPDATE Product SET Price = Price – 1.99 WHERE pname = ‘Gizmo’ COMMIT Client 2: START TRANSACTION UPDATE Product SET Price = Price*0.5 WHERE pname=‘Gizmo’ COMMIT Now it works like a charm

12. 12 ACID Transactions  ACID transactions are:  Atomic : Whole transaction or none is done.  Consistent : Database constraints preserved.  Isolated : It appears to the user as if only one process executes at a time.  Durable : Effects of a process survive a crash.  Optional: weaker forms of transactions are often supported as well.

13. 13 ACID: Atomicity Two possible outcomes for a transaction – It commits: all the changes are made – It aborts: no changes are made That is, transaction’s activities are all or nothing

14. 14 ACID: Consistency The state of the tables is restricted by integrity constraints – Account number is unique – Stock amount can’t be negative – Sum of debits and of credits is 0 Constraints may be explicit or implicit How consistency is achieved: – Programmer makes sure a txn takes a consistent state to a consistent state – System makes sure that the txn is atomic

15. 15 ACID: Isolation A transaction executes concurrently with other transaction Isolation: the effect is as if each transaction executes in isolation of the others

16. 16 ACID: Durability The effect of a transaction must continue to exists after the transaction, or the whole program has terminated Means: write data to disk Change on the horizon? Non-Volatile Ram (NVRam). Byte addressable.

17. 17 Challenges for ACID properties In spite of failures: Power failures, but not media failures Users may abort the program: need to “rollback the changes” Many users executing concurrently – Can be solved via locking Need to log what happened Performance!!

18. 18 Example: Consider two transactions: T1:BEGIN A=A+100, B=B-100 END T2:BEGIN A=1.06*A, B=1.06*B END T1 transfers $100 from B’s account to A’s account. T2 credits both accounts with a 6% interest payment.

19. 19 Gold Standard Serial Execution T1: A=A+100, B=B-100 T2: A=1.06*A, B=1.06*B Execute T1 and T2, one after the other. Or T2 then T1. Database allows either! Tricky!

20. 20 Database has freedom to interleave Consider a possible interleaving (schedule): T1: A=A+100, B=B-100 T2: A=1.06*A, B=1.06*B Seems OK… But what about: T1: A=A+100, B=B-100 T2: A=1.06*A, B=1.06*B The DBMS’s view of the second schedule: T1: R(A), W(A), R(B), W(B) T2: R(A), W(A), R(B), W(B)

21. 21 Scheduling Definitions Serial schedule schedule that does not interleave the actions of different transactions. Equivalent schedules: For any database state, the effect on DB of executing the first schedule is identical to the effect of executing the second schedule. Serializable schedule: A schedule that is equivalent to some serial execution of the transactions.

22. 22 Database has freedom to interleave T1: A=A+100, B=B-100 T2: A=1.06*A, B=1.06*B T1: A=A+100, B=B-100 T2: A=1.06*A, B=1.06*B Are these serializable? Yes, equivalent to T1 then T2 – even though actions of T1 & T2 are interleaved (a+100)*1.06 (b*1.06)-100 T1, T2  (x+100)*1.06 T2, T1  (x*1.06)-100 For both of x in (a,b) They differ!

23. 23 What else can go wrong? Anomalies Often Referred to by name

24. 24 Anomalies with Interleaved Execution Reading Uncommitted Data (WR Conflicts, “dirty reads”): Unrepeatable Reads (RW Conflicts): T1: R(A), W(A), R(B), W(B), Abort T2: R(A), W(A), C T1:R(A), R(A), W(A), C T2: R(A), W(A), C

25. 25 Anomalies (Continued) Overwriting Uncommitted Data (WW Conflicts): T1:W(A), W(B), C T2: W(A), W(B), C

26. 26 Famous anomalies Summary Dirty read – T reads data written by T’ while T’ is running – Then T’ aborts Lost update – Two tasks T and T’ both modify the same data – T and T’ both commit – Final state shows effects of only T, but not of T’ Inconsistent read – Task T sees some but not all changes made by T’

27. 27 Transaction Statements - Isolation Levels

28. 28 Example: Interacting Processes  Assume the usual Sells(bar,beer,price) relation, and suppose that Joe’s Bar sells only Bud for $2.50 and Miller for $3.00.  Sally is querying Sells for the highest and lowest price Joe charges.  Joe decides to stop selling Bud and Miller, but to sell only Heineken at $3.50.

29. 29 Sally’s Program  Sally executes the following two SQL statements called (min) and (max) to help us remember what they do. (max)SELECT MAX(price) FROM Sells WHERE bar = ’Joe’’s Bar’; (min)SELECT MIN(price) FROM Sells WHERE bar = ’Joe’’s Bar’;

30. 30 Joe’s Program  At about the same time, Joe executes the following steps: (del) and (ins). (del) DELETE FROM Sells WHERE bar = ’Joe’’s Bar’; (ins) INSERT INTO Sells VALUES(’Joe’’s Bar’, ’Heineken’, 3.50);

31. 31 Interleaving of Statements  Although (max) must come before (min), and (del) must come before (ins), there are no other constraints on the order of these statements, unless we group Sally’s and/or Joe’s statements into transactions.

32. 32 Example: Strange Interleaving  Suppose the steps execute in the order (max)(del)(ins)(min). Joe’s Prices: Statement: Result:  Sally sees MAX < MIN! {2.50,3.00} (del) (ins) {3.50} (min) 3.50 {2.50,3.00} (max) 3.00

33. 33 Fixing the Problem by Using Transactions  If we group Sally’s statements (max)(min) into one transaction, then she cannot see this inconsistency.  She sees Joe’s prices at some fixed time.  Either before or after he changes prices, or in the middle, but the MAX and MIN are computed from the same prices.

34. 34 Another Problem: Rollback  Suppose Joe executes (del)(ins), not as a transaction, but after executing these statements, thinks better of it and issues a ROLLBACK statement.  If Sally executes her statements after (ins) but before the rollback, she sees a value, 3.50, that never existed in the database.

35. 35 Solution  If Joe executes (del)(ins) as a transaction, its effect cannot be seen by others until the transaction executes COMMIT.  If the transaction executes ROLLBACK instead, then its effects can never be seen.

36. 36 START  The SQL statement START TRANSACTION begins a transaction.

37. 37 COMMIT  The SQL statement COMMIT causes a transaction to complete.  It’s database modifications are now permanent in the database.

38. 38 ROLLBACK  The SQL statement ROLLBACK also causes the transaction to end, but by aborting.  No effects on the database.  Failures like division by 0 or a constraint violation can also cause rollback, even if the programmer does not request it.

39. 39 Isolation Levels  SQL defines four isolation levels = choices about what interactions are allowed by transactions that execute at about the same time.  Only one level (“serializable”) = ACID transactions.  Each DBMS implements transactions in its own way.

40. 40 Choosing the Isolation Level  Within a transaction, we can say: SET TRANSACTION ISOLATION LEVEL X where X = 1.SERIALIZABLE 2.REPEATABLE READ 3.READ COMMITTED 4.READ UNCOMMITTED

41. 41 Serializable Transactions  If Sally = (max)(min) and Joe = (del)(ins) are each transactions, and Sally runs with isolation level SERIALIZABLE, then she will see the database either before or after Joe runs, but not in the middle.

42. 42 Isolation Level Is Personal Choice  Your choice, e.g., run serializable, affects only how you see the database, not how others see it.  Example: If Joe Runs serializable, but Sally doesn’t, then Sally might see no prices for Joe’s Bar.  i.e., it looks to Sally as if she ran in the middle of Joe’s transaction.

43. 43 Read-Commited Transactions  If Sally runs with isolation level READ COMMITTED, then she can see only committed data, but not necessarily the same data each time.  Example: Under READ COMMITTED, the interleaving (max)(del)(ins)(min) is allowed, as long as Joe commits.  Sally sees MAX < MIN.

44. 44 Repeatable-Read Transactions  Requirement is like read-committed, plus: if data is read again, then everything seen the first time will be seen the second time.  But the second and subsequent reads may see more tuples as well.

45. 45 Example: Repeatable Read  Suppose Sally runs under REPEATABLE READ, and the order of execution is (max)(del)(ins)(min).  (max) sees prices 2.50 and 3.00.  (min) can see 3.50, but must also see 2.50 and 3.00, because they were seen on the earlier read by (max).

46. 46 Read Uncommitted  A transaction running under READ UNCOMMITTED can see data in the database, even if it was written by a transaction that has not committed (and may never).  Example: If Sally runs under READ UNCOMMITTED, she could see a price 3.50 even if Joe later aborts.

47. 47 Implementation Mechanisms

48. 48 Motivation Atomicity: Transactions may abort (“Rollback”). Durability: What if DBMS crash? crash! Desired Behavior after system restarts:  T1, T2 & T3 should be durable.  T4 & T5 should be aborted (effects not seen). T1 T2 T3 T4 T5

49. 49 Two High-level Mechanisms 1. Use Logging to make sure we can undo operations. 1. Use Locking to make sure that each transaction “sees” a consistent view of the world Recovery uses 1. & 2. to ensure that the database stays consistent even after crashes or transactions abort.

50. 50 The Log Is a list of modifications Log is duplexed and archived on stable storage. – Assume we don’t lose it! Can force write entries to disk – A page goes to disk. All log activities handled transparently the DBMS.

51. 51 Basic Idea: (Physical) Logging Record UNDO information for every update! – Sequential writes to log – Minimal info (diff) written to log Log: An ordered list of actions – Log record contains: – Sufficient to UNDO any transaction!

52. 52 A picture of logging Data on Disk Main Memory Log on Disk Log T A=0 B=5 A=0 T: R(A), W(A)

53. 53 Write-Ahead Logging (WAL) DB uses Write-Ahead Logging Protocol. Each update is logged! Why not reads? 1. Must force log record for an update before the corresponding data page goes to storage. 1. Must write all log records for a TX before commit. #1 guarantees Atomicity. #2 guarantees Durability.

54. 54 A picture of logging Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1

55. 55 A picture of logging Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 NB: Logging can happen after modification, but not before disk!

56. 56 What can go wrong?

57. 57 DBMS Writes Back A

58. 58 A picture of logging Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 What if we crash now? Or T aborts?

59. 59 With WAL! Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 Now, if we crash can recover correct value of A !

60. 60 TX commit

61. 61 Transaction Commit FORCE Write commit record to log All log records up to last update from this TX are FORCED Commit() returns Transaction is committed once commit record is on stable storage

62. 62 Incorrect Commit Protocol Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 If we crash now, Is T durable? Ok, Commit ! Commit? Lost T’s update!

63. 63 Incorrect Commit Protocol Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 If we crash now, Is T durable? Ok, Commit ! Commit?

64. 64 Improved

65. 65 Commit Protocol Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 If we crash now? Is T durable? Ok, Commit ! Commit?

66. 66 Wrong Commit Protocol Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 Crash … MM is wiped! Are T’s effects visible?

67. 67 Commit Protocol Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 On Crash, MM is wiped! Are T’s effects visible? A=1

68. 68 With WAL! Data on Disk Main Memory Log on Disk Log T A=1 B=5 A=0 T: R(A), W(A) A=0  1 Now, if we crash can recover A!

69. 69 Write-Ahead Logging (WAL) DB uses Write-Ahead Logging Protocol. Each update is logged! Why not reads? 1. Must force log record for an update before the corresponding data page goes to storage. 2. Must write all log records for a TX before commit. #1 guarantees Atomicity. #2 guarantees Durability.

70. 70 Logging Summary If DB says TX commits, TX effect remains after database crash DB can undo actions and help us with atomicity. it’s only half the story…

71. 71 Lock-Based Concurrency Control Strict Two-phase Locking (Strict 2PL) Protocol: 1. Each Xact must obtain a S (shared) lock on object before reading, and an X (exclusive) lock on object before writing. 2. All locks held by a transaction are released when the transaction completes. 3. If an Xact holds an X lock on an object, no other Xact can get a lock (S or X) on that object.

72. 72 Picture of 2-Phase Locking (2PL) Time Strict 2PL 0 locks # Locks Lock Acquisition Lock Release

73. 73 Need to be able to perform Two Related Tasks 1. Abort a transaction. Ability to abort and clean up a task’s impact on the system’s state, 1. Recover from a crash. Main idea is called “repeating history” in the ARIES algorithm.

74. 74 2PL Locking & Serializability

75. 75 Conflict Serializable Schedules Two schedules are conflict equivalent if: – Involve the same actions of the same transactions – Every pair of conflicting actions of two transactions are ordered in the same way Schedule S is conflict serializable if S is conflict equivalent to some serial schedule Two actions conflict if they are in different TXs, on the same object, and one of them is a write.

76. 76 Example A schedule that is not conflict serializable: The cycle in the graph reveals the problem. The output of T1 depends on T2, and vice- versa. T1: R(A), W(A), R(B), W(B) T2: R(A), W(A), R(B), W(B) T1T2 A B Dependency graph

77. 77 Dependency Graph Dependency graph: – One node for each committed Xact T1…TN – edge from Ti to Tj if an actions in Ti precedes and conflicts with an action in Tj. Theorem: Schedule is conflict serializable if and only if its dependency graph is acyclic

78. 78 Strict 2PL Thm: Strict 2PL allows only schedules whose precedence graph is acyclic Therefore, Strict 2PL only allows serializable schedules Are all serializable schedules allowed by Strict 2PL?

79. 79 Serializable but not Conflict Serializable T1: R(A), W(A), C T2: W(A), C T3: W(A), C This is equivalent to T1 T2 T3, so serializable. But not conflict equivalent (T1 and T2s) writes are ordered differently.

80. 80 Summary So far If a schedule follows strict 2PL, it is serializable Not all serializable schedules are allowed by strict 2PL. So let’s use strict 2PL, what could go wrong?

81. 81 Deadlocks Deadlock: Cycle of transactions waiting for locks to be released by each other. Two ways of dealing with deadlocks: – Deadlock prevention – Deadlock detection

82. 82 Deadlock Prevention Assign priorities based on timestamps. Assume Ti wants a lock that Tj holds. Two policies are possible: – Wait-Die: It Ti has higher priority, Ti waits for Tj; otherwise Ti aborts – Wound-wait: If Ti has higher priority, Tj aborts; otherwise Ti waits If a transaction re-starts, make sure it has its original timestamp Issue: What if a transaction never makes progress?

83. 83 Deadlock Detection Create a waits-for graph: – Nodes are transactions – There is an edge from Ti to Tj if Ti is waiting for Tj to release a lock Periodically check for cycles in the waits-for graph

84. 84 Deadlock Detection (Continued) Example: T1: S(A), R(A), S(B) T2: X(B),W(B) X(C) T3: S(C), R(C) X(A) T4: X(B) T1T2 T4T3 T1T2 T3 In general, must search through this big graph. Sounds expensive! Is it?

85. 85 Locking Summary Locks must be atomic, primitive operation 2PL does not avoid deadlock Deadlock detection sounds more expensive than it is….

86. 86 Multiple-Granularity Locks Hard to decide what granularity to lock (tuples vs. pages vs. tables). Shouldn’t have to decide! Data “containers” are nested: Tuples Tables Pages Database contains

87. 87 Solution: New Lock Modes, Protocol Allow Xacts to lock at each level, but with a special protocol using new “intention” locks:  Before locking an item, Xact must set “intention locks” on all its ancestors.  For unlock, go from specific to general (i.e., bottom-up).  SIX mode: Like S & IX at the same time. -- ISIX -- IS IX      SX   S X      

88. 88 Multiple Granularity Lock Protocol

89. 89 Each Xact starts from the root of the hierarchy. To get S or IS lock on a node, must hold IS or IX on parent node. – What if Xact holds SIX on parent? S on parent? To get X or IX or SIX on a node, must hold IX or SIX on parent node. Must release locks in bottom-up order. Protocol is correct in that it is equivalent to directly setting locks at the leaf levels of the hierarchy.

90. 90 T1 scans R, and updates a few tuples: – T1 gets an SIX lock on R, then repeatedly gets an S lock on tuples of R, and occasionally upgrades to X on the tuples. T2 uses an index to read only part of R: – T2 gets an IS lock on R, and repeatedly gets an S lock on tuples of R. T3 reads all of R: – T3 gets an S lock on R. – OR, T3 could behave like T2; can use lock escalation to decide which. -- ISIX -- IS IX      SX   S X      

91. 91 Distributed Databases

92. 92 T1 scans R, and updates a few tuples: – T1 gets an SIX lock on R, then repeatedly gets an S lock on tuples of R, and occasionally upgrades to X on the tuples. T2 uses an index to read only part of R: – T2 gets an IS lock on R, and repeatedly gets an S lock on tuples of R. T3 reads all of R: – T3 gets an S lock on R. – OR, T3 could behave like T2; can use lock escalation to decide which. -- ISIX -- IS IX      SX   S X      

93. 93 Distributed Databases Desiderata Distributed Data Independence: Users do not need to know on which machine their data sits Distributed TX Isolation: Users write TXs that touch many sites just as they would for a single site. Think: 10s of distributed DBMSs

94. 94 Distributed Data Data may be horizontally or vertically fragmented To ensure that vertical fragments can be decomposed losslessly, may assign a tuple id.

95. 95 Replication 1. Increased Availability of Data We store several copies or replicas of a fragment of data redundantly. 2. Increased Performance of Queries.

96. 96 2 Phase Commit (2PC not 2PL), guarantee ACID in distributed RDBMSs.

97. 97 Challenges: What can go wrong in a Distributed Database? Everything that can go wrong in a single node RDBMS (we’ll refer to this as site failure) Link failure. Two machines may both be running, but are unable to communicate. Protocol must cope with both types of errors

98. 98 The Characters in our Story Each site maintains: – A log of the actions that are taken locally (by subtransactions) – We need the ability to force write records. For any TX, the site where it originates is called a coordinator. Other sites are called subordinates.

99. 99 ACK The 2PC Protocol User decides to commit. SubordinatesCoordinator Prepare Each subordinate decides: 1. Commit, FW a prepare record 2. Abort, FW an abort record Votes: Yes or No If all Yes, FW commit. If one No FW abort. FW = Force Write Outcome of vote Each subordinate force writes the outcome After all ACKs, write an end record NB: Log records always written before messages are sent.

100. 100 ACK The 2PC Protocol User decides to commit. SubordinatesCoordinator Prepare Each subordinate decides: 1. Commit, FW a prepare record 2. Abort, FW an abort record Votes: Yes or No If all Yes, FW commit. If one No FW abort. FW = Force Write Outcome of vote Each subordinate force writes the outcome After all ACKs, write an end record When is the TX officially committed?

101. 101 Restart after Node Failure At each node: A Recovery Process. For each TX T If we have a commit/abort record for T. Did we send the ACK? Why does this matter or not? If prepare record for T, but no commit/abort. Then, we need to contact the coordinator. Why? Then, complete TX If we have no prepare record. Then T could not have voted to commit, so we may abort T. Case I: We are a subordinate for T

102. 102 Restart after Node Failure Come back up. See the log. Have to redo/undo all the TXs Focus on the messages exchanged.

103. 103 Case II: Rise of the Coordinator If we have a commit/abort record, then we send all the subordinates the status until we get back ACKS. We then write an END Record. If we have no prepare, commit, or abort for T then we it could not have committed. The coordinator should respond by saying T is aborted if asked.

104. 104 ACK The 2PC Protocol User decides to commit. SubordinatesCoordinator Prepare Each subordinate decides: 1. Commit, FW a prepare record 2. Abort, FW an abort record Votes: Yes or No If all Yes, FW commit. If one No FW abort. FW = Force Write Outcome of vote Each subordinate force writes the outcome After all ACKs, write an end record Convince ourselves that failures are handled

105. 105 Some Observations about the Protocol

106. 106 ACK User decides to commit. SubordinatesCoordinator Prepare Each subordinate decides: 1. Commit, FW a prepare record 2. Abort, FW an abort record Votes: Yes or No If all Yes, FW commit. If one No FW abort. FW = Force Write Outcome of vote Each subordinate force writes the outcome After all ACKs, write an end record If a sub observes that the coordinator fails after sending a YES vote. What action must it take next?

107. 107 ACK User decides to commit. SubordinatesCoordinator Prepare Each subordinate decides: 1. Commit, FW a prepare record 2. Abort, FW an abort record Votes: Yes or No If all Yes, FW commit. If one No FW abort. FW = Force Write Outcome of vote Each subordinate force writes the outcome After all ACKs, write an end record Why are the ACK messages useful?

108. 108 Many optimizations possible! CW: biggest problem with 2PC is blocking which reduces availability. – There are variants to reduce blocking and to reduce # of Force Writes. Was decidedly out of fashion. – Then, GOOG wrote spanner, back in the big time. – Not really, but (nerdy) public perception…

109. 109 ACK User decides to commit. SubordinatesCoordinator Prepare Each subordinate decides: 1. Commit, FW a prepare record 2. Abort, FW an abort record Votes: Yes or No If all Yes, FW commit. If one No FW abort. FW = Force Write Outcome of vote Each subordinate force writes the outcome After all ACKs, write an end record If the coordinator crashes right after sending out a prepare, should it abort or commit? can it?