2. DBMS 2001Notes 7: Crash Recovery1 Principles of Database Management Systems 7: Crash Recovery Pekka Kilpeläinen (after Stanford CS245 slide originals by Hector Garcia-Molina, Jeff Ullman and Jennifer Widom)

3. DBMS 2001Notes 7: Crash Recovery2 PART II of Course Major consern so far: –How to manipulate database efficiently? Now shift focus to: –How to ensure correctness of database manipulation?

4. DBMS 2001Notes 7: Crash Recovery3 Integrity or correctness of data Would like data to be “accurate” or “correct” at all times EMP Name White Green Gray Age 52 3421 1

5. DBMS 2001Notes 7: Crash Recovery4 Integrity or consistency constraints Predicates that the data must satisfy Examples: - x is key of relation R - x  y holds in R - Domain(x) = {Red, Blue, Green} - no employee should make more than twice the average salary

6. DBMS 2001Notes 7: Crash Recovery5 Definition: Consistent state: satisfies all constraints Consistent DB: DB in a consistent state

7. DBMS 2001Notes 7: Crash Recovery6 Constraints ( as we use here) may not capture “full correctness” Example 1 Transaction constraints When salary is updated, new salary > old salary When account record is deleted, balance = 0 In general, various policies to ensure that the DB corresponds to "real world"

8. DBMS 2001Notes 7: Crash Recovery7 a 2 TOT.... 50.... 1000.... 150.... 1000.... 150.... 1100 Obs: DB cannot be consistent always! Example: a 1 + a 2 +…. a n = TOT ( constraint ) Deposit $100 in a 2 : a 2  a 2 + 100 TOT  TOT + 100

9. DBMS 2001Notes 7: Crash Recovery8 Transaction: collection of actions that preserve consistency (  process) Consistent DBConsistent DB’ T SQL: each query or modification statement Embedded SQL: Sequence of DB operations up to COMMIT or ROLLBACK ("abort")

10. DBMS 2001Notes 7: Crash Recovery9 Big assumption: If transaction T starts with consistent state + executes in isolation  T leaves consistent state Correctness (informally) If we stop running transactions, DB left consistent Each transaction sees a consistent DB

11. DBMS 2001Notes 7: Crash Recovery10 How can constraints be violated? Transaction bug DBMS bug Hardware failure e.g., disk crash alters balance of account Simultaneous transactions accessing shared data e.g.: T1: give 10% raise to programmers T2: change programmers  systems analysts

12. DBMS 2001Notes 7: Crash Recovery11 How can we prevent/fix violations? This lecture: due to system failures only [Chapter 8] Chapter 9: due to concurrency only Chapter 10: due to failures and concurrency

13. DBMS 2001Notes 7: Crash Recovery12 Will not consider: How to write correct transactions How to write correct DBMS Constraint checking & repair That is, solutions studied here do not need to know constraints

14. DBMS 2001Notes 7: Crash Recovery13 Events Desired Undesired Expected Unexpected Failure Model

15. DBMS 2001Notes 7: Crash Recovery14 Context of failure model processor memory disk - volatile, - nonvolatile disappears when power off CPU M D

16. DBMS 2001Notes 7: Crash Recovery15 Desired events: see product manuals…. Undesired expected events: System failures - SW errors, power loss -> memory and transaction state lost Undesired Unexpected: Everything else! that’s it!!

17. DBMS 2001Notes 7: Crash Recovery16 Examples: Disk data is lost Memory lost without CPU halt Aliens attack and destroy the building…. Undesired Unexpected: Everything else! (can protect against this by archive backups)

18. DBMS 2001Notes 7: Crash Recovery17 Is this model reasonable? Approach: Add low level checks + redundancy to increase probability that the model holds E.g., Replicate disk storage (stable store) Memory parity CPU checks

19. DBMS 2001Notes 7: Crash Recovery18 Interacting Address Spaces: Storage hierarchy Memory Disk x x t Input(x) Output(x) Read(x,t) Write(x,t) Input (x): block with x from disk to buffer Output (x): block with x from buffer to disk Read (x,t): transaction variable t:= X; (Performs Input(x) if needed) Write (x,t): X (in buffer) := t

20. DBMS 2001Notes 7: Crash Recovery19 Note on "database elements" X: Anything that can have value and be accessed of modified by transactions: –a relation (or extent of an object class) –a disk block –a record Easiest to use blocks as database elements

21. DBMS 2001Notes 7: Crash Recovery20 Key problem Unfinished transaction ExampleConstraint: A=B (*) T 1 : A  A  2 B  B  2 (*) simplification; a more realistic example: the sum of loan balances = the total debt of a bank

22. DBMS 2001Notes 7: Crash Recovery21 T 1 :Read (A,t); t  t  2 Write (A,t); Read (B,t); t  t  2 Write (B,t); Output (A); Output (B); A: 8 B: 8 A: 8 B: 8 memory disk 16 failure!  A != B 16

23. DBMS 2001Notes 7: Crash Recovery22 Need atomicity: either execute all actions of a transaction or none at all One solution: undo logging (immediate modification) Log: sequence of log records, recording actions of transactions T i : : transaction has begun : completed succesfully : could not complete succesfully : T i has updated element X, whose old value was v

24. DBMS 2001Notes 7: Crash Recovery23 T 1 :Read (A,t); t  t  2 Write (A,t); Read (B,t); t  t  2 Write (B,t); Output (A); Output (B); A:8 B:8 A:8 B:8 memory disk log Undo logging (Immediate modification) 16 16 16 constraint: A=B

25. DBMS 2001Notes 7: Crash Recovery24 One “complication” Log is first written in memory Not written to disk on every action memory DB Log A: 8 16 B: 8 16 Log: A: 8 B: 8 16 BAD STATE # 1

26. DBMS 2001Notes 7: Crash Recovery25 One “complication” Log is first written in memory Not written to disk on every action memory DB Log A: 8 16 B: 8 16 Log: A: 8 B: 8 16 BAD STATE # 2...

27. DBMS 2001Notes 7: Crash Recovery26 Undo logging rules (1) Generate an undo log record for every update action (with the old value) (2) Before modifying element X on disk, any logrecords for X must be flushed to disk (write-ahead logging) (3) All WRITEs of transaction T must be OUTPUT to disk before is flushed to log

28. DBMS 2001Notes 7: Crash Recovery27 Recovery rules: Undo logging For every Ti with in log: - If or in log, do nothing - Else For all in log: write (X, v) output (X ) Write to log  IS THIS CORRECT??

29. DBMS 2001Notes 7: Crash Recovery28 T 1 :A  A+1; B  B+1; T 2 :A  A+1; B  B+1; A:10 B:10 DB on disk log Recovery with Undo logging CRASH => A should be restored to 10, not 11 11 constraint: A=B

30. DBMS 2001Notes 7: Crash Recovery29 Undo logging Recovery : (more precisely) (1) Let S = set of transactions with in log, but no (or ) record in log (2) For each in log, in reverse order (latest  earliest) do: - if Ti  S then - write (X, v) - output (X) (3) For each Ti  S do - write to log

31. DBMS 2001Notes 7: Crash Recovery30 What if system fails during recovery? No problem! Undo idempotent - Can repeat (unfinished) undo, result remains the same

32. DBMS 2001Notes 7: Crash Recovery31 To discuss: Redo logging Undo/redo logging, why both? Checkpointing Media failures

33. DBMS 2001Notes 7: Crash Recovery32 Redo logging (deferred modification) T 1: Read(A,t); t t  2; write (A,t); Read(B,t); t t  2; write (B,t); Output(A); Output(B) A: 8 B: 8 A: 8 B: 8 memory DB LOG 16 output 16

34. DBMS 2001Notes 7: Crash Recovery33 Redo logging rules (1) For every disk update, generate a redo log record (with new value) (2) Before modifying DB element X on disk, all log records for the transaction that (including COMMIT) must be flushed to disk

35. DBMS 2001Notes 7: Crash Recovery34 For every Ti with in log: –For all in log: Write(X, v) Output(X) Recovery rules: Redo logging This time, the last updates should remain in effect

36. DBMS 2001Notes 7: Crash Recovery35 (1) Let S = set of transactions with in log (2) For each in log, in forward order (earliest  latest) do: - if Ti  S then Write(X, v) Output(X) Recovery with Redo Logging: (more precisely)

37. DBMS 2001Notes 7: Crash Recovery36 Recovery is very, very SLOW ! Redo log: FirstT1 wrote A,BLast RecordCommitted a year agoRecord (1 year ago)--> STILL, Need to redo after crash!!... Crash

38. DBMS 2001Notes 7: Crash Recovery37 Solution: Checkpointing (for redo-logging, simple version) Periodically: (1) Stop accepting new transactions (2) Wait all transactions to finish (commit/abort) (3) Flush all log records to disk (log) (4) Flush all buffers to disk (DB) (5) Write & flush a record on disk (log) (6) Resume transaction processing

39. DBMS 2001Notes 7: Crash Recovery38 Example: what to do at recovery? Redo log (on disk): Checkpoint Crash... Result of transactions completed prior to the checkpoint are stored permanently on disk -> Redo write(B, 17) output(B)

40. DBMS 2001Notes 7: Crash Recovery39 Drawbacks: Undo logging: –System forced to update disk at the end of transactions (may increase disk I/O) –cannot redo recent updates to refresh a backup copy of the DB Redo logging: –need to keep all modified blocks in memory until commit (may lead to shortage of buffer pool)

41. DBMS 2001Notes 7: Crash Recovery40 Solution: Undo/redo logging! Update record  Provides more flexibility to order actions

42. DBMS 2001Notes 7: Crash Recovery41 Rules for undo/redo logging (1) Log record flushed to disk before writing X to disk (2) Flush the log at commit –Element X can be written to disk either before or after the commit of Ti

43. DBMS 2001Notes 7: Crash Recovery42 Undo/Redo Recovery Policy To recover from a crash, –redo updates by any committed transactions (in forward-order, earliest first) –undo updates by any incomplete transactions (in backward-order, latest first)

44. DBMS 2001Notes 7: Crash Recovery43 Problem with Simple Checkpointing Simple checkpointing stops the system from accepting new transactions  system effectively shuts down Solution: "nonquiescent" checkpointing –accepts new transactions during a checkpoint

45. DBMS 2001Notes 7: Crash Recovery44 Nonquiescent Checkpointing Slightly different for various logging policies Rules for undo/redo logging: –Write log record (and flush log), where T1, …, Tk are the active transactions –Flush to disk all dirty buffers which contain changed database elements (Corresponding log records first!) => Effect of writes prior to the chkpt made permanent –Write & flush log record

46. DBMS 2001Notes 7: Crash Recovery45 Non-quiescent checkpoint L O G for undodirty buffer pool pages flushed <Start ckpt Ti,T2,…> <end ckpt>...

47. DBMS 2001Notes 7: Crash Recovery46 Examples what to do at recovery time? no T1 commit L O G... Ckpt T 1... Ckpt end......  Undo T 1 (restore Y to b and X to a)

48. DBMS 2001Notes 7: Crash Recovery47 Example LOGLOG... T1aT1a T1bT1b T1cT1c T 1 cmt... ckpt- end ckpt-s T 1  Redo T1: (redo update of elements to b and to c)

49. DBMS 2001Notes 7: Crash Recovery48 Recovery process: Backwards pass (end of log  latest checkpoint start) –construct set S of committed transactions –undo actions of transactions not in S Undo pending transactions –follow undo chains for transactions in (checkpoint active list) - S Forward pass (latest checkpoint start  end of log) –redo actions of transactions in set S backward pass forward pass start check- point

50. DBMS 2001Notes 7: Crash Recovery49 Media failure (loss of non-volatile storage) A: 16 Solution: Make copies of data! Disk crash!

51. DBMS 2001Notes 7: Crash Recovery50 Solution a: Use mirrored or redundant disks Mirroring: copies on separate disks –Output(X) --> three outputs –Input(X) --> three inputs + vote D1 D2 D3

52. DBMS 2001Notes 7: Crash Recovery51 Example #2 Redundant writes, Single reads Keep N copies on separate disks Output(X) --> N outputs Input(X) --> Input one copy - if ok, done - else try another one  Assumes bad data can be detected

53. DBMS 2001Notes 7: Crash Recovery52 Solution b: Archiving backup database active database log If active database is lost, – restore active database from backup (archive) – bring up-to-date using redo entries in log

54. DBMS 2001Notes 7: Crash Recovery53 When can we discard the log? check- point db dump last needed undo not needed for media recovery not needed for undo after system failure not needed for redo after system failure log time

55. DBMS 2001Notes 7: Crash Recovery54 Summary Consistency of data Recovery from system failures –Logging (undo/redo/undo-redo), checkpoints –Recovery: using log to reconstruct the DB Recovery form media failures –reducing risk through redundancy –backups / archiving Another source of problems: –Concurrent transactions that access shared DB elements.... NEXT