1. Transactional Recovery and Checkpoints

2. Difference How is this different from schedule recovery?
It is the details to implementing schedule recovery
It is at a lower level (transactional) – Operating System level
Writing out dirty data, etc.

3. Recovery is needed for atomicity and durability
Problems:
Reading
involves buffers, paging and LRU replacement (operating system)
Writing
when to write updates from memory to disk? 
After each update??? Called force approach
When LRU replaces (use dirty bit)?  
What if crash when update still in memory?
No immediate durability  
No atomicity
Solution?
Have a log buffer with log entries and a log file

4. Log buffer and Log file Log buffer stored in memory
Log file is a copy of the log buffer on disk (permanent storage)
Log buffer written to log file at intervals
Assume written when transaction commits
Log file used to perform recovery
Log buffer and log file guarantees atomicity and durability

5. Log Buffer Log buffer contains list of log entries
Log entries: Start, Commit, Write

6. Log Entries
In log entry keep track of: 1)  Start of Transactions (S, T#) S1
2)  committed T's ( C, T#) C1  3)  operations - enter writes but not reads:
(W, T#, data_item (or RID), before_value, after_value)
(W,1,A,10,20)
why before values?
in case must UNDO
why after values?
in case must REDO

7. Log files vs. Data file 2 different concepts:
1) Log buffer entries written to log file
vs.
2) Making changes to actual data on DB disk
DB data file is stored on disk (permanent storage)
Updated data pages (dirty pages) are cached in memory
Data file on disk not updated with dirty pages until some event (discussed later)

8. Durability Problems - WAL
When to write log buffer to log file
Durability - commits successful only once log file written on disk
if failure and log entry not written?
can never UNDO or REDO
Must write log buffer to log file

9. Dirty Pages When to update data on DB disk?
Write dirty pages to DB disk when log buffer full
Write dirty pages to DB disk when commit

10. Dirty Pages – option 1 Write dirty pages when full
How to ensure dirty pages not written until log buffer written to disk (REDO)
Write ahead log guarantee (WAL) log sequence number (LSN)-
every log entry keeps track of smallest LSN to log file since last write (lsn_buffmin)
keep track of updates to data pages (lsn_pgmax)
Can write pages to disk as long as lsn_pgmax < lsn_buffmin
Else?
Updates not written to log file yet

11. Example 1 2 lsn_pgmax lsn_buffmin 7 8 …
Can write pages 4, 5 (e.g. lsn_pgmax=5<lsn_buffmin)

12. Dirty Pages – option 2 Dirty data pages not written until commit
Atomicity Problem: Be sure dirty pages of data not written to DB disk before log entry – how?
Solution: Can modify LRU replacement to ensure data not written to DB disk before written to log file

13. Dirty Pages If updated data page not written to DB disk until commit
No UNDO processing ever
no before images needed in logs
What if lot of updates?  can't keep all in memory – so write whenever full?

14. When to write to disk When to write log buffer to log file?
Assume when a transaction commits
When to write updates to DB disk?
Explore options

15. How to recover from a crash
Must use information from non-volatile (permanent) storage
If not in permanent storage, it is lost
If ever restarted, assumed to be later and NOT part of the recovery
Method:
For each entry in log (move backwards in log) determine if should ignore or undo
Then (move forward in log) and redo relevant entries

16. Example Log entries: 1. (S1) 2. no entry 3. (W,1,A,50,20) 4. (S2)
R1(A,50) W1(A,20) R2(C,100)W2(C,50)C2 R1(B,50)W1(B,8)C1
1   2                     4 5           6             7    8            9          10
         Log entries:             1.  (S1)
2. no entry           
3. (W,1,A,50,20)             4. (S2)
5. no entry             6. (W,2,C,100,50)             7. (C2) write log buffer to log file             8.  no entry             9.  (W,1,B,50,80)            10. (C1) write log buffer to log file
   What if crash after operation 9 but before 10?

17. One way to recover from a crash
R1(A,50) W1(A,20) R2(C,100)W2(C,50)C2 R1(B,50)W1(B,8)T1 Crashes
T1 not committed, T2 committed – what to do?
Must undo T1 and redo T2
Rollback until log file empty  1.  C2 - T2 on committed list  2.  (W,2,C,100,50)  T2 committed, do nothing 3.  (S2)  T2 no longer active 4.  (W,1,A,50,20) T1 not on the committed list
UNDO this update only if written to data disk
(NOTE: Differs from textbook – says can always UNDO) 5.  (S1)  T1 no longer active

18. ACID properties
For atomicity Rollback - make list of committed T's and UNDO uncommitted T's actions if written to DB disk
For durability Rollforward - to REDO committed T's actions if not written to DB disk. This means load data (old value) from disk into memory and apply operation again
NOTE: can only UNDO, REDO what is in log file NOT log buffer

19. How to recover cont’d
Rollforward  6.  S1 - no action  7.  (W,1,A,50,20) T1 uncommitted - no action 8.  (S,2) no action 9.  (W,2,C,100,50) Redo update – assume not on data disk yet
10.  (C,2) no action
11. done
Note: T1 may be restarted later, but not part of recovery

20. If write log buffer to log file when commit
When to write dirty data to DB disk?

21. Common Way to guarantee ACID
Checkpoints: Provides an approach to address ACID properties
Always write buffer to log file when commit
Also determines when to write to data file on disk
Write dirty data to DB disk each time commit??
Maybe not

22. Checkpoints
Recovery Checkpoints used so only have to rollback to a certain point
A Checkpoint
Consistent snapshot of all of the database - values on durable disk
“A Checkpoint is a database event which synchronizes the modified data blocks in memory with the datafiles on disk.”

23. Checkpoint CKPT – my definition
A CKPT reflects current state of database
A CKPT includes data updates from all newly committed transactions
and possibly updates from uncommitted transactions
Assume dirty data only written at CKPT

24. Recovery
T's short-lived (take a few seconds) therefore rollback is quick - only a few T's active that need to be UNDONE
Rollforward takes longer - many T's to REDO, keep track of start T's, etc.

25. Checkpoint strategies
At checkpoint:
log buffer written to log file
Updated pages written to data file on disk
Still assume log buffer always written to log file at commit
3 different approaches for when to take checkpoints
Commit consistent
Cache consistent
Fuzzy consistent

26. Commit consistent
1.  Commit consistent -needed when count of log events exceeds some limit
Enter checkpoint state:
a)  no new T's start until checkpoint complete
b)  DB processing continues until all existing T's commit and log entries out on disk
c)  current log buffer written to log file, all dirty pages written to DB disk all from committed T’s
d)  when a)-c) complete, special log entry CKPT entered  (these steps are the same as an orderly shutdown of the system ) R1(A,1) W1(A,3) R3(B,4) C1 W3(B,6) CKPT R3(C,7) R4(Z,1) W3(C,11) C3 R5(Y,3) W4(Z,2) C4 W5(Y,4) CRASH

27. Commit consistent cont’d
To recover from a failure or termination:
Rollback?
No-steal approach – never need to UNDO
Rollforward?
start at CKPT, then REDO committed since CKPT
Problems 
But what if some transactions are long-lived or lots of transactions?
must wait a long-time for them to finish, with no new T's active

28. Cache-consistent – Oracle, Informix
2.  Cache-consistent checkpoint - aim to reduce time if long transactions
a)  No new T's permitted to start
b)  existing T's cannot start any new ops
c)  current log buffer written to disk, all dirty data pages to DB disk (even it not committed)
d)  log entry (CKPT, list of active T's) written on log file on disk
R1(A,1) W1(A,3) R4(Z,1) W4(Z,2) C1 R3(B,4) CKPT R3(C,7) R5(Y,3) W3(C,11) C3 W5(Y,4) CRASH

29. Cache-consistent cont’d
To recover:
must rollback past CKPT
Rollback - Starting at last CKPT, keep rolling back until UNDO uncommitted (in memory) in active list in CKPT
Rollforward – Starting at last CKPT, REDO all updates by committed T's

30. Cache-consistent cont’d
Problems:
  Time to flush dirty pages to disk

31. Fuzzy - Informix 3. Fuzzy checkpoint
aim to reduce time to perform a checkpoint
makes use of 2 checkpoint events;  CKPTn-1 and CKPTn
a)  no new T's start - existing T's no new ops
b)  current log buffer written to disk with CKPTn
c)  set of dirty pages in buffer that accumulated since CKPTn-1 is noted
background process makes sure pages written out on data file disk by next checkpoint CKPTn+1
e) Note: prior to CKPTn, remaining pages dirty between CKPTn-2 and CKPTn-1 forced to disk
R1(A,1) CKPTn-1 W1(A,3) R4(Z,1) W4(Z,2) C1 R3(B,4) CKPT R3(C,7) R5(Y,3) W3(C,11) C3 W5(Y,4) CRASH

32. Fuzzy
For our class we decided since we do not know if DB updated yet, always REDO committed and UNDO uncommitted (complicated in reality)
Rollback?
Starting at CKPTn UNDO committed
Continue rollingback to CKPTn-1
Rollforward?
starts with first log entry following 2nd to last checkpoint log        CKPTn   CKPTn     (start at CKPTn-1)
REDO committed transactions starting at CKPTn-1
Rollforward all the way to crash

33. Important observations
Only UNDO
if data written to data disk at CKPT before transaction committed (didn’t commit before checkpoint)
Only REDO
if committed transaction’s data not written to data disk (committed after checkpoint)

34. Log buffer, Log file, Data file and Checkpointing
Log buffer contains list of operations (Start, Commit, Write); it is stored in memory.
Log file is a copy of the log buffer on disk (permanent storage).  
Data file is stored on disk (permanent storage). Updated data pages (dirty pages) are cached in memory. Data file on disk not updated with dirty pages until CKPT.
A CKPT reflects current state of database. It includes data updates from all newly committed transactions, and even updates from uncommitted transactions (if cache or fuzzy checkpoint).

35. Events At CKPT: At a COMMIT: UNDO: REDO:
Updated pages written to data file on disk and log buffer written to log file.  
At a COMMIT:
log buffer also written to log file, but updated data not written to disk.
UNDO:
when data written to data disk and transaction doesn’t commit. 
REDO:
when transaction committed but not written to data disk yet.

36. Rollback to where? Cache consistent: Commit consistent:
To CKPT
Cache consistent:
Past CKPT until no active transactions if cache consistent
Fuzzy consistent:
To CKPTn-1 if fuzzy consistent
Which strategy does UNDO?
cache consistent
Our version of fuzzy.

37. Rollforward
REDO when?
only if Commit has occurred after a CKPT (so dirty data not written to disk yet).
Can only REDO operations entered in log file.
If operation in log buffer, but not written to log file yet, do not REDO, assume lost.

38. After recovery After recovery using CKPTs:
Data file on disk in a consistent state
What is on the data disk?
All committed transactions?
updates recorded on data disk
All updates from transactions not committed?
not recorded on data disk

39. Checkpoints in Oracle
The mechanism of writing modified blocks on disk in Oracle is not synchronized with the commit of the corresponding transactions.
all database changes up to the checkpoint are recorded in the data disk, making it unnecessary to apply redo log entries prior to the checkpoint.
Which consistency?
cache consistent

40. Checkpoints in Oracle Logs
Oracle writes the dirty buffers to disk only on certain conditions:
Every three seconds
When a checkpoint is produced

41. Checkpoints in Oracle
A checkpoint is realized on five types of events:
At each switch of the redo log files
When the delay for LOG_CHECKPOINT_TIMEOUT is reached.
When the size in bytes corresponding to:  (LOG_CHECKPOINT_INTERVAL* size of IO OS blocks) is written on the current redo log file.
Directly by the ALTER SYSTEM SWITCH LOGFILE command.
Directly with the ALTER SYSTEM CHECKPOINT command

42. Checkpoints in Oracle During a checkpoint the following occurs:
The database writer (DBWR) writes all modified database blocks in the buffer cache back to data disk (cache consistent)
Log writer (LGWR) updates both the controlfile and the datafiles to indicate when the last checkpoint occurred

43. Informix Used to checkpoint when:
Administrative events, for example, archives, adding a dbspace,
A 75 percent full physical log
?? One checkpoint in the logical log space
?? The CKPTINTVL configuration parameter in the ONCONFIG file

44. Informix Checkpoints in Informix Now uses non-blocking
Used to use blocking checkpoint
Fuzzy Checkpoint – still blocked when flushed buffer
Now uses some special type of checkpoint