1. © 2009 IBM Corporation March 1, 2009 Log File Management in DB2 for Linux, UNIX, and Windows Ron Castelletto IBM Canada Lab castelle@ca.ibm.com

2. Information Management – DB2 © 2009 IBM Corporation 2 Agenda  How does DB2 ensures no data loss on crash ?  Log file management  Log file archiving  Removing single points of failure

3. Information Management – DB2 © 2009 IBM Corporation 3 Database Fundamentals  The ACID properties –Atomicity – all actions in the TXN happen or none happen –Consistency – if each TXN is consistent, and the DB starts as consistent, the DB ends up consistent –Isolation – execution of one TXN is isolated from that of other TXNs –Durability – if a TXN commit, its effects persists  The Recovery Manager ensure Atomicity and Durability

4. Information Management – DB2 © 2009 IBM Corporation 4 Atomicity and Durability  Atomicity –Transactions may abort (rollback)  Durability –What if DB2 stops running (eg, power failure) ?  After system crashes –T1, T2, T3 should be durable –T4 and T5 should abort

5. Information Management – DB2 © 2009 IBM Corporation 5 Properties of Database Transactions  Concurrency control is in effect –Share (read) and exclusive (write) locking  Updates happen in place –Data is overwritten on disk  A simple solution for atomicity and durability ? –Use a “force” and “no steal” buffer pool

6. Information Management – DB2 © 2009 IBM Corporation 6 The Buffer Pool  “Force” write of data pages to disk at commit –Provides durability –But poor performance  “No stealing” of pages from uncommited Txns –However, “no steal” implies poor performance –But, if 'steal' allowed, how to ensure atomicity ? No Steal Steal Force No Force

7. Information Management – DB2 © 2009 IBM Corporation 7 Forcing and Stealing Pages  No Force (why Durability is hard) –Reminder: “Force” is writing all modified pages for this TXN at commit time –What is system crashes before all pages written ? –Need to know what changes were lost before I/O completed so can redo them.  Steal (why enforcing Atomicity is hard) –Reminder: “Steal” is allowing a new TXN to steal a slot in the buffer pool, ie: write a modified page to disk before the TXN commits –What is the TXN aborts ? Need to know what modification were written to disk so can undo the change.  Solution: write log records to support redo/undo of changes

8. Information Management – DB2 © 2009 IBM Corporation 8 Logging  DB2 records all modifications for every update in log records –Log records are written sequentially to log files(s)‏ –Should be placed on a different disk –Log record should contain as little possible (old/new data)‏ –Multiples log records fit on a single log page –Log records represent an ordered list of update TXNs

9. Information Management – DB2 © 2009 IBM Corporation 9 Write Ahead Logging  Write Ahead Logging Protocol –1) Log records for an update must be written to disk, before the corresponding data page is written –2) write all log records for a TXN when it commits  1) guarantees Atomicity  2) guarantees Durability

10. Information Management – DB2 © 2009 IBM Corporation 10 300 old new Buffer pools 200 new 300 old 300 new 100 101 201 old new 201 new301 old Table space Containers Read Only Committed Update Uncommitted Update Page Cleaners I/O Servers Logger DB2 Agent S0000020.log S0000021.log Active Log Directory Min Buff LSN = Oldest Changed Page in Buffer pool Low Tran LSN = Oldest Uncommited Log Record Min Buff LSN Low Tran LSN 201 new 301 new Bufferpool and Log buffer – all log records flushed SQLOGCTL.LFH.1 SQLOGCTL.LFH.2 Log Control File Log Buffer 301 old new 200 old new 300 old new 200 old new commit 301 old new 300 old new 201 old new commit

11. Information Management – DB2 © 2009 IBM Corporation 11 300 old new Buffer pools 200 old 200 new 300 old 300 new 100 101 201 old301 old Table space Containers Read Only Committed Update Uncommitted Update Page Cleaners I/O Servers Logger DB2 Agent S0000020.log S0000021.log Active Log Directory Min Buff LSN = Oldest Changed Page in Buffer pool Low Tran LSN = Oldest Uncommited Log Record Min Buff LSN Low Tran LSN 201 new 301 new Bufferpool and Log buffer – some logrecs flushed SQLOGCTL.LFH.1 SQLOGCTL.LFH.2 Log Control File Log Buffer 301 old new 200 old new 300 old new 200 old new commit 301 old new 300 old new 201 old new commit

12. Information Management – DB2 © 2009 IBM Corporation 12 300 old new Buffer pools 200 new 300 new 100 101 201 new301 new Table space Containers Read Only Committed Update Uncommitted Update Page Cleaners I/O Servers Logger DB2 Agent S0000020.log S0000021.log Active Log Directory Min Buff LSN = Oldest Changed Page in Buffer pool Low Tran LSN = Oldest Uncommited Log Record Min Buff LSN Low Tran LSN 201 new 301 new ERROR - commit not flushed SQLOGCTL.LFH.1 SQLOGCTL.LFH.2 Log Control File Log Buffer 301 old new 200 old new 300 old new 200 old new commit 301 old new 300 old new 201 old new

13. Information Management – DB2 © 2009 IBM Corporation 13 300 old new Buffer pools 200 new 300 new 100 101 201 new301 new Table space Containers Read Only Committed Update Uncommitted Update Page Cleaners I/O Servers Logger DB2 Agent S0000020.log S0000021.log Active Log Directory Min Buff LSN = Oldest Changed Page in Buffer pool Low Tran LSN = Oldest Uncommited Log Record Min Buff LSN Low Tran LSN 201 new 301 new ERROR – page flushed before log record (row 201)‏ SQLOGCTL.LFH.1 SQLOGCTL.LFH.2 Log Control File Log Buffer 301 old new 200 old new 300 old new 200 old new commit 301 old new 300 old new 201 old new commit

14. Information Management – DB2 © 2009 IBM Corporation 14 S0000020.log S0000021.log S0000023.log Active log directory Other Transactions (committed)‏ 1000 S0000025.log S0000022.log 1001 S0000024.log Log Space Reservation 1001 1000 Transaction 1000

15. Information Management – DB2 © 2009 IBM Corporation 15 S0000020.log S0000021.log S0000023.log Active log directory Other Transactions (committed)‏ 1000 S0000024.log S0000022.log 1001 S0000024.log Out Of Log Space 1001 Transaction 1000 and 1001 (reserved)‏ 1000 Reserved Transaction 1000 Reserved

16. Information Management – DB2 © 2009 IBM Corporation 16 S0000020.log S0000021.log S0000023.log Active log directory Transaction 1000 12 MB Limit Other Transactions 1000 S0000024.log S0000022.log 1000 S0000024.log max_log * (logfilsiz * 4096 *logprimary) / 100 50 * (1000 * 4096 * 6 ) / 100 = 12 MB -964 sqlcode Database CFG logfilsiz 1000 logprimary 6 max_log 50 MAX_LOG database configuration parameter

17. Information Management – DB2 © 2009 IBM Corporation 17 NUM_LOG_SPAN database cfg parameter S0000020.log S0000021.log Active log directory Transaction 1000 Other Transactions 1000 S0000023.log S0000022.log 1000 num_log_span = 2 db2 Force application -964 SQLCODE Database CFG logfilsiz 1000 logprimary 4 num_log_span 2

18. Information Management – DB2 © 2009 IBM Corporation 18 Archived logs Database Configuration Archived logs LOG 100 Active log directory Active Logs LOGSECOND = -1 LOGPRIMARY= 6 LOGARCHMETH1=DISK:/db/archive OVERFLOWLOGPATH = /db/tp1/ologs FIRST ACTIVE LOG = S0000100.LOG Log CTL File Low Tran LSN Log Manager LOG 101LOG 102 LOG 103 LOG 104 LOG 116 LOG 118 LOG 119 LOG 117 LOG 120 LOG 98 LOG 99 LOG 107 LOG 108LOG 109 LOG 110 LOG 106 LOG 105 LOG 111 LOG 112LOG 113LOG 114 LOG 115 LOG 121 LOG 122LOG 123LOG 124 LOG 126 LOG 125 Infinite active logs - LOGSECOND = -1

19. Information Management – DB2 © 2009 IBM Corporation 19 Controlling Minbuff - SOFTMAX  Specifies the percentage of the logfilsiz when a soft checkpoint will: –Write the log control file to disk (Sqlogctl.lfh)‏ –Call an asynchronous page cleaner (Log space page cleaners)‏  Range: 1 to 100 * number of primary logs.  Default = 100 (soft checkpoint in every logfile).  Lower value will reduce the time required to restart a database during crash recovery.  The smaller the number, the greater overhead of normal database logging activity due to page clean activity.

20. Information Management – DB2 © 2009 IBM Corporation 20 Tablespace Containers Buffer Pools S0000020.log S0000021.log S0000022.log Min Buff LSN 1006 1000 1007 1001 1004 10081010 1005 1001 1009 Current LOGFILSIZ = 1000 SOFTMAX = 200 200% * ( 1000 * 4K ) = 8MB 1001 1003 1004 1005 100610071008 10091010 1003 Active log directory Log CTL File Page Cleaner 1002 xxxx Change NOT Written to Disk xxxx Updates to Log CTL Change Written to Disk Database crash recovery - Softmax = 200

21. Information Management – DB2 © 2009 IBM Corporation 21 Table space Containers Buffer Pools S0000020.logS0000021.logS0000022.log Min Buff LSN 1006 1000 1007 1004 1008 1010 1005 1001 1009 Current LOGFILSIZ = 1000 SOFTMAX = 50 50% * ( 1000 * 4K ) = 2MB 1001 1003 1004 1005 100610071008 10091010 1003 Active log directory Log CTL File Page Cleaner 1002 1008 xxxx Change NOT Written to Disk xxxx Updates to Log CTL Change Written to Disk Database crash recovery - Softmax = 50

22. Information Management – DB2 © 2009 IBM Corporation 22 S0000020.log S0000021.log S0000022.log Active log directory transaction 1000 transaction 1001 transaction 1002 transaction 1003 1000 1001 10021003 1 23 DB2 Log Manager 4 ActivityAction Taken Point 1Log S0000020 fullLog Manager archives Log 20. Transaction 1000 is oldest current transaction (lowtran lsn)‏. Cannot rename log 20. Point 3 Transaction 1000 Commits Point 2Transaction 1000 commits Log 20 no longer an active log, can be renamed to Log 23. Point 4Transaction 1002 commitsLog S0000021 can be renamed to Log 24 1003 DB2 log archive processing 1002 Log S0000021 fullLog Manager archives Log 21. Transaction 1002 is oldest current transaction (lowtran lsn)‏. Cannot rename log 21.

23. Information Management – DB2 © 2009 IBM Corporation 23 S0000020.log S0000021.log S0000022.log Active log directory transaction 1000 transaction 1001 transaction 1002 transaction 1003 1000 1001 10021003 1 23 DB2 Log Manager 4 ActivityAction Taken Point 1Log S0000020 fullLog Manager begins to Archive Log 20 but FAILS. Transaction 1000 is oldest current transaction (lowtran lsn)‏ Point 2Log S0000021 fullLog Manager does not Archive log 21 until 20 is done. Point 3Transaction 1000 CommitsLog 20 not active log and not archived. Txn 1003 is lowtran lsn. Log 21 is first active log. Log 23 is created Point 4Active log dir contains 4 logsLog 20 is not active, and unarchived. Logs 21-23 are active logs. 1003 DB2 log archive processing - Backlog

24. Information Management – DB2 © 2009 IBM Corporation 24 What if Archiving does not work for a long time?  Eventually, the filesystem becomes full....  DB2 cannot pre-allocate log files to maintain LOGPRIMARY...  Later, run out of disk space provided by pre- allocated log files.  This scenario is 'out of disk space', distinct from 'out of log space'  Default behaviour, database comes down with severe error  How to avoid this ? 2 solutions...

25. Information Management – DB2 © 2009 IBM Corporation 25 DB2 Database Log Buffer Active Log Files Active log path Inactive Unarchived Logs Offline Archive Log Files DB2 Log Manager DISK FULL! If NO (default)‏ Database comes down with severe error (out of disk space)‏ If YES Update transactions wait for new log Read only transactions continue Write message to db2diag.log Wait 5 minutes and retry creating new log Check for completed log archive and rename log..... BLK_LOG_DSK_FUL = YES..... CFG Log disk full condition handling

26. Information Management – DB2 © 2009 IBM Corporation 26 S0000020.log S0000021.log S0000022.log Active log directory transaction 1000 transaction 1001 transaction 1002 transaction 1003 1000 1001 10021003 1 23 DB2 Log Manager 4 ActivityAction Taken Point 1Log S0000020 fullLog Manager begins to archive Log 20 but FAILS. Transaction 1000 is oldest current transaction (lowtran lsn)‏ Point 2Log S0000021 fullLog Manager does not archive log 21 until 20 is done. Point 3Transaction 1000 CommitsLog 20 no longer an active log, but is not archived. Txn 1003 is lowtran lsn. Log 21 is first active log. Log 20 is moved to FAILPATH. Log 23 created. Point 4Archiving works again.Log 20 is archived from FAILPATH, Log 21 is archived from active log dir. 1003 DB2 log archive processing - FAILARCHPATH FAILARCH directory S0000020.log 1000 1001

27. Information Management – DB2 © 2009 IBM Corporation 27 Log Buffer DB2 Database Primary Disk Operating System Mirroring or Hardware RAID Mirror Disk DB2 Logs Mirror Copy No protection from accidental deletion of active logs. Recovery from Log media failure: –Operating System facilities - Software RAID and Disk Mirrors 1.RAID Hardware: RAID 1 - Mirroring RAID 5 - Parity Based Protecting the DB2 logs

28. Information Management – DB2 © 2009 IBM Corporation 28 Log Buffer DB2 Database DB2 Active Logs copy 1 Protection from accidental deletion of active logs. CFG..... LOGPATH = /database/tp1/logs MIRRORLOGPATH = /database/tp1/logmir..... /database/tp1/logs /database/tp1/logmir DB2 Active Logs copy 2 Protecting the DB2 Active logs - Log mirroring

29. Information Management – DB2 © 2009 IBM Corporation 29 db2 update db cfg for salesdb using logarchmeth1 disk:/dbarchlcl db2 update db cfg for salesdb using logarchmeth2 disk:/dbarchdr salesdb Database Table Spaces Database Directory Active Logs db2 Log Manager DB CFG Remote Archive logs Local Archive logs /dbarchdr/INST1/SALESDB/NODE0000 /dbarchlcl/INST1/SALESDB/NODE0000 SYS2 SYS1 Recovery History Path Includes: –Instance Name –Database Name –Database Partition (NODExxxx)‏ Archiving logs to Local and Remote disks

30. © 2009 IBM Corporation March 1, 2009 Log File Management in DB2 for Linux, UNIX, and Windows THE END Ron Castelletto IBM Canada Lab castelle@ca.ibm.com