Understanding and Managing SQL Server's Transaction Log Virendra Yaduvanshi Principal Data Architect

Understanding and Managing SQL Server's Transaction Log

About me Principal Data Architect at Manipal Business Solutions (20 years experience) Data Mining and Analytics from IIT Roorkee Master in Computer Science PMI - PMP MCSE, MCITP, MCTS ITIL PASS Speaker Blog : https://wikidba.net/ E-Mail : virendra.yaduvanshi@manipalgroup.info v_yaduvanshi@hotmail.com

Thanks to our Sponsors and Organizers

Understanding and Managing SQL Server's Transaction Log Agenda Understanding Transaction Log Virtual Log Files (VLFs) Dealing with Excessive Log Growth Transaction Consistency Optimizing Log Throughput Q&A

Why Transaction Log is so Important? The Transaction log is just like a black-box / Unknown to many people – Internal level Its very misunderstood: why it is necessary? Why and how its grow? Reason : Misconfiguration, Deleted, Grown unexpectedly, shrunk The Transaction log is most important part of live database. Its allow transactions durable and recoverable in any event of disaster Database may be corrupted due without logging and transaction would be inconsistent Without logging, how transaction rollback will work Without logging how High Availability – Replication , Log Shipping, Availability group The performance of the transaction is the paramount to many workload.

Transaction Log A transaction log is a file in which SQL Server stores a record of all the transactions performed on the database with which the log file is associated. In the event of a disaster that causes SQL Server to shut down unexpectedly, such as an instance-level failure or a hardware failure, SQL Server uses the transaction log in order to recover the database to a consistent state, with data integrity intact. Upon restart, a database enters a crash recovery process in which the transaction log is read to ensure that all valid, committed data is written to the data files (rolled forward) and the effects of any partial, uncommitted transactions are undone (rolled back). SQL Server also uses the transaction log during normal operation, to identify what it needs to do when a transaction rolls back, due to either an error or a user-specified ROLLBACK statement. In short, the transaction log is the fundamental means by which SQL Server ensures database integrity and the ACID properties of transactions, notably durability. The transaction log, or T-Log, is a record of all transactions happening to the database. All these transactions are actually first written to the log, and then after a checkpoint, is written to the data file via the lazy writer. Uses: Point in time recovery ( full recovery model). Start and end of each transaction. Every data modification(insert, update, delete) including system SP’s DDL statements to any table including system tables. Every extent and page allocation and de-allocation. Creating or dropping tables and indexes.

Transaction Log SQL Server uses a write-ahead log, this guarantees that no data modifications is written to the disk before the associated log record is written to disk. This maintains the ACID properties of a transaction. Modified pages in the buffer stays until a checkpoint or commit tran Associated log records are created in the log cache SQL determines where data page is in the log, and if copy is not made from disk to the cache The T-Log is written to sequentially and in a round-robin fashion. The T-Log is one of the things we overlook to optimize. Yes, put the log on separate SAN or SSD’s in a RAID 1+0 or 0+1 and give it a 100MB growth rate because that is better than 1MB.

Virtual Log Files (VLFs) Every transaction log is composed of smaller segments called virtual log files. Every time a growth event occurs new segments, virtual log files, are created at the end of your transaction log file. The T-Log is written to sequentially and in a round-robin fashion. A T-Log will have 2 VLF’s when created, and depending on the auto grow size, it will create a various amount of VLF’s as follows: < 64MB = 4VLF’s >= 64MB < 1GB = 8 VLF’s > 1GB = 16VLF’s

VLF Size Matters When SQL Server is writing to VLFs there can be two issues.  First, the VLFs are very small and SQL Server is jumping all over the place maintaining small files and figuring out where it should write the next entry – this is common and a big performance hit.  Second, the VLFs are too large and SQL Server is reserving a ton of space when only a small part of the tail end of the VLF is in use, then trying to clear it all at once – this is less common and less of an issue.

VLF Usage We can use below DBCC command to see VLF information. DBCC LOGINFO(DB_NAME) This command is undocumented but is safe to run as it only reads the T-Log. Important columns to look for is the following: FileSize – VLF size in bytes Status – 0 - Not in use, 2 - In use CreateLSN – This can be used to determine how many VLF’s was created in a log file growth, 4, 8 or 16. This also indicates the Log Sequence Number.

Dealing with Excessive Log Growth The most common problems and forms of mismanagement that lead to excessive growth of the transaction log, including: Operating a database in FULL recovery model, without taking log backups Performing index maintenance Long-running or uncommitted transactions that prevent space in the transaction log from being reused. Lack of log space reuse

Investigating heavy log-writing transactions Taking full database backups only protects the contents of the data file, not the log file. The only way to fully protect the data that has changed since the last full or differential backup, which will be required for point-in-time restores, is to perform a log backup. Full database backups do not truncate the transaction log. Only a log backup will cause the log file to be truncated. Without the latter, space in the log file is never marked for reuse, and the log file will constantly grow in size.

Explicit Transactions Defined by the user Automatically abort only in case of fatal error TSQL Begin Tran Commit Tran/Commit Work Rollback Tran/Rollback Work OLEDB ITransactionLocal::StartTransaction ITransaction::Commit/ITransaction::Abort ADO.NET SqlConnection object BeginTransaction Commit()/Rollback() ADO Connection object BeginTrans CommitTrans/Rollbacktrans

Uncommitted transactions There are a number of causes of orphaned transactions in SQL Server, and it's beyond the scope of this chapter to investigate them in full detail. However, some of the most common causes are: Application timeouts caused by a long-running transaction Incorrect error handling in T-SQL or application code Failure during trigger execution Linked server failures resulting in orphaned distributed transactions No corresponding COMMIT/ROLLBACK statement to a BEGIN TRANSACTION command.

Identifying orphaned or long-running transactions using the DMVs

Optimizing Log Throughput Physical Architecture The correct physical hardware and architecture will help ensure that you get the best possible log throughput,. Only need one log file There is no performance gain, in terms of log throughput, from multiple log files. SQL Server does not write log records in parallel to multiple log files. There's one situation when SQL Server will write to all the log files, and that's when it updates the header in each log file. The log header is a mirror of the database header page, and SQL Server writes to it to update various LSNs, such as the last checkpoint, the oldest open transaction, the last log backup, and so on. This sometimes tricks people into thinking that SQL Server is logging to all the log files, when it is only updating the header. Use a dedicated drive/array for the log file There are several reasons why it's a good idea to house the data files on separate disks from the log file. Firstly, this architecture offers better chances of recovery in the event of disk failure. For example, if the array housing the data files suffers catastrophic failure, the log file doesn't go down with the ship. We may still be able to perform a tail log backup, and so recover the database to a point very close the time of failure.

Optimizing Log Throughput Use RAID 10 for log drives, if possible RAID 1 using 2 x 600 GB 15K RPM disks =>185 IOPS at 11.5 MB/sec. RAID 5 using 5 x 146 GB 15K RPM disks =>345 IOPS at 22 MB/sec. RAID 10 using 14 x 73 GB 15K RPM disks => 1609 IOPS at 101 MB/sec. Log Fragmentation and Operations that Read the Log SQL Server internally breaks down a transaction log file into a number of sub-files called virtual log files (VLFs). SQL Server determines the number and size of VLFs to allocate to a log file, upon creation, and then will add a predetermined number of VLFs each time the log grows, based on the size of the auto-growth increment. Monitoring Tools Windows Perfmon Physical Disk\Disk Reads/sec and Physical Disk\Disk Writes/sec Physical Disk\Avg. Disk sec/Read and Physical Disk\Avg. Disk sec/Write Dynamic Management Views and Functions sys.dm_io_virtual_file_stats - provides statistics on utilization of all database data and log files. It is an excellent resource for discovering hot spots, and identifying opportunities to spread I/O over different channels. sys.dm_io_pending_io_requests – provides a list of all I/O operations that SQL Server is currently waiting to complete. The "sys.dm_os_" category of DMVs also provides sys.dm_os_performance_ counters, which exposes the performance counters, and therefore the "queues" in our system Using sys.dm_db_log_space_usage

Transaction Consistency SQL SERVER uses locks to manage data consistency and concurrency. The database manager sets exclusive locks on the modified rows and shared locks or update locks when data is read, according to the isolation level . The locks are held until the end of the transaction. When multiple processes want to access the same data, the latest processes must wait until the first finishes its transaction or the lock timeout occurred. The locking strategy of SQL SERVER is row locking with possible promotion to page or table locking. SQL SERVER dynamically determines the appropriate level at which to place locks for each Transact-SQL statement. Consistency means that all data remains in a consistent state after a transaction finishes, regardless of whether the transaction failed or was completed successfully. Before a transaction begins, the database must be in a consistent state, After a transaction occurs, the database must be in a consistent state—a new state if the transaction succeeded or, if the transaction failed, the same consistent state it was in before the transaction started.

Thanks to our Sponsors and Organizers

Thank for attending. Nepal SQLSaturday#692 feedback: ug@Sqlpassnepal Thank for attending Nepal SQLSaturday#692 feedback: ug@Sqlpassnepal.org