Optimized Transaction Time Versioning Inside a Database Engine Intern: Feifei Li, Boston University Mentor: David Lomet, MSR

Transaction Time Support  Provide access to prior states of a database:  Auditing the database  Querying the historical data  Mining the pattern of changes to a database  General approach:  Build it outside the database engine  Build it inside the database engine

Overview of A Versioned Database Page Header 0 Dynamic Slot Array Record A Record B 1 A.1 A.0 B.1 B.2 B.0

Key Challenges  Timestamping  Eager timestamping vs. lazy timestamping  Record takes the transaction commit timestamp  Recovery of timestamping information when system crashes  Indexing both current versions and historical versions simultaneously  Storage utilization  Query efficiency

Talk Outline  Even “lazier” timestamping  Deferred-key-split policy in the TSB tree  Auditing the database

Talk Outline  Even “lazier” timestamping  Deferred-key-split policy in the TSB tree  Auditing the database

Lazy Timestamping  When do we timestamp records affected by a transaction?  Maintain a list of updated records and timestamp them when transaction commits  may lead to additional I/Os  Timestamp records when they are accessed by other queries, updates, page reads and writes later on.  Where to get the timestamping information?

Volatile timestamp table (VTT) and Persistent timestamp table (PTT) Transaction 23 begins Insert a record A Transaction commits Insert a record B Disk Main memory TIDTtimeRefcnt ……… ……… 23NA0 VTT Record A Timestamp= TID.23 Record B Timestamp= TID.23 23NA1 23NA TIDTtime …… …… PTT Ensure that we can recover the timestamping information if system crashes (VTT is gone!)

Timestamping the Record Transaction 45 begins Insert a record C Transaction commits Update record A Disk Main memory TIDTtimeRefcnt ……… ……… 23NA0 VTT Record A Timestamp= TID.23 Record B Timestamp= TID.23 23NA1 23NA TIDTtime …… …… PTT NA0 Record C Timestamp= TID.45 Record A Timestamp= Record D Timestamp= TID Update record D Record D Timestamp= NA

The Checkpointing Process Time kth checkpointk-1th checkpointk-2th checkpoint EOLLSN(U)LSN(P) All the log records have been removed from the log and it is impossible to recover information earlier than LSN(P). The dirty pages between LSN(P) and LSN(U) have been all flushed into the disk prior to our current checkpoint The current checkpoint may not finish yet and log records with LSNs between LSN(U) and EOL are not guaranteed to be stable yet. LSN(P)LSN(U) EOL k+1th checkpoint

Garbage Collection

Let’s Be Even More Lazier  Don’t write an entry to PTT when transaction commits  Piggyback timestamping information to the commit log record so that we still can recover if necessary  Batch updates entries from VTT to PTT at the checkpoint  Why this is better?  Batch update using one transaction is faster than write to PTT on a per transaction basis;  A lot of entries have their Refcnt down to zero by the time of checkpointing  less number of writes to PTT

The New Story Transaction 23 begins Insert a record A Transaction commits Disk Main memory TIDTtimeRefcnt ……… ……… 23NA0 VTT Record A Timestamp= TID.23 23NA TIDTtime …… …… PTT Transaction 76 begins Insert a record B Transaction commits Record B Timestamp= TID.76 76NA0 76NA Update A Record A Timestamp= Checkpoint

Be Careful When Updating the VTT and PTT at the Checkpoint

Improvement  Each record is 200 bytes  The database is initialized with 5,000 records  Generate workload containing up to 10,000 transactions  Each transaction is an insert or an update (to a newly inserted record by another transaction)  One checkpoint every 500 transactions  Cost metrics:  Execution time  Number of writes to PTT  Number of batched updates

Execution Time Audit Mode: Always keep everything in PTT

Number of Writes to PTT

Batched Update Analysis

Talk Outline  Even “lazier” timestamping  Deferred-key-split policy in the TSB tree  Auditing the database

Time Split B (TSB) Tree  Indexing both the current version pages and historical version pages simultaneously  Time split:  Create a new page and historical records in the current page is pushed into the new page  Key split:  Proceed as the normal B + tree key split  When to do time split and key split?

What Happens Now Page Header 0 Dynamic Slot Array 1 A.1 A.0 B.2 B.1 B.0 Record C Insert C but page is full Current page 2 What if the current page exceeds the key split threshold? Record C Page Header Dynamic Slot Array 0 1 Current page Page Header 0 Dynamic Slot Array 1 A.1 A.0 B.2 B.1 B.0 Historical page

Why We need a Key Split Threshold?  Wait till the page is full then do the key split:  Leads to too many time splits and hence lots of replicas in the historical versions  What is the best value for the key split threshold?  Too high: overall utilization drops  Too low: current version utilization is reduced  Find a balance

Could We Do Better?  Key split immediately follows the time split  Leads to two pages with utilization 0.5thresh ksplit  If the new pages are not filled up quickly, storage utilization is wasted for no good reason  A fix  Deferring the key split until the next time that the page requires a key split  Simulate as if a key spit has been performed on previous occasion as it is in the current situation

Deferring the Key Split Page Header 0 Dynamic Slot Array 1 A.1 A.0 B.2 B.1 B.0 Record C Insert C but page is full Historical page Current page 2 What if the current page exceeds the key split threshold? Record C Page Header Dynamic Slot Array 0 1 Current page We still insert the record A.0’ B.1’ D 2 Page is full again. Update D Now we key split if last time the page has already satisfied the key split requirement. D.1 D.0 2 Page Header 0 Dynamic Slot Array 1 A.1 A.0 B.2 B.1 B.0 We use the key split value from the last occasion when a key split should has happened. 3

Analytical Result  We can show the following: Where in is the insertion ratio, up is the update ratio and cr is the compression ratio.

The Goal of Our Design  To ensure that for any particular version the version utilization is at least kept above a specified threshold value.

Experiment  50,000 transactions  Each transaction inserts or updates a record  Varying the insert / update ratio in the workload  Each record is 200 bytes  Utilize the delta-compression technique to compress the historical versions (as they share a lot of common bits with newer version)

Single Version Current Utilization (SVCU)

Multi-Version Utilization (MVU)

Talk Outline  Even “lazier” timestamping  Deferred-key-split policy in the TSB tree  Auditing the database

Auditing A Database  Transaction versioning support enables the check of any prior state of a database  Store the user id in PTT for each transaction entry  Any change to the database is traceable  User id is grabbed from the current session that a transaction belongs to

Conclusion  Transaction versioning support inside a database engine is one step closer to be even more practical  Other interesting applications that will become possible now with transaction versioning support?

Thanks!