1. Failure-Atomic Slotted Paging for Persistent Memory
ASPLOS’17,
Failure-Atomic Slotted Paging for Persistent Memory
Jihye Seo, Wook-Hee Kim, Woongki Baek, Beomseok Nam, Sam H. Noh
UNIST (Ulsan National Institute of Science and Technology)

2. Contents Introduction Slotted Page Structure in DBMS
Leveraging Persistent Memory in DBMS
Slotted Page Structure in DBMS
Failure-Atomic Slotted Paging
Failure-atomic in-place commit scheme
Failure-atomic slot-header logging scheme
Evaluation
Conclusion

3. Persistent Memory (PM)
Persistent memory is expected to replace both DRAM & NAND
NAND
STT-MRAM
PCM
DRAM
Non-volatility o x
Read (ns)
2.5 X 104
5 - 30
20 – 70 10
Write (ns)
2 X 105
Byte-addressable
Density
185.8 Gbit/cm2
0.36 Gbit/cm2
13.5 Gbit/cm2
9.1 Gbit/cm2
K. Suzuki and S. Swanson. “A Survey of Trends in Non-Volatile Memory Technologies: ”, IMW 2015
Non-volatile
High performance
Persistent Memory

4. How can DBMS benefit from persistent memory?
Traditional Database Management System
Rely on redundant write operations
Copies to volatile memory first and then to persistent secondary storage
Query
Volatile (DRAM) Buffer Cache
Block Device Storage
update(EAST) N O R T H N O R T H
DB File
WAL File
Volatile Memory

5. How can DBMS benefit from persistent memory?
Traditional Database Management System
Rely on redundant write operations
Copies to volatile memory first and then to persistent secondary storage
Query
Volatile (DRAM) Buffer Cache
Block Device Storage
update(EAST) N O R T H E O R T H E A R T H E A S T H E A S T N O R T H
DB File
WAL File

6. How can DBMS benefit from persistent memory?
Traditional Database Management System
Rely on redundant write operations
Copies to volatile memory first and then to persistent secondary storage
Query
Volatile (DRAM) Buffer Cache
Block Device Storage
update(EAST)
commit E A S T E A S T N O R T H
DB File
WAL File

7. How can DBMS benefit from persistent memory?
Traditional Database Management System
Rely on redundant write operations
Copies to volatile memory first and then to persistent secondary storage
Query
Volatile (DRAM) Buffer Cache
Block Device Storage
update(EAST)
commit
checkpointing E A S T N O R T H
DB File E A S T
Redundant Copies required in legacy DBMS
WAL File

8. How can DBMS benefit from persistent memory?
Considering PM as main memory
Do we need a Log file when DB buffer cache is non-volatile?
Query
Persistent (PM) Buffer Cache
Block Device Storage
update(EAST)
commit
Goal
Let’s keep a single copy and eliminate any redundant write operations E A S T N O R T H
DB File E A S T
WAL File

9. How can DBMS benefit from persistent memory?
Challenge
How to guarantee ACID with persistent buffer caching?
E.g.) Updates must be invisible until the transaction commits
Query
Persistent (PM) Buffer Cache
Block Device Storage
update(EAST) N O R T H N O R T H
DB File E A R T H
If system crash is caused,
committed data is lost!

10. Contents Introduction Slotted Page Structure
The most widely used database page format for variable-length records
Failure-Atomic Slotted Paging
Failure-atomic in-place commit scheme
Failure-atomic slot-header logging scheme
Evaluation
Conclusion

11. Slotted Page Structure
Slot Header
# of records
record offset array for the ordering of keys
Record Content Area
record contents (Append only)
Free Space
Slot Header
Metadata
Free space
Free space
1024
Number of Records

12. Slotted Page Structure
Slot Header
# of records
record offset array for the ordering of keys
Record Content Area
record contents (Append only)
Free Space
Sorted Keys 30
Slot Header
Slot Header
Metadata
Metadata
Record Offset Array
Record Content Area 1
Free space
1000
Free space
Free space
Key = 30
1000
1024
Number of Records

13. Slotted Page Structure
Slot Header
# of records
record offset array for the ordering of keys
Record Content Area
record contents (Append only)
Free Space
Sorted Keys 30 50
Slot Header
Slot Header
Metadata
Record Offset Array
Record Offset Array
Record Content Area
Record Content Area 1 2
1000
900
Free space
Free space
Key = 50
Key = 30
900
1000
1024
Number of Records

14. Slotted Page Structure
Slot Header
# of records
record offset array for the ordering of keys
Record Content Area
record contents (Append only)
Free Space 40
Sorted Keys 30 50
Slot Header
Slot Header
Metadata
Record Offset Array
Record Offset Array
Record Content Area
Record Content Area 1 2 3
1000
1000
1000
900
900
800
Free space
Free space
Free space
Free space
Key = 40
Key = 50
Key = 30
800
900
1000
1024
Number of Records

15. Contents Introduction Slotted Page Structure
Failure-Atomic Slotted Paging
Failure-atomic in-place commit scheme
Failure-atomic slot-header logging scheme
Evaluation
Conclusion

16. Failure-atomic In-place Commit Scheme
Single write operation guarantees the failure-atomicity for DB
When a record (key = 20) is inserted
Slot Header
900
1000
Free space
Record Content Area
Key = 10
Key = 30 2
invisible
Free space
Key = 20
800
① Writing the record into the record content area
Since the slot header is not updated,
the dirty record is invisible.

17. Failure-atomic In-place Commit Scheme
Single write operation guarantees the failure-atomicity for DB
When a record (key = 20) is inserted
Slot Header
900
Free space
Record Content Area
Key = 10
Key = 30 3
1000
visible
800
Free space
Free space
Key = 20
800
② Writing to the slot header
Slot header is used as a commit mark
PM is expected to guarantee failure-atomic writes of 8-bytes.
How do we atomically update the slot header?

18. Failure-atomic In-place Commit Scheme
Hardware Transactional Memory (HTM)
We guarantee failure-atomic cache line write operation using HTM
Persistent Memory
Dirty Record of Slotted Page
Slot Header
DB Buffer Cache
Write combining store buffer (64B)
in Private L1 Cache
Insert
XBEGIN
Slot header
XEND 3
CLWBs/
MFENCE
CLWB/
MFENCE

19. What if multiple pages are modified?
A DB transaction may modify multiple records in multiple tables
With HTM, we guarantee the failure-atomic write operations for only a single cache line
When a transaction updates multiple pages, in-place commit scheme is not enough for inserting data atomically
Bank account
Transaction
Seller_account += 100;
Buyer_account -= 100;
Sales_item_in_stop -= 1;
Sales item

20. Failure-atomic Slot-header Logging Scheme
Example: Move data with key 20 from page A to page B
Page A
Slot Header
900
800
Record Content Area
Key = 10
Key = 30
Key = 20
Free space
1000 3
Page B
Slot Header
1000
900
Record Content Area
Key = 50
Key = 40
Free space 2

21. Failure-atomic Slot-header Logging Scheme
Example: Move data with key 20 from page A to page B
Page A
Slot Header
900
800
Record Content Area
Key = 10
Key = 30
Key = 20
Free space
1000 3
Page B
① Writing the record
Slot Header
1000
900
Record Content Area
Key = 50
Key = 40
Free space 2
Key = 20
invisible

22. Failure-atomic Slot-header Logging Scheme
Example: Move data with key 20 from page A to page B
Page A
② Updating the slot header A
Slot Header
900
1000
Record Content Area
Key = 10
Key = 30
Key = 20
Free space 2
invisible
Page B
Slot Header
1000
900
Record Content Area
Key = 50
Key = 40
Free space 2
Key = 20
invisible

23. Failure-atomic Slot-header Logging Scheme
Example: Move data with key 20 from page A to page B
Page A
Slot Header
900
1000
Record Content Area
Key = 10
Key = 30
Key = 20
Free space 2
invisible
Page B
③ Updating the slot header B
Slot Header
800
900
Record Content Area
Key = 50
Key = 40
Free space 3
1000
Key = 20
visible

24. Failure-atomic Slot-header Logging Scheme
Same example in system failure
Page A
Slot Header
900
800
Record Content Area
Key = 10
Key = 30
Key = 20
Free space
1000 3
Page B
Slot Header
1000
900
Record Content Area
Key = 50
Key = 40
Free space 2

25. Failure-atomic Slot-header Logging Scheme
Same example in system failure
Page A
Slot Header
900
800
Record Content Area
Key = 10
Key = 30
Key = 20
Free space
1000 3
Page B
① Writing the record
Slot Header
1000
900
Record Content Area
Key = 50
Key = 40
Free space 2
Key = 20
invisible

26. Failure-atomic Slot-header Logging Scheme
Same example in system failure
Page A
② Updating the slot header A
Slot Header
900
1000
Record Content Area
Key = 10
Key = 30
Key = 20
Free space 2
invisible
If system crash is caused in the second step,
data whose key is 20 is lost!
Page B
Slot Header
1000
900
Record Content Area
Key = 50
Key = 40
Free space 2
Key = 20
invisible

27. Failure-atomic Slot-header Logging Scheme
We propose small slot-header logging w/o duplicating the records
Persistent (PM) Buffer Cache A 3 20 10 30 B 2 20 50 40
dirty record
Slot Header Log A 2 B 3
commit

28. Failure-atomic Slot-header Logging Scheme
We propose small slot-header logging w/o duplicating the records
Persistent (PM) Buffer Cache A 3 20 10 30 10 30 B 2 20 50 40 20 50 40
dirty record
Slot Header Log A 2 B 3
commit

29. Failure-atomic Slot-header Logging Scheme
We propose small slot-header logging w/o duplicating the records
How slot-header logging works
CLWB/
MFENCE
Persistent Memory
Dirty Slotted
Page A
Dirty Slotted
Page B
Slot Header
of Page A
Slot Header
of Page B
DB Buffer Cache
Slot-Header Log A B 2 3
commit
Insert()
logHeader()
logHeader()
commit()
checkpoint()
Recovery
Ignore dirty records
Ignore slot-header logs
Do checkpointing

30. Contents Introduction Slotted Page Structure
Failure-Atomic Slotted Paging
Failure-atomic in-place commit scheme
Failure-atomic slot-header logging scheme
Evaluation
Conclusion

31. Experimental Environment
Testbed
Intel Xeon Haswell-EX E v3 (2.2GHz)
256 GB DDR3 Memory
We implemented Failure-Atomic Slotted Paging in SQLite 3.8
FASH : Failure-Atomic Slot-Header logging
FAST : Failure-Atomic Slot-header with in-place commiT
with HTM(RTM)
We compared FASH and FAST with NVWAL*
NVWAL proposed for use in hybrid memory (DRAM+PM)
We used software persistent memory emulator - Quartz for read latency
For write latency, we injected additional delay after clflush instruction
* NVWAL: Exploiting NVRAM in Write-Ahead-Logging. ASPLOS, 2016.

32. Experimental Environment
Differences between FAST, FASH and NVWAL
NVWAL
FASH
FAST
Single page update
Differential logging
Slot-header logging
In-place commit
Multiple page update
Buffer cache
In DRAM
In PM
Log
DRAM
Persistent Storage
Hybrid memory architecture
PM-only architecture
Persistent Memory
DB File
WAL File
Volatile Buffer Cache
Persistent Buffer Cache VS

33. Breakdown of Time Spent for B-tree Insertion
2.1x 2.6x
Both failure-atomic slotted paging schemes outperform NVWAL
NVWAL incurs considerable commit overhead due to multiple copies

34. Breakdown of Time Spent for B-tree Insertion
1.2x
Both failure-atomic slotted paging schemes outperform NVWAL
NVWAL incurs considerable commit overhead due to multiple copies
FAST is faster than FASH
FAST doesn’t have logging overhead if overflow or underflow does not happen

35. Insertion Performance
FAST and FASH consistently outperform NVWAL
FAST and FASH do not duplicate write operations for records
NVWAL generates large log frames for large records
FASH calls more clflush instructions for small record sizes
The reason is that with smaller records, the slotted-page can hold more records
FAST calls about 3 clflush instructions when the record is smaller than 64 bytes
The slot-header size of FAST must be less than 64bytes.

36. Insertion Performance
FAST and FASH consistently outperform NVWAL
FAST and FASH do not duplicate write operations for records
NVWAL generates large log frames for large records
FASH calls more clflush instructions for small record sizes
The reason is that with smaller records, the slotted-page can hold more records
FAST calls about 3 clflush instructions when the record is smaller than 64 bytes
The slot-header size of FAST must be less than 64bytes

37. Transaction Throughput
31%
15%
33%
Real world effect – mobibench
Compared to NVWAL, the transaction throughput of FASH and FAST are 31% and 33% higher, respectively
These results show database transactions are less sensitive to PM latency

38. Contents Introduction Slotted Page Structure
Failure-Atomic Slotted Paging
Failure-atomic in-place commit scheme
Failure-atomic slot-header logging scheme
Evaluation
Conclusion

39. Conclusion We proposed a novel “failure-atomic slotted paging” for PM
In-place commit scheme
Minimizes redundant write operations
Slot-header logging scheme
Eliminates unnecessary redundant copies
Failure-atomic slotted paging outperforms NVWAL
In-place commit scheme shows optimal performance
Only 3 cache line flushes for database transactions that insert a single record ( < 64B )
Slot-header logging reduces the logging overhead
At least 1/4 compared to NVWAL
PM-only memory systems can perform faster than hybrid memory systems that consist of both PM and DRAM

40. Data Intensive Computing Lab
Thank You
Data Intensive Computing Lab