0. Evaluating a Trade-Off between DRAM and Persistent Memory for Persistent-Data Placement on Hybrid Main Memory
Satoshi Imamura, Mitsuru Sato, Eiji Yoshida
Fujitsu Laboratories Ltd.
Min-Move 2017

1. Persistent Memory (PM)
DRAM
SSD
Higher capacity
Non-volatile
Byte addressable
Much higher access speed
Higher write endurance
Lower access speed (esp. write)
Lower write endurance
Lower capacity

2. Future Computer Systems
Processor
Hybrid main memory
Persistent
memory
DRAM
Used to reduce #writes to PM
Storage
Volatile
Used to store a huge amount of data
Non-volatile

3. Data Placement Optimization on Hybrid Main Memory
Write-heavy data is placed on DRAM to reduce #writes to PM [Mogul+, 2009] [Ramos+, 2011] [Li+, 2012] [Lee+, 2014]
Performance and write endurance can be improved
Viewpoints: the differences of write speed and endurance b/w DRAM and PM
Persistent
memory
DRAM
Write frequency
Data

4. Short Summary
Objective: to maximize performance by optimizing data placement on hybrid main memory
Target: applications which guarantee data persistence with logging
E.g., in-memory databases
Record logs of data updates on DRAM to non-volatile devices
Contribution: evaluation of a new trade-off b/w DRAM and PM
We focus on the difference of volatility

5. New Trade-Off between DRAM and Persistent Memory
Processor
Storage
Persistent
memory
DRAM
Faster accesses
Slower accesses
Logging overhead
Data
Data
No logging
Volatile
Log
Non-volatile
Our finding: it is better for performance to place write-heavy data on PM rather than DRAM!

6. Case Study: In-Memory Database (Tarantool)

7. Tarantool Open-source NoSQL in-memory database
Processes transactions on a single thread
Consistency and isolation
Applies write-ahead logging (WAL)
Atomicity and durability

8. Write-Ahead Logging (WAL)
Traditional logging mechanism for databases
2. Update
Log entry
(A, B) A B
DRAM
1. Read
4. Write
3. Commit
Log entry
(A, B)
Storage B A

9. Data Management on Persistent Memory
Three requirements:
Data persistence → Cache flushing instructions (e.g., CLWB)
Write order → Memory fence instructions (e.g., sfence)
Transaction atomicity → Snapshot to roll-back data updates on a transaction abort
We modify Tarantool with libpmemobj library (
Details in our paper
Instructions in this library incur performance overhead

10. Evaluation

11. PM emulation region with DAX support in Linux
PM Emulation Platform
Intel Xeon
Channel 0-1
Channel 2-3
PM emulation region with DAX support in Linux
384 GB
DDR3
64 GB
DDR3
PM emulation parameters*:
PM latency: 4x of DRAM latency
PM bandwidth: 1/4 of DRAM bandwidth
* Both reads and writes are affected
1 TB HDD
The 384 GB region is used as PM emulation region.
The accesses to this region is provided by a kernel module, which exposes the region as a block device.
This block device is formatted with an ext4 file system and mounted with the direct access support enabled.
It allows direct load/store accesses at byte granularity to the contents of the files.

12. Micro-Benchmark Inserts 1 M records to Tarantool database at first
Each record contains a key and ten 100 B fields
Executes 1 M operations to randomly selected records
Read or write operation is randomly selected for each operation
Throughput (ops/sec) is measured during 1 M operations
Write ratio is changed from 0% to 100%

13. Evaluation Results
Logging overhead
Larger overhead

14. Evaluation Results Placing data on DRAM with logging is better
PM overhead
Logging overhead
Larger overhead
Placing data on DRAM with logging is better
Placing data on PM without logging is better

15. Evaluation Results Overhead of log creation PM overhead
Logging overhead
Larger overhead

16. DRAM w/ PM-based logging
Discussion
Commit
Transaction
Log entry
creation
Write(2)
(I/O
wait)
Storage
cache flush
DRAM w/ WAL
Time
Can hide slower
writes to PM
Commit
*Write pending queue on memory controller (asynchronously flushed to PM by a power-fail safe domain)
Snapshot w/
libpmemobj
Transaction
Cache
flush
WPQ* flush
PM w/o WAL
Time
Can save PM capacity
for log entries
Commit
Transaction
Log entry
creation
Cache
flush
WPQ* flush
DRAM w/ PM-based logging
Time

17. Conclusions
Future computer systems will apply hybrid main memory with storages
Target: applications which guarantee data persistence with logging
Trade-off b/w faster DRAM with logging and slower PM w/o logging
It is better for performance to place write-heavy data on PM rather than DRAM!
Data movement for logging is reduced
Future work:
We will implement and evaluate PM-based logging
We will devise a dynamic technique to optimize data placement

19. Database on Hybrid Main Memory [Oukid+, 2014-2015]
Stores primary data on PM
No need to copy data from storages
Logging is unnecessary
Instant recovery
Persistent
memory
DRAM
Stores index data either on DRAM or PM
Trade-off b/w performance and recovery time
Data on DRAM can be accessed faster
Data on PM can be recovered instantly
Index data
Primary data
Our work
Assumes platforms which apply hybrid main memory with storages
Primary data can be placed on either DRAM w/ logging or PM w/o logging

20. Reduction in Logging Overhead
Command logging [Malviya+, 2014]
Records only executed transactions instead of all data updates
Much lower overhead than traditional logging
Longer recovery time
PM-based logging [Fang+, 2011] [Gao+, 2015]
Records logs to PM instead of storages
Eliminates storage I/O times
Needs to create log entries and write them to PM
Our work
Eliminates logging itself by placing data on PM

21. Data Placement Optimization on Hybrid Main Memory
Place write-heavy data on DRAM to reduce #writes to PM [Mogul+, 2009] [Ramos+, 2011] [Li+, 2012] [Lee+, 2014]
Performance and write endurance can be improved
Viewpoints: differences of access speed and write endurance b/w DRAM and PM
Persistent
memory
DRAM
Write frequency
Data
Our work
Focuses on difference of volatility b/w DRAM and PM
Shows that placing write-heavy data on DRAM causes a large logging overhead

22. Data Management on Persistent Memory
Three requirements:
Data persistence → Cache flushing instructions (e.g., CLWB)
Write order → Memory fence instructions (e.g., sfence)
Transaction atomicity → Roll-back functionality on a transaction abort
We modify Tarantool with libpmemobj library (
TX_BEGIN (mempool) { …
pmemobj_tx_add_range_direct(addr, size);
// Data updates in the memory region
} TX_ONCOMMIT {
// pmemobj_persist() is called automatically
} TX_END
Takes a snapshot of the memory region → Automatic roll-back on a transaction abort
CLWB and sfence are called internally

23. Methodology We compare four types of executions to evaluate Label
Logging overhead
Overhead of placing data on PM (PM overhead)
Longer PM latency and additional instructions in libpmemobj library
Label
Data placement
Logging
DRAM w/o WAL
DRAM -
DRAM w/ WAL-HDD
WAL to HDD
DRAM w/ WAL-nowrite*
WAL w/o writing logs PM
*DRAM w/ WAL-nowrite assumes PM-based logging but does not write log entries to PM

24. Various PM latencies
Better
38%↑
25%↑