1. Efficient Migration of Large-memory VMs Using Private Virtual Memory
I’m Yuji Muraoka from Kyushu Institute of Technology.
I’m going to talk about Efficient Migration of Large-memory VMs Using Private Virtual Memory.
Yuji Muraoka and Kenichi Kourai
Kyushu Institute of Technology　

2. Large-memory VMs IaaS Clouds are widely used
Provide users with virtual machines (VMs)
Users can build their systems flexibly in VMs
VMs with a large amount of memory are emerging
E.g., VMs with 12 TB of memory in Amazon EC2
VMs with 24 TB of memory are planned in 2019
Used for big data processing
Recently, Infrastructure-as-a-Service clouds are widely used.
They provide users with virtual machines.
Users can build their systems flexibly in VMs.
IaaS clouds also provide VMs with a large amount of memory.
For example, Amazon EC2 provides VMs with 12 TB of memory.
It plans VMs with 24 TB of memory in 2019.
Such large-memory VMs are used for big data processing. VM
12TB VM
4GB

3. VM Migration Move a VM to another host on host maintenance
Transfer all the memory of the VM
Retransfer the memory updated during migration
Require sufficient free memory at the destination host
Not cost-effective to always preserve such a host for a large-memory VM
When a host running a VM is maintained, the execution of the VM can be continued by migrating the VM to another host.
VM migration transfers all the memory of the VM via the network.
If the memory of the VM is updated during the transfer, VM migration retransfers the updated memory data.
After the transfer of the memory, VM migration stops the VM and transfers the VM core such as virtual CPUs and devices.
Then, the VM is restarted at the destination host.
VM migration requires free memory that can accommodate the entire memory of a VM in a destination host.
However, it is not cost-effective to always preserve such hosts as the destination of VM migration for large-memory VMs even in clouds.
source host
destination host
VM core
memory
migration 1 2 3 3 4 5

4. VM Migration Using Virtual Memory
VM migration is possible by using virtual memory
Store part of VM's memory in swap space on secondary storage
Perform paging when necessary
Page-in: data in swap space to physical memory
Page-out: data in physical memory to swap space
Even if a destination host does not have sufficient free memory, VM migration is possible by using virtual memory.
Using virtual memory, VM migration can transparently store part of the memory of a VM in swap space on secondary storage to use a necessary amount of memory.
When free memory runs out during VM migration at the destination host, a page-out is performed.
It moves memory data in physical memory to the swap space to reserve free memory.
In contrast, when the VM requires memory data in the swap space, a page-out is first performed to reserve free memory.
Then, a page-in is performed to move the data in the swap space to physical memory.
source host
VM core
destination host
physical
memory
page-in
migration
swap
space
memory 1 2 3 4 5
page-out

5. Long Migration Time The first memory transfers cause frequent paging
Page-outs always occur after physical memory is full
The following memory retransfers can cause paging
Page-ins are needed if memory data to be updated exists in swap space
Page-outs are needed in exchange for the page-ins
However, traditional virtual memory is incompatible with VM migration.
VM migration first transfers the entire memory data of a VM to physical memory in the destination host.
After the physical memory becomes full, the transfers of the remaining memory data always cause page-outs.
This is because received memory data has to be stored in physical memory at first.
This large number of page-outs lead to the increase in migration time.
Also, the following retransfers of updated memory data can cause paging.
If the memory data to be updated exists in swap space,
VM migration has to first page-in that memory data to physical memory and then update it.
At the same time, page-outs are necessary because physical memory is already full.
source host
VM core
destination host
physical memory
migration
swap space
memory 1 1 2 3 4 5
updated memory

6. Long Downtime & Low VM Performance
Paging in the final phase increases the downtime
The memory of virtualization software is often paged out during memory transfers
Page-ins frequently occurs on resuming virtual devices
Paging also occurs after VM migration
Frequently accessed data is often paged out
Since the final phase of VM migration stops a VM, the downtime of the VM increases if paging occurs frequently.
The main reason is paging of virtualization software at the destination host.
The memory of virtualization software is often paged out during VM migration.
Virtual devices provided by virtualization software are not used until the final phase.
Therefore, the memory for them becomes a target of page-outs.
As a result, when the virtual devices are restored, page-ins occur.
After VM migration, paging also occurs.
Frequently updated memory data is often stored in physical memory, but memory data that are just read frequently is often paged out.
source host
VM core
destination host
physical memory
migration
swap space
memory 3 4 5 1 2 2
memory of virtualization software

7. Split Migration [Suetake et al.’18]
Divide the memory of a VM and transfer them to a main host or sub-hosts
Likely accessed memory to the main host
No paging occurs during VM migration
Perform remote paging between the hosts
Frequent paging does not occur just after VM migration
To counteract these problems, split migration has been proposed using multiple destination hosts.
Split migration divides the memory of a VM and transfers the memory fragments to multiple smaller hosts.
It transfers the VM core and likely accessed memory data to a main host.
The other memory data is directly transferred to sub-hosts.
Since no paging occurs during VM migration, split migration can improve the migration performance.
If the VM requires memory data in sub-hosts after the migration, the data is transferred to the main host via the network by a technique called remote paging.
Since necessary memory data is transferred to the main host at the migration time, frequent paging does not occur just after the migration.
source host
VM core
main host
sub-host
VM core
migration
memory
remote paging 1 2 3 4 5

8. Issues in Split Migration
More costly than VM migration using virtual memory
Require extra sub-hosts and high-speed network
E.g., remote paging with InfiniBand [Gu et al.'17]
Migrated VMs are subject to host and network failures
However, split migration is more costly than VM migration using virtual memory.
It requires at least one extra sub-hosts.
For efficient remote paging, high-speed network such as InfiniBand is also required.
These costs are usually higher than that of secondary storage used as swap space.
In addition, migrated VMs are subject to host and network failures.
If only one host or the network stops, the entire VM cannot continue to run.
sub-host 1
main host
memory
VM core
memory
sub-host 2
memory
remote paging

9. Our Approach：VMemDirect
Achieve efficient migration of large-memory VMs using private virtual memory
Integrate virtual memory per VM with VM migration
Bypass traditional system-wide virtual memory
Transfer VM's memory without paging
Reduce paging just after VM migration
For efficient migration of large-memory VMs, we propose VMemDirect using private virtual memory and the technology of split migration.
VMemDirect prepares virtual memory per VM and integrates it with VM migration.
This enables VM migration to bypass the traditional system-wide virtual memory.
Using private virtual memory, VMemDirect divides the memory data of a VM and directly transfers them to physical memory and swap space.
Like split migration, paging does not occur between them at the migration time.
Since likely accessed memory data is transferred to physical memory, paging can be reduced just after VM migration.
The execution of the migrated VM is not affected by a network failure.
destination host
source host
VM core
memory
private virtual memory
migration
physical memory
swap space

10. Private Virtual Memory
Handle only the memory of one VM
Perform no paging for virtualization software
Manage memory data in the physical memory allocated to a VM and private swap space
Use NVMe SSDs for private swap space
The cost can be lower than using sub-hosts and high- speed network
Private virtual memory handles only the memory data of one VM.
Therefore, it can prevent performance degradation due to paging of the memory of virtualization software itself.
During memory transfers in VM migration, the memory of virtualization software is not paged out at the destination host.
Private virtual memory manages memory data in the physical memory allocated to a VM and its own private swap space.
To reduce the overhead of paging, VMemDirect uses fast NVMe SSDs for private swap space.
Since recent NVMe SSDs are becoming more inexpensive, the cost can be lower than using multiple sub-hosts and expensive high-speed network.
virtualization
software VM
ディスク
paging 1 2 3 4 5 6
memory
physical memory
private swap space

11. Private Swap Space Use a sparse file as a swap file
Make one-to-one mapping between memory pages of a VM and blocks of the swap file
Make it easy to find the corresponding block for a page
Allow empty blocks that have no memory data called holes
Efficiently use disk space
For private swap space, VMemDirect creates a sparse file with the same size as the memory of a VM and uses it as a swap file.
A sparse file is a special file that can contain empty blocks without actual data, which are called holes.
Using this swap file, VMemDirect makes one-to-one mapping between memory pages of the VM and blocks of the swap file, as shown in this figure.
Memory data of the VM is stored in either physical memory or the swap file.
This straight mapping makes it easy to find the corresponding file block for a memory page.
When memory data is stored in physical memory, VMemDirect makes the corresponding block of the swap file a hole to use disk space efficiently.
VM memory
swap file
memory
page
file block
hole

12. Access to Private Swap Space
Access the swap file using direct I/O
Allocate no page cache in memory for the file blocks
Prevent page-outs by memory pressure from the page cache
Access by the chunk larger than the 4-KB page
Use the bandwidth of NVMe SSDs as much as possible
Currently use 256-page chunks
VMemDirect accesses the swap file using direct I/O.
Direct I/O does not allocate the page cache in memory when data is read from and written to storage.
In VMemDirect, most of the physical memory is used to store memory data of a VM.
If the page cache were allocated, other data in physical memory would be paged out by memory pressure from the page cache.
To use the bandwidth of NVMe SSDs as much as possible, VMemDirect accesses the swap file by the chunk larger than the 4-KB page.
In the current implementation, VMemDirect uses 256-page chunks.
chunk
private swap space

13. Direct Memory Transfer
Transfer VM's memory directly to physical memory or private swap space
Determine data locations when starting VM migration
Directly update memory data on memory retransfers
Prevent paging during memory transfers
No page-outs even when physical memory is full
No page-ins when updating data in private swap space
VMemDirect directly transfers VM's memory to either physical memory or private swap space in a destination host, instead of relying on paging of traditional virtual memory.
At the source host, VMemDirect determines locations where each memory data is stored at the destination host when it starts VM migration.
When VMemDirect retransfers memory data, it directly updates data in physical memory or private swap space.
This direct memory transfer can prevent paging from occurring during VM migration.
No page-outs occur even when physical memory is full.
No page-ins occur when updating data in private swap space during VM migration.
source host
VM core
destination host
private
swap space
migration
physical memory
memory 1 2 3 4 5

14. Using Memory Access History
Determine data locations on the basis of the memory access history of a VM
Obtain the history from memory access bits
Assign likely accessed data to physical memory
The other data to private swap space
Reduce updating data in private swap space
The locations of memory data in the destination host are determined on the basis of the memory access history of a VM.
The history is obtained from memory access bits recorded by CPUs.
VMemDirect assigns likely accessed memory data to physical memory and the other data to private swap space.
This increases the probability that retransferred memory data is stored in physical memory.
Unlike VM migration using traditional virtual memory, not only frequently updated memory data but also frequently read data is stored in physical memory.
Therefore, the occurrence of paging can be suppressed after the VM is resumed.
source host
destination host
memoryaccess
history
VM core
migration
private
swap space
physical memory 1 2 3 4 4 5 14

15. Page-in for Private Virtual Memory
Use the userfaultfd mechanism in Linux
Register the memory of a VM to userfaultfd after VM migration
Detect an access to a non-existent memory page
Read memory data from blocks in the swap file
Write the data to the memory pages of the VM
Remove the blocks in the swap file and make them holes
VMemDirect achieves paging for private virtual memory using the userfaultfd mechanism in Linux.
After VM migration, it registers the entire memory of the VM to userfaultfd.
When the VM accesses a non-existent memory page and a page fault occurs, that event is notified to the virtualization software by userfaultfd.
Next, VMemDirect reads memory data from the block corresponding to the faulting address in the swap file.
Then, it writes the data to the corresponding memory page of the VM.
In addition, it performs the same operations for the other pages in the same chunk.
At the same time, it removes these blocks from the swap file and makes the blocks holes. VM
page fault
page in
VM memory
private swap space

16. Page-out for Private Virtual Memory
Select a chunk including the least likely accessed memory pages
Use the memory access history of the VM
Write memory data of the selected chunk to the corresponding blocks in the swap file
Remove the mapping of the pages from the VM
In exchange for page-ins, VMemDirect performs page-outs.
First, it selects a chunk including the least likely accessed memory pages.
Such pages are found by using the memory access history of the VM.
Then, VMemDirect writes data of the selected pages to the corresponding blocks in the swap file.
At the same time, it removes the mapping of the pages from the VM.
page out
VM memory
private swap space

17. Experiments
We examined the performance of VM migration and a migrated VM in VMemDirect
Comparison
Ideal VM migration with sufficient memory
Naive VM migration with traditional virtual memory
Split migration
Adjusted free memory of the destination host to 6 GB
To examine the performance of VM migration and a migrated VM in VMemDirect, we conducted several experiments.
For comparison, we examined the performance for ideal VM migration with sufficient memory in the destination host.
In addition, we executed naive VM migration with traditional virtual memory and split migration with two hosts.
We used 10 Gigabit Ethernet between the source and destination hosts.
We ran a VM with 12 GB of memory and an NVMe SSD as swap space.
We adjusted the size of free memory of the destination host to 6 GB, which was the half of the memory size of the VM.
CPU: Intel Xeon E v3
Memory：16 GB
SSD: Samsung NVMe SSD 970 PRO
Network：10 Gigabit Ethernet
OS: Linux 4.11
Virtualization software：QEMU-KVM 2.4.1
virtual CPU: 1
Memory：12 GB VM
source/
destination host

18. Migration Time Naive migration was 1.6x longer than ideal one
Due to frequent paging
VMemDirect was almost the same as ideal and split migration
Thanks to direct memory transfer without paging
First, we measured the time needed for VM migration.
This figure shows the migration time.
The migration time of naive migration with traditional virtual memory was 1.6 times longer than that of ideal migration with sufficient memory.
This is because frequent paging occurred in naive migration.
For VMemDirect, in contrast, the migration time was almost the same as ideal and split migration.
This is thanks to direct memory transfer that does not cause paging.

19. Downtime Naive migration was 4.2x longer than ideal one
Due to paging of virtualization software
VMemDirect was almost the same as ideal one
Thanks to private virtual memory only for VM's memory
Split migration was 1.2x longer than ideal one
Next, we measured the downtime during VM migration.
The downtime in naive migration was 4.2 times longer than that in ideal migration.
This reason was that many page-ins occurred when virtual devices were resumed in the final phase.
In contrast, the downtime in VMemDirect was almost the same as that in ideal migration because VMemDirect uses private virtual memory only for VM’s memory.
The downtime in split migration was 1.2 times longer than that in ideal migration.

20. VM Performance after Migration (High Localization)
We ran an in-memory database with 5-GB data
VMemDirect was 33x higher than after naive migration
Thanks to migration using memory access history
1.1x higher than after split migration
28% lower than after ideal migration
Due to paging
To examine the performance of private virtual memory, we ran an in-memory database in a VM and measured the VM performance after VM migration.
We first ran a benchmark before VM migration and stored data in the database.
After VM migration, we measured the throughput using the same benchmark.
In this experiment, we allocated 5GB of memory to the database so that the size was less than the size of free memory.
The throughput in VMemDirect was 33 times higher than after naive migration.
This is because the data was kept in physical memory thanks to VM migration using memory access history.
Compared with after split migration, the throughput increased by 1.1 times.
However, that was 28% lower than after ideal migration due to paging.

21. VM Performance after Migration (Low Localization)
We ran an in-memory database with 6-GB data
VMemDirect was 1.7x higher than after naive migration
Thanks to paging by the chunk
1.3x higher than after split migration
Much lower than after ideal migration
Finally, we allocated 6GB of memory to in-memory database so that the size was more than the size of free memory.
The throughput in VMemDirect was still 1.7 times higher than after naive migration thanks to paging by the chunk.
Compared with after split migration, the throughput increased by 1.3 times.
However, that was much lower than after ideal migration due to frequent paging.

22. Related Work Agile live migration [Deshpande et al.'16]
Page out most of VM's memory to networked swap space and transfer only a small amount of memory
Paging overhead is much larger
FlashVM [Saxena et al.'10]
Virtual memory considering characteristics of SSDs
The prefetching technique can be used in VMemDirect
VSwapper [Amit et al.'14]
Improve the performance of VMs using virtual memory
Applicable to VMs after VM migration with VMemDirect
Agile live migration pages out most of VM’s memory to networked swap space and transfers only a small amount of remaining memory.
This method improves migration performance, but the paging overhead is much larger because most of the data is paged out.
FlashVM is virtual memory considering characteristics of SSDs.
It pages out more memory pages at once than when using HDDs.
The prefetching technique can be used in VMemDirect.
VSwapper improves the performance of VMs using virtual memory.
It proposes various optimizations and achieves 10 times performance improvement.
Those optimizations are applicable to VMs after VM migration with VMemDirect.

23. Conclusion
We proposed VMemDirect for achieving efficient migration of large-memory VMs
Cooperate with private virtual memory per VM
Directly transfer VM's memory to physical memory or private swap space
Improved the performance of VM migration and migrated VMs dramatically
Future work
Improve VM performance after migration
More clarify trade-offs between using high-speed network and NVMe SSDs for paging
In conclusion, we proposed VMemDirect for achieving efficient migration of large-memory VMs.
VMemDirect cooperates with private virtual memory per VM instead of traditional system-wide virtual memory.
It directly transfers VM’s memory to physical memory or private swap space to avoid paging.
Compared with VM migration using traditional virtual memory, VMemDirect improved the performance of VM migration and migrated VMs dramatically.
Our future work is to improve VM performance after migration.
We will implement asynchronous paging in VMemDirect.
We are also planning to more clarify trade-offs between using high-speed network and NVMe SSDs for paging.
Thank you for listening.