1. Virtualization Technology
Tal Maoz
07/05/2009

2. Outline Background Hardware Virtualization Virtual Machines
The XEN Virtual Machine
Virtual Machine Migration

3. Virtual: Webster dictionary defines “virtual” as:
being such in essence or effect though not formally recognized or admitted
of, relating to, or using virtual memory
of, relating to, or being a hypothetical particle whose existence is inferred from indirect evidence

4. Background 1 - IBM
1963 – IBM builds M44/44X to research time sharing technologies at MIT. Each VM (44X) is an image of the main machine (M44)
1964 – IBM announces System 360. Each VM is a copy of the underlying hardware, running it’s own OS. A hardware component called VMM (Virtual Machine Monitor) managed the VM’s directly on the real hardware.

5. Background 1 - IBM

6. Background 2 – Software
At the end of the 1960’s, the VMM took the form of a software abstraction layer
Each VM was similar enough to the underlying hardware to run unmodified software
Ever growing mainframes gave users the ability to multiplex applications but the cost grew as well
For a while, it was good

7. Background 3 - Downhill
The 1980’s and 1990’s brought modern multitasking operating systems and strong, faster minicomputers and PC’s
Large mainframe VMM’s faded away and gave way to small cheap computers
By the late 1980’s neither academics nor industry viewed VMM’s as much more than a historical curiosity
In the 1990’s, Stanford researchers looked at VM’s as a way to create cheaper MPPs – Massively Parallel Processing machines

8. Background 4 - Revival
Modern stronger machines gave the ability to run more than a few applications at a time
VM’s harness the full power of modern CPU’s
During the 1990’s VM’s were built for almost every architecture – Emulators were born
The biggest step was the emulation or virtualization of the PC’s themselves – VMware Inc. started supplying VMMs for commodity platforms
The VMM got a new name: Hypervisor
VMM – Running over an existing OS
Hypervisor – Running on the bare-metal

10. XEN running 3 XenoLinux VM’s and a NetBSD machine

11. VMWare running 3 virtual machines

12. Background – What Now?
Sun’s Sparc and IBM’s G4 have long had support for virtualization technology
As the market for VMM’s grew, hardware manufacturers (Intel, AMD …) decided to support VMM’s at the hardware level
By late 2005, Intel announced VT (Virtual Technology), codenamed Vanderpool (VT-x & VT-I / VMX)
AMD announces the superior Pacifica (AMD-V / SVM), incompatible with Intel’s VT, spelling trouble for developers…
HP currently offers it’s HP_UX partitioning technology on Itanium servers
Sun introduces the xVM technology based on Xen for x86 and on “Logical Domains” for Sparc

13. Hardware Virtualization
Started with IBM’s System 360 followed by the famous VM/370 – Full Hardware Virtualization
Over the years, other architectures such as Alpha, MIPS, PA-RISC and SPARC introduced virtualization friendly features such as software managed TLB’s (Translation Lookaside Buffer)
The IA-32 and x86 are less friendly but give 4 privilege “rings”, of which only 2 are in use by current OS’s.
Virtualization on x86 can be made efficient by reducing the OS’s privilege mode from ring 0 to ring 1 and running a software VMM in ring 0

14. Intel’s VT (Vanderpool)
First, there were 4 privilege rings, 0 through 3
On the move from 32 bit to 64 bit, Intel decided to drop the extra privilege rings and only keep the 2 in use
To solve the problem one has to change the host OS so it only uses ring 3, same as user-level programs.
VT introduces a new mode: VMX. Hypervisor runs in VMX root mode, OS and programs run in VMX mode 3 …
VMM
VMX Mode 1 2 3 3
Normal Mode

15. Intel’s Vanderpool – Contd.
Special commands facilitate the creation of new OS’s (VMPTLRD, VMCLEAR) and switching of OS’s (VMLAUNCH, VMRESUME)
A new VMCS (Virtual Machine Control Structure) stores info on the running VMs, including event flags that tell the CPU when to return control to the hypervisor (like a PCB for VMs)

16. AMD’s Pacifica
Pacifica does everything VT does but adds 2 key features: MMU modes & a DEV
AMD’s MMU (Memory Management Unit) has been moved into the CPU. The MMU manages the Virtual Memory at the hardware level
AMD moved much of the MMU’s job from hardware to software when running VM’s. The MMU now gives the VMM the ability to trap VM’s memory calls via the CR3 register, and manage an SPT (Shadow Page Table) for each VM. This provides each guest OS with it’s own little world. Effective but costly!
AMD introduces the NPT (Nested Page Tables). The CR3 register is now fully virtualized for each VM by the MMU and each machine’s SPT is controlled by the MMU at the hardware level, under the hypervisor’s supervision. Hardware works much faster!

17. AMD’s Pacifica – Contd.
Paged Real Mode – an MMU mode for running guest OS’s that use Real Mode (no virtual memory)
TTLB (Tagged Transition Lookaside Buffer) – ID’s are added to TLB addresses so the MMU knows to which entries belong to each VM and can save costly hypervisor flushing work. In effect you have several multiplexed TLB’s, one per VM.

18. AMD’s Pacifica – Contd.
DEV (Device Exclusion Vector) is a safety mechanism used when several guest OS’s want to use DMA (Direct Memory Access)
DMA allows devices to write directly to the memory with no CPU intervention, This can cause many problems when several VM’s want to use the same device – the device can end up writing to the wrong OS’s page table!
The DEV tells each device if it is allowed to access a page of memory or not, thus it can dynamically bind devices to VM’s
DEV also helps prevent VM collisions when using HyperTransport (Device-CPU bus) and Cache Coherent HyperTransport (CPU-CPU bus)
A new instruction called SKINIT, which is intended for running secure VM’s that are totally invisible to other VM’s

19. Intel – The Next Generation
The Core-i7 microarchitecture moved the MMU from the chipset into the CPU
Next, Intel plans to insert a GPU into the same die
Hyper-Threading is back…
QuickPath replaces the FSB (Front Side Bus)
Support for non-ECC DDR3 only! (up to tri-channel)

20. Trends Microsoft included the WoW technology in it’s Windows OS’s
Apple offers integrated virtualization technology to lure Windows users over to OSX (Parallels Desktop)
VMware offers “Fusion” for Mac and “Unity” for PC’s – Run graphic applications from various OS’s simultaneously and transparently!
Windows 7 will include transparent XP virtualization on hardware-virtualization enabled CPUs
Cloud computing farms (Amazon, Yahoo, …) offer virtual clusters on-demand

21. Virtual Machines – Why?
Home users who want several OS’s with ease (Windows, Linux, OSX, etc.)
Developers who need to develop for several OS’s and want a dedicated development machine
Security – one can run a non-secure machine for browsing and games, a more secure machine for work and a very secure machine for sensitive mail and information

22. Virtual Machines – Why?
Server farms can use virtual machines to make administration easier
Each client gets a dedicated VM where his application can run with it’s own configuration and environment with no compatibility issues
VM’s can be migrated from server to server and thus applications can continue running uninterrupted while servers go down for maintenance
VM’s allow for much more powerful and efficient Grid economy algorithms and management systems

23. Virtual Machines - Brands
Hardware
OS Virtualization:
Bochs
FAUmachine
Microsoft Virtual PC
Open CoLinux
Parallels
QEMU
SVISTA
TRANGO
PL’s
twoOStwo
UML (User-Mode Linux)
Virtuozzo
VM from IBM (S360 OS)
Denali
VMWare
XEN
MOSIX Virtual OS

24. Open Source Projects
Xen
VirtualBox
QEMU
KVM
Vserver
OpenVZ
UML …

25. Virtualization Approaches
Full Virtualization – All the system aspects are virtualized, often while the guest OS’s code is being dynamically rewritten to allow for the VMM’s control (e.g. older VMware ESX Server) – works nicely with x86, allows hosting of unmodified systems but not very efficient and requires shadowing of various system structures
Paravirtualization – The hypervisor runs in ring 0 of an x86 architecture while the guest OS is modified to run in ring 1 and use hypercalls so that interrupts are processed through the hypervisor – works with x86, very efficient but requires modified guest OS’s

26. The Xen Hypervisor
Developed by the University of Cambridge and supported, amongst others, by Microsoft
The Xen virtual machine runs on the x86 architecture and uses Paravirtualization
Xen is designed to run up to 100 VM’s simultaneously and efficiently
Each VM is called a “Domain”, Control and management software are run on a VM called Domain0 running XenoLinux
The Xen project has already ported the Linux OS (XenoLinux) and NetBSD. XenoXP (Windows XP port) was still under work but was abandoned due to the large XP code-base and to Microsoft legal limitations. XP can run unmidified as an HVM (Hardware VM)
OpenSUSE, Fedora Core, Debian and NetBSD all include support (standard kernels and utility packages) for various versions of Xen

27. Xen 3.0 Architecture AGP ACPI PCI SMP VT-x 32/64bit
Domain0
VM1
VM2
VM3
Device
Manager &
Control s/w
Unmodified
User
Software
Unmodified
User
Software
Unmodified
User
Software
GuestOS
(XenLinux)
GuestOS
(XenLinux)
GuestOS
(XenLinux)
Unmodified
GuestOS
(WinXP))
AGP
ACPI
PCI
Back-End
Back-End
SMP
Native
Device
Driver
Native
Device
Driver
Front-End
Device Drivers
Front-End
Device Drivers
VT-x
Control IF
Safe HW IF
Event Channel
Virtual CPU
Virtual MMU
32/64bit
Xen Virtual Machine Monitor
Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE)

28. VMware 5.5 Architecture

29. Paravirtualization in Xen
The Architecture is called xen_x86, similar to x86 but requires Xen Hypercalls for privileged operations
Guest OS’s kernel is modified to understand the virtualized environment:
Wall-Clock time vs. virtual processor time
Expose real resource availability to enable the OS to optimize it’s behavior

30. CPU Virtualization
Xen runs in ring 0 (most privileged) while the guest OS runs in rings 1 or 2. Guest apps run in ring 3
Page Faults are generated if the guest OS tries to use privileged instructions
TLB Flushed for every VM context switch
All exceptions are caught by Xen and forwarded to handlers registered with Xen by the guest OS’s
Guest OS’s can register ‘fast’ exception handlers, verified by Xen only when installed and do not have to be indirected via ring 0
Scheduling is done using the BVT (Borrowed Virtual Time) algorithm so to insure fair-sharing

31. Device I/O
Xen does NOT emulate hardware devices (e.g. VMware). Xen uses a set of device abstractions
I/O data in transferred via Xen using shared-memory asynchronous buffer-rings, Xen performs validation checks
Xen uses an event mechanism instead of interrupts
The notifications can be held by the guest OS to avoid frequent wake-up notifications

32. Memory Xen does NOT keep shadow page tables! (e.g. VMware)
Guest OS’s use the real page table where updates go through Xen for verification
Guest OS’s can send several updates together for optimizations
Physical memory is statically partitioned for each VM at the time of creation
XenoLinux’s Balloon Driver adjusts a VM’s memory usage by passing pages back and forth between Xen and the page allocator.

33. Network
Xen provides a virtual firewall-router (VFR) and guest OS’s can register virtual network interfaces (VIF’s)
Domain0 can manipulate the VFR rules
I/O rings are used to buffer packet transmission and reception
To send packets the guest OS enqueues a buffer descriptor onto the transmit ring. Xen verifies VFR rules and pinnes the relevant page frames
Packet reception is done by exchanging unused page frames for each packet

34. Disk Only Domain0 has direct unchecked access to physical disks
VM’s see virtual block devices (VBDs), created and configured from domain0
Xen reorders requests from VBDs for optimization – VBDs look like SCSI devices, guest OS’s can pass down reorder barriers to prevent reordering if needed.
Xen performs permission checks on VBD requests and uses Zcopy using DMA for optimization
Requests a serviced in batches and in a simple round-robin fashion

35. Performance Relative performance of native Linux (L),
XenoLinux (X), VMware 3.2 (V) and User-Mode Linux (U)

36. Xen 3.0 – The goodies!
I/O Spaces – Drivers run in dedicated domains, can enforce security and stability
Event channels are used to buffer interrupts
New hardware features (Pacifica’s DEV and similar in Silvervale) allow for very efficient security and exclusion of devices, can eliminate paravirtualization and thus run unmodified OS’s
Using device channels to pass requests and responses, Xen let’s many VM’s use the same driver instance
Xen 3.0 supports SMP 32/64bit hosts and guests, PCI, ACPI and AGP/DRM devices, supports VT-x and Pacifica
PPC and IA64 port are now in progress

37. Xen – Use the source, Luke!
The Xen source is a BIG (~150KLOC) patch to the Linux kernel
It compiles 3 kernels, combined into a single binary by default:
The Xen Hypervizor
The Dom0 kernel
A DomU kernel
The source packages of the standard SUSE, Debian, Ubuntu, Fedora Code and RedHat kernels include the Xen patch
The standard Xen distribution is built around the kernel (the standard kernel for SUSE Enterprise, RedHat Enterprise etc.)

38. VM Migration – Why?
Avoid many difficulties faced by process-level migration (such as the need for the home node)
In-memory state can be transferred consistently
A group of mutual processes can be moved together
Server farms operator can let the users take care of the VM’s while they take care of the hardware
VM Migration can help with cluster load balancing with greater ease

39. VM Migration - Overview
Daneli migrates VM’s by stopping, copying and resuming
VMware has developed it’s VMotion OS migration technology
Zap migrates process domains via partial OS virtualization
NomadBIOS is a virtualization and migration system built on top of the L4 microkernel, using pre-copy
MOSIX migrates processes but keeps them connected to the home node’s deputy process

40. Migrating Memory Can be divided to 3 phases:
Push phase – Pages are pushed to the destination machine while the VM keeps running, changed pages must be resent
Stop-and-copy-phase – The source VM stops, pages are copied to the destination VM, new VM is started
Pull phase – The new VM pulls non-copied pages when they are needed

41. VM Migration – Xen’s way

42. VM Migration - Network
A migrating VM retains it’s protocol states (e.g. TCP PCB)
Within a subnet, the new host will issue an unsolicited ARP reply advertising the VM’s new location
Alternatively, switches can sense that the VM’s MAC has moved to a new port and correct themselves
For WAN’s, Xen suggests an indirection layer such as Mobile IP
Xen does not handle local storage migration and relies on a NAS (network-attached storage)
For block devices, Xen suggests mirroring

43. VM Migration – Optimizations
Xen uses a Dynamic Rate-Limiting algorithm to apply a smart bandwidth usage
The algorithm compensates for large WWS’s (Writable Working Sets) by dynamically increasing the used bandwidth with each iterative pre-copy to a certain size threshold
Once the threshold has been achieved, Xen moves to the Stop-and-copy phase

46. Summary
One can say that the industry is re-embracing the idea of virtualization
Virtualization can solve many problems for home users, developers and businesses
Virtualization can help with security issues
VM migration can solve process migration difficulties
VM migration helps with cluster load balancing and maintenance