1. Xen and the Art of Virtualization
Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt & Andrew Warfield
Presented by Anthony So
November,
CS533 - Concepts of Operating Systems
Fall 2013

2. Presentation Overview
Introduction
Xen approach
Overview
Implementation
Evaluation
Summary
CS533 - Concepts of Operating Systems
Fall 2013

3. CS533 - Concepts of Operating Systems
Introduction
CS533 - Concepts of Operating Systems
Fall 2013

4. CS533 - Concepts of Operating Systems
Monolithic kernel
User Apps
Non-Privilege
File System
Virtual
Memory
IPC
Scheduler
Device
Driver
Privilege
Hardware (CPU, Physical Memory, Storage, I/O, … etc)
CS533 - Concepts of Operating Systems
Fall 2013

5. CS533 - Concepts of Operating Systems
Virtualization
VM/Domain
File System
Virtual
Memory
IPC
Scheduler
Device
Driver
User Apps
Non-Privilege
VMM (Virtual CPU, Virtual Physical Memory,
Virtual Network, Virtual Block Device … etc)
Privilege
Hardware (CPU, Physical Memory, Storage, I/O, … etc)
CS533 - Concepts of Operating Systems
Fall 2013

6. Non, Full, and Para-Virtualization
User Apps
User Apps
User Apps
Non-Privilege OS
Modified OS OS
VMM
VMM
Privilege
Hardware
Hardware
Hardware
CS533 - Concepts of Operating Systems
Fall 2013

7. CS533 - Concepts of Operating Systems
Xen - Overview
CS533 - Concepts of Operating Systems
Fall 2013

8. Xen Architecture Overview
CS533 - Concepts of Operating Systems
Fall 2013

9. CS533 - Concepts of Operating Systems
Control Transfer
Synchronous calls from a domain to Xen may be made using a hypercall
Notification are delivered to domains from Xen using an asynchronous event mechanism
Domain
Synchronous
Hypercall
Asynchronous
Event
VMM
CS533 - Concepts of Operating Systems
Fall 2013

10. CS533 - Concepts of Operating Systems
Xen – Implementation
CS533 - Concepts of Operating Systems
Fall 2013

11. CPU – Privilege Instruction
How x86 architecture handles privileged instructions?
Non
Full
Para
User Apps
User Apps
User Apps
Non-Privilege OS
Modified OS OS
VMM
VMM
Privilege
Hardware
Hardware
Hardware
CS533 - Concepts of Operating Systems
Fall 2013

12. CS533 - Concepts of Operating Systems
Memory Management
Tagged TLB vs No Tagged TLB
Tagged TLB is ideal for virtualization because each TLB entry associated with an address-space identifier to allows hypervisor and guest OS entries to coexist even with context switch, thus, avoid complete TLB flush.
x86 – No Tagged TLB and must flush after a context switch.
Xen exists in a 64MB section a the top of every address space, thus avoiding a TLB flush when entering and leaving the hypervisor.
CS533 - Concepts of Operating Systems
Fall 2013

13. CS533 - Concepts of Operating Systems
Memory Management
S/W managed vs H/W managed TLB
x86 uses H/W managed TLB. Therefore, TLB management and handling TLB faults are done entirely by the MMU hardware.
S/W managed TLB is ideal for virtualization because TLB misses are serviced by the OS.
CS533 - Concepts of Operating Systems
Fall 2013

14. CS533 - Concepts of Operating Systems
Memory Management
Xen register guest OS page tables directly with the MMU but restricted guest OS to read-only access.
Page Table updates are passed to Xen via hypercall.
Request are validated before being applied.
Type: writable, page table … etc.
Reference count: Must be 0 to switch task type.
To minimize hypercall, guest OS locally queue updates before applying an entire batch with a single hypercall.
CS533 - Concepts of Operating Systems
Fall 2013

15. CS533 - Concepts of Operating Systems
Memory Management
Shadow Page Table.
VMM lookup
the real
address
in memory
and update
the table
VMM lookup
the real
address
in memory
and update
the table
Shadow
Page
Table
Machine To
Virtual
Guest
Page
Table
Physical To
Virtual
Pmap
Machine To
Physical
Guest OS
wants to
update page
table
Guest OS
wants to
update page
table
CS533 - Concepts of Operating Systems
Fall 2013

16. CS533 - Concepts of Operating Systems
Memory Management
Xen
Page
Table
Machine To
Virtual
Guest OS
has direct read
access to page
table
Read
Guest OS
want to
update
page table
Hypercall
VMM do the update on behave
of guest OS
Write
CS533 - Concepts of Operating Systems
Fall 2013

17. CS533 - Concepts of Operating Systems
Memory Management
[3]
Balloon Driver is a mechanism to adjust a domain’s memory usage.
CS533 - Concepts of Operating Systems
Fall 2013

18. Exception / System Calls / Interrupt
Exception: A table describing the handler for each type of exception is registered with Xen for validation. The handler are identical to real x86 hardware (except page faults).
System Calls: Xen allows each guest OS to register & install a fast handler to enable direct calls from user apps into its guest OS and avoid routing through Xen on every calls.
Interrupt: Hardware interrupts are replaced with a lightweight event system.
CS533 - Concepts of Operating Systems
Fall 2013

19. CS533 - Concepts of Operating Systems
Time and Timers
Xen provides guest OS the following notion of time:
Real Time:
Time that is maintained continuously since machine boot.
Virtual Time:
Time that a particular domain has executed. It will not advance if the domain is not executing.
Wall-Clock Time:
Current Real Time + an offset.
CS533 - Concepts of Operating Systems
Fall 2013

20. CS533 - Concepts of Operating Systems
I/O Ring
An asynchronous I/O rings is used for data transfer between Xen and guest OS. (Circular queue)
Guest OS
Xen
CS533 - Concepts of Operating Systems
Fall 2013

21. CS533 - Concepts of Operating Systems
Network
Xen provides the following abstraction:
Virtual firewall-router (VFR)
Virtual network interfaces (VIF) – Like a modem network interface card
Two I/O rings: transmit and receive.
Round-Robin packet scheduler.
Page flipping: require guest OS to exchange an unused page frame for each packet it receives to avoid copying between Xen and the guest OS (but require page-alignment).
CS533 - Concepts of Operating Systems
Fall 2013

22. CS533 - Concepts of Operating Systems
Disk
Domain0 has unchecked access to physical disks.
All other domains access persistent storage through Virtual block device (VBD).
Domain0 manages VBDs.
Ownership and access control information are accessed via the I/O ring.
Round-round scheduler.
Batching of requests for better access performance.
CS533 - Concepts of Operating Systems
Fall 2013

23. CS533 - Concepts of Operating Systems
Xen - Evaluation
CS533 - Concepts of Operating Systems
Fall 2013

24. CS533 - Concepts of Operating Systems
Hardware
Dell 2650 dual processor 2.4GHz Xeon server
2GB RAM
Broadcom Tigon 3 Gigabit Ethernet NIC
Hitachi DK32EJ 146GB 10k RPM SCSI disk
Linux version
RedHat 7.2
CS533 - Concepts of Operating Systems
Fall 2013

25. Virtualization Comparison
Native Linux
Compiled for i686
XenoLinux
Compiled for Xeno-i686 for Xen
VMware Workstation
User-mode Linux (UML)
Compiled for um for UML
CS533 - Concepts of Operating Systems
Fall 2013

26. CS533 - Concepts of Operating Systems
Relative Performance
Computation Intensive:
Processor & memory
w/ minimal I/O or O/S
Database:
Sync. Disk operation
Web server:
File Server:
Compiling kernel:
I/O, scheduler, memory management
CS533 - Concepts of Operating Systems
Fall 2013

27. CS533 - Concepts of Operating Systems
Concurrent
Higher overhead from single domain is due to lack of support to SMP guest OS
CS533 - Concepts of Operating Systems
Fall 2013

28. CS533 - Concepts of Operating Systems
Conclusion
Xen is a paravirtualization
Xen exposes an hypercall interface to Guest OS. Guest OS use it to communicate with Xen to do privileged instructions.
As a result, Xen can not use unmodified guest OS.
Performance is comparable to native Linux.
CS533 - Concepts of Operating Systems
Fall 2013

29. CS533 - Concepts of Operating Systems
Learn More
The Xen Project at
CS533 - Concepts of Operating Systems
Fall 2013

30. Type-1 vs Type-2 Hypervisor
Guest OS Apps
Guest OS Apps
Guest OS Apps
Guest OS Apps
Guest OS
Guest OS
Guest OS
Guest OS
Type-2 Hypervisor
Host OS Apps
Type-1 Hypervisor
Host OS
Hardware
CS533 - Concepts of Operating Systems
Fall 2013

31. CS533 - Concepts of Operating Systems
References
Edouard Bugnion , Scott Devine , Mendel Rosenblum, Disco: running commodity operating systems on scalable multiprocessors, Proceedings of the sixteenth ACM symposium on Operating systems principles, p , October 05-08, 1997, Saint Malo, France
Carl A. Waldspurger, Introduction to Virtual Machines, VMware Labs
Carl A. Waldspurger, Memory resource management in VMware ESX server, Proceedings of the 5th symposium on Operating systems design and implementation, December 09-11, 2002, Boston, Massachusetts
Kenneth J. Duda , David R. Cheriton, Borrowed-virtual-time (BVT) scheduling: supporting latency-sensitive threads in a general-purpose scheduler, Proceedings of the seventeenth ACM symposium on Operating systems principles, p , December 12-15, 1999, Charleston, South Carolina, United States
Fiuczynski , M. E. (2009). Virtual Machine Monitor. Retrieved from Princeton University website:
Gelas, J. D. (2008, March 17). AnandTech | Hardware Virtualization: the Nuts and Bolts. Retrieved November 11, 2013, from
Kurth, L. (2013). 10 Years of Xen and beyond. Linux Foundation Collaborative Projects. (n.d.). Retrieved November 11, 2013 from
CS533 - Concepts of Operating Systems
Fall 2013

32. CS533 - Concepts of Operating Systems
References
Memory management unit - Wikipedia, the free encyclopedia. (n.d.). Retrieved November 11, 2013, from
Microkernel - Wikipedia, the free encyclopedia. (n.d.). Retrieved November 11, 2013, from
Paul Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Rolf Neugebauer , Ian Pratt , Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
Smith, J.E.; Nair, R., "The architecture of virtual machines," Computer , vol.38, no.5, pp.32,38, May 2005
Tanenbaum, A. (2008). Modern operating systems. Upper Saddle River, N.J: Pearson Prentice Hall.
Timeline of virtualization development - Wikipedia, the free encyclopedia. (n.d.). Retrieved November 11, 2013, from
Understanding Full Virtualization, Paravirtualization, and Hardware Assist. (2007). Retrieved from VMware website:
x86 virtualization - Wikipedia, the free encyclopedia. (n.d.). Retrieved November 11, 2013, from
CS533 - Concepts of Operating Systems
Fall 2013