Cloud Operating System Unit 09 Cloud OS Core Technology M. C. Chiang Department of Computer Science and Engineering National Sun Yat-sen University Kaohsiung, Taiwan, ROC Cloud Operating System

Outline  Main Characteristics of Cloud Computing  Why Virtualization?  Hardware Virtualization  Full virtualization  Paravirtualiztion  Hardware-assisted virtualization  Device Driver Virtualization  Memory Virtualization  Local vs. Distributed 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-2

Main Characteristics of Cloud Computing  On-demand self-service  Broad network access  Resource pooling  Rapid elasticity  Measured service 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-3

Main Characteristics of Cloud Computing  On-demand self-service  System demands resource automatically  Broad network access  Resource pooling  Virtualizing resources such as memory, storage, CPU into pools.  Rapid elasticity  Provisioning or releasing resources at anytime  Measured service 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-4

History of Virtualization (1)  1964 – IBM CP-40.  1965 – IBM System/360-67, with virtual memory hardware.  1999 – Full Virtualization. VMware.  2003 – Para-Virtualization. Xen.  2005 – HW-Assisted Virtualization. Intel VT-x, AMD-V.  2006 – QEMU.  2007 – KVM (Kernel-based Virtual Machine). Linux /11/15 Cloud Operating System - Uint 09: Core Technique U01-5

History of Virtualization (2)  From another perspective, the development of virtualization can be divided into the following stages:  Mainframe Virtualization  Desktop Virtualization  Server Virtualization  Cloud Computing  Nowadays.  Mobile Virtualization  On the way. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-6

History of Virtualization (3)  The reasons of x86 virtualization:  Low infrastructure utilization.  Increasing physical infrastructure costs.  Increasing IT management costs.  Insufficient failover and disaster protection.  High maintenance end-user desktop.  x86 is widespread. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-7

Applying Virtualization to Cloud  With virtualizing resources, they can easily be put into pools.  Virtualized CPU and memories can be distributed dynamically to virtual machines in real time.  Leads to Rapid elasticity and On-demand self- service  Imagine ‘hot plug memories/CPU’ into machines. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-8

Hardware Virtualization (1)  How to realize x86 virtualization?  What’s the problem encountered?  How to deal with the problem? 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-9

Hardware Virtualization (2)  How to realize x86 virtualization?  What’s the problem encountered?  How to deal with the problem? 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-10

Hardware Virtualization (2) 2015/11/15 Cloud Operating System - Uint 09: Core Technique  Traditionally, OS is at Ring 0 for hardware resource management.  When realizing virtualization, VMM is at Ring 0, and OS is downgraded to Ring 1. U01-11

Hardware Virtualization (2)  According to Popek and Goldberg, CPU instructions can be classified into three sets.  Privileged instructions.  Sensitive instructions.  Critical instructions. *1: GJ Popek and RP Goldberg, “Formal requirements for virtualizable third generation architectures”, Communication of the ACM, vol 17, pp , July, /11/15 Cloud Operating System - Uint 09: Core Technique U01-12

Hardware Virtualization (3)  How to realize x86 virtualization?  What’s the problem encountered?  How to deal with the problem? 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-13

Hardware Virtualization (3)  Instruction which will modify the configuration of resources should either be executed in Ring 0 (privileged mode) or trap.  There are 17 instructions in x86 architecture may cause problem. Why?  They are unprivileged. Won’t cause trap when execution.  They expose physical processor state. Reveal some details which shouldn’t be. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-14

Hardware Virtualization (4)  How to realize x86 virtualization?  What’s the problem encountered?  How to deal with the problem? 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-15

Hardware Virtualization (4)  There are three approaches for virtualization available nowadays.  Full Virtualization.  ParaVirtualization.  Hardware-Assisted Virtualization. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-16

Full Virtualization  Binary Translation.  Replace nonvirtualizable instructions with new sequences of instructions which have the same effect as original on the virtual hardware.  Many instructions replace one, so if the instruction is frequently used, the overhead is huge.  Huge cost on resources.  Better compatibility.  VMware, QEMU. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-17

ParaVirtualization (1)  Also named OS-assisted virtualization.  OS kernel adaptation.  Performing corresponding hypercalls instead of the original privileged instructions.  Low cost of CPU and I/O.  Compatibility worse than Full Virtualization.  Meet problems when the kernel is not open.  Citrix Xenserver, Microsoft hyper-V. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-18

ParaVirtualization (2)  Hypercall is quite similar to system call.  Implementation of Xen.  System call: an interrupt (0x80) from user space to kernel space.  Hypercall: an interrupt (0x82) from guest domain (Ring 1) to hypervisor (Ring 0).  Both system call table and hypercall table are defined in file “arch/x86/kernel/entry_64.S”. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-19

Hardware-Assisted Virtualization (1)  Re-design the hardware architecture.  Add new instruction sets for x86 architecture to trap those nonvirtualizable instructions.  Conceptually, hypervisor stays at “Ring -1”.  No need to de-privilege the guest OS.  AMD AMD-V, Intel VT-x. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-20

Hardware-Assisted Virtualization (2)  Intel VT-x:  First supported by Pentium 4 (Model 662 and 672) on  Two operating mode: VMX Root Operation (for VMM). Non-Root Operation (for Guest OS).  Extented Page Tables. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-21

Hardware-Assisted Virtualization (3)  AMD AMD-V:  On 2006, first supported by Athlon 64, Athlon 64 X2 and Athlon 64 FX.  Two modes for hypervisor to handle memory partitioning: Shadow Page Tables. Nested Page Tables. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-22

Device Driver Virtualization – the Idea  Virtual machines need their own device drivers.  For guest OSs to communicate with physical hardware.  Most virtualization solutions emulated simple devices that the drivers already existed. Like IDE interface and NE2000 (a very old nic). 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-23

Device Driver Virtualization – in Practice  Xen Hypervisor’s paravirtulization solution:  Requests from front-end drivers in DomU writes into a ring-buffer shared with Dom0.  Dom0 handles the request by the back-end drivers in kernel, and access the hardware by real drivers.  Then back-end write response back and consumed by guest. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-24

Device Driver Virtualization  The Split Driver 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-25

Memory Virtualization  Modern Operating system and x86 hardware mostly support memory virtualization.  In user-space applications access to virtual memory with paging mechanism.  VM instance hosts with a OS support virtual memory means an addition layers of memory mapping.  Newer CPUs have optimizations for addition layers of mapping.  Additional Tagged TLBs in case that when switching VMs the TLB will be flushed. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-26

Memory Virtualization – in Practice  In full virtualization, hypervisor traps every action try to update page table.  At once hypervisor traps the action, hypervisor can take over.  In Xen paravirtualization, para-virtualized OS directly invoke hypercalls to ask hypervisor update the page table.  Much faster. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-27

Distributed Memory Virtualization – the Idea  Pooling the virtualized memory  We can add nodes into our cloud to extend the memory capacity.  Breaking the bound of physical machines  If an VM instance in a node need more memories, it can get memories from other nodes. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-28

Distributed Memory Virtualization – in Practice  Performance issue. Relatively very very high latency compare to local memories.  Latency of Infiniband or 10G Ethernet : 1.07 ~ 2.6 µs.  Latency of DDR3 SDRAM: 10.5 ns.  Page table for distributed environment adds even more overhead.  Therefore, mainstream hypervisors don’t support distributed memory virtualization, for now. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-29

Summary  Some characteristics of cloud computing need virtualization to achieve.  Virtualization technology leads to better utilization and elastic deployment. It also leads to better business model. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-30

Summary  Virtualization Approaches  Full Virtualization : High compatibility, low performance.  Para-Virtualization : Need OS modified for VMM, best performance.  HAV : Need newer hardware, good compatibility, good performance 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-31

Summary  Device Drivers needs virtualization as well.  Rather than emulation a whole device, we can implement a simple front-end in guest, and communicate with back-end device driver.  Memory Virtualization  Performance critical.  Lot’s new designs in CPU in order to optimization memory related operations. 2015/11/15 Cloud Operating System - Uint 09: Core Technique U01-32