0. CertiKOS Implementation Progress
Liang Gu
Yale University

1. Content Overview Virtual Disk Port to Intel platform
Reduce virtual device code
Virtual device at user mode
Pass through device with IOMMU
Demo

2. Progress after Boston PI meeting
Virtualization
Virtual devices: PCI, Virtual Disk ( virtio ,virtio block)
Support Multiple VM guests
Move virtual devices to execute in user mode
IOMMU based device pass-through to guest OS
Port CertiKOS to work on bare metal
PCI and AHCI
Port to Intel platform with VT-x
Green- done
Blue- almost done
Red- ongoing

3. CertiKOS Architecture
APP
(uncertified)
APP
(certified)
Mgmt Shell
Mgmt OS
(Linux)
Commodity OS
Application
Master
Master Syscall
Slave
Slave Syscall
Process Management
Virtual Machine Management
Memory
V-Interrupt
V-Devices
Hypercall
Context
IPC
CertiKOS
Bottom is the commodity hardware.
Maybe we need to mention CertiKOS requires CPU provides hardware virtualization support, i.e. AMD SVM in latest CertiKOS.
The medium layer is CertiKOS kernel.
Hardware abstraction layer (HAL) provides some device drivers, e.g. PCI adaptor, hard disk drive, ethernet adaptor, etc. It’s almost empty in latest CertiKOS.
The layer upon HAL provides basic kernel primitives.
SMP management, memory management, virtual memory management, interrupt handling and spinlock are similar to those in other operating systems.
The extension is the virtualization part, which is composed of SVM primitives, virtual devices and virtualization abstraction.
SVM primitives provide functions to control the underlying AMD SVM CPU.
Virtual Devices provide necessary device emulation and passthrough, e.g. CertiKOS emulates interrupt controller, programmable timer, keyboard controller, etc, blocks PCI adaptor and devices, and passes through other devices.
Virtualization abstraction provides an abstract interface to other parts of CertiKOS kernel, so that they don’t need to take care which the underlying hardware virtualization is and if the access to devices from the guest should be emulated or just passed through.
CertiKOS provides two kinds of runnable objects: process and virtual machine.
Process is similar to those in other operating systems except that CertiKOS uses a much simple process model: each physical process core runs at most one process.
Context contains the information used in process schedule, trap handling, etc.
IPC provides message-passing style communication mechanism between processes.
Virtual machine management is to
Create a virtualization environment for the guest, which contains virtualizing CPU, memory, interrupt mechanism and a minimal set of devices.
Control the execution of the virtual machine, e.g. start, stop, resume, shutdown. (functions in bold have not been done yet)
Handle the hypercals. The details will be shown in the following slides.
CertiKOS provides two kinds of system calls:
Master system calls are for the management shell.
Slave system calls are for the certikos-based applications.
We can run four kinds of applications upon CertiKOS.
Management shell, which launches and controls other applications.
We plan to move it to the guest and implement it as a userspace application of the guest OS. In this way the management shell can leverage the libraries in the guest OS to provide a better user interface, and use hypercalls provided by CertiKOS to complete the management work.
Virtual machine, or guest, in which runs a commodity operating system and its applications. No modifications are required for both the commodity operating system and the applications, except when they want to use hypercalls provided by CertiKOS. Except the hypercall, CertiKOS is transparent for them.
Uncertified CertiKOS-based applications.
Certified CertiKOS-based applications, which are formally certified and doing security-concerned jobs.
Only objects below the red line are in the trusted computing base.
SMP Management
Virtual Memory
Interrupt Handling
Vconsole
Virtualization Abstraction
SVM Primitives
Virtual Devices
Memory Management
Hardware Abstraction Layer
Hardware
CPU Cores
Memory
Hard Disk
Network
. . .

4. CertiKOS Architecture
APP
(uncertified)
APP
(certified)
Virtual Devices
Mgmt Shell
Mgmt OS
(Linux)
Commodity OS
Application
Master
Master Syscall
Slave
Slave Syscall
Process Management
Virtual Machine Management
Memory
V-Interrupt
V-Devices
Hypercall
Context
IPC
CertiKOS
Bottom is the commodity hardware.
Maybe we need to mention CertiKOS requires CPU provides hardware virtualization support, i.e. AMD SVM in latest CertiKOS.
The medium layer is CertiKOS kernel.
Hardware abstraction layer (HAL) provides some device drivers, e.g. PCI adaptor, hard disk drive, ethernet adaptor, etc. It’s almost empty in latest CertiKOS.
The layer upon HAL provides basic kernel primitives.
SMP management, memory management, virtual memory management, interrupt handling and spinlock are similar to those in other operating systems.
The extension is the virtualization part, which is composed of SVM primitives, virtual devices and virtualization abstraction.
SVM primitives provide functions to control the underlying AMD SVM CPU.
Virtual Devices provide necessary device emulation and passthrough, e.g. CertiKOS emulates interrupt controller, programmable timer, keyboard controller, etc, blocks PCI adaptor and devices, and passes through other devices.
Virtualization abstraction provides an abstract interface to other parts of CertiKOS kernel, so that they don’t need to take care which the underlying hardware virtualization is and if the access to devices from the guest should be emulated or just passed through.
CertiKOS provides two kinds of runnable objects: process and virtual machine.
Process is similar to those in other operating systems except that CertiKOS uses a much simple process model: each physical process core runs at most one process.
Context contains the information used in process schedule, trap handling, etc.
IPC provides message-passing style communication mechanism between processes.
Virtual machine management is to
Create a virtualization environment for the guest, which contains virtualizing CPU, memory, interrupt mechanism and a minimal set of devices.
Control the execution of the virtual machine, e.g. start, stop, resume, shutdown. (functions in bold have not been done yet)
Handle the hypercals. The details will be shown in the following slides.
CertiKOS provides two kinds of system calls:
Master system calls are for the management shell.
Slave system calls are for the certikos-based applications.
We can run four kinds of applications upon CertiKOS.
Management shell, which launches and controls other applications.
We plan to move it to the guest and implement it as a userspace application of the guest OS. In this way the management shell can leverage the libraries in the guest OS to provide a better user interface, and use hypercalls provided by CertiKOS to complete the management work.
Virtual machine, or guest, in which runs a commodity operating system and its applications. No modifications are required for both the commodity operating system and the applications, except when they want to use hypercalls provided by CertiKOS. Except the hypercall, CertiKOS is transparent for them.
Uncertified CertiKOS-based applications.
Certified CertiKOS-based applications, which are formally certified and doing security-concerned jobs.
Only objects below the red line are in the trusted computing base.
SMP Management
Virtual Memory
Interrupt Handling
Vconsole
Virtualization Abstraction
SVM /VMX Primitives
SVM Primitives
Virtual Devices
Memory Management
Hardware Abstraction Layer
Hardware
CPU Cores
Memory
Hard Disk
Network
. . .

5. CertiKOS Architecture
APP
(uncertified)
APP
(certified)
Virtual Devices
Mgmt Shell
Mgmt OS
(Linux)
Commodity OS
Application
Master
Master Syscall
Slave
Slave Syscall
Process Management
Virtual Machine Management
Memory
V-Interrupt
V-Devices
Hypercall
Context
IPC
CertiKOS
Bottom is the commodity hardware.
Maybe we need to mention CertiKOS requires CPU provides hardware virtualization support, i.e. AMD SVM in latest CertiKOS.
The medium layer is CertiKOS kernel.
Hardware abstraction layer (HAL) provides some device drivers, e.g. PCI adaptor, hard disk drive, ethernet adaptor, etc. It’s almost empty in latest CertiKOS.
The layer upon HAL provides basic kernel primitives.
SMP management, memory management, virtual memory management, interrupt handling and spinlock are similar to those in other operating systems.
The extension is the virtualization part, which is composed of SVM primitives, virtual devices and virtualization abstraction.
SVM primitives provide functions to control the underlying AMD SVM CPU.
Virtual Devices provide necessary device emulation and passthrough, e.g. CertiKOS emulates interrupt controller, programmable timer, keyboard controller, etc, blocks PCI adaptor and devices, and passes through other devices.
Virtualization abstraction provides an abstract interface to other parts of CertiKOS kernel, so that they don’t need to take care which the underlying hardware virtualization is and if the access to devices from the guest should be emulated or just passed through.
CertiKOS provides two kinds of runnable objects: process and virtual machine.
Process is similar to those in other operating systems except that CertiKOS uses a much simple process model: each physical process core runs at most one process.
Context contains the information used in process schedule, trap handling, etc.
IPC provides message-passing style communication mechanism between processes.
Virtual machine management is to
Create a virtualization environment for the guest, which contains virtualizing CPU, memory, interrupt mechanism and a minimal set of devices.
Control the execution of the virtual machine, e.g. start, stop, resume, shutdown. (functions in bold have not been done yet)
Handle the hypercals. The details will be shown in the following slides.
CertiKOS provides two kinds of system calls:
Master system calls are for the management shell.
Slave system calls are for the certikos-based applications.
We can run four kinds of applications upon CertiKOS.
Management shell, which launches and controls other applications.
We plan to move it to the guest and implement it as a userspace application of the guest OS. In this way the management shell can leverage the libraries in the guest OS to provide a better user interface, and use hypercalls provided by CertiKOS to complete the management work.
Virtual machine, or guest, in which runs a commodity operating system and its applications. No modifications are required for both the commodity operating system and the applications, except when they want to use hypercalls provided by CertiKOS. Except the hypercall, CertiKOS is transparent for them.
Uncertified CertiKOS-based applications.
Certified CertiKOS-based applications, which are formally certified and doing security-concerned jobs.
Only objects below the red line are in the trusted computing base.
SMP Management
Virtual Memory
Interrupt Handling
Vconsole
Virtualization Abstraction
SVM Primitives
SVM /VMX Primitives
IOMMU
Memory Management
Hardware Abstraction Layer
Hardware
CPU Cores
Memory
Hard Disk
Network
. . .

6. Content Overview Virtual Disk Port to Intel platform
Reduce virtual device code
Virtual device at user mode
Pass through device with IOMMU
Demo

7. Virtual Disk … Motivation Virtual PCI Virtual disk based on virtio
Enable CertiKOS to boot guest OS on bare metal
Separate the storage of guest OS from CertiKOS physically
Virtual PCI
Virtual disk based on virtio
a. Mgmt shell
Linux
Legacy OS,
e.g., Linux
c. CertiKOS-based APP …
d.Mgmt tool in Linux
CertiKOS
Virtual Disk
disk0
disk1

8. Virtio
Virtio
Rusty Russell, “virtio: Towards a De-Facto Standard For Virtual I/O Devices”
Available in both Linux and Windows
A simple and efficient framework to provide virtual devices to guest OS
Virtio is an abstraction for a set of common virtual devices
Legacy OS,
e.g., Linux
Add an example to show the virtqueue and virtring
Virtqueue
Front-end driver
Back-end driver
CertiKOS …
Disk driver
disk1

9. Boot CertiKOS on Bare Metal
Multiple settings for booting CertiKOS on bare metal
Boot CertiKOS and Guest on the same disk
Boot CertiKOS and Guest on different disks
Boot CertiKOS on USB and boot the guest on disk

10. Content Overview Virtual Disk Port to Intel platform
Reduce virtual device code
Virtual device at user mode
Pass through device with IOMMU
Demo

11. Port to Intel platform Motivation Modularized implementation LOCs
Another widely supported Hardware-based Virtualization solution
Widely available VT-d support
Modularized implementation
Separate architecture dependent modules
Integrated by interfaces in the abstraction layer
LOCs
Sys/virt/svm
Sys/virt/vmx
VMX uses more sophisticated methods to control the virtualization
Access memory region for control data structures by special instructions, instead of direct memory read and write
More sophisticated setup
Virtual Machine Management
Virtualization Abstraction
SVM Primitive
VMX Primitive
Virtual Devices

12. Content Overview Virtual Disk Port to Intel platform
Reduce virtual device code
Virtual device at user mode
Pass through device with IOMMU
Demo

13. Virtual Device LOCs in previous version at Boston PI meeting
Sys/virt/ *
Sys/virt/dev/ *
With Virtual PIC, KBD, PIT, text mode VGA
LOCs in current clean_code branch
Sys/virt/ *#
Sys/virt/dev/ *
Added virtual PCI, Virtio, Virtio-blk
Considering more devices, such as USB, Network, …
Moving virtual device to execute in user mode
Securely pass through device with IOMMU
( * counted by cloc 1.56)
( # with Intel vt-x )

14. Virtual Device at User Mode
For untrusted guest domains, their virtual devices don’t have to be trusted
Process model extension
Multiple processes on a single core based on round-robin scheduling
Message passing via channels among processes
a. Idle …
d. Legacy Linux
V-KBD
V-PIC
V-PIT …
CertiKOS
CPU0
CPU1

15. Virtual Device at User Mode
Support multiple VM guests with VM session extension
VM Session 1
VM Session 2
a. Idle … …
d. Legacy Linux
Guest Linux 2 … … …
CertiKOS …
CPU0
CPU1
CPU2

16. Pass Through Device
Exclusively used devices can be directly exposed to guest VM, without introducing device virtualization code
However, malicious DMA operations are capable of attacking memory spaces
IOMMU / VT-d
allow a guest OS running under a VMM to have direct control of a device
Provide fine-grain control of device access to system memory

17. IOMMU
from AMD IOMMU specification Revision 2

18. IOMMU Main memory IOMMU MMU Device CPU
Physical Address
IOMMU
MMU
Device Address
Virtual Address
Device
CPU
from

19. IOMMU IOMMU MMU Device CPU
Page Table 1
Page Table 2
IOMMU
MMU
Device Table …
Device Address
Virtual Address … … …
Device
CPU …
Interrupt Remapping Table
Based on image from

20. Pass through device with IOMMU
Legacy OS,
e.g., Linux
NPT
Device Table
CertiKOS …
Interrupt Remapping Table
IOMMU
device

21. Content Overview Virtual Disk Port to Intel platform
Reduce virtual device code
Virtual device at user mode
Pass through device with IOMMU
Demo

22. CertiKOS Demo Setting For Previous Version … … BSP AP AP KVM
a. Mgmt shell
Legacy OS,
e.g., Linux
c. CertiKOS-based APP …
master
slave
slave
CertiKOS
Dotted line of mgmt Linux means it is under developing. …
Qemu
BSP AP AP
Linux
KVM
AMD processor with SVM
BSP- Boot Strap Processor
AP-Application Processor

23. AMD processor with SVM/ Intel with VT-x
CertiKOS Demo Setting
a. Mgmt shell
Legacy OS,
e.g., Linux
c. CertiKOS-based APP …
master
slave
slave
CertiKOS
Dotted line of mgmt Linux means it is under developing.
AMD processor with SVM/ Intel with VT-x
BSP- Boot Strap Processor
AP-Application Processor

24. Thank you!