1. The Design & Implementation of Hyperupcalls
Nadav Amit & Michael Wei
July 2018

2. Hardware Virtualization
Common use-case: server consolidation
Mature technology
Challenged by alternatives (e.g., containers) since it is:
Relatively inefficient
Hard to provide services for VMs
virtual machine
virtual machine
process
process OS OS
guest
host
hypervisor
hardware

3. The Semantic Gap hypervisor discard swap
low on memory?
The VM does not know the physical hardware constraints
sensitive architectural events
The hypervisor is oblivious to the VM OS state
guest
host
Degrades performance
Prevents VM introspection
is page free?
hypervisor
discard
swap
Makes the hypervisor robust

4. Paravirtualization: Extended hypervisor and VM interface
page X is free
hypercalls
hypervisor
is page X free?
upcalls
hypervisor
pre-virtualiztion
hypervisor code
page X is free
guest
host

5. Paravirtual Interfaces
For hypervisor & virtual machine (VM) coordination
execution context
initiator
logic VM
hypervisor
pre-virtualization [ LeVaasseur ’05 ]
hypercalls
upcalls
no context switch overhead
can run privileged operations
“pull” mechanism
hyperupcalls ?

6. Hyperupcalls: A new Paravirtualization Mechanism
Short encapsulated programs
Provided by the VM to the hypervisor, registered to certain hypervisor events
Invoked on hypervisor events
Query VM state or notify it on events
Reuses OS code
Can run while the VM is suspended
hypervisor
is page X free?
hyperupcalls
guest
host
VM code

7. Hyperupcalls Safety The VM cannot be trusted
Isolation is key feature of virtualization
Must ensure safety properties:
No privileged instructions
Safe memory accesses
Bounded runtime
Solution: verifiable code - eBPF
Originates from Berkley Packet Filter
Bytecode with provable safety
AoT compilation to native code
LLVM compiles C to eBPF
Supported by Linux, DPDK, etc.
program/kernel interaction resembles virtual-machine/hypervisor interaction
eBPF can verify hyperupcalls

8. Using Verifiable Code / eBPF
guest
host
compiler
AoT
assembler
event
h-upcall code
h-upcall bytecode
safety checker
h-upcall native code
helper functions
compilation (once)
registration (boot)
execution (event)

9. Memory Mappings for Hyperupcalls
virtual machine view
hypervisor view
hyperupcall memory
guest virtual
host virtual
host physical
might be occupied
both should point to the same data

10. Memory Mappings for Hyperupcalls
virtual machine view
hypervisor view
guest virtual
host virtual
guest base
host base
host physical
Cannot use native pointers
[ guest base ] is only known after boot due to address space randomization
[ hyperupcall address ] = [ address ] – [ guest base ] + [ host base ]
Extend compiler to transparently adjust the pointer
Do not adjust host pointers by annotating them

11. Additional Issues Hardware interfaces Interrupts generation
Accessing VCPU registers
Accessing descheduled VCPUs
Solutions:
Helper functions
Synchronization points
eBPF limitations
No loops, atomic operations, static variables, etc.
Frequent verification failures
Native assembly is unsupported
No linker – no symbols
Solution:
A framework as an in-place replacement for common OS function

12. Use-cases New features Hypervisor event tracing
Kernel security hardening
Performance enhancements
Free memory discarding
TLB shootdowns to inactive cores

13. Hyperupcalls Performance

14. Hypervisor Event Tracing
Performance analysis requires tracing and profiling tools
Only virtual machine events are traced
On the cloud hypervisor events cannot be traced
virtual machine trace
OS event 1
time gap?
OS event 2
virtual machine descheduled
hypervisor

15. Tracing with Hyperupcalls
trace buffer
OS tracing service
VM OS tracing code
[x86]
VM ev
ent
hypervisor event
guest
host
OS hyperupcall
VM OS tracing code
[eBPF  x86]
The hypervisor is oblivious
Virtual machine and hypervisor are decoupled
hypervisor
VM-Exit (context switch)

16. Free Memory Reclamation
Swap (no paravirtualization)
hypervisor
Ballooning (upcall)
hypervisor
Free memory discard (hyperupcall)
guest
host
hypervisor

17. When both memory and CPUs are overcommitted
Memory Reclamation
When both memory and CPUs are overcommitted

18. Conclusions Hyperupcalls Alternative hyperupcall designs are possible
Provide a flexible interface for VM—hypervisor cooperation
Decouples VM-hypervisor
Alternative hyperupcall designs are possible
Programmability is the key for flexible interfaces