1. INFSO-RI-223782 An On-Demand Dynamic Virtualization Manager Øyvind Valen-Sendstad CERN – IT/GD, ETICS Oeyvind.valen-sendstad@cern.ch Virtual Node bootstrapper (VNB)

2. INFSO-RI-223782 Contents Brief introduction to ETICS ETICS build infrastructure Improving the build/test process The Virtual Node Bootstrapper Achievements Status, Lessons learned and Further work Summary 2

3. INFSO-RI-223782 What ETICS is It’s a software engineering management system It’s a build and test infrastructure It provides tools and resources to configure, manage and analyse build and test runs It provides a common interface to diverse projects to facilitate knowledge sharing and operations management It has an open repository of configuration metadata, packages, reports. The goal is to share information, but also to reliably store and preserve information It has a plugin-based architecture and APIs to allow integrating ETICS into existing processes and extending it with custom actions It’s multi-platform and independent from any specific build or test tool 3

4. INFSO-RI-223782 4 Architecture ETICS is not ‘just’ a build system

5. INFSO-RI-223782 ETICS build infrastructure - static platforms X 5 SL5 / 64 X 5 Mac OSX X 5 SL5 / 32 X 5 SL5 / 64 X 5 SLC4 X 5 Debian X 5 Ubuntu 7 X 5 SL5 New SL5 Job Physical node Virtual static nodes Problem 1: The composition of the pool is fixed. If we have 20 SL5 nodes all busy and a new request for SL5 arrives, it is queued even if many nodes are available with other platforms

6. INFSO-RI-223782 ETICS build infrastructure - static platforms X 5 SL5 / 64 X 5 Mac OSX X 5 SL5 / 32 X 5 RH7 X 5 SLC4 X 5 Debian X 5 Ubuntu 7 X 5 SL5 New RH7 Job Physical node Virtual static nodes Problem 2: If a platform is very rarely used, but we want to make it available, one or more nodes will do nothing most of the time

7. INFSO-RI-223782 The proposed solution: dynamic virtual platforms New job Available X 5 Physical machine Virtual machine Available X 5 Available X 5 Available VM repository X Available X 5 Available X 5 Available X 5 Available X 5

8. INFSO-RI-223782 The bootstrap process 8 1.User submits a build/test for a specific platform 2.Request arrives to VNB 3.VNB determines platform and hypervisor support 4.Downloads a suitable VM from repository (or local cache) 5.Deploys the VM 6.Executes job 7.Returns results 8.Cleans up

9. INFSO-RI-223782 Deploying the bootstrapper

10. INFSO-RI-223782 Deploying the bootstrapper

11. INFSO-RI-223782

13. Deploying the bootstrapper

14. INFSO-RI-223782 Deploying the virtual machine

15. INFSO-RI-223782 Deploying the virtual machine

16. INFSO-RI-223782 Deploying multiple virtual machines on the same node

17. INFSO-RI-223782 Deploying multiple virtual machines on the same node

18. INFSO-RI-223782 Deploying multiple virtual machines on the same node

19. INFSO-RI-223782 Achievements Perform platform deployment on demand Increase availability Increase capability Provide reproducible environments Reduce maintenance New releases Upgrades Security patches Accommodate for specific software dependencies Offer privileged access (root access to the VM, not the host) Enable post build analysis (VM snapshot can be saved with results) Virtual machine image customization (users can provide their own) 19

20. INFSO-RI-223782 Status Working Bootstrap process, tested on VMware and KVM Remaining Testing on Xen Support for NT-based operating systems Caching VM images locally 20

21. INFSO-RI-223782 Lessons learned Privileges The bootstrapper runs in the user-space and need privileges to use Xen and Vmware Compatibility The kernel in a virtual machine created in VMware is not necessarily able to run in a Xen or KVM environment 21

22. INFSO-RI-223782 Further work Deploy virtual machine image into a folder and chroot in order to run natively on a worker node Update VNB on VM during startup Support for NT-based operating systems

23. INFSO-RI-223782 Open issues Reproducibility Different implementations of the virtual machine hardware can cause different behaviour of a virtual machine Firewall Ephemeral port range can be blocked by strict firewalls 23

24. INFSO-RI-223782 Summary The VNB gives Better service availability Better resource utilization The VNB has Generic design Emulates when no hypervisors are present The VNB allows Running multiple bootstrappers on a single host