1. Jaeyoung Yoon Computer Sciences Department University of Wisconsin-Madison jyoon@cs.wisc.edu http://www.cs.wisc.edu/condor Virtual Machine Universe in Condor

2. www.cs.wisc.edu/condor 2 What is VM universe? › A job user can submit a virtual machine to Condor › Condor runs the virtual machine and sends back a result virtual machine › support VMware server and Xen

3. www.cs.wisc.edu/condor 3 Big picture Submit machine Schedd Virtual Machine Shadow Execute machine Startd Starter VM GAHP

4. www.cs.wisc.edu/condor 4 Benefits of VM universe › platform independence › environment independent on host machine › checkpoint › networking in a virtual machine › snapshot disk › input CDROM image

5. www.cs.wisc.edu/condor 5 Snapshot disk › All modified data will be stored into snapshot disks without changing original VM disk files. › VM disk files in a shared file system can be safely shared among multiple jobs › Can reduce disk space for result and checkpoint

6. www.cs.wisc.edu/condor 6 Submit description file with shared file system › universe = vm › executable = WindowsXP › vm_type = vmware › vm_memory = 256 › vm_checkpoint = TRUE › vm_networking = TRUE › vm_networking_type = dhcp › vmware_dir = /shared/windows_vm › vmware_should_transfer_files = FALSE › vmware_snapshot_disk = TRUE › initialdir = /result1 › Queue › initialdir = /result2 › Queue

7. www.cs.wisc.edu/condor 7 Snapshot disk with shared file system Submit machine Execute machine 1 Shared file system Execute machine 2 /windows_vm Job 1 Snapshot Disk Job 2 Snapshot Disk /result1 /result2

8. www.cs.wisc.edu/condor 8 Submit description file without shared file system › universe = vm › executable = WindowsXP › vm_type = vmware › vm_memory = 256 › vm_checkpoint = TRUE › vm_networking = TRUE › vm_networking_type = dhcp › vmware_dir = /windows_vm › vmware_should_transfer_files = TRUE › initialdir = /result1 › vmware_snapshot_disk = TRUE › Queue › initialdir = /result2 › vmware_snapshot_disk = FALSE › Queue

9. www.cs.wisc.edu/condor 9 Snapshot disk without shared file system Submit machine Execute machine 1 (Job 1) Job 1 submit description... vmware_snapshot_disk = TRUE Initialdir = /result1 Job 2 submit description... vmware_snapshot_disk = FALSE Initialdir = /result2 /windows_vm Execute machine 2 (Job 2) snapshot disk

10. www.cs.wisc.edu/condor 10 Submit machine Job 1 /result1 Job 2 /result2 /windows_vm Execute machine 1 (Job 1) snapshot disk Execute machine 2 (Job 2) Snapshot disk without shared file system

11. www.cs.wisc.edu/condor 11 Input CDROM image › VM universe can not use input or argument parameter in a job submit description file as other universes do › With input CDROM images, a job user may run the same VM several times on different input data sets

12. www.cs.wisc.edu/condor 12 Submit description file with input CDROM image › universe = vm › executable = WindowsXP › vm_type = vmware › vm_memory = 256 › vm_checkpoint = TRUE › vm_networking = TRUE › vm_networking_type = dhcp › vmware_dir = /windows_vm › vmware_should_transfer_files = FALSE › vmware_snapshot_disk = TRUE › initialdir = /result1 › vmware_cdrom_files = a.iso › Queue › initialdir = /result2 › vmware_cdrom_files = a.txt, b.txt › Queue

13. www.cs.wisc.edu/condor 13 Input CDROM image Submit machine Execute machine 1 VM a.iso Job 1 submit description... vmware_cdrom_files = a.iso Job 2 submit description... vmware_cdrom_files = a.txt, b.txt Execute machine 2 VM a.txt b.txt

14. www.cs.wisc.edu/condor 14 VMware VM universe › Snapshot disk › Input CDROM image › Can be used on either Linux host or Windows host

15. www.cs.wisc.edu/condor 15 Xen VM universe › No support of snapshot disk  VM disk file in a shared file system can not be shared among multiple job unless it is read-only. › Input CDROM image › Can be used on only Linux host

16. www.cs.wisc.edu/condor 16 Checkpoint › Periodic checkpoint and vacate checkpoint › All modified VM disk files and a file for VM memory will be transferred back to a submit machine › When snapshot disks are used, snapshot disk files and a file for VM memory will be transferred.

17. www.cs.wisc.edu/condor 17 Suspend › Hard suspend: Memory being used by a VM will be released and the memory will be saved into a file › Soft suspend: Memory being used by a VM will not be released and the VM will be just paused like SIGSTOP

18. www.cs.wisc.edu/condor 18 Networking issues when restarting from checkpoint › MAC and IP address for VM are also preserved when checkpointed › When restarting the checkpointed VM, MAC and IP address don’t change. › If we use NAT for VM networking, different execution machines may have different MAC and IP address of NAT gateway. › In VMware, if we install VMware tool inside VM, the tool program will automatically execute DHCP renew when a VM is restarted.

19. www.cs.wisc.edu/condor 19 Future work › Support snapshot disks in Xen VM universe › For result, get only output files from VM instead of all VM files. › Support another Virtual machine program (e.g. QEMU)

20. www.cs.wisc.edu/condor 20 Summary › We are testing VM universe. › Hopefully VM universe will be included in Condor 6.9.x. Questions?