1. Virtualization: Towards More Flexible and Efficient Grids Kate Keahey keahey@mcs.anl.gov Argonne National Laboratory

2. 13/09/05Kate Keahey, PPAM 2005 The Grid Metaphor How do we store energy? How do we charge for energy? How do we reliably deliver energy? What happens if a power station fails? How do we ensure quality of service? What elements make for a safe and efficient power Grid? How do we make sure that supply meets demand?

3. 13/09/05Kate Keahey, PPAM 2005 Computational Grids How do we store computing? How do we charge for computing? How do we reliably deliver cycles? What happens if a power station fails? How do we ensure quality of service? What elements make for a safe and efficient power Grid? How do we make sure that supply meets demand? What is the unit of resource usage? How can we manage different computing environments? How can we ensure that disk, CPUs, network are all available? How can we negotiate for computation? NCSA ANLCaltech SDSC Tera Grid Grid Middleware We need a computon that will combine environment and enforcement aspects as well as a way of managing the multi-dimensional nature of the Grid

4. 13/09/05Kate Keahey, PPAM 2005 Grids Today l Grid Middleware Tools u Security, Data Management, Resource Management & Scheduling, Monitoring u Standards: GGF, OASIS u Implementations: Globus Toolkit, Condor and others u Many new services are being developed l Significant deployments and use of Grid infrastructure u TeraGrid, Open Science Grid (OSG), Grid 3, many European deployments u Multiple projects making production use of Grid infrastructure. l Still issues: heterogeneity, lack of satisfactory control and accounting, no on-demand computing

5. 13/09/05Kate Keahey, PPAM 2005 The Virtualization Layer l Virtual Grids: virtualize computers, networks, disks, memory u Overlay networks, virtual storage… l Use middleware to map the virtualized constructs onto physical hardware u Trust middleware to map and remap the virtual environment as needed u Trust market forces to ensure that physical resources are plentiful when you need them

6. 13/09/05Kate Keahey, PPAM 2005 Virtual Workspace l For now: focus on virtual workspace u Unit of enforcement, a computon for the Grid u Representation of a desired environment l Later: put all elements of the system together into a virtual Grid l We need progress in the following areas: u Protocols to dynamically negotiate and describe a workspace l Ongoing work at GGF: WS-Agreement, JSDL spec u A unit of enforcement l A critical mass implementation l Recent revival in virtual machine technologies provides potential for such an implementation

7. 13/09/05Kate Keahey, PPAM 2005 Virtual Machine Basics Hardware Virtual Machine Monitor (VMM) / Hypervisor Guest OS (Linux) Guest OS (NetBSD) Guest OS (Windows) l A VM can serialize all of its state (including RAM) u A VM image is simply a collection of files l Disk partitions, RAM, configuration file u Such image can be easily moved (migrated) between hypervisors of the same type u Such image can also be saved and used for rollbacks VM App

8. 13/09/05Kate Keahey, PPAM 2005 Different Hypervisor Implementations l Depending on the layer you virtualize you will end up with a different VM u API: language VMs (JVM) u ISA: system VMs (VMware) l Different types of system virtual machines u Full virtualization (VMware) l Run multiple unmodified guest OSs u Para-virtualization (Xen, UML, Denali) l Run multiple guest OSs ported to a special architecture u Single OS image (Vserver) l What is the cost of using VMs? u Paper: From Sandbox to Playground: Dynamic Virtual Environments in the Grid, Grid 2004

9. 13/09/05Kate Keahey, PPAM 2005 The Need for Speed LXVU SPEC INT2000 (score) LXVU Linux build time (s) LXVU OSDB-OLTP (tup/s) LXVU SPEC WEB99 (score) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 Benchmark suite running on Linux (L), Xen (X), VMware Workstation (V), and UML (U) Paper: Xen and the Art of Virtualization, SOSP 2003

10. 13/09/05Kate Keahey, PPAM 2005 Licensing and Distribution l License u Open source (Xen, UML) l Visible effects of open source community at work u Commercial (VMware) l Also, XenSource l Distribution/Installation u Para-virtualization requires kernel modifications l Yes, but … everything else stays the same l Xen is (or soon to be) part of multiple distributions: Fedora Core 4, Debian, inofficial: Gentoo, Mandrake and SUSE distributions l Work on making Xen part of the Linux kernel u Privilege l Xen (root, patch kernel, domain 0 privileges setup) l VMware Workstation (root, installation only) l UML: user-level

11. 13/09/05Kate Keahey, PPAM 2005 What Makes VMs Great l Summary of VM properties: u Good isolation properties l Generally enhanced security, audit forensics u Excellent enforcement potential l Details depend on implementation u Customizable software configuration l Library signature, OS, maybe even 64/32-bit architectures u Serialization property l VM images (include RAM), can be copied u The ability to pause and resume computations l Allow migration l How do we make VMs available over the network and manage them so as to leverage this potential? u Challenges: security, enforcement, protocols

12. 13/09/05Kate Keahey, PPAM 2005 Grid Services l Web Service Resource Framework u An extension of Web Services u Provides standard mechanisms for l Creation l Lifetime Management l State management, inspection (notification) u Globus Toolkit 4 l Implementation of the WSRF framework u Available since April 2005 l Provides secure authentication, authorization as well as tools for fast transfer, replica management, monitoring, and others.

13. 13/09/05Kate Keahey, PPAM 2005 What are Virtual Workspaces? l Virtual Workspaces: environments that can be made available dynamically the Grid u well-defined properties in terms of environment definition and resource usage enforcement l Examples: u A physical cluster booted to a desired configuration (e.g. Cluster on Demand) u A Grid3 node dynamically configured using Pacman u A cluster partition configured with a hypervisor u A VM representing an OSG configuration enforcing memory and CPU usage l Workspaces can be implemented using a variety of technologies u VMs are the most promising

14. 13/09/05Kate Keahey, PPAM 2005 Virtual Workspace l Environment Aspect (workspace meta-data) u Information/state that outlives its deployment l Generic information (name, time to live) l Attested software partition information: OS, OSG configuration, application installation, etc. l Services: ssh, GRAM, pre-configured job l Resource allocation request (deployment time) u Flexibly negotiated within desired constraints l See GGF WS-Agreement standard u Memory, disk, networking, etc. l See GGF JSDL standard u On deployment the actual resource allocation information becomes available for inspection l Atomic workspaces and virtual clusters u Clusters are simply aggregate workspaces

15. 13/09/05Kate Keahey, PPAM 2005 Deploying Workspaces in the Grid l Define workspace environment l Manage workspace l Negotiate workspace deployment characteristic Workspace Wizard (VW Factory) Workspace Management Service (VW Repository) Workspace Service (VW Manager) request a workspace workspace meta-data manage workspace environment workspace metadata Workspace terminate workspace deployment negotiate workspace deployment manage/monitor/renegotiate workspace deployment manage activities within the workspace

16. 13/09/05Kate Keahey, PPAM 2005 Current Implementation l Current prototype using Globus Toolkit 4 u Leveraging standard Grid Service features l Workspace Wizard u Returns workspace meta-data u Very rudimentary implementation l Workspace Service u Create: takes workspace meta-data and a deployment descriptor u Manage: l renegotiate resource allocation l Also traditional Grid Service management: TTL, etc. u Destroy l Different options: pause, shutdown or destroy l First tech preview release expected later this month

17. 13/09/05Kate Keahey, PPAM 2005 How dynamic is the deployment? l Automatic u Protocol-based u Moving towards better articulation of migration u Renegotiation of resource allocation l How fast is this deployment? u Deployment of workspace for EMBOSS suite: l Manual: ~45 minutes l Based on pre-configured Vmware VMs: ~6 minutes l Based on pre-configured Xen VM: < 1 second l How much overhead does workspace deployment add over what we have today?

18. 13/09/05Kate Keahey, PPAM 2005 How much deployment overhead are we adding? l Using a paused VM allows us to save on initiation time a)GRAM job execution b)GRAM job execution in a paused Xen VM c)job execution in a booted Xen VM (pre-configured job)

19. 13/09/05Kate Keahey, PPAM 2005 Workspace Service: Virtual Clusters

20. 13/09/05Kate Keahey, PPAM 2005 Workspace Deployment Across Technologies l Basic node configuration (+/-boot from image) u Cluster on Demand, PXE, bcfg u On the order of many minutes (~30 minutes) l Refining configuration, creating access u Dynamic account with workspace service: < 1s (mostly GT4 request processing time) u Refining Installation: ~2 hours to configure an ATLAS node using Pacman l Virtual machines u Deploying images l Xen: ~100 ms l VMware Workstation: ~ several seconds

21. 13/09/05Kate Keahey, PPAM 2005 Nested Workspaces Physical machine procure hardware program … VM Hypervisor/OS deploy hypervisor/OS workspace VM deploy VM workspace (with hypervisor/OS) It is easier to maintain a few hypervisor configurations than thousands of user configurations. Those can be deployed in virtual machines.

22. 13/09/05Kate Keahey, PPAM 2005 Virtual Playgrounds Application Virtual Grid

23. 13/09/05Kate Keahey, PPAM 2005 Ongoing Work on Workspaces l Dynamic resource management with VMs u Virtual clusters, fine-grained resource mangement, migration, moving towards economic management l X. Zhang, T. Freeman l IP overlay network for virtual machines u Management infrastructure for VM IP addresses l T. Freeman & L. Chen l Secure management of VM images u Image attestation and verification u Handling image distribution u Managing workspace identity l W. Lu, T. Freeman, F. Siebenlist l Deployment u Edge Services for OSG: with F. Wuertherwein & A. Rana

24. 13/09/05Kate Keahey, PPAM 2005 Related Work l In-Vigo u VM-based infrastructure for the Grids u VM deployment, virtual storage, virtual networks u Renato Figueiredo, Jose Fortes l Virtuoso u VNET: virtualizing networks u Peter Dinda & lab l VIOLIN u Isolated, virtual networks for VMs u Dongyan Xu & lab l Cluster on Demand u Clusters of VMs on demand, also networking, resource management u Jeff Chase and lab

25. 13/09/05Kate Keahey, PPAM 2005 The Challenges that Lie Ahead l Deployment u How do I prepare a cluster for VM execution? l Reserve and publish u Site-specific versus Grid-specific u What security trade-offs are acceptable? u How will VM usage change site configuration? u And many, many others l Environment configuration management u How to configure and manage a VM? l GGF CDDLM working group u Packaging infrastructures l Security u Huge potential: how are we going to leverage it? l Economics, Grid markets, and many others

26. 13/09/05Kate Keahey, PPAM 2005 Conclusions l Virtual is the new real! u Virtualization is emerging as an important abstraction layer in the Grids u Virtual workspaces are cornerstone of this new abstraction layer l Rapidly developing VM technology has the potential to implement a computon for the Grids u Fast, accessible VMs u critical mass implementation for virtual workspaces l Two sides to providing computation on tap u Abstractions and enforcement mechanisms u Protocols l There is much ongoing work in VMs but even more challenges still like ahead

27. If you like a challenge, give us a call keahey@mcs.anl.gov