Virtual Workspaces State of the Art and Current Directions Borja Sotomayor University of Chicago (Dept. of CS) Kate Keahey ANL/UC Tim Freeman ANL/UC

Index Virtual Workspaces Use Cases GT4 Workspace Service Current Directions

Index Virtual Workspaces Use Cases GT4 Workspace Service Current Directions

What is a workspace? What do I do if I want to run something in the Grid? We have to encapsulate our work in a job abstraction. This abstraction imposes a lot of restrictions on the resource consumer. For example: Resource consumers can specify what libraries their jobs require, but they (usually) can't have those libraries dynamically installed when the job is submitted. If the library is not already available in the resources, they generally have to make a prior arrangement with resource providers to make sure that software is installed. However, these restrictions are usually perfectly acceptable. But... what if they're not?

Magic! What is a workspace? In some cases, resource consumers need execution environments to be dynamically deployed on remote resources. The virtual workspace is an abstraction for such an execution environment. Dream up an ideal environment for your jobs (or anything else!) 2GB memory 500MB scratch disk Software: FOO, BAR, BAZ Ideal environment is deployed and can be managed Run jobs in ideal execution environment

What is a workspace? A virtual workspace is an execution environment that can be deployed dynamically and securely on the Grid. Dynamic: Started and stopped on an on-demand basis. Secure: Environment encapsulated inside the workspace and the user submitting the workspace are both trustworthy. Of course, this is not an entirely new idea... It is possible to create custom execution environments by: Dynamically setting up cluster nodes: CoD: bcfg: Providing access to existing installation Dynamic Accounts: Refining site configuration Pacman:

What is a workspace? Main two aspects of workspaces: Environment definition: Users get exactly the (software) environment they need. Resource allocation: Provision and guarantee all the resources the workspace needs to function correctly (CPU, memory, disk, bandwidth, availability), allowing for dynamic renegotiation to reflect changing requirements and conditions Existing implementations either don't provide both, or... Quality of life: Setting up a new software environment takes a long time, and still doesn't give the resource consumer full control. Quality of service: Little or no enforcement. Quality of Life Quality of Service

What is a workspace? What magic spell can we use? Virtual Machines are a promising vehicle to achieve higher quality workspaces Magic!

Virtual Machines VM technology allows multiple virtual machines to run on a single physical machine. Hardware Virtual Machine Monitor (VMM) / Hypervisor Guest OS (Linux) Guest OS (NetBSD) Guest OS (Windows) VM App Xen VMWare UML Denali etc. Performance: Paravirtualization (e.g. Xen) is very close to raw physical performance

Virtual Machines and Grid Computing VMs have several qualities that make them an appealing technology in Grid systems: Security and isolation Customization of execution environment Resource control Site independence Workspaces can be encapsulated in VM images, and dynamically deployed on VMM- enabled sites.

Index Virtual Workspaces Use Cases GT4 Workspace Service Current Directions

Virtual labs A CS department wants to teach a course on Parallel Programming. Unfortunately, they do not have a computing cluster the students can use. And, even if they do, the cluster admin won't give the students root access. Solution: Deploy a virtual workspace that provides the students with a ready-to-use cluster.

Event-driven applications Some applications need large amounts of computational power the instant a specific event arrives. Science comes at you fast! Current solutions include job preemption or allowing urgent requests to skip to the front of the queue. Solution: VMs allow seamless suspend/resume of execution environments. An urgent environment can be placed on standby, activated when the event arrives, and all other VMs can be paused. Less draconian example: Fusion experiments

Strict software requirements Resource consumers can have strict software requirements: Jobs that will only run with a specific version of a library. Legacy software. Special access privileges Managing multiple software requirements can be problematic for the resource provider Conflicting software requirements. Switching between software environments can be costly (e.g., reimaging cluster nodes) Avoid malicious use of resources. Solution: Encapsulate desired software environment inside a VM.

Index Virtual Workspaces Use Cases GT4 Workspace Service Current Directions

GT4 Workspace Service The GT4 Virtual Workspace Service (VWS) is a VM-based workspace implementation. GT4 WSRF frontend Xen-based, but other VMMs could potentially be used.

GT4 Workspace Service Architecture VWS Node Workspace Service Pool node Resource Provider Image Node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node The workspace service has a WSRF frontend that allows users to deploy and manage virtual workspaces The VWS manages a set of nodes (typically a cluster). This is called the node pool. Each node must have a VMM (Xen) installed, along with the workspace backend (software that manages individual nodes) VM images are stored in a separate node.

GT4 Workspace Service Typical Deployment Pool node Image Node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Resource Provider VWS Node Workspace Service Users can interact directly with their workspaces the same way the would with a physical machine.

GT4 Workspace Service Workspace Service is a Globus incubator project. Current release TP1.2.3 (04/20/07) Provides the functionality described in the previous slides (managing a pool of nodes, and deploying single-machine workspaces to those nodes) New in this release: Multiple partitions per VM (with some restrictions) HTTP staging Improved documentation Multiple bug fixes

Index Virtual Workspaces Use Cases GT4 Workspace Service Current Directions

Workspaces + Batch Scheduling (I) Workspaces allow us to dynamically deploy execution environments on a grid. But... how do they fit into the model of batch-oriented computation? The job is the most widely used execution abstraction on grids. In other words: Will it play in Peoria? Workspace Service LRM Option #1 – Use a workspace-aware LRM. This would be ideal, but there is no such thing. There are plenty of commercial data center solutions, but they focus on resource leasing, not batch systems. Option #2 – Use workspaces to deploy virtual compute clusters. Could be a valid option in certain cases (e.g., gliding in your own scheduler). However, cluster admins have to give up their LRM, and the responsibility of providing an LRM rests on the job submitter. Option #3 – Have LRMs and workspaces as separate components, but working together. Workspace- aware LRM

Workspaces + Batch Scheduling (II) We want the LRM to be able to create a workspace for jobs requiring a specific software environment. There is a naïve way of doing this: Set the job prolog to start VM and the job epilog to stop VM. This adds overhead which can be excessive if not adequately managed [1] Questions we want to answer: How can we efficiently combine resource leasing and batch scheduling? Can we use the suspend/resume functionality of VMs to improve cluster utilization, despite the overhead of using VMs? (partially addressed in [1]) How can we do this in such a way that cluster admins will be happy? [1] A Resource Management Model for VM-Based Virtual Workspaces, Sotomayor, Borja. Masters paper, University of Chicago

Configuration Management Virtual workspaces without a collection of VM images to choose from is like iTunes without the music. However, VM images cannot be created just once and then used out-of-the-box. VM images need to be configured to match the configuration of the site they are being deployed to (IP address, certificates, software in the image that depends on that information,...) Site admins need to establishing that a VM image is trustworthy. VM images are large, and can use up valuable storage resources. We need mechanisms that address these concerns. Otherwise, only site-specific VM images will be deployable in practice. Bradshaw et al. [2] discuss requirements and services (including Bcfg2 and Workspace Service) necessary to ensure scalable management and deployment of appliances (implemented as VM images) [2] A Scalable Approach To Deploying And Managing Appliances, Bradshaw, R., N. Desai, T. Freeman, K. Keahey. TeraGrid 2007

STAR (I) STAR is an experiment at the Relativistic Heavy Ion Collider (RHIC) at the Brookhaven National Laboratory. Their software is an example of an application that requires a very specific software environment to work correctly, and which cannot easily be deployed on Grid sites. Doug Olson (LBL): tends to push the boundaries on what will actually compile using rarely used features of the language even just validating a new platform is a big job even when it all compiles. The STAR offline analysis software is about 1.3M lines of code, 2/3 C++, a bit under 1/3 Fortran, and a bit of C.

STAR (II) rBuilder to the rescue rPath was founded by ex-RedHat luminaries Software Appliances (and not just VMs) Stu Gott, Ken Vandine, and Marty Wesley worked with OSG and Doug Olson to produce a STAR appliance with rBuilder STAR image successfully deployed on Teraport Xen nodes. We have also experimented with the AliEn software (from the ALICE experiment)

GridWay + Workspaces (I) The GridWay Metascheduler ( is a very successful dev.globus project (first to escalate to a full project) The GridWay team is interested in using workspaces for deployment of Virtual Infrastructures across sites. Dynamic deployment and reallocation of VMs between different administration domains. GridHypervisor Project: We are collaborating with the GridWay team, although the lack of common funding is an impediment.

GridWay + Workspaces (II) Workspace Service GridFT P Grid Hypervisor Physical Infrastructure Virtual Infrastructure Middleware Applications Virtual Machine Control & Monitor Globus Services Service Overlay Virtualizer Service Interaction Users VM Request Grid Site Management of Virtual Machines in Grid Infrastructures, Rubén S. Montero. OGF20.

GT4 Workspace Service On the implementation side, we are working on adding the following to the Workspace Service in the next releases: Support for virtual clusters EC2 backend Workspace Marketplace Improved usability and documentation

Questions? Borja Sotomayor University of Chicago (Dept. of CS) Kate Keahey ANL/UC Tim Freeman ANL/UC