Working Spaces: Virtual Machines in the Grid Kate Keahey Argonne National Laboratory Tim Freeman, Frank Siebenlist

Slides:



Advertisements
Similar presentations
Open Science Grid Living on the Edge: OSG Edge Services Framework Kate Keahey Abhishek Rana.
Advertisements

A Scalable Approach to Deploying and Managing Appliances Kate Keahey Rick Bradshaw, Narayan Desai, Tim Freeman Argonne National Lab, University of Chicago.
Virtual Playground: from Quality of Service to Quality of Life in the Grids Katarzyna Keahey Argonne National Laboratory.
On-Demand Virtual Workspaces: Quality of Life in the Grid Kate Keahey Argonne National Laboratory.
Virtual Appliances for Scientific Applications Kate Keahey Argonne National Laboratory University of Chicago.
Virtualization: Towards More Flexible and Efficient Grids Kate Keahey Argonne National Laboratory.
From Sandbox to Playground: Virtual Environments and Quality of Service in the Grids Kate Keahey Argonne National Laboratory.
Enabling Cost-Effective Resource Leases with Virtual Machines Borja Sotomayor University of Chicago Ian Foster Argonne National Laboratory/
Workspaces for CE Management Kate Keahey Argonne National Laboratory.
The VM deployment process has 3 major steps: 1.The client queries the VM repository, sending a list of criteria describing a workspace. The repository.
Wei Lu 1, Kate Keahey 2, Tim Freeman 2, Frank Siebenlist 2 1 Indiana University, 2 Argonne National Lab
Division of Labor: Tools for Growing and Scaling Grids Tim Freeman, Kate Keahey, Ian Foster, Abhishek Rana, Frank Wuerthwein, Borja Sotomayor.
From Sandbox to Playground: Dynamic Virtual Environments in the Grid Kate Keahey Argonne National Laboratory Karl Doering University.
Virtual Workspaces in the Grid Kate Keahey Argonne National Laboratory Ian Foster, Tim Freeman, Xuehai Zhang, Daniel Galron.
Remus: High Availability via Asynchronous Virtual Machine Replication
Secure Virtual Machine Execution Under an Untrusted Management OS Chunxiao Li Anand Raghunathan Niraj K. Jha.
Ljubomir Ivaniš CPU d.o.o.
Virtualization Dr. Michael L. Collard
Xen and the Art of Virtualization Ian Pratt University of Cambridge and Founder of XenSource Inc. Computer Laboratory.
Trusted Ring: A Security Enhancing Software Architecture Michael DiRossi, Inventor The Johns Hopkins University Applied Physics Laboratory.
Virtualization and Cloud Computing. Definition Virtualization is the ability to run multiple operating systems on a single physical system and share the.
High Performance Computing Course Notes Grid Computing.
Xen , Linux Vserver , Planet Lab
Towards High-Availability for IP Telephony using Virtual Machines Devdutt Patnaik, Ashish Bijlani and Vishal K Singh.
CS-3013 & CS-502, Summer 2006 Virtual Machine Systems1 CS-502 Operating Systems Slides excerpted from Silbershatz, Ch. 2.
Xen and the Art of Virtualization A paper from the University of Cambridge, presented by Charlie Schluting For CS533 at Portland State University.
Virtual Machines Measure Up John Staton Karsten Steinhaeuser University of Notre Dame December 15, 2005 Graduate Operating Systems, Fall 2005 Final Project.
Virtual Machine approach to Security Gautam Prasad and Sudeep Pradhan 10/05/2010 CS 239 UCLA.
Virtualization for Cloud Computing
5205 – IT Service Delivery and Support
Jaeyoung Yoon Computer Sciences Department University of Wisconsin-Madison Virtual Machines in Condor.
Kate Keahey Argonne National Laboratory University of Chicago Globus Toolkit® 4: from common Grid protocols to virtualization.
Methodologies, strategies and experiences Virtualization.
Tanenbaum 8.3 See references
An Introduction to Xen Prof. Chih-Hung Wu
Virtual Infrastructure in the Grid Kate Keahey Argonne National Laboratory.
A Cloud is a type of parallel and distributed system consisting of a collection of inter- connected and virtualized computers that are dynamically provisioned.
+ CS 325: CS Hardware and Software Organization and Architecture Cloud Architectures.
Xen Overview for Campus Grids Andrew Warfield University of Cambridge Computer Laboratory.
การติดตั้งและทดสอบการทำคลัสเต อร์เสมือนบน Xen, ROCKS, และไท ยกริด Roll Implementation of Virtualization Clusters based on Xen, ROCKS, and ThaiGrid Roll.
Improving Network I/O Virtualization for Cloud Computing.
Large Scale Sky Computing Applications with Nimbus Pierre Riteau Université de Rennes 1, IRISA INRIA Rennes – Bretagne Atlantique Rennes, France
High Performance Computing on Virtualized Environments Ganesh Thiagarajan Fall 2014 Instructor: Yuzhe(Richard) Tang Syracuse University.
Copyright © cs-tutorial.com. Overview Introduction Architecture Implementation Evaluation.
Presented by: Reem Alshahrani. Outlines What is Virtualization Virtual environment components Advantages Security Challenges in virtualized environments.
Virtual Workspaces Kate Keahey Argonne National Laboratory.
EVGM081 Multi-Site Virtual Cluster: A User-Oriented, Distributed Deployment and Management Mechanism for Grid Computing Environments Takahiro Hirofuchi,
Topic 5a Operating System Fundamentals. What is an operating system? a computer is comprised of various types of software device drivers (storage, I/O,
6/23/2005 R. GARDNER OSG Baseline Services 1 OSG Baseline Services In my talk I’d like to discuss two questions:  What capabilities are we aiming for.
A. Frank - P. Weisberg Operating Systems Structure of Operating Systems.
Security Vulnerabilities in A Virtual Environment
Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Tools and techniques for managing virtual machine images Andreas.
Full and Para Virtualization
Globus and PlanetLab Resource Management Solutions Compared M. Ripeanu, M. Bowman, J. Chase, I. Foster, M. Milenkovic Presented by Dionysis Logothetis.
Dynamic Creation and Management of Runtime Environments in the Grid Kate Keahey Matei Ripeanu Karl Doering.
© 2010 VMware Inc. All rights reserved Why Virtualize? Beng-Hong Lim, VMware, Inc.
Towards Dynamic Database Deployment LCG 3D Meeting November 24, 2005 CERN, Geneva, Switzerland Alexandre Vaniachine (ANL)
Workspace Management Services Kate Keahey Argonne National Laboratory.
Harvesting Free Windows CPU Cycles for Linux Applications using Sandboxing Rasmus Andersen Dept. of Computer Science, University of Copenhagen, Denmark.
Open Source Virtualization Andrey Meganov RHCA, RHCX Consultant / VDEL
XEN – The Art of Virtualisation. So what is Virtualisation? ● Makes use of spare capacity ● Run multiple instances of OSes simultaneously ● Multitasking.
Dynamic Accounts: Identity Management for Site Operations Kate Keahey R. Ananthakrishnan, T. Freeman, R. Madduri, F. Siebenlist.
Virtualization for Cloud Computing
Virtualization.
Virtualization Review and Discussion
Virtualization Dr. Michael L. Collard
Management of Virtual Machines in Grids Infrastructures
Management of Virtual Machines in Grids Infrastructures
1. 2 VIRTUAL MACHINES By: Satya Prasanna Mallick Reg.No
20409A 7: Installing and Configuring System Center 2012 R2 Virtual Machine Manager Module 7 Installing and Configuring System Center 2012 R2 Virtual.
Presentation transcript:

Working Spaces: Virtual Machines in the Grid Kate Keahey Argonne National Laboratory Tim Freeman, Frank Siebenlist

Globus World /10/2005 Towards Realizing the Grid Vision l Quality of Service u Dynamically controlled enforcement of various qualities u Not just per-process enforcement l Quality of Life u Being able to find the right configuration on the Grid

Globus World /10/2005 We Need a Workspace! l A configurable execution environment, container u Good isolation properties u Good enforcement potential u Customizable software configuration l Library signature, OS, maybe even 64/32-bit architectures l We need to be able to create, manage and deploy it u We need to be able to negotiate/renegotiate an environment shape with a VO u A broker would then be able to negotiate resource allocations and map these workspaces onto

Globus World /10/2005 Virtual Machines (VMs) l VMs happen to have: u Good isolation properties l Generally enhanced security, audit forensics u Good enforcement potential u Customizable software configuration l Library signature, OS, maybe even 64/32-bit architectures u Serialization property l VM images (include RAM), can be copied u The ability to pause and resume computations l Allow migration l Common concern: u Overhead: application, startup, resource usage

Globus World /10/2005 Virtual Machines Primer l Different types of virtual machines u Full virtualization (VMware) l Run multiple unmodified guest OSs u Para-virtualization (Xen, UML, Denali) l Run multiple guest OSs ported to a special architecture u Single OS image (Vserver) l Paper: From Sandbox to Playground: Dynamic Virtual Environments in the Grid, Grid 2004 Hardware Virtual Machine Monitor (VMM) Guest OS (Linux) Guest OS (NetBSD) Guest OS (Windows)

Globus World /10/2005 The Need for Speed LXVU SPEC INT2000 (score) LXVU Linux build time (s) LXVU OSDB-OLTP (tup/s) LXVU SPEC WEB99 (score) Benchmark suite running on Linux (L), Xen (X), VMware Workstation (V), and UML (U) Paper: Xen and the Art of Virtualization, SOSP 2003

Globus World /10/2005 TCP results LXVU Tx, MTU 1500 (Mbps) LXVU Rx, MTU 1500 (Mbps) LXVU Tx, MTU 500 (Mbps) LXVU Rx, MTU 500 (Mbps) TCP bandwidth on Linux (L), Xen (X), VMWare Workstation (V), and UML (U)

Globus World /10/2005 Scalability LX 2 LX 4 LX 8 LX Simultaneous SPEC WEB99 Instances on Linux (L) and Xen(X)

Globus World /10/2005 Other Concerns l License u Open source (Xen, UML) l Visible effects of open source community at work u Commercial (VMware) l Also, XenSource l Distribution/Installation u Para-virtualization requires kernel modifications l Yes, but … everything else stays the same l Xen is soon to be part of the Linux kernel, Fedora Core 4 (May 05), is in Debian unstable, unofficial: Gentoo, Mandrake and SUSE distributions u Privilege l Xen (root, patch kernel, domain 0 privileges setup) l VMware Workstation (root, installation only)

Globus World /10/2005 New Technology, New Challenges l How can we leverage the benefits of VMs in Grid technology? l How efficient will be this new technology? l How can we ensure a secure environment under these new assumptions? l How well will it all work in practice? l What new scenarios will this enable? l How will it change Grid computing? l What new problems will it create?

Globus World /10/2005 Integrating VMs into the Grid Architecture Client request VM EPR inspect and manage deploy & suspend use existing VM image Create VM image VM Factory VM Repository VM Manager create new VM image Resource VM start program

Globus World /10/2005 Supporting Services l Factory u Creates VM images l Eventually it may have to support negotiation u Images may be created based on an already existing image l VM Repository u Access to state describing a VM u Allows inspection, management, termination, potentially renegotiation, etc. l VM Manager u Service deploying VMs on nodes u Operations: stage, start, pause, stop, checkin l Once deployed, jobs may be executed in the virtual machines in a variety of ways

Globus World /10/2005 Workspace Structure l Elements of a workspace u Workspace description (meta-data) l EPR/name, category, state, etc., also hardware, network, software configuration, etc. u Workspace implementation (VM image) l Workspace types u Workspaces conforming to set configurations u Provenance of VM images l Workspace instances u Workspace meta-data contains a name u Instance equality u Copying operation: copy image, meta-data, create a new name u Signing instances

Globus World /10/2005 Security: New Opportunities, New Problems l VMs introduce a new layer of trust u VM monitor needs to be trusted as a technology u Trusted computing l VMs can be serialized and transferred as data u The integrity of a VM image needs to be protected (signing) u Private information on a VM image need to be protected (encryption) l VM private key: should a VM be able to assert its identity? l Application private keys, security context l VMs can be migrated (source --- > target) u Trust management: a VM image may be moved between parties that dont trust each other l A popular problem u Key management: target VM needs to verify the integrity and identity of a VM image in ways acceptable to the client: key management u Security context has to be preserved or renegotiated

Globus World /10/2005 Migrating Securely target VM image in transfer: Signed and encrypted VM vouched for by sourceVM vouched for by target reestablish security context

Globus World /10/2005 Security: New Twist on Old Problems l Deployment problem u Protecting the VM from the world l VMs are only as secure as the software they run l Who maintains all those VMs? Local administrators would have to maintain too many images yet need to protect against vulnerabilities u Protecting the world from the VM l One could use ones privileges as root on a VM (for example to generate harmful network traffic) l Although audit works great by the time the damage is done it is too late! l Deployment solutions u VO certification and authorization l Certification Authority to certify VM image l Site policies may require VMs to conform to site policies u Detection: Intrusion Detection Systems u Actions l Restricting network traffic: putting the good guys in jail l Right to take action

Globus World /10/2005 Job Startup l c) job startup using a pre-configured job l b) job startup using an unpaused VM l a) job startup using GRAM

Globus World /10/2005 Things We Didnt Talk About l Security u Processing of encryption and signing l Moving images u Image size: starting at 1MB, more typically 200 MB and upwards u Image diff (Rosenblum) u Proximity of image as matching criterion (VMPlant) u Mounting partitions (general on-site assembly) l Scalability u Distribute processes among existing VMs u Lightweight VMs: Denali l Clusters u Currently in progress: work on virtual cluster, collaboration with the COD team at Duke

Globus World /10/2005 How does it work in practice? l Recent project: combining VMs and Grids to create a platform for bioinformatics applications l Some of the conclusions: u Use of virtual machines can significantly broaden the resource base u Saves installation time l EMBOSS installation: ~45 minutes l Deploying a 2GB VM image: ~6.5 minutes l Peace of mind: priceless! u Enforcement capabilities l Depend on the implementation but are generally better than what we have now l SC04 poster: u Quality of Life in the Grids: VMs meet Bioinformatics Applications, D. Galron et al

Globus World /10/2005 Virtual Playgrounds l Define a virtual Grid in terms of requirements u Virtual workspaces u Networking requirements, virtual network u Storage and other requirements l Provide mechanisms to create a Grid l Provide services for the deployment of such virtual playgrounds on real resources l Ephemeral Grids built for a special purpose: u Scientists getting up a Grid for the purposes of a specific experiment run u A scientific simulation that gets discarded or interrupted but can potentially be restored later

Globus World /10/2005 Conclusions l For Grids to scale we need a way to create and manage remote environments in the dynamically and effortlessly l Virtual is the new Real! u VMs present a very compelling solution l Efficiency, flexibility, migration, etc. u …and introduce some new challenges l New services, different models of sharing, security, etc. l Watch out for emergent properties and behaviors! l We need to work with the VM community to fine-tune requirements and features u Open Source helps! l A growing role for Virtual Organizations l Policy, Policy, Policy… u Policy of resource owners, VOs, users… l Closer to the dream on seamlessly negotiating, provisioning, renegotiating… l Status: a GT4 prototype that we keep evolving!

Globus World /10/2005 Related Efforts l Condor u ClassAds, glide-ins l In-Vigo (VMPlant) l Virtuoso (VNET) l VIOLIN (UML + private networks) l Cluster on Demand l Workspaces Project u

Globus World /10/2005 Credits l Actively working: u Tim Freeman u Frank Siebenlist u Xuehai Zhang l Past contributions: u Karl Doering (UCSD) u Daniel Galron (OSU)