Presentation is loading. Please wait.

Presentation is loading. Please wait.

First Steps in the Clouds Kate Keahey University of Chicago Argonne National Laboratory.

Similar presentations

Presentation on theme: "First Steps in the Clouds Kate Keahey University of Chicago Argonne National Laboratory."— Presentation transcript:

1 First Steps in the Clouds Kate Keahey University of Chicago Argonne National Laboratory

2 Virtual Workspaces: http// Why Clouds? l Resource consumers u Individual users or Virtual Organization u Requirements l Customized environments for their services/applications l Services/applications can be short-lived l New environments/services deployed quickly and often l Resource providers u Own and operate physical resources u Requirements l Ability to monitor and control their resources l Provide resources at reasonable operational cost l Protection from activities performed by resource consumer l Consumers need to be able to lease (potentially for short-term) platforms that they can customize and control

3 Cloud Computing for Grid Communities: The STAR Application Use Case

4 Virtual Workspaces: http// The STAR Application l Complex experimental application codes u Developed over more than 10 years, by more than 100 scientists, comprises ~2 M lines of C++ and Fortran code u l Require complex, customized environments u Rely heavily on the right combination of compiler versions and available libraries u Dynamically load external libraries depending on the task to be performed l Environment validation u To ensure reproducibility and result uniformity across environments l Why do we need a cloud? u Resources with the right configuration are hard to find u A VM-based cloud gives us the required control

5 Virtual Workspaces: http// Running STAR in a Cloud l First Challenge: finding VM-enabled resources u Amazon Elastic Compute Cloud (EC2) l More Challenges: u Can we use X.509 certs to submit to a cloud? Can we use Grid access protocols? How much manual configuration do we need to do for a cluster that we need for 4 hours? How do we integrate the cluster into the Grid infrastructure? l Workspace Service u X.509 certificates are mapped to a project account u Grid access protocols u Creating a virtual cluster dynamically l Contextualization (cluster context): the cluster node VMs find out about each other and integrate that information at boot time u Integrating the cluster into the Grid l Contextualization (grid context): cluster is configured with appropriate host certs, gridmapfiles, etc.

6 Virtual Workspaces: http// Running jobs : 300 PDSF Fermi VWS/EC2BNL Running jobs : 230 Running jobs : 150 Running jobs : 50 Running jobs : 150 Running jobs : 300Running jobs : 282Running jobs : 243Running jobs : 221Running jobs : 195Running jobs : 140Running jobs : 76Running jobs : 0 Running jobs : 200 Running jobs : 50 Running jobs : 150Running jobs : 142Running jobs : 124Running jobs : 109Running jobs : 94Running jobs : 73Running jobs : 42 Running jobs : 195Running jobs : 183Running jobs : 152Running jobs : 136Running jobs : 96Running jobs : 54Running jobs : 37Running jobs : 0 Running jobs : 42Running jobs : 39Running jobs : 34Running jobs : 27Running jobs : 21Running jobs : 15Running jobs : 9Running jobs : 0 Job Completion : File Recovery : WSU with thanks to Jerome Lauret and Doug Olson of the STAR project with thanks to Jerome Lauret and Doug Olson of the STAR project, presented at CHEP07

7 Virtual Workspaces: http// Nersc PDSF EC2 (via Workspace Service) WSU Accelerated display of a workflow job state Y = job number, X = job state with thanks to Jerome Lauret and Doug Olson of the STAR project with thanks to Jerome Lauret and Doug Olson of the STAR project, presented at CHEP07

8 Virtual Workspaces: http// What Did We Learn? l Performance was not an issue l The real comparison is having a resource to run on vs not having a resource to run on l Contextualization is key for dynamic virtual cluster deployment l Next steps: a more challenging application

9 Cloud Computing for Grid Providers: Building the Science Cloud at the University of Chicago

10 Virtual Workspaces: http// Challenges l Virtualization adoption has been relatively slow among Grid Providers l Challenge: integrating VMs into current provisioning models u Integrate into a site without disrupting the current operation of resources l I.e., be able to run jobs as well as VMs u Non-invasive from the perspective of currently used tools l E.g., no modification to the currently used schedulers and resource managers u Can be used alongside the current mode of operation l Batch jobs u Represent as small a change as possible l Operate within familiar metaphors l Avoid error-generating complexity

11 Virtual Workspaces: http// Roll Your Own Cloud l The Workspace Pilot u Operates on resources that can support jobs as well as VMs l E.g., have been booted into Xen domain 0 u Non-invasive extension to batch schedulers (e.g., PBS) l Wrappers for submission operation, scheduler signals to operate on VMs u Glidein approach: submits a pilot program that prepares a resource slot for VM deployment l E.g., adjusts Xen domain 0 memory u Comes with administrator tools l E.g., kill-all

12 Virtual Workspaces: http// Workspace Pilot in Action Workspace Service LRM/PBS Xen dom0 VM Level 1: provision raw resources Level 2: provision VMs VMs are decomissioned raw resources are decomissioned

13 Virtual Workspaces: http// The Pilot Program l Uses Xen balloon driver to reduce/restore domain0 memory so that guest domains (VMs) can be deployed l Secure VM deployment u The pilot requires sudo privilege and thus can be used only with site administrators approval u The workspace service provides fine-grained authorization for all requests l Signal handling u SIGTERM: pilot exceeded its allotted time l Notifies VWS, allows it to clean up l After a configurable time period takes things into its hands. l Default policy: one VM per physical node l Available for download u Workspace Release 1.3.1: u

14 Virtual Workspaces: http// UC l What is it? u The Science Cloud at University of Chicago u UC TeraPort cluster configured with the workspace pilot u Currently 16 nodes l What can it do for me? u Allow you to lease out a cluster of VMs l Who can use it? u Members of scientific community l In as much as usage policies will allow l What do I need to do if I want to use it? u Contact us: u You will need a VM image (we can help and know others who can), a certificate, and a simple client

15 Virtual Workspaces: http// Cloud Interoperability l Moving an app from a hardware platform to a cloud is relatively hard u Need to develop a VM image, learn about cloud computing, figure our logistics u Moving between clouds l E.g., STAR app EC2->Science Cloud and vice versa is very easy l Rough consensus on the interfaces needed to provision resources in the cloud l OGF gridvit-wg u Chairs: Erol Bozak, Wolfgang Reichert u Define the requirements for integration of Grid architecture with system virtualization platforms u Exploring the impact of virtualization on Grid use cases u Exploring the relationship with standards (DMTF, etc.)

Download ppt "First Steps in the Clouds Kate Keahey University of Chicago Argonne National Laboratory."

Similar presentations

Ads by Google