1. Cloud Computing with Nimbus
FNAL, January 2009
Kate Keahey
University of Chicago
Argonne National Laboratory

2. Science Clouds Cloud Computing Elastic computing, Pay-as-you-go,
Capital expense
operational expense
Science Clouds
10/20/ The Nimbus Toolkit: http//workspace.globus.org

3. Everything-as-a-Service
SaaS
PaaS
IaaS
10/20/ The Nimbus Toolkit: http//workspace.globus.org

4. The Quest Begins Code complexity Resource control
10/20/ The Nimbus Toolkit: http//workspace.globus.org

5. “Workspaces” Dynamically provisioned environments
Environment control
Resource control
Hardware implementations vs virtualization
10/20/ The Nimbus Toolkit: http//workspace.globus.org

6. A Brief History of Nimbus
STAR production
runs on EC2
Xen released
EC2 goes online
Nimbus Cloud
comes online
2003
2006
2009
Research on
agreement-based
services
First
Workspace Service
release
EC2 gateway
available
Support for
EC2 interfaces
10/20/ The Nimbus Toolkit: http//workspace.globus.org

7. Nimbus Overview
Goal: open source, extensible, IaaS implementation and tools
Specifically targeting scientific community
A platform for experimentation with features for scientific needs
Set up private clouds (privacy, expense considerations)
Tools
IaaS layer (Workspace Service)
Orchestration layer (Context Broker, gateway)
10/20/ The Nimbus Toolkit: http//workspace.globus.org

8. The Workspace Service VWS Service
Pool
node
Pool
node
Pool
node
VWS
Service
Pool
node
Pool
node
Pool
node
Pool
node
Pool
node
Pool
node
Pool
node
Pool
node
Pool
node
10/20/ The Nimbus Toolkit: http//workspace.globus.org

9. The Workspace Service Trusted Computing Base (TCB)
The workspace service publishes
information on each workspace
as standard WSRF Resource
Properties.
Pool
node
Pool
node
Pool
node
VWS
Service
Pool
node
Pool
node
Pool
node
Users can query those
properties to find out
information about their
workspace (e.g. what IP
the workspace was bound to)
Pool
node
Pool
node
Pool
node
Pool
node
Pool
node
Pool
node
Users can interact directly with their workspaces the same way the would with a physical machine.
Trusted Computing Base (TCB)
10/20/ The Nimbus Toolkit: http//workspace.globus.org

10. Workspace Service Interfaces and Clients
Web Services based
Web Service Resource Framework (WSRF)
GT-based
Elastic Computing Cloud (EC2)
Supported: ec2-describe-images, ec2-run-instances, ec2-describe-instances, ec2-terminate-instances, ec2-reboot-instances, ec2-add-keypair, ec2-delete-keypair
Unsupported: availability zones, security groups, elastic IP assignment, REST
Used alongside WSRF interfaces
E.g., the University of Chicago cloud allows you to connect via the cloud client or via the EC2 client
10/20/ The Nimbus Toolkit: http//workspace.globus.org

11. Security GSI authentication and authorization Secure access to VMs
PKI credential required
Works with Grid proxies
VOMS, Shibboleth (via GridShib), custom PDPs
Secure access to VMs
EC2 key generation or accessed from .ssh
Validating images and image data
Collaboration with Vienna University of Technology
10/20/ The Nimbus Toolkit: http//workspace.globus.org

12. Networking Network configuration
External: public IPs or private IPs (via VPN)
Internal: private network via a local cluster network
Each VM can specify multiple NICs mixing private and public networks (WSRF only)
E.g., cluster worker nodes on a private network, headnode on both public and private network
10/20/ The Nimbus Toolkit: http//workspace.globus.org

13. The Back Story Workspace back-end: Trusted Computing Base (TCB)
Workspace WSRF front-end
that allows clients
to deploy and manage
virtual workspaces
VWS
Service
Pool
node
Pool
node
Pool
node
Workspace back-end:
Pool
node
Pool
node
Pool
node
Resource manager for
a pool of physical nodes
Deploys and manages
Workspaces on the nodes
Pool
node
Pool
node
Pool
node
Each node must have a
VMM (Xen)‏ installed, as
well as the workspace control
program that manages
individual nodes
Pool
node
Pool
node
Pool
node
Trusted Computing Base (TCB)
10/20/ The Nimbus Toolkit: http//workspace.globus.org

14. Workspace Components workspace resource manager workspace workspace
WSRF
workspace
service
workspace
control
EC2
workspace
pilot
workspace
client
10/20/ The Nimbus Toolkit: http//workspace.globus.org

15. Workspace Control VM image propagation
Image management and reconstruction
Creating blank partitions, sharing partitions
VM control
Starting, stopping, pausing, etc.
Integrating a VM into the network
Assigning MAC addresses and IP addresses
DHCP delivery tool
Building up a trusted (non-spoofable) networking layer
Contextualization information management
Talks to the workspace service via ssh
Standalone component
Some functionality overlap with libvirt
Implementations in Xen and KVM (queued up for release)
10/20/ The Nimbus Toolkit: http//workspace.globus.org

16. The Workspace Resource Manager
Basic slot fitting
Implements “immediate leases”
Extensible vehicle to experiment with different leases
Open source resource manager for multiple different VMMs
Datacenter technology equivalent
Can be replaced by OpenNebula or other datacenter technologies
Deployment
University of Chicago, University of Florida, Purdue, Masaryk University and all the other Science Cloud sites
10/20/ The Nimbus Toolkit: http//workspace.globus.org

17. The Workspace Pilot
Challenge: how can I provide a virtualization solution without disrupting the current operation of my cluster?
Flying Low: the Workspace Pilot
Integrates with popular LRMs (such as PBS, SGE)
Implements “best effort” leases
Glidein approach: submits a “pilot” program that claims a resource slot
Includes administrator tools
Deployment
U of Victoria (Atlas), Ian Gable and collaborators
Adapting for the use of the Atlas CERN, Omer Khalid
10/20/ The Nimbus Toolkit: http//workspace.globus.org

18. Cloud Closure storage service workspace resource manager workspace
WSRF
workspace
service
workspace
control
EC2
workspace
pilot
cloud
client
workspace
client
10/20/ The Nimbus Toolkit: http//workspace.globus.org

19. IaaS Gateway Goals Released as service, not as code
Access to different IaaS infrastructures
Account management
Facilitate movement between academic and commercial clouds and creation of meta-clouds
Combine higher-level tools and IaaS
Released as service, not as code
First online in June 2007, currently in a rewrite
Used to move e.g., HEP STAR experiments between Science Clouds and EC2
10/20/ The Nimbus Toolkit: http//workspace.globus.org

20. The IaaS Gateway storage service workspace resource manager workspace
WSRF
workspace
service
workspace
control
EC2
workspace
pilot
IaaS
gateway
EC2
Enabling migration between commercial and academic resources
100 node STAR run
potentially other providers
cloud
client
workspace
client
10/20/ The Nimbus Toolkit: http//workspace.globus.org

21. One-click Virtual Clusters
Parameterizable appliance
Tightly-coupled clusters
IP1
HK1
IP2
HK2
IP3
HK3
MPI
Reciprocal exchange of information:
networking and security
10/20/ The Nimbus Toolkit: http//workspace.globus.org

22. Context Broker Context Broker IP1 HK1 IP1 IP2 HK2 HK1 IP1 IP3 HK1 HK3
10/20/ The Nimbus Toolkit: http//workspace.globus.org

23. Goals for Context Broker
Can work with every appliance
Appliance schema, can be implemented in terms of many configuration systems
Can work with every cloud provider
Simple and minimal conditions on generic context delivery
Can work across multiple cloud providers, in a distributed environment
10/20/ The Nimbus Toolkit: http//workspace.globus.org

24. Status for Context Broker
Release history:
In alpha testing since August ‘07
First released summer July ‘08 (v 1.3.3)
Latest update January ‘09 (v 2.2)
Used to contextualize 100s of nodes for EC2 STAR runs
Contextualized images on workspace marketplace
Working with rPath to make contextualizatin easier for the user
10/20/ The Nimbus Toolkit: http//workspace.globus.org

25. End of Nimbus Tour storage service workspace resource manager
WSRF
workspace
service
workspace
control
EC2
workspace
pilot
context broker
IaaS
gateway
EC2
potentially other providers
context
client
cloud
client
workspace
client
10/20/ The Nimbus Toolkit: http//workspace.globus.org

26. Science Clouds
Make it easy for scientific projects to experiment with cloud computing
Can cloud computing be used for science?
Evolve software in response to the needs of scientific projects
Start with EC2-like functionality and evolve to serve scientific projects: virtual clusters, diverse resource leases
Federating clouds: moving between cloud resources in academic and commercial space
What applications can use cloud computing and which ones cannot? What are the problems in practice? 2)
NOT SOMETHING WE CAN RUN A SIMULATION FOR
10/20/ The Nimbus Toolkit: http//workspace.globus.org

27. Science Cloud Resources
University of Chicago (Nimbus):
first cloud, online since March 4th 2008
16 nodes of UC TeraPort cluster, public IPs
University of Florida
Online since 05/08
16-32 nodes, access via VPN
Other Science Clouds
Masaryk University, Brno, Czech Republic (08/08), Purdue (09/08)
Installations in progress: IU, Grid5K, others
Using EC2 for overflow
Minimal governance model
10/20/ The Nimbus Toolkit: http//workspace.globus.org

28. Cloud Use ~100 DNs Utilization: Requests rejected: Overall: 16%
Peak pw: 86% (week of 7/14)
Requests rejected:
None till 7/14
Lots afterwards ;-)
Data scaled to the nubmer of days
10/20/ The Nimbus Toolkit: http//workspace.globus.org

29. Who Runs on Nimbus?
Project diversity: Science, CS, education, build&test…
10/20/ The Nimbus Toolkit: http//workspace.globus.org

30. Hadoop over Many Clouds
U of Florida
U of Chicago
ViNE
router
ViNE
router
CS research: investigate latency-sensitive apps, e.g. hadoop
Need access to distributed resources, and high level of privilege to run a ViNE router
Virtual workspace: ViNE router + application VMs
Paper: “CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications” by Andréa Matsunaga, Maurício Tsugawa and José FortesFirst, accepted to eScience 2008.
10/20/ The Nimbus Toolkit: http//workspace.globus.org

31. Alice HEP Experiment at CERN
CHEP paper in preparation
10/20/ The Nimbus Toolkit: http//workspace.globus.org

32. STAR STAR: a high-energy physics experiment
Need resources with the right configuration
Complex environments: correct versions of operating systems, libraries, tools, etc all have to be installed.
Consistent environments: require validation
A virtual OSG STAR cluster
OSG cluster
OSG CE (headnode), gridmapfiles, host certificates, NSF, PBS
STAR worker nodes: SL4 + STAR conf
Requirements
One-click virtual cluster deployment
Migration: Science Clouds -> EC2
10/20/ The Nimbus Toolkit: http//workspace.globus.org

33. STAR (cntd) From proof-of-concept to production runs Performance
~2 years ago: proof-of-concept
Last September: EC2 runs of up to 100 nodes (production scale, non-critical codes)
Testing for critical production deployment
Performance
Within 10% of expected performance for applications
Work by Jerome Lauret, Doug Olson, Leve Hajdu, Lidia Didenko
10/20/ The Nimbus Toolkit: http//workspace.globus.org

34. Scalability Testing Motivation Workspaces Requirements
Test scalability of various Globus components
Test on a different platforms
Workspaces
Globus others
Requirements
very short-term but flexible access to diverse platforms
Work by various members of the Globus Toolkit (Tom Howe and John Bresnahan)
Resulted in provisioning a private cloud for Globus
Typically very short-lived communities of one
10/20/ The Nimbus Toolkit: http//workspace.globus.org

35. Montage Workflows Evaluating a cloud from user’s perspective
Paper: “Exploration of the Applicability of Cloud Computing to Large-Scale Scientific Workflows”, C. Hoffa, T. Freeman, G. Mehta, E. Deelman, K. Keahey, SWBES08: Challenging Issues in Workflow Applications
10/20/ The Nimbus Toolkit: http//workspace.globus.org

36. Cloud Computing Ecosystem
Appliance Providers
marketplaces
commercial providers
communities
Deployment Orchestrator
orchestrate the deployment of
environments across possibly
many cloud providers
VMM/datacenter/IaaS
User Environments
10/20/ The Nimbus Toolkit: http//workspace.globus.org

37. Open Source IaaS Implementations
Eucalyptus
Open source implementation of EC2
UCSB, R. Wolski & team, 06/2008
OpenNebula
Open source datacenter implementation
University of Madrid, I. Llorente & team, 03/2008
Cloud-enabled Nimrod-G
Monash University, MeSsAGE Lab, 01/2009
Industry efforts
openQRM, Enomalism
10/20/ The Nimbus Toolkit: http//workspace.globus.org

38. Friends and Family
Committers: Kate Keahey & Tim Freeman (ANL/UC), Ian Gable (UVIC)
A lot of help from the community, see:
Collaborations:
Cumulus: S3 implementation (Globus team)
EBS implementation with IU
Appliance management: rPath and Bcfg2 project
Virtual network overlays: University of Florida
Security: Vienna University of Technology
10/20/ The Nimbus Toolkit: http//workspace.globus.org

39. To the Future and Beyond
Increasing Importance of Appliance Providers
Cloud computing tools
Increased interest in cloud interoperability
Standards: “rough consensus & working code”
Image formats, contextualization capabilities, cloud interfaces, etc.
Cloud markets
10/20/ The Nimbus Toolkit: http//workspace.globus.org