Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Grid and Virtualization Orran Krieger Sr. Technical Staff Member Vmware.

Similar presentations

Presentation on theme: "The Grid and Virtualization Orran Krieger Sr. Technical Staff Member Vmware."— Presentation transcript:

1 The Grid and Virtualization Orran Krieger Sr. Technical Staff Member Vmware

2 2 A level of indirection between hardware and software Virtual Machines: a full abstraction of a physical machine Safely and efficiently multiplex virtual hardware on physical hardware Benefits: resource sharing, portability, compatibility Physical machine Virtual machine What is Virtualization?

3 3 Before Server Virtualization: Single OS image per machine Software and hardware tightly coupled Running multiple applications on same machine often creates conflict Underutilized resources Fault/security vulnerability Server Virtualization

4 4 Before Server Virtualization: Single OS image per machine Software and hardware tightly coupled Running multiple applications on same machine often creates conflict Underutilized resources Fault/security vulnerability After Server Virtualization: Virtual machines (VMs) break 1-to-1 dependency between OS and HW Manage OS and application as single unit by encapsulating them into VMs Hardware-independent: they can be provisioned anywhere Strong isolation between VMs Server Virtualization

5 5 VMM VMM VM Encapsulation and Mobility Suspend Resume Snapshot Clone Migrate Record Replay etc. Virtual Machines are more versatile than Physical Machines Log

6 6 VMkernel HAL and Device Drivers Resource Management NetworkingStorage RHEL3-based Service Console Helpers VMM AgentAgent … VMM Disk Footprint: 2 GB 32 MB RPM VMM Disk Footprint: Traditional ESX Server Percent of Patches >50% 98%2%

7 7 ESX Server 3i: Thin Virtualization! VMkernel Resource Management NetworkingStorage RHEL3-based Service Console VMM AgentAgent … VMM RPM VMM Helpers HAL and Device Drivers Disk Footprint: 2 GB 32 MB Disk Footprint: Percent of Patches >50% 98%2% Small footprint with OS-independence Virtualization should have the reliability, security, and performance of hardware Less code and fewer interfaces: Compact 32MB footprint CIM- and remote CLI-based management Aids both security and reliability (no legacy) Integrated in server systems: Hardware is certified and ready-to-run Simple start up experience Launched with IBM, HP, Dell, FSC, NEC. In the next few years, virtualization will be pervasive embedded in all HW

8 8 Grid and grid-like technologies are the future Web applications being developed with scale-out frameworks MapReduce and related frameworks…, Data Intensive Supercomputing Utilities inside companies Hosting companies like Rackspace Shared utilities like Amazons EC2 The rise of SAAS like HPC capacity and capability systems Are both technologies really going to be pervasive? Grid and Virtualization are complementary: Grid: motivated by analogy of power gridGrid: motivated by analogy of power grid Virtualization: converts computation into a fungible commodity; hugely simplifies realization of gridVirtualization: converts computation into a fungible commodity; hugely simplifies realization of grid

9 9 Traditional Grid Adoption Challenges Security Grid jobs can potentially compromise security on host machine Intruders may piggy-back on Grid jobs to attack the system Isolation Grid jobs may eat into host machine's resources Buggy grid job may crash host system OSes often cant control resource usage between Grid jobs Forces single job per host models OS heterogeneity limits number of Grid nodes available for a job Service level guarantees Enterprise grid jobs need guarantees on completion time In many cases, applications must be re-written for Grid

10 10 Traditional Grid Adoption Challenges (Contd) Provisioning Increased activity on host system can slow down Grid jobs Utilization Dedicated machines for Grid jobs lead to severe under-utilization Legacy applications fail to utilize multi-core features Management Grids lead to increase in number of nodes Management of these nodes is a big challenge Infrastructure for application virtualization Grid middleware lack support from infrastructure layer for on-demand deployment of application instances Grid implementation limited by having to use commodity OSes Large attack surface, high overhead, …

11 11 Virtualization in Grid Computing Any problem in computer science can be solved with another level of indirection (David Wheeler) Virtualization provides a natural level of indirection to address Grid Computing challenges Inherent cures to Grid adoption issues Same solution applies to broader utility, hosting, enterprise, web … marketplace

12 12 Benefits of Virtualization in Grids Virtual machines (VM) provide robust security A compromised VM can be isolated without affecting the host VMs provide excellent isolation A Virtual Grid node (VGN) crash has no effect on other VGNs or host VGNs can guarantee service level times VMs can be provisioned with required explicit resource reservation CPU bandwidth, memory, disk space, disk bandwidth, etc Ensures the grid job has the muscle power to complete within the SLA Virtualization reduces data centre complexity Abstracts out infrastructure heterogeneity

13 13 Benefits of Virtualization in Grids (Contd) No resource usage creep with VMs Provisioning places limit on resources that can be consumed Very useful for resource scavenging Grids Host performance does not deteriorate Multiple VGNs on a single physical machine Increases utilization of Grid boxes Enhances workload management capability of the Grid Additional VGNs can be started and deployed instantly On-demand provisioning of VGNs eliminates OS heterogeneity issues As many VGNs of a particular OS as needed

14 14 Benefits of Virtualization in Grids (Contd) Dynamic resource management (1) Vary resource allocated to a VGN based on load/requirement (2) Move VGN to a more/less powerful host Provisioning additional grid nodes becomes trivial Maintain repository of pre-created VM images Choose and bring-up VGN based on requirement Ease of management Infrastructure can be managed through virtual machine management solutions Maintenance and upgrade of VMs easier than physical servers Custom OS support Special purpose OSes, specialized for Grid applications can be developed, resulting in major performance, reliability, simplicity advantages. Lets the Grid application get much closer to the metal.

15 15 Grid Computing GridMiddlewareGridMiddleware Grid Job Jobs IN/OUT

16 16 Virtualization based Grid Grid job Other Job Grid Middleware Jobs IN/OUT Grid middleware can focus on core Grid issues. Virtual infrastructure provides simplified abstraction used by Grid middleware and other applications.

17 17 VMwares GRID enablement features Base features: low-overhead, strong isolation… Integrated management of virtual and physical infrastructure. Guaranteed quality of service: VMotion to move tasks between physical machines Distributed resource scheduling Reliability: Snapshotting for checkpointing of long running jobs High availability Cheap creating of VMs memory overcommit templates & clones Special purpose appliances

18 18 Towards Distributed Virtualization RESOURCE POOL RP 1B Machine Virtualization Physical Machines Distributed Virtualization RP1A OS OS

19 19 VMotion Migrate running VMs from one server to another Encapsulation and HW independence are the key enablers To minimize service disruption: Iteratively pre-copy VM memory from source to destination Store virtual disks on networked storage: FC, iSCSI, NAS VGNs moved across ESX hosts with zero downtime; No need to restart Grid jobs Allows efficient usage of hardware resources and improves overall hardware utilization Similar VGNs with can be grouped together for better performance

20 20 Storage VMotion Storage independent migration of virtual machine disks Zero downtime to virtual machines LUN independent Supported for Fibre channel SANs

21 21 Automatic load balancing across hosts Distributed Resource Scheduling (DRS) Dynamic Balancing Continuous Optimization

22 22 Adding and removing hosts Hot-plug machines Add/remove capacity on demand Distributed power optimization Improve application availability DRS X

23 23 HWHWHWHWHW HWHWHWHWHW HWHW HWHW HW HW HWHW Transforming capacity management VIRTUALIZED PHYSICAL POOLED RP1RP1RP1RP1 RP2RP2RP2RP2 Logical Resource Pooling (RP) Distributed Resource Scheduler (DRS)

24 24 Firewall SW Linux FIREWALL Virtual Appliance Apache Linux Tomcat mySQL CRM Virtual Appliance What is a Virtual Appliance Pre-built, pre-configured and ready-to-run software application packaged with the OS inside a Virtual Machine. Or packaged inside multiple Virtual Machines

25 25 Appliance OSes can be simplified or customized Dont need complex hardware management Dont need broad application support Highly-customizable, application- specific OS Include only what application needs. Look at hardware appliance operating systems for examples Hardware Mgmt. Application Support Application Operating System Virtualization Layer Virtual Appliance OS

26 26 Virtualization and Grid will both be pervasive technologies Virtualization will increasingly be embedded on all server platforms. Various forms of utility computing are increasingly dominating many markets. Virtualization and Grid are complementary. Its time to start working together: What is the right interface between Grid middleware and virtual infrastructure? What are the requirements of the Grid? Determinism for collective operations? Fast start up of VM? Fast Clone… Its time to write Grid appliances/OSes. How tiny can they be? Plan9? Does a Library OS model make sense? Libra/Prose? Is there any paravirtualization support that would be useful? Orran Krieger Concluding remarks

27 27 Thanks Hariprasad Nellitheertha Chinmay Albal Radhika Kulkarni Larry Rudolph; Steve Strassman; Scott Davis; Jack Lo

28 28

29 29 VMware Memory Overcommit Unique VMware Feature Host system memory is usually the resource bottleneck VMware memory management technology allows an average of 2:1 memory overcommit (and oftentimes greater) Transparent Page Sharing Memory ballooning

30 30 HW Independence & Transparency Run a virtual machine on any PC without modification. Break dependence between OS & HW Partitioning Run multiple virtual machines simultaneously on a single PC Isolation The virtual machine is isolated from Host OS, other VMs on the same PC Encapsulation Entire virtual machine is saved in files. Can be moved and copied like any other file. Key Properties of Virtualization

Download ppt "The Grid and Virtualization Orran Krieger Sr. Technical Staff Member Vmware."

Similar presentations

Ads by Google