Alleviating Constraints with Resource Pools & Live Migration with Enhanced VMotion* Breakout Session# 2823 Raghu Yeluri Sr. Architect Intel Corporation Date: 09/16/2008 Alan Bumgarner Program Manager Intel Corporation Jim Blakley Director, DC Virtualization Intel Corporation

Contents Problem  VMotion* Overview  VMotion* CPU Compatibility Constraints  Why we care Solution: “Enhanced VMotion*” & Intel® VT FlexMigration Case Study: Enabling “Enhanced VMotion*” with Multi-Generation Intel Processors Wrap Up

Core Enabling Technology for Virtualization 2.0 Use cases:  Dynamic Load balancing, planned maintenance/server upgrades, High Availability, Power-aware load balancing Key Requirements for Migration  Part of the same Datacenter (should have the same subnet)  Connected to the same storage (SAN, NAS, iSCSI)  Recommended: dedicated Gigabit network for VMotion  Destination host must have enough resources  Source and destination hosts must have compatible CPU models. Capability provided by VMware to migrate a live Virtual Machine (VM) from one physical server to another. VMotion* Overview Key Requirement: Source and destination hosts must have compatible CPU models

VMotion* Steps VM running on server esx01 to be moved to esx02 Provision VM on esx02, pre-copy memory & CPU state from esx01 to esx02, log changes to a bitmap Delete virtual machine from esx01 VMotion Successfully Completed… 1 25 Memory and CPU State copied to the target host! Suspend the VM on esx01 and copy changed memory to esx02 3 Resume VM on esx02 4

Why Compatible CPUs? VMotion* requires all processors in “Live VM Migration Pool” be “compatible”  Compatible = same ring 3 visible instructions (e.g. visible to all application software)  E.g. 64-bit, SSE3, SSE4.1, SSE4.2, etc  Caches, speeds, #of cores, #of sockets can vary  CPU Vendor has to be same (not the OEM) Why?  User mode features are not intercepted… execute directly on the CPU from the application.  If an application starts using an instruction…  …it wants to keep using it even when an application is migrated to a new server  If the instruction suddenly isn’t present…the application “breaks Example 1: SSE4.1 (Penryn) instruction support  Running application which uses SSE4.1  Migrate to a CPU which does not support SSE4.1 (Merom) => crash

** Shows a DC environment with Intel-based Processors. 45nm (SSE4.1) Intel Core2™ Micro architecture (Penryn) VMotion Future Intel (SSE4.2. Nehalem & Beyond) Multiple Resource Pools. Data Center Fragmentation Why we care: DC Environment** for VMotion* Compatibility Quad-Core Dual-Core 4S 2S 1S 4S+ 2S 1S Dual-Core, Quad-Core Multi-Core 45nm+ 4S+ 2S 1S 65nm (SSE3) Intel Core2™ Micro architecture (Merom) Pool of Compatible Machines XX VMotion

Why we care: Impact of Multiple Resource Pools  Islands of Clusters/Pools, Limited Fungibility of resources  In-effective utilization of resources; Cannot leverage available capacity in a different Pool  Reduced benefit of VMotion* & DRS  Cannot add newer servers incrementally to a VMotion compatible pool  Cannot take advantage of the faster performance and lower power processors..  Customers need simpler ways to determine if new hardware are compatible with current environment

Solution: Enhanced VMotion* What:  Capability in VMWare ESX3.5U2* to increase VMotion* Compatibility  Automatically ensures hosts in a cluster present same CPU feature set, even with different generation CPUs on hosts. (CPUs must be from the same vendor)  Will enable resource pools to include hosts with multiple generation of processors How:  Defines extension to a resource cluster called Enhanced VMotion Compatibility (EVC).  Defines baseline feature/instruction set for cluster that host processors present to VMs.  For Intel Processors:  Baseline feature set: Intel Core Micro Architecture (Merom-based) - SSE3  Leverages Intel® VT FlexMigration Assist to present baseline feature set to VMs.  Hosts with out ‘baseline’ feature set or VT FlexMigration hardware assist – cannot be added to the EVC cluster.  EVC capability can be enabled/disabled from Virtual Center

Solution: Intel® VT FlexMigration Built-in hardware assist starting in 45nm Intel® Core™ Micro architecture that increases live migration compatibility pool  VT-based VMMs (64bit VMWare* ) : “Intel VT-x execution control”  Non VT-based VMMs (BT – 32Bit VMWare*) : “CPU-ID Masking ” Allows VMWare’s ESX to report a consistent set of available instructions to guest software running within a VM. 45nm Intel ® Core™ (SSE4.1 ) Xeon 7400 Xeon 5400 Xeon 3300 Penryn based Dual Core 5200 Dual Core Next Generation Intel (SSE4.2 ) Future Nehalem based Intel ® VT FlexMigration Assist 4P+ 2P 1P Merom based µarch Dual Core 5100 Dual Core 7200 Dual Core 3000 Intel ® Core™ (SSE3 ) Xeon 5300 Xeon 7300 Xeon 3200 Investment Protection and Future Proofing Resource Pools

Case Study: Enhanced VMotion* with multi-generation Intel Processors Proof of Concept Goal:  Test ride and demonstrate Enhanced VMotion* & Intel® VT Flex Migration  Identify integration & configuration issues to use Enhanced VMotion* Approach:  Realistic VMs & workloads – vConsolidate benchmark  VConsolidate 1.1  2 CSUs (10VMs )  CSU – 5 VMs (DB, Java, Web, Mail, Idle)  EVC Cluster with multi-generational Intel® Xeon® processor hosts  Intel® Core™ (merom) and Intel® Core2™ (penryn) micro architecture servers  Manual & Automatic Migration with VMware DRS * vConsolidate Consolidate Stack Unit (CSU)

Case Study : EVC Cluster setup EVC Cluster with Intel® Core™ (Merom) and Intel® Core2™ (Penryn) architecture Servers Intel® Xeon® 5365 (2S/4C, Merom-based) Intel® Xeon® 5460(2S/4C, Penryn-based) Intel® Xeon® 7350(4S/4C, Merom-based) Intel® Xeon® 7400 (4S/6C, Penryn-based) Intel® Xeon® 3200 (1S/4C, Merom-based) (Second OEM) Initial Hosts - Added later 1 2 Baseline Feature set: SSE3 (Merom)

Issue 1: In compatible hardware. Certain CPU features not enabled on physical host  Execute Disable (XD) Bit  VT flag Fix: Enable XD Bit and VT flag from the BIOS. Need an AC recycle for the VT flag to take into effect. EVC Cluster setup: Adding physical hosts

EVC Cluster setup: Adding physical hosts -2 Issue 3: Incompatible host being added into a EVC Cluster. All the hosts in a EVC cluster should have the ESX3.5U2 software.  Fix: Upgrade the host to ESX3.5 U2 Issue 2: Having active VMs on a host in the cluster  Fix 1: Power down all the active VMs on the physical host.  Fix 2: Migrate VMs to a host with same baseline feature set, Power- down VMs, add host to the cluster and migrate VMs back to the cluster.

DB Java Web Mail Idle DB Java Web Mail Idle DRS: Conservative DB Web Java Web Mail Idle DB Web DB Mail Java Mail Idle Case Study: Migration Results Intel Xeon 5460 (2S/4C, Penryn-based) Intel Xeon 5365(2S/4C, Merom-based) Intel Xeon 7350(4S/4C, Merom-based) Intel Xeon 5460(2S/4C, Penryn-based) DRS: Aggressive DRS: Medium DB Web Java Web Mail Idle Manual Migration Before Enhanced VMotion* After Enhanced VMotion* Successful migration between multi-generation Intel-based servers within the same pool!!

EVC Requirements - Summary  All hosts in the cluster should have ESX3.5 U2.  Intel VT and Execute Disable (XD) flag needs to be enabled in BIOS for systems in EVC enabled clusters.  Hosts should have Intel ®Xeon® 3200, Intel ®Xeon® 5300, Intel ®Xeon® 7300 or later processors.  No VMs should be active on a host when being added to the EVC cluster.  EVC can be enabled/disabled from Virtual Center.  Need the latest version of Virtual Center (VC2.5 U2) to enable EVC.

Wrap Up  VMotion* enabling new usages – HA, Power-aware load-balancing, etc.  CPU compatibility requirements of VMotion* creating Islands of clusters/pools – DC fragmentation  Enhanced VMotion* addresses “Multiple Resource Pool Constraints” in Data Center.  For Intel Processors: Enhanced VMotion* leverages Intel® VT FlexMigration hardware assist.  Intel® VT FlexMigration provides Investment Protection and future proofing of resource pools

Backup

Creating Enhanced VMotion Compatible clusters Scenarios 1Creating a brand new EVC cluster, and add physical hosts 2Enabling an existing resource cluster for Enhanced VMotion with active VMs on the physical hosts Enabling an existing resource cluster for Enhanced VMotion with hosts running prior version of ESX server software

Create a new cluster and enable for Enhanced VMotion  Create a regular resource cluster from VI client; You will not see the ‘Enhanced VMotion’ options during the creation of the resource cluster.  “Edit Settings” to see ‘VMWare EVC’ option and select ‘Enable EVC for Intel Hosts’  Decide baseline ‘feature set/instruction set’ for the cluster – Default: “Merom- based” processors  Advanced Usage: Change the baseline instruction set

Enabling EVC for an existing resource cluster Issue 1: In compatible hardware in the cluster for Enhanced VMotion. Certain CPU features not enabled  Execute Disable (XD) & VT flag Disabled with 32bit OS and ESX3.5  Fix: From the BIOS, enable the XD and VT flags.. Issue 2: Having active VMs on a host in the cluster  Fix: Power down all the active VMs on the physical host.  Migrate VMs to a host with same baseline feature set, VMotion VMs, migrate back to cluster once setup Issue 3: Incompatible Host being added into a EVC Cluster. All the hosts in a EVC cluster should have the ESX3.5U2 software. Upgrade host to ESX3.5 U2