4/6/2017 Cisco Live 2013.

4/6/2017 Cisco Live 2013

Cisco Live 2013 4/6/2017 Virtual Machine Fabric Extension (VM-FEX) Bringing the Virtual Machines Directly on the Network BRKCOM-2005 Dan Hanson, Technical Marketing Manager, Data Center Group, CCIE #4482 Timothy Ma, Technical Marketing Engineer, Data Center Group

The Session will Cover FEX Overview &History VM-FEX Introduction
VM-FEX Operational Model VM-FEX General Baseline on UCS VM-FEX with VMware on UCS VM-FEX with Hyper-V on UCS VM-FEX with KVM on UCS VM-FEX General Details on Nexus 5500 Summary

FEX Overview & History

Fabric Extender Evolution
One Network Parent Switch to Top of Rack Network Administrator FEX Architecture Consolidates network management FEX managed as line card of parent switch Uses Pre-standard IEEE 802.1BR IEEE BR* Many applications require multiple interfaces FEX Today *IEEE 802.1BR Pre-Standard

Adapter FEX Consolidates multiple 1Gb interface into a single 10Gb interface Extends network into server Uses Pre-standard IEEE 802.1BR One Network Parent Switch to Adapter Network Administrator IEEE BR * Many applications require multiple interfaces FEX IEEE BR * Adapter FEX Today *IEEE 802.1BR Pre-Standard

One Network Virtual Same As Physical VM-FEX Consolidates virtual and physical network Each VM gets a dedicated port on switch Uses Pre-standard IEEE 802.1BR Network Administrator IEEE BR * FEX IEEE 802.1BR * IEEE 802.1BR * Adapter FEX Hypervisor VM network managed by Server administrator Today VM-FEX *IEEE 802.1BR Pre-Standard

One Network Parent Switch to Application Single Point of Management FEX Architecture Consolidates network management FEX managed as line card of parent switch Adapter FEX Consolidates multiple 1Gb interface into a single 10Gb interface Extends network into server VM-FEX Consolidates virtual and physical network Each VM gets a dedicated port on switch Network Administrator IEEE BR* Manage network all the way to the OS interface – Physical and Virtual FEX IEEE BR* IEEE BR* Hypervisor Today Adapter FEX VM FEX * IEEE 802.1BR Pre-Standard

Key Architectural Component #1: VNTAG
“Intra-Chassis” Bus Header FEX architecture LAN Application Payload Switch Frame TCP VNTAG Ether type IP VNTAG Frame VNTAG D P Destination Virtual Interface Ethernet L R ver Source Virtual Interface FEX VNTAG mimics forwarding vectors inside a switch D: Direction, P: Unicast/Multicast, L: Loop Policy associated with the Virtual Interface NOT port VLAN member ship, QoS, MTU, Rate limit etc Ethertype[16] – Defines the type field used to determine that a frame is carrying a VNTAG. This value is currently undefined, awaiting assignment from the IEEE. direction[1] – A 0 indicates that the frame is sourced from an Interface Virtualizer (Adapter) to the Virtual Interface Switch. Likewise, a 1 indicates that the frame is sourced from the Virtual Interface Switch headed to one or more Interface Virtualizers. pointer[1] – Indicates that Destination (v)interface is an index into a (v)interface list table; this case is used for multicast frame delivery. When direction is 0, these fields must be 0. Destination (v)interface[14] – The DVIF select the downlink interface(s) that receive a frame when sent from the Nexus 5000 to Fabric Extender or the Interface Virtualizer. As such, they are evaluated when direction is set to 1. A 0 in pointer indicates that Destination (v)interface selects a single (v)interface, typically used for unicast frame delivery. A 1 in pointer indicates that Destination (v)interface is an index into a virtual interface list table; this case is used for multicast frame delivery. When direction is 0, these fields must be 0. looped[1] – Indicates that the frame is headed back towards the Interface that it originated from (on the same L2 network that it was injected). In this situation, the Fabric Extrender or Interface Virtualizer must ensure that the (v)interface indicated by Source (v)interface does not receive the frame; this exclusion occurs only when both direction and looped are set to 1. reserved[1] – This field is reserved for future use. version[2] – Indicates the version of the VNTAG protocol carried by this header. All implementations of future protocol extensions must be compatible with all those prior. The current and only, defined version is 0. Source (v)interface[12] – The SVIF Indicates the (v)interface that sourced the frame. Source (v)Interface must be set appropriately for all frames from a Fabric Extender or Interface Virtualizer to the Nexus 5000 (direction set to 0). In addition, Source (v)interface may be set for frames headed towards the Fabric Extender or Interface Virtualizer (direction set to 1) as indicated by looped.

FEX Data Forwarding Revisiting Traditional Modular Switches (Example Catalyst 6500) Constellation Bus had 32 byte header for fabric switching Vast majority of modular switch vendors have an internal “Tag” for fabric communications Originally, Centralized forwarding ASICs Line cards fed into these ASICs directly When we needed higher performance – we added faster Switch Fabrics, and Distributed Forwarding Capabilities to system What this really meant – adding more ASIC forwarding capacity to the system to minimize the number of devices a flow had to traverse

FEX Data Forwarding Decoupling the Modular Switch
Think the original C6k Satellite Program for VSL and RSL The Constellation Bus now is smaller header – 6 Byte VNtag header Core to FEX technology and being standardized as 802.1BR This is NOT a 1:1 mapping to VEPA/802.1bg which is designed to offer an enhanced forwarding mechanism between peer devices via a single upstream device Keep the ASIC counts for high performance but put them on the Central controlling switch instead of all these line cards Latency and bandwidth were more a function of the layers of ASICs to traverse in a tree – rather than the location of these ASICs (the fiber/copper paths for a packet to propagate) Add protocols for configuration and firmware management of these remote cards (Satellite Control Protocol, Satellite Discovery Protocol) Allows us to get away from manual firmware code management per (remote) line-card Move from Store-and-Forward behavior to Cut-Through switching to make latency actually better

Fabric Extension (FEX) Concept
Virtualising the Network Port Multi-tier architecture FEX architecture Switch port extended over Fabric Extender LAN LAN Switch Switch Logical Switch Switch FEX Collapse network tiers, fewer network management points

FEX Technology for Unified I/O
Virtual Switch Ports, Cables, and NIC Ports Mapping of Ethernet and FC Wires over Ethernet Service Level enforcement Multiple data types (jumbo, lossless, FC) Individual link-states Fewer Cables Multiple Ethernet traffic co-exist on same cable Fewer adapters needed Overall less power Interoperates with existing Models Management remains constant for system admins and LAN/SAN admins Possible to take these links further upstream for aggregation DCB Ethernet Blade Management Channels (KVM, USB, CDROM, Adapters) Individual Ethernets Individual Storage (iSCSI, NFS, FC)

Key Architectural Component #2 : UCS VIC
256 PCIe devices Devices can be vNICs or vHBAs Each device has a corresponding switch interface Bandwidth 2x4x10 Gb Uses 4x10 Ether Channel, HW 40Gb Capable vNICs/vHBAs NOT limited to 10Gb PCIe Gen-2 x 16 Mezzanine and PCIe 256 PCIe devices vHBAs vNIC vNIC vNIC vFC vEth vEth vEth Dual 4x10Gb

VM-FEX Introduction

Server Virtualization Issues
1. When VMs move across physical ports—the network policy must follow Live Migration Hypervisor Hypervisor 2. Must view or apply network/security policy to locally switched traffic Port Profile 3. Need to maintain segregation of duties while ensuring non-disruptive operations Security Admin Server Admin Network Admin

Cisco Virtual Networking Options
Extend networking into hypervisor (Cisco Nexus 1000V Switch) Cisco UCS VM-FEX Server Extend physical network to VMs (Cisco UCS VM-FEX) Hypervisor Cisco Nexus 1000V Generic Adapter

UCS VM-FEX Distributed Modular System
Removing the Virtual Switching Infrastructure to a FEX UCS Fabric Interconnect Parent Switch LAN SAN + MDS N7000/ C6500 UCS IOM-FEX = Access Layer + UCS 6100 Distributed Modular System Cisco UCS VIC UCS IOM UCS IOM Distributed Modular System UCS VIC . . . UCS VIC 1 160 VM-FEX: Single Virtual-Physical Access Layer Collapse virtual and physical switching into a single access layer VM-FEX is a Virtual Line Card to the parent switch Parent switch maintains all management & configuration Virtual and Physical traffic treated the same App OS App OS App OS App OS App OS App OS App OS App OS App OS App OS App OS App OS

Extending FEX Architecture to the VMs
Cascading of Fabric Extenders Virtualized Deployment VM-FEX architecture LAN LAN Switch Switch Logical Switch Logical Switch Logical Switch Switch port extended over cascaded Fabric Extenders to the Virtual Machine FEX FEX Hypervisor Hypervisor vSwitch App OS VM-FEX App App App OS OS OS

Nexus 5000/2000 VM-FEX Distributed Modular System
Removing the Virtual Switching Infrastructure to a FEX Nexus 5500 Parent Switch LAN SAN + MDS N7000/ C6500 Nexus 2000 FEX = Access Layer + Nexus 5500 Distributed Modular System Cisco UCS VIC Nexus 2000 Nexus 2000 Distributed Modular System . . . UCS VIC UCS VIC 1 160 VM-FEX: Single Virtual-Physical Access Layer Collapse virtual and physical switching into a single access layer VM-FEX is a Virtual Line Card to the parent switch Parent switch maintains all management & configuration Virtual and Physical traffic treated the same App OS App OS App OS App OS App OS App OS App OS App OS App OS App OS App OS App OS

Nexus 5000 + Fabric Extender
Single Access Layer Nexus 5000 Parent Switch LAN SAN + = MDS Cisco Nexus® 2000 FEX N7000/ C6500 Distributed Modular System Access Layer N5000 Distributed Modular System N2232 . . . N2232 1 12 Distributed Modular System Nexus 2000 FEX is a Virtual Line Card to the Nexus 5000 Nexus 5000 maintains all management & configuration No Spanning Tree between FEX & Nexus 5000 Over 6000 production customers Over 5 million Nexus 2000 ports deployed

IEEE-802.1BR vs. IEEE802.1Qbg FEX based on IEEE 802.1BR
VM- FEX Logical Switch FEX Switch FEX based on IEEE 802.1BR VEPA based on IEEE 802.1Qbg Management complexity: each VEPA is an independent point of management Doesn’t support cascading Reflective Relay (used in basic VEPA) Vulnerable: ACLs based on source MAC (can be spoofed) Resource intensive: Hypervisor component consumes CPU cycles Inefficient bandwidth : separate copy of each Mcast and Bcast packets on the wire Ease of management: one switch manages all Port Extenders (adapters/switches/virtual interfaces) Supports cascading of Port Extenders (multi-tier, single point of management) Virtual Machine aware FEX Secure: ACLs based on VN-TAG Scalable: Mcast and Bcast replication performed in HW at line rate Efficient: no impact to server CPU Virtual Embedded Bridge (VEB) is an IEEE standard ! It’s a common terminology that involves the interaction between virtual switching environments in a hypervisor and the first layer of the physical switching infrastructure. IEEE-802.1Qbh Par is dead ! IEEE-802.1Qbh is alive on best way to become to be a standard, has been renamed to IEEE-802.1BR ! VEPA Standard ... VEPA is a HP proprietary implementation IEEE-802.1Qbg Standard ? Qbg like BR, are in Sponsor Ballot Phase, very close to be finished and published ! The EVB enhancements are following 2 different paths: 802.1Qbg and 802.1BR.The two proposals are parallel efforts, meaning that both will become standards and both are "optional" for any product being IEEE compliant. The standards are finished, be published early next Year.

Deployments of Cisco’s FEX Technology
Nexus 5000/5500/7000 + Nexus 2200 UCS 6100/ IOM 2k B22H with Nexus 5500 (HP) Server Rack Rack Server Rack FEX FEX Switch Chassis FEX Switch / FI Blade Server Blade Server Chassis 1 2 UCS 6100/ VIC 1 or 2 Nexus VIC P81E Blade/Rack Server Adapter OS FEX Adapter FEX Switch / FI 3 1 2 4 n Port 0 UCS 6100/ VIC 1 or 2 + VM Mgmt Link Nexus VIC P81E vCenter/VMM Management Plane Integration UCS Manager VM Host Hypervisor VM FEX VM-FEX Switch / FI RedHat KVM 4 1 2 n

VM-FEX Operations Model
Pre-Boot Configuration Step 1: Preboot UCS defined PCIe devices and enumerations Host discovers PCIe devices Hypervisor Hypervisor

VM-FEX Operational Model
Defining “Port Profiles” on the UCS or Nexus 5000 Port Profiles Definition Step 1: Preboot UCS defined PCIe devices and enumerations Host discovers PCIe devices Step 2: Port Profile Folder of Network Policy defined WEB Apps HR DB Compliance VLAN Web VLAN HR VLAN DB UCSM or Nexus 5500 VLAN Comp Hypervisor Hypervisor

Pushing Port Profiles to the Hypervisor System Step 1: Preboot UCS defined PCIe devices and enumerations Host discovers PCIe devices Step 2: Port Profile Folder of Network Policy on UCS or Nexus 5500 defined Step 3: Port Profile Export Port Profile name list exported to virtualization manager UCSM or Nexus 5500 exports Port Profiles VLAN Web Hypervisor Manager VLAN HR VLAN DB UCSM or Nexus 5500 VLAN Comp Hypervisor Hypervisor

Mapping of Port Profiles to VM Virtual Adapters Step 1: Preboot UCS defined PCIe devices and enumerations Host discovers PCIe devices Step 2: Port Profile Folder of Network Policy on UCS or Nexus 5500 defined Step 3: Port Profile Export Port Profile name list exported to virtualization manager Step 4: VM Definition Named Policy in VM VLAN Web Hypervisor Manager VLAN HR Network Manager VLAN DB VLAN Comp VM Hypervisor Hypervisor VM VM VM

Simplifying the Access Infrastructure Physical Network Unify the virtual and physical network Same Port Profiles for various hypervisors and bare metal servers Consistent functions, performance, management Hypervisor Hypervisor Virtual Network VETH VNIC VM VM VM VM VM VM VM VM VM-FEX Basics Fabric Extender for VMs Hypervisor vSwitch removed Each VM assigned a PCIe device Each VM gets a virtual port on physical switch VM-FEX: One Network Collapses virtual and physical switching layers Dramatically reduces network management points by eliminating per host vSwitch Virtual and Physical traffic treated the same Host CPU Cycles Relief Host CPU cycles relieved from VM switching I/O Throughput improvements Speaker notes VM-FEX: Even though Distributed virtual switch addresses some of the scaling concerns of virtual network (Eg – instead of having a network management point within each physical host you get to manage groups of physical hosts together in the Nexus 1Kv case upto 64 hosts), it adds a few new complications Virtual network management is opaque to the physical network management Features available in your physical network infrastructure may not be consistent with the features available in your virtual network infrastructure With VM-FEX collapses the physical and virtual network into “ONE” network. All VMs get assigned a PCIe device on the Virtual Interface Card. Virtual switching inside the hypervisor is removed <click> - The virtual port on the vSwitch (or vETH) that used to be inside the hypervisor now moves to the physical switch and the VM gets its own presence on the physical network using the vEth - Traffic between VMs on the same host need to go the physical switch. - VM-FEX eliminates the additional complexity introduced by virtual network Network management points are dramatically reduced – instead of managing a switch per host (or even one per 64 hosts in case of distributed switches) you manage a single switch that is responsible for both physical and virtual networking - Virtual and physical traffic is treated consistently. You don’t need 2 sets of policies for virtual and physical network The 2nd benefit from VM-FEX comes from the fact that switching is completely off-loaded to the physical switch’s ASIC. The Host CPU is relieved from this effort and we see improvements upto 12% for standard (emulated) mode and upto 30% for high performance (VMDirectPath or PCIe pass thru) mode. Lets talk about the different modes of VM-FEX now

Traffic Forwarding Physical Network Removing performance dependencies from VM location Offloading software switching functionalities from host CPU More on this in upcoming slides Hypervisor Hypervisor VETH VNIC VM VM VM VM VM VM VM VM

VM-FEX Modes of Operation
VMware ESX vSphere 5 Emulated Mode vEth dvNIC dvNIC VMDirectPath vSphere 5 vEth High Performance Mode Co-exists with Standard mode Bypasses Hypervisor layer ~30% improvement in I/O performance Appears as distributed virtual switch to hypervisor Currently supported with ESXi 5.0 + LiveMigration supported Emulated Mode Each VM gets a dedicated PCIe device Appears as distributed virtual switch to hypervisor LiveMigration supported A standard vSwitch or regular DVS is an emulated switch /device. We offer VM-FEX in emulated mode but we are not emulating the switch, we are just doing a pass through, a PCIe device that is discovered as VM-FEX interface just pass through the hypervisor. We have emulated mode since ESX4. Each VM get a dedicate PCI device, when we say a dedicate device what really mean each VM-FEX interface would a dedicate hardware resource including queue counts, interrupt number and ring size. The VM-FEX emulate mode would present itself as stand DVS to ESXi but really it is just doing PCI mapping Customer who want the performance boost would get the benefit from DirectPath I/O, however there is a trade off and the trade off is those VM would loss all hypervisor feature such as DRS, FT or even basic as VMotion and become isolated. With VM-FEX technology, we are the only vendor that support vmotion with DirectPath I/O by simply allow virtual machine to go from bypass mode to emulated mode without service interruption and allow hypervisor to complete vmotion process All the vNIC we talk about here is dynamic vNIC Standard Mode: In the standard the mode each VM is associated one to one with a PCIe device presented by the VIC. All VM traffic is sent to the upstream Fabric Interconnect for switching In our tests performance improvement over industry standard NICs was between 12%-15% From a functionality perspective – VM-FEX presents itself as a distributed network solution to the hypervisor and integrates with it seamlessly – standard functionality such as vMotion are all supported while enjoying the performance improvement benefits <click> High performance Mode: In the high performance mode the hypervisor is completely bypassed and VM I/O is directly placed on the VIC’s descriptors – this is possible since the VIC’s descriptors are a perfect match to the descriptors of the emulated NIC in the VM. The performance benefit comes primarily from the fact that one extra memory copy is eliminated – Typically buffers are copied from the VM’s memory to the hypervisor and finally placed on the physcial NICs descriptors In VMDirectPath mode – the buffers are placed directly to the VIC’s descriptors thereby eliminating the memory copy in the hypervisor layer In our testing the I/O performance improvements are close to 30% and CPU performance improvements are upto 50% for I/O bound applications. Again from a functionality perspective this mode appears as a first class distributed virtual switch to the hypervisor and with ESX 4.1 U1 VMware is enabling vMotion with VMDirectPath – thereby enabling dynamic workload management.

VMDirectPath: How is Works
Dynamic VIC device Config: Used by PCI mgmt layer Mgmt Bar: Used by Vmkernel and PTS Data Bar : Vmxnet3 compliant Rings and Registers Vmxnet3 OS PCI subsystem Control Path Emulation <-> PT transitions Port Events PCIe events VM Vmxnet3 Driver Ethernet Device Driver Cisco DVS Data Path There are several components I would like to introduce under VMware VM-FEX architecture. Leverage ESX native driver – VMXNET3 and DirectPath I/O feature with user configurable parameters such as RQ/TQ number, available interrupt and ring size. Depends on the type of Guest OS and its configuration such as enable RSS or not, will have different recommendation. A dedicate Cisco DVS will be installed on each ESX host through the form of vib file and its main purpose is to overwrite the default DVS function and enable the VM traffic to bypass hypervisor stack. As mentioned in previous slides, in emulation mode, VM traffic would still pass-through the hypervisor stack but offload the policy management piece to the bridge controller hardware. Whereas, in bypass mode, the VM traffic complete bypass the hypervisor stack and eliminate additional layer of memory copy. - Dynamic vNIC device is configured through UCSM Service Profile and present to the host OS as individual PCI device, however only the virtual machine interface would leave dynamic interface

UCS VM-FEX Modes of Operation
Windows Hyper-V & Red Hat KVM with SR-IOV dvNIC Hyper-V 2012 vEth VF SvNIC PF Hyper-V 2012 vEth dvNIC VF PF SvNIC Emulated Mode Hypervisor Bypass High Performance Mode Co-exists with Standard mode Bypasses Hypervisor layer ~30% improvement in I/O performance Appears as a Virtual Function to Guest OS Currently supported through SR-IOV with Hyper-V 2012 & RHEL KVM 6.3 Live Migration supported Emulated Mode Each VM gets a dedicated PCIe device Appears as a Virtual Function to Guest OS LiveMigration supported Start with Windows Server 2012 and Red Hat 6.3, we also support both emulate mode and hypervisor bypass mode with VM-FEX deployment. However the architecture is different, VM-FEX interface is no longer present itself as DVS but as a virtual function within the Guest OS. We leverage the industry SR-IOV stand to implement hyperv and redhat VM-FEX and our goal is the same, to achieve hypervisor pass for VM traffic with low packet latency and host CPU utilization but preserve the hypervisor functionality such Live Migration. The same methodology is used by converting back to emulated mode to complete Live Migration process and promoted back hypervisor bypass mode without service interruption. SR-IOV Primer (Delmar Functionality) SR-IOV – PCI-SIG standard that allows creation of lightweight PCI devices (VF – virtual functions) “under” a parent PCI device (PF – physical function). HyperV, KVM support SR-IOV. ESX does not. A VF is associated to a VM VNIC. PF is the host uplink. Intel VT-d provides hardware assisted techniques to allow direct DME transfers to and from VM (hypervisor bypass). VM-FEX extends VM VNIC connectivity to the FI via the VIC protocol.

Hyper-V Host Hyper-V Host
SR-IOV: How is Works 4/6/2017 2:13 PM Hyper-V Host Hyper-V Host Root Partition Virtual Machine Root Partition Virtual Machine Hyper-V Switch Hyper-V Switch Virtual Function Virtual NIC Switching VLAN Filtering Data Copy Switching VLAN Filtering Data Copy VMBUS SR-IOV Physical Function Physical NIC Network I/O path without SRIOV Network I/O path with SRIOV In the traditional network data path, when a packet arrive in the underlying physical adapter, it would then do a memory copy into the Virtual Machine Manager (VMM) in this case, is the hyperV and hyperV vSwitch and based on the destination information such as MAC address or VLAN, the VMM or hypervisor would then do another memory copy and place data packet in the proper virtual machine queue for further process. As you could see, a lot of host resource is wasted through this double memory copying. In SR-IOV technology, we introduce two components, - Physical Functions (PFs): These are full PCIe devices that include the SR-IOV capabilities. PFs are discovered, managed and configured as normal PCIe devices which is configured as Static VNIC in UCSM - Virtual Functions (VFs): These are light-weight PCIe functions that contain minimal and dedicated resources necessary for data movement, such as queues and interrupts. A set of VFs are managed by a given PF. One or more VFs can be assigned to a Virtual Machine which is configured as dynamic VNIC in UCSM. With paring of PF and VF, now we move the virtual machine queue to the physical SR-IOV capable adapter such Palo or VIC1280 and with Intel Virtualization direct I/O technology, each date packet would now directly transfer or copy from the SRIOV enable physical adapter into the virtual machine adapter. The performance difference comes from eliminating the additional memory copy with Virtual machine manager / hypervisor to achieve higher I/O throughput, high CPU utilization and lower data latency. Microsoft Confidential

Live Migration with Hypervisor Bypass Temporary transition from to standard I/O Hypervisor Live Migration to secondary host vSphere 4 Hypervisor vNIC vNIC vNIC vEth vEth 1 sec silent period vEth - Always use Live Migration VM Sending TCP stream (1500MTU) UCS B200 M2 blades with UCS VIC card

VM-FEX Modes of Operation
Enumeration vs. Hypervisor Bypass VM-FEX Mode VMware Hyper-V KVM Emulation Pass Through (PTS) Hyper-V Switch SR-IOV with MacVTap Hypervisor Version ESX 4.0 U1 + Window Server 2012 RHLE 6.2 + UCSM Version 1.4 + 2.1 VMotion / Live Migration Support VM-FEX Mode VMware Hyper-V KVM Hypervisor Bypass SR-IOV SR-IOV with PCI Passthrough Hypervisor Version ESX 5.0 + Window Server 2012 RHEL 6.3 UCSM Version 1.4 + 2.1 VMotion / Live Migration Support N/A

ESX Kernel Module / Libvirt / HyperV Extendable Switch
UCS VM-FEX System View Deploying on a UCS B or C Series Infrastructure UCS Fabric Interconnect A (port profiles) UCS Fabric Interconnect B (port profiles) Fiber Channel Uplink Ports Fiber Channel Uplink Ports 7 8 5 6 1 2 1 2 5 6 7 8 Mgmt Uplink veth10 veth1 veth2 veth3 veth4 vfc0 Virtual Interface Control Logic Virtual Interface Control Logic vfc1 veth1 veth2 veth3 veth4 veth10 VN 10Gbe Mgmt Uplink Internal Connections 1 2 3 4 5 6 1 2 3 4 5 6 UCS 6x00 Physical Ports UCS 6x00 Physical Ports Chassis IOM Ports Chassis IOM Ports 1 2 3 4 1 2 3 4 Chassis IO Module A Chassis IO Module B Server Ports VM-FEX 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 Server Ports Cisco Adapter VIC CPU 1 UCS B or C Series Server CIMC KVM etc. vHBA0 vNIC1(s) vNIC2(s) vHBA1 HBA 0 ESX Kernel Module / Libvirt / HyperV Extendable Switch HBA 1 For this slide, we are trying to show a system view of how individual component work and connect together to realize VM-FEX deployment. Apologize for this busy slides, I am actually not that artistic and inherent this slides from my boss. At the very bottom is the server view before implement VM-FEX, there is bare metal component including the adapter and hypervisor component which could ESX, Libvirt for KVM virtualziation library and Hyper-V extension switch. Next layer up, we have the IO Module where the VNTag comes into play between Fabric Interconnect and individual server adapter. At Fabric Interconnect Level, user configure desired number of static vNIC for Host OS and dynamic for VM interface through service profile. During server association and boot process, Fabric Interconnect instructs the adapter processor to enumerate the corresponding interface configuration through VIC protocol on internal connection. vHBA interface could be also generate through the same process to enable FCoE capability.

ESX Kernel Module / Libvirt / HyperV Extendable Switch
UCS VM-FEX System View Deploying on a UCS B or C Series Infrastructure UCS Fabric Interconnect A (port profiles) UCS Fabric Interconnect B (port profiles) Fiber Channel Uplink Ports Fiber Channel Uplink Ports 7 8 5 6 1 2 1 2 5 6 7 8 Mgmt Uplink veth10 veth1 veth2 veth3 veth4 vfc0 Virtual Interface Control Logic Virtual Interface Control Logic vfc1 veth1 veth2 veth3 veth4 veth10 VN 10Gbe Mgmt Uplink Internal Connections 1 2 3 4 5 6 1 2 3 4 5 6 UCS 6x00 Physical Ports UCS 6x00 Physical Ports Chassis IOM Ports Chassis IOM Ports 1 2 3 4 1 2 3 4 Chassis IO Module A Chassis IO Module B Server Ports VM-FEX 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 Server Ports Cisco Adapter VIC CPU 1 UCS B or C Series Server CIMC KVM etc. vHBA0 vNIC1(s) d-vNIC1 d-vNIC2 d-vNIC3 d-vNIC4 vNIC2(s) vHBA1 HBA 0 ESX Kernel Module / Libvirt / HyperV Extendable Switch HBA 1 One thing to note is that dynamic vNIC is slightly different from static vNIC in terms of fail over function. For static vNIC, failover between fabrics is optional but for dynamic vNIC fabric failover is requirement since we want to guarantee the policy consistency for the given virtual machine no mater where it moves within the UCS fabric. - Moreover, the pin group feature is also available to virtual machine with VM-FEXX, it is defined in the port profile that attach to the dynamic interface and you could traffic engineer the VM traffic and pin it to the specific boarder port on fabric interconnect to external network no matter where it Live Migrate within the domain Also we only shows one to one matching between Virtual Machine and dynamic interface, in fact you could have multiple dynamic vNIC assigned to a single VM and configure the VM-FEX mode of operation individually. Kernel Service Console

VM-FEX Scalability Number of VIF Supported per Hypervisor
Cisco UCS 6100 / 6200 Series Hypervisor Half-Width Blade with Single VIC Full-Width Blade with Dual VIC ESX 4.0 – 4.1 (DirectPath I/O) 56 (54 vNIC + 2 vHBA) ESXi 5.0 – 5.1 (DirectPath I/O) 116 (114 vNIC + 2 vHBA) 116 (114 vNIC + 2 vHBA)* Windows 2012 (SR-IOV) 232 (228 vNIC + 4 vHBA) KVM 6.1 – 6.3 (SR-IOV) * Additional VIC will NOT increase the total VIF count due to OS limitation * Multiple VIC is Supported for full width blade and B200M3 Nexus Series Hypervisor Adapter FEX VM-FEX ESX 4.1 – ESXi 5.1 96 vNIC This slides provides the basic scalability information for VM-FEX deployment. These number are not the software / hardware limitation but more as QA testing effort. As newer UCSM version release, we are looking to increase the scalability matrix accordingly. Go through slides * Only one VIC (P81E/VIC1225) is Supported for each C series rack server

VM-FEX Advantage Simpler Deployments Robustness Performance Security
Unifying the virtual and physical network Consistency in functionality, performance and management Robustness Programmability of the infrastructure Troubleshooting, traffic engineering virtual and physical together Performance Near bare metal I/O performance Improve jitter, latency, throughput and CPU utilization Security Centralized security policy enforcement at controller bridge Visibility down to VM to VM traffic VM-FEX Advantage comparing to virtualized switch layer So why we develop VM-FEX technology, obviously, Performance is main driver, however we are not just boost the VM throughput to bare metal level but in the same also improve packet jitter, latency and low host CPU utilization. With VM-FEX, you also have a programmable access layer which no longer depends on the physical port, you could decide the size of the DVS and how individual VM utilize these access ports through port profile definition. Lastly being able to have visibility for VM to VM traffic is another security advantage when deploy VM-FEX Also being able to manage your virtual infrastructure as physical one is another advantage and offer a consistent functionality, performance and policy management.

VM-FEX General Baseline on UCS

UCS General Baseline #1: Dynamic vNICs Policy
Setting a Dynamic Adapter Policy Up Policies are to automatically provision dynamics on Servers Dependent on the number of Fabric Interconnect to IO Module connections (# IOM to FI links * 63) - 2 for Gen 2 Hardware (62xx, 22xx and VIC12xx) (# IOM to FI links * 15) - 2 for Gen 1 Hardware (61xx, 21xx and Palo) In the next couple slides, we are going walk through several VM-FEX configuration baseline which is independent of the hypervisor, with the understanding of baseline setting, we will then diving into hypervisor specific configuration for more details. In order to get the benefits of VM-FEX, there are some trade off that we need to make and one of it is scalability. Comparing to software switching, there is a limit of hardware resource that is available for VM-FEX within single UCS domain. The number of available dynamic vNIC per server is one of it. As shown in this slides, based on the number of physical links between FI and IOM and the generation of hardware, we layout formula at your reference to calculate the interface. Basically for generation 1 hardware, the maximum dynamic interface you could have is 58 and in generate 2 is 502 which actually exceeds the current support matrix of 116 per adapter and 232 per server.

UCS General Baseline #1: Creating Dynamic vNICs Fabric Interconnect VIF calculation
2 3 4 5 6 In port-channel mode, when you cable between FEX and the FI, the available virtual interface (VIF) namespace varies, depending on where the uplinks are connected to the FI ports:When port-channel uplinks from the FEX are connected only within a set of eight ports managed by a single chip, Cisco UCS Manager maximizes the number of VIFs used in service profiles deployed on the servers. If uplink connections are distributed across ports managed by separate chips, the VIF count is decreased. For example, if you connect seven members of the port channel to ports 1–7, but the eighth member to port 9, this port channel can only support VIFs as though it had one member.

UCS General Baseline #2: Building Service Profile
Adding the Dynamic Policy and Static Adapters 2 Statics – 1 to each UCS Fabric Change dynamic vNIC connection policy to setup dynamics Additionally you would also noticed that there is no mac address association to dynamic vNIC during service profile configuration and the reason is that each dynamic vNIC would get assigned dedicated hardware resource but present as a vellina interface to hypervisor. The virtual machine manager such as vCenter would actually provide the mac address during VM creation

UCS General Baseline #2: Building Service Profile
Static and Dynamic Adapter Policy Adapter Policy Static vNIC Dynamic vNIC VMware ESXi VMware VMwarePassThru Window Hyper-V SR-IOV Windows RedHat KVM Linux The location and the adapter policy for dynamic vNIC are vary based the hypervisor. For VMware scenario, there is a parallel relationship between static and dynamic vNIC. Whereas, the PF and VF relationship would be form in service profile for Hyper-V and KVM deployment

UCS General Baseline #3: Building Port Profiles
Creating Folders of Network Access Attributes Creating Port Profiles Includes: VLAN(s) Native and/or Tagging allowed QoS Weights and Flow Rates Upstream Ports to always use As we discuss earlier, Port Profile is a logical container for your network and security policy, in this slides shows some of the feature that you could configure within a profile include what are vlan and vlans that this virtual machine would have access, do we enable CDP, what is the QoS policy, line rate or rate limited for bandwidth, what is the maximum port that could be associated to this port profile and last but the least, how do you want to traffic engineer the traffic

UCS General Baseline #4: Building Port Profiles
Enhanced Options like VMDirectPath with VM-FEX Selecting High Performance will only Impact VMware deployment today No problem if selected and used on other hypervisors We only toggle the Host Network I/O Performance for Vmware deployment today. Port Profile with high performance enable will operate in bypass mode or otherwise in emulate mode

UCS General Baseline #5: Communication with Manager
Establishing Communication to Hypervisor Manager Tool discussed later to simplify the whole integration process Once we have port profile and network infrastructure lays out in UCSM, we need to find a way to propagate these information to virtual machine manager. For VMware in particular, we currently have a build-in wizard that guide you through the integration step. If you familiar or have experience with nexus 1000v, we use the mechanism to build a secure communication between UCSM and vCeter through the export of UCSM extension file, which is in XML format and would register in vCenter as a plug-in

UCS General Baseline #5: Communication with Manager
6100 6200 Hypervisor Manager DVS ESXi 1 vCenter per UCS doman 1 DVS per vCenter 4 DVS per vCenter Hyper-V 1 MMC Instance 5 DVS per MMC Instance 5 DVS per MMC Instance KVM N/A 1 DVS per UCS Domain Port Profile per UCS Domain 512 Dynamic Ports per port profile 4096 Dynamic Ports per DVS This slides summarize the current supporting matrix for various Virtual machine manager. These number are by no means hardware / software limitation but more for QA testing effort with current UCSM release. We understand there is a high demand regarding to cluster communication between UCSM and VMM, for example we would like to have multiple VC instances connect a single UCSM and leverage the same set of port profile across your vmware infrastructure or you would have multiple UCSM connect to a single VC and vmotion across UCSM domain. We support both scenarios today and we have customer actively deploy both configurations in their production environment today. If you have a specific need for cluster communication, please reach out to me after session we could talk more in details on how we achieve and support it.

UCS General Baseline #6: Publishing Port Profiles
Exporting Port Profiles to these to Hypervisor Manager Publish Port Profiles to Hypervisors and virtual switches within

VM-FEX Implementation with VMware on UCS

VMware VM-FEX: Infrastructure Requirements
Versions, Licenses, etc. VMware VM-FEX is available B series, integrated and standalone C series Each VIC card supports up to 116 Virtual Machine Interface (OS limitation) Enterprise Plus License required (as is for any DVS) on Host Standard License and above is required for vCenter Hypervisor features are supported for both emulated and hypervisor bypass mode vMotion, HA, DRS, Snapshot and Hot add/remove virtual device, Suspend/Resume VMDirectPath (Hypervisor Bypass) with VM-FEX is supported with ESXi 5.0+ VM-FEX upgrade is supported from ESXi4.x to ESXi5.x with Customized ISO and VMware Update Manger

VM-FEX and VMware SR-IOV Comparison
VM-FEX is the hypervisor bypass solution with vMotion capability VMware SR-IOV is incompatible with hypervisor features including vMotion, HA, DRS … VM-FEX has the highest Virtual Machine interface per host With UCSM 2.1 release, each ESXi host support up to 116 VM interface With ESX5.1, SR-IOV supports up to 64 VF with Emulex and 41VF with Intel adapter VM-FEX is available on both UCS blade and rack severs Blade Server, Integrated rack server with UCSM and Standalone rack server with Nexus 5500 With ESX5.1, SR-IOV is only available on PCIe adapter and standalone rack server VM-FEX enable centralized network management and policy enforcement Network policy is configured as port profile in UCSM / N5K and push to vCenter as network label Clean separation between Network and Server responsibility VM-FEX Configuration is fully automated – Easy VM-FEX tool VM-FEX provides inter VMs traffic visibility through network tool before we get into the configure workflow, I would like to spend some time and talk about the SR-IOV feature with ESX5.1 release. Just like other hypervisor vendors who implement SR-IOV, the goal is to provide hardware like performance into your virtual machine. Secondly, the whole idea of manage your virtual switch port as physical one and coverage both virtual and physical infrastructure does not exist with standalone SR-IOV. There is no concept of port profile and bridge controller and there is no grantee on policy consistent throughout your infrastructure. There are also some SR-IOV limitation regarding to scale number and support form factor but we expect these to change with future ESX release. However there are some cavets I would like to point out with Vmware SR-IOV. First and foremost, as of today, SR-IOV is incompatible with majority of the hypervisor features including basic function such as vmotion, HA and DRS. The moment you enable SR-IOV, your VM would be isolated on the hypervisor since it would not able to vmotion around. You would have to make a choice between performance or hypervisor features but with VM-FEX you would enjoy both.

VMware VM-FEX Configuration Workflow
1. Configure Service Profile and adapter 4. Install Cisco VEM software bundle and plugin in vCenter VMW ESX Server vCenter UCS DVS (PTS) VM #1 #4 #3 #2 #5 #8 #7 #6 UCS exports Port Profiles to VC UCS administrator Network Administrator 2. Creating Port Profile and Cluster in UCSM 5. Add ESX host into DVS cluster in vCenter 3. Configure USCM VMware Integration wizard 6. Configure Virtual Machine setting to enable hypervisor bypass 7. Verify VM-FEX configuration in both UCSM and vCenter Up to this point, we have touched based on a lot VM-FEX feature and individual steps. Lets connect the dots together and see what is complete stream of workflow looks like. The workflow could be divided into two parts. First is UCSM side of configuration which generally fall under network administration and second VC configuration which is responsible by server team. The entire workflow is a collaboration between network and server team but with define task for each. Throughout these 7 steps of workflow, only step 2 – configure port profile and step 6 attaching port profile are day to day operation for the administrator, the remain steps are only a execute during step up. - Use service profile template People know what is bring up and day to day operation 54

VMware VM-FEX Demo Topology
Server Cisco UCS VM-FEX VMware ESX 5.1 Vware ESX 5.1 Cisco UCS Fabric Interconnect Virtual Ethernet ports (vEth) DirectPath I/O Active VMXNET 3 Adapter VM-FEX NTTTCP Sender vSwitch NTTTCP Sender VM-FEX NTTTCP Receiver vSwitch NTTTCP Receiver 55

VMware VM-FEX Demo

VMware VM-FEX Best Practice
Pre-provision the number of dynamic vNIC for future usage Changing the quantity and the adapter policy require server reboot Select “High Performance Mode” in Port Profile to enable hypervisor bypass Utilize ESX Native VMXNET3 Driver User configurable parameter including queue, interrupt, ring size through policy Recommend to have Num (vCPU) = Num (TQ) = Num(RQ) to enable DirectPath I/O Other consideration to deploy VM-FEX ESX heap memory size : MTU size ESX available interrupt vectors : Guest OS and adapter policy Dedicated spreadsheet for VM-FEX calculation and sizing

Easy VM-FEX Tool VMware solution only today with UCS
Quick System Bringups Assumption of 1 management interface per ESX host Optional vMotion / FT logging also handled All supported versions of VMware that VM-FEX supports Enterprise+ or Evaluation Can define some defaults in text file Server needs Dynamic vNICs on Service Profile (will check) Deployment name limited to 8 characters in tool UCSM respository for ESX kernel model, or separate tool to pull from VMware online to a dedicated directory locally

VM-FEX Implementation with Nexus 5K

ESX Kernel Pass Through Module
Nexus VM-FEX System View Nexus VM-FEX System View UCS VM-FEX System View Deploying on a UCS C Series with Nexus 5500 Infrastructure Deploying on a UCS C Series with Nexus 5500 Infrastructure Nexus 55xx A (port profiles) Nexus 55xx B (port profiles) Fiber Channel Uplink Ports Fiber Channel Uplink Ports 7 8 1 2 1 2 7 8 Mgmt Uplink 47 47 veth10 veth1 veth2 veth3 veth4 vfc0 Virtual Interface Control Logic 48 48 Virtual Interface Control Logic vfc1 veth1 veth2 veth3 veth4 veth10 VN 10Gbe Mgmt Uplink Internal Connections 1 2 3 4 5 6 1 2 3 4 5 6 Nexus 55xx Physical Ports vPC Connections Nexus 55xx Physical Ports 2232 Fabric Ports 2232 Fabric Ports 1 2 8 1 2 8 2232 FEX A 2232 FEX B VM-FEX 1 2 3 4 5 6 32 1 2 3 4 5 6 32 2232 Server Ports 2232 Server Ports Cisco Adapter VIC CPU 1 UCS C Series Server CIMC KVM etc. vHBA0 vNIC1(s) vNIC2(s) vHBA1 HBA 0 ESX Kernel Pass Through Module HBA 1 Nexus VM-FEX follows the same fashion as UCS. The minor difference is that Nexus utilize the vPC peer link to synchronize the virtual Ethernet and port profile database across Nexus 5500 cluster rather than the L1/L2 links on fabric interconnect. The vPC domain and peer links are purely for database synchronization and not actively participate in forwarding virtual machine traffic. The virtual interface control logic is now on Nexus 5500 but the mechanism in terms of VN Tage and VIC control protocol works exactly the same as UCS.

ESX Kernel Pass Through Module
Nexus VM-FEX System View UCS VM-FEX System View Deploying on a UCS C Series with Nexus 5500 Infrastructure Nexus 55xx A (port profiles) Nexus 55xx B (port profiles) Fiber Channel Uplink Ports Fiber Channel Uplink Ports 7 8 5 6 1 2 1 2 5 6 7 8 Mgmt Uplink 47 47 veth10 veth1 veth2 veth3 veth4 vfc0 Virtual Interface Control Logic 48 48 Virtual Interface Control Logic vfc1 veth1 veth2 veth3 veth4 veth10 VN 10Gbe Internal Connections 1 2 3 4 5 6 1 2 3 4 5 6 Mgmt Uplink Nexus 55xx Physical Ports vPC Connections (veth’s not a vPC at FCS) Nexus 55xx Physical Ports 2232 Fabric Ports 2232 Fabric Ports 1 2 8 1 2 8 2232 FEX A 2232 FEX B VM-FEX 1 2 3 4 5 6 32 1 2 3 4 5 6 32 2232 Server Ports 2232 Server Ports Cisco Adapter VIC CPU 1 UCS C Series Server CIMC KVM etc. vHBA0 vNIC1(s) d-vNIC1 d-vNIC2 d-vNIC3 d-vNIC4 vNIC2(s) vHBA1 HBA 0 ESX Kernel Pass Through Module HBA 1 Kernel Service Console

Nexus 5500 VM-FEX Demo Topology
Nexus 5548/C22 M3/ VIC1225/ESXi 5.1 Nexus 5548-A is pre-configured, focus on B with the same configuration in the demo Uplink Redundancy – Each static vNIC configure as Active/Standby No need for OS teaming software Required for hypervisor uplink Each dynamic vNIC attach to uplink in Round Robin fashion vPC Doman and Peer Link is configured to synchronize veth numbering for the same VM Not used for the forwarding plane vEth vEth vEth vEth vEth vEth Nexus 5548 vPC Peer Link Fabric Extenders VIC 1225 Port 1 Port 2 C22 M3 vNIC 1 CH1 vNIC 2 CH2 dNIC 1 dNIC 2 dNIC 3 ESXi 5.1 VM 1 vNIC 0 VM 2 vNIC 0 VM 3 vNIC 0 62

Nexus 5500 VM-FEX Configuration Workflow
6. Download Extension and register plugin 8. Add Sever into DVS cluster 9. VM created and attach port profile Server administrator Nexus 5500 0. Set Up Connection 2. Install N5K Feature License 3. Configure Static Binding interface and enable VNTag on host interfaces 4. Configure Port Profile 5. Configure DVS & Extension 10. Verify VM-FEX status Network Administrator Cisco Adapter 1. VM vNICs provisioned and VNTag Mode enable 7. Install VEM on ESXi host UCS C-series CIMC Server administrator The Nexus VM-FEX work flow follow pretty much the same fashion as UCS including Instead of just between UCSM and VC, it would be three components involved including both Nexus 5500 and CIMC. Since Nexus5500 does not have the capability to manage C series racks server directly, we would connect directly to the server CMIC and provision the dynamic policy Think of CIMC as a strip down version of UCSM for individual rack server, instead of configure -You would need to enable VM-FEX license, the license itself is free and include within the system but VM-FEX feature is no enable by default. - We would also need to enable VNTag capability on the host interface. By default Step 1. Enable NIV (Network Interface Virtualization) on the P81E Adapter in the CIMC CIMC is the management Interface for the Cisco C2xx servers Choose the number of dynamics to configure (next slide) Setting a Group of Dynamics on C2xx Servers Step 2. Enable vNICs then view the VM-FEXs Tab in the CIMC UCS Standalone CIMC version 1.4 or greater required Minimum of 2 static vNICs defined Numbers of VM FEX’s (dynamic vNICs) are dependent on links from 5500 to 2232 if using FEX (Limit at 96 today) Nexus 5500 version 5.1(3)N1(1) or later required Step 3. Configuring a static profile for the fixed interfaces One to each N5k in a pair Select the vNIC, and the port profile to assign to it These will be initially in the vSwitch in the out of box VMware configuration Step 4. Downloading and registration of plug-in per the other VM-FEX topologies 63

Nexus 5K VM-FEX Demo

Nexus 5500 VM-FEX Best Practice
VM-FEX supports single N5k topology, vPC Topology is recommended In vPC Topology, ensure both N5k have the same port-profile configuration In vPC Topology, need to configure the same SVS connection on both N5k but only Primary switch has active connection to vCenter When secondary switch takeover primary role, seamlessly activate the connection to vCenter Enable “vethernet auto-create” feature Automatically create vEth port for dynamic vNIC during server boot up Auto created vEth are numbered > = 32769 DirectPath I/O is active with “high-performance host-netio” cmd in port profile

VM-FEX Implementation with Hyper-V on UCS

Hyper-V Scale Comparison
4/6/2017 2:13 PM Vmware vSphere 5 .1 Windows Server 2008 R2 Windows Server 2012 HW Logical Processor Support 160 LPs 64 LPs 320 LPs Physical Memory Support 2 TB 1 TB 4 TB Cluster Scale 32 Nodes up to 4000VMs 16 Nodes up to 1000 VMs 64 Nodes up to 4000 VMs Virtual Machine Processor Support Up to 64 VPs Up to 4 VPs VM Memory Up to 1 TB Up to 64 GB Live Migration Concurrent vMotion 128 per datastore Yes, one at a time Yes, with no limits. As many as hardware will allow. Live Storage Migration Concurrent Storage vMotion 8 per datastore, 2 per host No. Quick Storage Migration via SCVMM Servers in a Cluster 32 16 64 VP:LP Ratio 8:1 8:1 for Server 12:1 for Client (VDI) No limits. As many as hardware will allow. Microsoft Confidential

Hyper-V Host Hyper-V Host
SR-IOV Overview 4/6/2017 2:13 PM Hyper-V Host Hyper-V Host Root Partition Virtual Machine Root Partition Virtual Machine Hyper-V Switch Hyper-V Switch Virtual Function Virtual NIC Switching VLAN Filtering Data Copy Switching VLAN Filtering Data Copy VMBUS SR-IOV Physical Function Physical NIC Network I/O path without SRIOV Network I/O path with SRIOV In the traditional network data path, when a packet arrive in the underlying physical adapter, it would then do a memory copy into the Virtual Machine Manager (VMM) in this case, is the hyperV and hyperV vSwitch and based on the destination information such as MAC address or VLAN, the VMM or hypervisor would then do another memory copy and place data packet in the proper virtual machine queue for further process. However with SR-IOV technology, we introduce two components, - Physical Functions (PFs): These are full PCIe devices that include the SR-IOV capabilities. PFs are discovered, managed and configured as normal PCIe devices which is configured as Static VNIC in UCSM - Virtual Functions (VFs): These are light-weight PCIe functions that contain minimal and dedicated resources necessary for data movement, such as queues and interrupts. A set of VFs are managed by a given PF. One or more VFs can be assigned to a Virtual Machine which is configured as dynamic VNIC in UCSM. With paring of PF and VF, now we move the virtual machine queue to the physical SR-IOV capable adapter such Palo or VIC1280 and with Intel Virtualization direct I/O technology, each date packet would now directly transfer or copy from the SRIOV enable physical adapter into the virtual machine adapter. The performance difference comes from eliminating the additional memory copy with Virtual machine manager / hypervisor to achieve higher I/O throughput, high CPU utilization and lower data latency. Microsoft Confidential

Hyper-V Extensible Switch Architecture
4/6/2017 2:13 PM Hyper-V extensible switch architecture is an open API model that enhance vSwtich feature Three types of extension is defined by Hyper-V Capture Extension Filtering Extension (Window Filtering Platform) Forwarding Extension (VM-FEX) Multiple extension is allowed Still need to verify with Vendor for compatibility Several extension is incompatible with WFP Extension state is unique for each vSwitch Leverage SCVMM to centrally configure extension Cisco also provides both PF and VF Drivers Virtual Machine Virtual Machine Root Partition VM NIC Host NIC VM NIC VF Driver VF Driver Hyper-V Switch Extension Miniport Extension Protocol Capture Extensions Filtering Extensions WFP Extensions Forwarding Extension VM-FEX Forwarding Extension In the previous slide we mentions about the VM-FEX forwarding extension. HyperV has introduced several types of switch extension as plugin to vSwitch, comparing to Vmware where we have to install a VEM (virtual Ethernet module) and completely replacing the vSwtich /DVS functionality. The HyperV extension architecture is an open API model that enhance vSwtich feature but not replacing/remove the underlying switch function. It is a more flexible architecture and different type of extension could plug into the same vSwitch. Extension state/configuration is unique to each instance switch, in this case we would need to install VM-FEX forwarding extension on each participate vSwitch. Several extension type includes, Capture extensions can inspect traffic and generate new traffic for report purposes, but cannot modify traffic, could have multiple capture extensions on single vSwitch Filtering Extensions can inspect, drop, modify, and insert packets but can’t direct the packet, for example Windows Filter Platform is an example Forwarding extensions could direct traffic, defining the destination(s) of each packet, Forwarding extensions can capture and filter traffic but only one forwarding extension per vSwitch Extensible, not replaceable - Added features don’t remove other features Pluggable switch - Extensions process all network traffic, including VM-to-VM 1st class citizen of system - Live Migration and offloads just work; Extensions work together Open & public API model - Large ecosystem of extensions Logo certification and rich OS framework - High quality extensions Physical NIC PF Driver Microsoft Confidential

SCVMM Management of Switch Extensions
UCS Virtualization VF Driver Root Partition SCVMM Server Capture Extension VMM Service VMM Agent Filtering Extension UCSM Provider Plugin Forwarding Extension VM-FEX Forwarding Extension UCS Manager API Physical NIC UCS VIC PF Driver - Slide is too busy Install Cisco Provider DLL on SCVMM Host Install Provider DLL Setup Registry Keys Create VSEM (Virtual Switch Extension Manager) Assign one or more Host Groups Specify a connect-string : example Upon successful connection, it will fetch FND, FN, UP-Link PP, VPP, IP-Pools etc. from UCS (Apache/LUA , NSM component). Create Logical Switch Associate to a VSE Create Native VPP (alias to VPP) Create Native UPP (alias to UPP) Service Profile Network and Security Policy database

Hyper-V VM-FEX: Infrastructure Requirements
VM-FEX is available for Hyper-V on UCS B series and Integrated C series Standalone C series support is on the road map Each VIC card supports up to 116 Virtual Machine Interface Install additional VIC will double VM interface (B200M3 with 2 VIC -> 232 per host) Windows Server 2012 is required for both Host and Guest OS (same level of Kernel) Do Not Support Windows Server Core and Hyper-V standalone server VM-FEX with Live Migration fully supported Various options for share storage – Failover Cluster, SMB, share nothing storage Full PowerShell library support for automation PowerShell Commandlet for UCSM, Hyper-V and SCVMM

Hyper-V VM-FEX: Infrastructure Requirements
Support Two Management Approaches Microsoft Management Console (MMS) for Standalone Hyper-V deployment System Center Virtual Machine Manager (SCVMM SP1) for Integrated Hyper-V deployment UCS Manager full Integration with Systems Center Virtual Machine Manager (SCVMM) SP1 Expect to release with UCSM 2.2 UCS Provider Plugin includes VM-FEX switch forwarding extension SCVMM use UCSM Provide Plugin to pull information from UCSM Cisco provides both VIC drivers and VM-FEX switch forwarding extension VIC Utility Tool is provided (MSI) The same Windows Driver for both Physical Function (Host) and Virtual Function (Guest)

Hyper-V VM-FEX: SCVMM Network Definition
HOST GROUP: SJ HOST GROUP: NYC Fabric Network (FN) – A network abstraction representing a logical network composed of network segments (VLANs) spanning across multiple sites Fabric Network Definition (FND) – A network abstraction composed of site-specific network segments VM Network Definition (VMND) – A sub-network abstraction composed of a single network segment (and an IP pool) at a specific site VM Network (VMN) – A sub-network abstraction composed of network segments spanning across multiple sites. Used by a tenant’s VM Uplink Port-Profile (UPP) – Carries a list of allowed FNDs for a pNIC Virtual Port Profile (VPP) – Profile defining QoS/SLA characteristics for a vNIC Logical Switch – Microsoft’s native DVS and define Live Migration Boundary VM 1 VM 2 VM 4 VM 5 WEB Gold-VPP WEB, Silver--VPP Logical Switch (DVS) vSwit ch vSwit ch FN: PUBLIC Live Migration Boundary VMN: WEB Uplink PP-SJC Gold-VPP Silver-PP Bronze-PP Uplink PP-NYC FND: PUBLIC-SJC FND: PUBLIC-NYC VMND: WEB, VLAN: 55 VMND: WEB, VLAN: 155 - Fabric Network is a network abstraction representing a logical network composed of network segment across multiple sites. -Fabric Network Definition is a network abstraction composed of site-specific networking such as VLAN and subnet information. -VMND – is the minimum network segment defined by SCVMM, each virtual machine interface would get assigned to a VMND with specific subnet and IP information. A group of VMND becomes a Fabric Network Definition and a group of FND becomes fabric network. -The uplink port profile is one of the two port profile assigned by SCVMM to specify the allowed Fabric Network Definition or Site for individual physical NIC The virtual port profile defines the QoS and SLA information for the individual VM nic. Lastly, the logical switch which define the VM Live Migration boundary and the available switch extension for each Hyper-V host. VM Network: Span the VMND segment across multiple sites (FND)

Hyper-V VM-FEX Configuration Workflow
3. Install UCSM Provider Plugin 4. Configure SCVMM Switch Extension Manager 1. Configure Service Profile 2. Setup SCVMM and Create Port Profile 5. Configure SCVMM Logical Switch 6. Associate Native VM Network to External Provided VM Network 8. Verify VM-FEX Connectivity in UCSM The workflow for hyper-V VM-FEX is similar what we have been seen on VMware. Two parties are involved, network administrator majority would perform the task directly on FI and Server administrator who would be focusing on SCVMM configuration. The first step, we would configure service profile with appropriate number of dynamic vnic. Next the network admin would layout the infrastructure based on the SCVMM architecture mentioned in the previous slides including detailed information of FN, FN site, VMND (minimum subnet with IP pool), two types of port profile (uplink port profile and virtual port profile) and mostly switch extension. Then we will go sever admin and install the UCSM Provide Plugin DLL on SCVMM server and enable the communication between SCVMM and UCSM. 7. Assign Hyper-V Host to Logical Switch and attach port Classification

Hyper-V VM-FEX Demo Topology
Server Cisco UCS VM-FEX Microsoft Hyper-V NTTTCP/Server Cisco UCS Fabric Interconnect Virtual Ethernet ports (vEth) SR-IOV enabled adapter NTTTCP/Client 75

Hyper-V VM-FEX Demo

Hyper-V VM-FEX Best Practice
Always utilize SCVMM to configure Hyper-V and Virtual Machine property Utilize NTTTCP as performance benchmark tool in Windows Platform NTTTCP is Microsoft internal testing tool Latest version available – NTTTCP v5.28 with Windows Server 2012 (April 2013) Optimized for 10GE interface Enable Receive Side Scaling (RSS) Use Powershell Command - Set-VMNetworkAdapter –VMName “Server” – IovQueuePairsRequested 4 Need to shutdown VM to apply RSS iSCSI boot is NOT support for PF as an overlay iSCSI vNIC

VM-FEX Implementation with KVM on UCS

RHEL KVM VM-FEX: Infrastructure Requirements
VM-FEX is available for Hyper-V on UCS B series and Integrated C series Standalone C series support is on the road map Each VIC card supports up to 116 Virtual Machine Interface Install additional VIC will double VM interface (B200M3 with 2 VIC -> 232 per host) UCS Manager 2.1 release is required to supported SR-IOV in KVM Install Red Hat as Virtualization Host RHEL 6.2 for VM-FEX emulation mode (SR-IOV with MacVTap) RHEL 6.3 for VM-FEX hypervisor bypass mode (SR-IOV with PCI passthrough) MacVTap Direct (Private) mode is no longer supported with UCSM release 2.1 Live migration feature only supported in emulation mode Guest Operating System RHEL 6.3 Required to support SR-IOV with PCI passthrough RHEL 6.3 inbox driver supports SR-IOV with PCI passthrough Scripted nature of configuration at FCS No current RHEV-M for RHEL KVM 6.x Virtual Machine interface management via editing of VM domain XML file

VM-FEX with KVM Architecture
Guest 1 Guest 2 Libvirt – Open source management tool is used for managing virtual machines provides a generic framework supports for a various virtualization A Virtual Machine in Libvirt is represented as a domain XML file and store under QEMU user space Virsh is GUI interface built on top of Libvirt API MacvTap - Linux Macvtap driver provides a mechanism to connect a VM interface directly to a host physical device Libvirt uses macvtap to provide direct attach of VM NIC to host physical device VM-FEX bypass MacvTap Interface KVM Application Application Virsh Port Profile1: Qos1, vlan1 Guest OS Guest OS User Libvirt Port Profile2: Qos2, vlan2 virtio-net virtio-net Netlink Socket vhost-net vhost-net Macvtap Interface Kernel Macvtap 1 Macvtap 2 … … Netdev Interface eth0 eth1 eth2 ethn … … PF VF1 VF2 VFn Cisco VIC Adapter Port Switch Port - KVM uses x86 hardware virtualization extensions (Intel VT or AMD-V) - KVM kernel driver manages the virtualization hardware - KVM uses QEMU user-space for device emulation, a Virtual machine is simply a process. Hence all process management commands like kill etc work on VM’s Libvirt Libvirt an open source management tool is used for managing virtual machines. Libvirt provides a generic framework and supports management of a various virtualization technologies like KVM, Xen, Vmware ESX and others. It runs as a service named libvirtd - A virtual machine in libvirt is represented in the form of a domain xml file Virsh tool built on top of the libvirt API provides an interface to manage virtual machines (GUI interface of Libvirt) MacVTap - The Linux macvtap driver provides a mechanism to connect a virtual machine tap interface directly to a host physical device Libvirt uses macvtap to provide direct attach of virtual machine NIC to host physical device Guest 1 is configured with SR-IOV macvtap where all the VM traffic is passing through vhost-net and macvtap in user space, VM migration is supported in this mode Guest 2 is configured with SR-IOV PIC pass through which VM traffic is directly copy from VM enic to the VIC hardware and achieve better performance by bypass macvtap However VM migration is not supported in this mdoe. Veth 1 Veth 2 Port Profile1: Qos1, vlan1 UCS Switch Port Profile2: Qos2, vlan2

VM-FEX on KVM Configuration Steps
Upgrade UCSM to 2.1+ firmware (Del Mar release support SR-IOV) Configure Service Profile with static (PF) and dynamic (VF) adapter policy Creating Port Profile and Port Profile Client (only single default DVS support) Install VM OS with RHEL 6.3 to support SR-IOV Modify Virtual Machine Domain XML file to enable VM-FEX function Connect VM to Virtual Machine Manager (GUI interface of Virsh) Verify VM-FEX configuration in both UCSM and RHEL host

KVM VM-FEX Demo

VM-FEX Customer Benefit
Simplicity One infrastructure for virtual & physical resource provisioning, management, monitoring and troubleshooting Consistent features, performance and management for virtual & physical infrastructure Performance VMDirectPath and SR-IOV enabled near bare metal I/O performance Line rate traffic to the virtual machine Robustness Trouble shooting & traffic engineering VM traffic holistically from the physical network Programmability, ability to re-number VLANs without disruptive changes FEX technologies can reduce managed device count FEX technologies will greatly reduce cabling overhead VM-FEX is terminating these virtual links directly on the VMs Closely maps to the physical server model for operations and management Multiple Hypervisors are supported with advanced features Bandwidth can be engineered identically to physical infrastructures today Latency can surpass local virtualized switching by moving away from virtual switching store and forward buffering, “tree’s” of ASIC traversals, to a uniform port controller and switch fabric model

UCS Advantage Videos on YouTube
BRKCOM-2005 Recommended Viewing UCS Advantage Videos on YouTube Playlist UCS Technical Videos Overview Cisco UCS Advantage

BRKCOM-2005 Recommended Viewing Category Title URL UCS server
Service Profiles and Templates Organizations and Roles Extended Memory Technology Server Pre-Provisioning BIOS Policies RAID Policies Firmware Policies Server Pools and Qualification Policies Maintenance Policies High Availability During Upgrades Monitoring with BMC BPPM Microsoft Hyper-V on UCS

BRKCOM-2005 Recommended Viewing Category Title URL UCS I/O
Adapter Templates Network Interface Virtualization Adapter Fabric Failover Extend the Network to the Virtual Machine Traffic Analysis of All Servers Ethernet Switching Modes Fibre Channel and Switch Modes FC Port Channels and Trunking

BRKCOM-2005 Recommended Viewing Category Title URL UCS Infrastructure
Lights-Out Management Easy VM-FEX Deployment Server Power Grouping Blade and Rack-Mount Management Manager Platform Emulator Cisco Developer Network and Sandbox

4/6/2017 Cisco Live 2013.

Similar presentations

Presentation on theme: "4/6/2017 Cisco Live 2013."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

4/6/2017 Cisco Live 2013.

Similar presentations

Presentation on theme: "4/6/2017 Cisco Live 2013."— Presentation transcript:

Similar presentations

About project

Feedback