Advanced XenServer Training

Advanced XenServer Training
CCAT Q

Coursework Agenda XenServer Overview XenServer Components Deep-Dive
Lab 1: Navigating Dom0, Logs and the CLI XenServer Storage Primitives (Caveman-ish?) Lab 2: Playing with Storage XenServer Networking Lab 3: NICs, NICs and more NICs XenServer Troubleshooting Lab 4: How do I fix this stuff?

Hypervisor Overview What is XenServer?

Topics Covered Comparisons to other hypervisors
VMware vSphere 5.1 Microsoft Hyper-V 2012 Real world applications for XenServer XenApp XenDesktop PVS

Hypervisor Overview XenServer vs. The Competition

Market Share and Growth Comparison
Company Hypervisor Platform 2011 Growth Current Estimated Market Share Microsoft Hyper-V 62% 25% Citrix XenServer 15 – 20% VMware ESX 21% 50%

VMware vSphere Current Version – 5.1
Consists of two components: VMware ESX and vCenter Server ESXi is the hypervisor and installs on bare metal hardware. vCenter Server provides centralized management and allows administrators to configure and monitor ESXi hosts, provision virtual machines, storage, networking, and much more. Management mechanism: vSphere Client Windows application that acts as a single pane of glass to manage either a standalone ESXi host directly or an entire datacenter though vCenter.

vSphere Architecture diagram

vSphere VM Architecture diagram
The vmkernel handles CPU and memory directly, using scan-before-execution (SBE) to handle special or privileged CPU instructions and the SRAT (system resource allocation table) to track allocated memory. Access to other hardware (such as network or storage devices) takes place using modules. At least some of the modules derive from modules used in the Linux kernel. To access these modules, an additional module called vmklinux implements the Linux module interface. According to the README file, "This module contains the Linux emulation layer used by the vmkernel.”

Hyper-V Current Version – Hyper-V 2012
Originally released in Server 2008 Bundled with Windows Server licenses Behind VMware, Hyper-V retains the most market share in server virtualization Consists of single component: Windows Server with Hyper-V role Windows Server is the base hypervisor Management mechanism: SCVMM SCVMM (System Center Virtual Machine Manager) provides centralized management and allows administrators to configure and monitor hosts, provision virtual machines, storage, networking, etc. Can tie in directly with MOM/SCOM and Systems Management Server

Hyper-V Architecture diagram

Hyper-V VM Architecture diagram
Enlightened – has an integration services pack installed which provides extra features. Enlightened I/O is a specialized virtualization-aware implementation of high level communication protocols (such as SCSI) that utilize the VMBus directly, bypassing any device emulation layer. This makes the communication more efficient but requires an enlightened guest that is hypervisor and VMBus aware. Hyper-V enlightened I/O and a hypervisor aware kernel is provided via installation of Hyper-V integration services. Integration components, which include virtual server client (VSC) drivers, are also available for other client operating systems. Unenlightened –a virtual machine that is running on its own with no special features. The equivalent of the integration packs for VMware is VMware tools. Having these integration packs or VMware tools installed is highly desirable, unless there is some particular reason that you should run without them, as they help you to realize the full benefits of virtualization. APIC – Advanced Programmable Interrupt Controller – A device which allows priority levels to be assigned to its interrupt outputs. Child Partition – Partition that hosts a guest operating system - All access to physical memory and devices by a child partition is provided via the Virtual Machine Bus (VMBus) or the hypervisor. Hypercall – Interface for communication with the hypervisor - The hypercall interface accommodates access to the optimizations provided by the hypervisor. Hypervisor – A layer of software that sits between the hardware and one or more operating systems. Its primary job is to provide isolated execution environments called partitions. The hypervisor controls and arbitrates access to the underlying hardware. IC – Integration component – Component that allows child partitions to communication with other partitions and the hypervisor. I/O stack – Input/output stack MSR – Memory Service Routine Root Partition – Manages machine-level functions such as device drivers, power management, and device hot addition/removal. The root (or parent) partition is the only partition that has direct access to physical memory and devices. VID – Virtualization Infrastructure Driver – Provides partition management services, virtual processor management services, and memory management services for partitions. VMBus – Channel-based communication mechanism used for inter-partition communication and device enumeration on systems with multiple active virtualized partitions. The VMBus is installed with Hyper-V Integration Services. VMMS – Virtual Machine Management Service – Responsible for managing the state of all virtual machines in child partitions. VMWP – Virtual Machine Worker Process – A user mode component of the virtualization stack. The worker process provides virtual machine management services from the Windows Server 2008 instance in the parent partition to the guest operating systems in the child partitions. The Virtual Machine Management Service spawns a separate worker process for each running virtual machine. VSC – Virtualization Service Client – A synthetic device instance that resides in a child partition. VSCs utilize hardware resources that are provided by Virtualization Service Providers (VSPs) in the parent partition. They communicate with the corresponding VSPs in the parent partition over the VMBus to satisfy a child partitions device I/O requests. VSP – Virtualization Service Provider – Resides in the root partition and provide synthetic device support to child partitions over the Virtual Machine Bus (VMBus). WinHv – Windows Hypervisor Interface Library - WinHv is essentially a bridge between a partitioned operating system’s drivers and the hypervisor which allows drivers to call the hypervisor using standard Windows calling conventions WMI – The Virtual Machine Management Service exposes a set of Windows Management Instrumentation (WMI)-based APIs for managing and controlling virtual machines.

Why is Hyper-V important?
Bundled in with Windows Server 2012, Hyper-V provides free virtualization, and enterprise features at no additional cost Bridging functionality gap with VMware Tight integration with Windows infrastructure Microsoft has had much success with midmarket companies (<1000 users), winning 30 – 40% of the time

XenServer Current Version – 6.1
Consists of single component: XenServer Core CentOS-based hypervisor Management mechanism: XenCenter XenCenter console which is Windows only is the management tool. It directly connects to the current Pool master in order to send commands to the rest of the XS pool Has APIs for monitoring and management (To be discussed in more detail later).

XenServer VM Architecture diagram

Limitations Comparison
Parameter Hyper-V 2012 XenServer 6.1 vSphere 5.1 Host Cores 320 160 RAM 4 TB 1 TB 2 TB VMs 1024 150 (60 w/HA) 512 vCPUs 900 2048 (25/core) Cluster/Pool 4000 1024 (600/SR) Hosts 64 16 32 VM CPUs 128 GB NUMA Yes Host Only HyperV ESX XenServer

Feature Comparison Feature Hyper-V 2012 XenServer 6.1 vSphere 5.1
Incremental Backups Yes Yes (VM Prot & Reco) NIC Teaming Integrated HA OS App Monitoring No API-Only Failover Prioritization Affinity & Rules API-Only (SCVMM) Cluster-Aware Updating 50/50 (Pool Upgrade is not very smart) Main point of this slide is to show how close the Hypervisors have become in terms of feature set and that MSFT is trying to push Hyper-V hard by adding features that are Windows-specific as they have that advantage over the competition. ESX: VMware have made APIs publicly available, but actual application monitoring is not included XenServer Affinity Rules wuign SCVMM

Hypervisor Overview Key Topics/Components

Hyper-V | XenServer | vSphere – VM Limits
Explain NUMA and Guest NUMA – maybe a separate slide is ned Guest NUMA may have performance improving potential on high-vCPU XenApp workloads vCPU sweet spot for virtual XenApp is between 4-8 vCPUs

CPU Scheduling XenServer Hyper-V vSphere

Hypervisor Overview XenServer 6.1 - What’s new?
-Thanks to Ryan McClure for donating content

XenServer 6.1 – Key features
Link Aggregation Control Protocol (LACP) support Storage XenMotion Integrated StorageLink (iSL) support for EMC VNX series arrays Release Notes:

XenServer 6.1 – Virtual Appliances
Demo Linux Virtual Appliance Workload Balancing Virtual Appliance vSwitch Controller Virtual Appliance Web Self Service Virtual Appliance Citrix License Server VPX v11.10 Citrix XenServer Conversion Manager Allows up to 500 VM conversions from VMware vSphere to XenServer

What’s New? – Management and Support
Storage XenMotion Live VDI Migration Emergency Network Reset XenServer Conversion Manager Enhanced Guest OS Support Supported VM Count Increased to 150 Storage XenMotion - Allows running VMs to be moved from one host to another. For use cases see slide 6. Live VDI Migration – Allows running VM .vhds to be moved from one storage location to another. For use cases see slide 6. Emergency Network Reset - Provides a simple mechanism to recover and reset a host's networking, allowing system administrators to revert XenServer hosts to a known good networking state. XenServer Conversion Manager - Enables batch import of VMs created with VMware products into a XenServer pool to reduce costs of converting to a XenServer environment. Monitor progress Logs Cancel active jobs Retry failures Unattended Leaves original intact Enhanced Guest OS Support – See slide 8. Supported VM Count Increase – Storage and network driver domains.

What’s New? – Networking
LACP Support VLAN Scalability Improvements IPv6 Guest Support Updated OvS LACP Support - Enables the use of industry-standard network bonding features to provide fault-tolerance and load balancing of network traffic. Was unsupported but could be set on Linux Bridge. Now supported on vSwitch. See slide 7. VLAN Scalability Improvements – Removes issues with creating large numbers of VLANs. Removed database lookups Networking configured in dom0 on behalf of xapi Uses Linux and OvS interfaces IPv6 Guest Support – Supports use of IPv6 addresses for guests. Updated OvS (open vSwitch) – Software switch similar to Vmware distributed vSwitch. Linux bridge to be deprecated Faster packet forwarding Automated control - OpenFlow Support (CloudStack is an OpenFlow controller)

Storage XenMotion and VDI Migration
Live VDI Migration Storage XenMotion: 1.) Upgrade storage array 2.) Upgrade a pool with VMs on local storage 3.) Provide tiered storage arrays 4.) Rebalance VMs between XenServer pools or CloudStack clusters Live VDI Migration: 1.) Move a VM from cheap, local storage to fast, resilient, array-backed storage; 2.) Move a VM from a development to a production environment; 3.) Move between tiers of storage when a VM is limited by storage capacity; 4.) Perform storage array upgrades.

XenServer 6.1 – Feature(s) that didn’t get lime light
Performance Monitoring Enhancements Supplemental Pack Extended XenServer monitoring available (stored in RRD – Round Robin Databases) Metric Availability Average physical CPU usage over all CPUs (%) New Time physical CPU spent in [C,P]-state (%) Host memory reclaimed by dynamic memory control (B) Maximum host memory that could be reclaimed by dynamic memory control (B) Host aggregate, physical network interface traffic received/sent (B/s) I/O throughput read/write/total per PBD (MiB/s) Average I/O latency (milliseconds) I/O average queue size Number of I/O requests currently in flight IOPS, read/write/total (requests/s) Time spent waiting for I/O (%) RDD reference :

XenServer Roadmap Sarasota Jupiter Tallahassee – Private Release
Dom0 Disaggregation Increased Scalability Fewer Reboots due to Hotfixes vGPU Jupiter Storage QoS Tallahassee – Private Release NetScaler SDX Update XenDesktop Interoperability and Usability Enhancements Hardware Monitoring and Reporting Clearwater Windows 8 Guest Support UI Enhancements Naples User Experience and Third Party Enhancements VHDX Disk Format Support Dom0 Disaggregation – Moving functionality from dom0 to system domains ‘virtualizing dom0’. - Separating storage and network driver domains Results: Improved VM density Remove performance bottlenecks Cloud scale Isolation for multi-tenancy vGPU – GPU pass-through 4 GPUs per host Storage QoS – Storage tiering based on performance agreements etc. (Moving towards the ‘Cloud’) User Experience and Third Party Enhancements New SDK Workload balancing extensions

Additional Resources CTX118447 XenServer Administrator’s Guide
CTX XenServer Installation Guide CTX XenServer Virtual Machine Installation Guide CTX XenServer Release Notes CTX XenConvert Guide CTX XenServer Software Development Guide • CTX Deploying Citrix XenServer 5.0 with Dell EualLogic PS Series Storage • CTX Citrix XenServer 5.0 and NetApp Storage Best Practices • CTX XenServer Demo and Evaluation Guide • CTX XenServer Virtual Machine Installation Guide • CTX XenConvert Installation Guide • CTX XenServer 4.1 and 5.0 Licensing Overview • CTX XenServer Pool Replication – Disaster Recovery • CTX Current XenServer Technical Support Information • CTX XenServer Storage Overview • CTX Understanding XenServer Networking • CTX Multipathing Overview for XenServer 5.0 • CTX How to Run a Status Report in XenCenter 4.1/5.0 and XenCenter 4.1/5.0 • CTX How to assign a Static IP Address to a XenServer Host • CTX How to Configure High Availability in XenServer 5.0 • CTX How to Change the Default Storage Repository to File-based VHD-on-EXT3 • CTX How to Use a USB Disk as a Local Storage Repository on XenServer

XenServer Under The Hood

Agenda Core Components Beyond Single Server Workload Balancing
Hardware Hypervisor Dom0 DomU Beyond Single Server Resource Pools HA Meta Data XenMotion/Live Migration Workload Balancing

XenServer Overview Core Components

Terminology to Remember
Host Resource Pool Pool Master Dom0 XAPI (XENAPI) XenServer Tools Live VDI Migration Storage XenMotion Host A server-class machine devoted to hosting multiple VMs. Resource Pool A collection of a set of XenServer hosts managed as a unit. Typcially, a resource pool will share some amount of networked storage to allow VMs to be rapidly migrated from one host to another within the pool. Pool Master One host in the pool is specified to be the “Pool Master”. The Pool Master controls all administrative activities required on the pool. This means that there is no external single point of failure. If the Pool Master fails, other hosts in the pool will continue to operate, and resident VMs will continue to run as normal. If the Pool Master cannot be brought back online, one of the other hosts in the pool can be promoted to master to regain control of the pool. This process is automated with the High Availability feature. Refer to the XenServer Administrator's Guide for more details. A copy of the configuration data is stored on every host in the resource pool. If the current pool master fails, this enables any host in the resource pool to become the new pool master. Dom0 Control Domain (Dom0), is a secure, privileged VM that runs the XenServer management toolstack known as xapi. Besides providing XenServer management functions, dom0 also runs the physical device drivers for networking, storage, etc. XAPI XAPI is the software stack that controls VM lifecycle operations, host and VM networking, VM storage, user authentication, and allows the management of XenServer resource pools. XAPI provides the publicly documented XenAPI Management Interface which is used by all tools that manage VMs and resource pools. Examples of internal and external tools that make use of the XenAPI include: XenCenter management GUI xs management console xe command line interface Automated Workload Balancing virtual appliances Apache Cloudstack Cloud orchestration software XenServer Tools The XenServer Tools are software packages for Windows and Linux-based guest operating systems. The Tools include high-performance disk and network paravirtualized drivers (PV drivers) for Windows operating systems, and a Guest Agent for Linux-based operating systems that provides additional information about the VM to the XenServer host. Refer to the XenServer VM User's Guide for more details. Live VDI Migration Live VDI Migration allows system administrators to relocate a VM's Virtual Disk Image (VDI) without shutting down the VM. This enables system administrators to: Move a VM from cheap, local storage to fast, resilient, array-backed storage; Move a VM from a development to a production environment; Move between tiers of storage when a VM is limited by storage capacity; Perform storage array upgrades. Storage XenMotion Storage XenMotion allows running VMs to be moved from one host to another. This includes the case where (a) VMs are not located on storage shared between the hosts and (b) hosts are not in the same resource pool. This enables system administrators to: Rebalance or move VMs between XenServer pools – for example promoting a VM from a development environment to a production environment; Perform software maintenance – for example upgrading or updating standalone XenServer hosts without VM downtime; Perform hardware maintenance – for example upgrading standalone XenServer host hardware without VM downtime; Reduce deployment costs by using local storage. For more information, refer to the XenServer Virtual Machine User's Guide and the XenCenter online help. < <

Behind The Scenes Native 64-bit bare metal hypervisor – Is this true?
Cent OS Based on Linux kernel Drivers are SUSE Linux Enterprise Server v11 SP1 Xen hypervisor and Domain 0 manage physical server resources among virtual machines Well, Dom0 is 32Bit… Linux, many drivers and tools can be potentially utilized; XenServer drivers are based on SUSE Linux Enterprise Server v11 SP1 Device Driver Kit. Introducing an add-on driver or tool into the Control Domain should be carefully assessed and tested to mitigate its potential negative impact. Installing a new driver outside of the Citrix Supplemental Pack will require opening a Citrix Technical Support case to ensure its Citrix supportability.

Legacy Virtualization Architecture
Presentation Title Goes Here Insert Version Number Here Legacy Virtualization Architecture Legacy Hypervisor User Apps User Apps “Binary Translation” Slow, expensive software emulation layer HALT SAFE HALT A virtualization solution that relies on emulation must monitor all CPU instructions and rewrite unsafe instructions as it runs to safe instructions. This technology works, but it comes with a price: complexity and higher levels of performance overhead. For example, a single OS on a physical box can call a halt instruction telling the CPU to stop executing. This is useful for suspending or shutting down the system. However, when running multiple virtual machines, this instruction cannot be allowed because it would affect all the other VMs on the box. The HALT command must be rewritten to a SAFE HALT command. This is how our largest competitor achieves virtualization. Hardware © 2007 Citrix Systems, Inc.—All rights reserved.

Presentation Title Goes Here
Insert Version Number Here Paravirtualization XenServer Relies on “enlightened” operating systems Kernel and I/O paths know they are being virtualized Cooperation provides best performance User Apps User Apps HALT Paravirtualized guests make high-speed calls directly to the hypervisor The Xen hypervisor implements a technique called paravirtualization that allow virtual machines to cooperate with the hypervisor and the underlying system. This cooperation eliminates the need to monitor and translate the CPU instructions, significantly reducing the performance overhead typically associated with virtualization. This is what allows Xen to run with near-native performance. Xen pioneered paravirtualization on the x86 platform. XenServer VMs use paravirtualized drivers for storage and network devices. Paravirtualized drivers drastically improve performance over emulated drivers (from our competition). This is what is installed with Citrix Tools and why it is critical to be installed in every VM. Linux VMs on XenServer use fully paravirtualized kernels. Instead of calling the halt instruction, it uses a quick safe halt hypercall to the Xen layer. The paravirtualized Linux hypercall removes the need to scan instructions of machines and rewrite instructions. Windows Server 2003 is not enlightened, Server 2k8 has some enlightenments. HALT HYPERCALL VT/AMD-V Hardware © 2007 Citrix Systems, Inc.—All rights reserved.

Hardware-Assisted Virtualization
Presentation Title Goes Here Insert Version Number Here Hardware-Assisted Virtualization XenServer Hardware-assist allows high performance without emulation User Apps User Apps HALT Other guests benefit from hardware-accelerated call translation Xen was designed from the ground up to leverage hardware virtualization assist technologies delivered by both Intel and AMD in their CPUs. Intel Research was involved early on with the Xen project. The Xen Hypervisor was designed to be very thin and easily take advantage of both current virtualization assist technologies and the slate of newer technologies being delivered in the chips in the coming years. Windows VMs require hardware virtualization assist because modifying the kernel is not permitted. Windows VMs call the halt instruction, but the CPU catches and redirects the call to the Xen layer to handle and return a valid result to the VM. The Xen hypervisor removes the need to scan all executing instructions from guest VMs. The result is increased performance over emulation technology. HALT HYPERCALL VT/AMD-V Hardware © 2007 Citrix Systems, Inc.—All rights reserved.

Understanding Architectural Components
Presentation Title Goes Here Insert Version Number Here Understanding Architectural Components The Xen hypervisor and Domain 0 manage physical server resources among virtual machines This shows a blowup of the Xen virtualization engine and the virtualization stack “Domain 0” with a Windows and Linux virtual machine. The green arrows show memory and CPU access which goes through the Xen engine down to the hardware. In many cases Xen will get out of the way of the virtual machine and allow it to go right to the hardware. Xen is a thin layer of software that runs right on top of the hardware, Xen is only around 50,000 lines of code. The lines show the path of I/O traffic on the server. The storage and network I/O connect through a high performance memory bus in Xen to the Domain 0 environment. In the domain 0 these requests are sent through standard Linux device drivers to the hardware below. © 2007 Citrix Systems, Inc.—All rights reserved.

Understanding the Hardware Component
Presentation Title Goes Here Insert Version Number Here Understanding the Hardware Component Hardware layer contains the physical server components, including memory, CPU and storage The physical platform. © 2007 Citrix Systems, Inc.—All rights reserved.

Understanding the Hypervisor Component
Presentation Title Goes Here Insert Version Number Here Understanding the Hypervisor Component Xen hypervisor is a thin layer of software that runs right on top of the hardware The Xen hypervisor is a thin layer of software that runs right on top of the hardware. Xen provides an abstraction layer that allows each physical server to run one or more virtual servers, effectively decoupling the OS and its applications from the underlying hardware © 2007 Citrix Systems, Inc.—All rights reserved.

Understanding the Domain 0 Component
Presentation Title Goes Here Insert Version Number Here Understanding the Domain 0 Component Domain 0, or the Control Domain, is a Linux VM that manages the network and storage I/O of all guest VMs Domain 0 is a Linux VM with higher priority to the hardware than the guest operating systems. Domain 0 manages the network and storage I/O of all guest VMs, and because it uses Linux device drivers, a broad range of physical devices are supported © 2007 Citrix Systems, Inc.—All rights reserved.

Dom0 Resourcing Linux-based guests virtual network interfaces
virtual disks virtual CPUs Windows-based guests Resource Control Processor, network, disk I/O priority Machine Power-States Adjust virtual resources on the fly.

Why This Model Kinda Sucks
Control Requests - XS Console / Direct CLI Queuing Confusion Dom0 Overburdening Distributed Domino Effect Limited space due to pre-allocation Log Bug Not built to Scale out of the Box Dom0 Memory Dom0 CPU

DOM0 Tuning Supports 4 vCPUs (Default 1 Prior to 6.0)
More is better IRQBALANCE ( Memory Tweaking (Default 752) 2940M Tested for 130/16.25 VMs Warning: Single Server! XenHeap (5.5 Only) Set xenheap_megabytes to (12 + max-vms/10) Xc.Error("creating domain failed: hypercall 36 fail: 12: Cannot allocate memory (ret -1)") Adjusting Dom0 and Xenheap Settings in XenServer How to Configure Dom0 Memory in XenServer 5.6 XenServer Single Server Scalability with XenDesktop Nitro’s Blog Irqbalance is a lightweight Linux daemon developed by some smart folks at Intel that dynamically distributes interrupt queues to all available vCPUs All traffic for a specific VIF of a guest is processed by a statically-allocated netback process. Distributing Guest Traffic Over Physical CPUs in XenServer

Understanding the DomU Linux VM Component
Presentation Title Goes Here Insert Version Number Here Understanding the DomU Linux VM Component Linux VMs include paravirtualized kernels and drivers Linux VMs include paravirtualized kernels and drivers. Storage and network resources are accessed through Domain 0, while CPU and memory are accessed through Xen to the hardware © 2007 Citrix Systems, Inc.—All rights reserved.

Understanding the DomU Windows VM Component
Presentation Title Goes Here Insert Version Number Here Understanding the DomU Windows VM Component Windows VMs use paravirtualized drivers to access storage and network resources through Domain 0 Windows VMs use paravirtualized drivers to access storage and network resources through Domain 0. XenServer is designed to utilize the virtualization capabilities of Intel VT and AMD-V enabled processors. Hardware virtualization enables high performance virtualization of the Windows kernel without using legacy emulation technology © 2007 Citrix Systems, Inc.—All rights reserved.

Real-time Resource Adjustment
Linux-based guests Add/remove virtual network interfaces Add virtual disks Add/remove virtual CPUs Windows-based guests Add/remove virtual disks Resource QoS Control Processor, network, disk I/O priority Adjust virtual resources on the fly.

Why This is Cool Paravirtualization is key to performance
Hypervisor and OS cooperation provides best performance Utilizes hardware-assisted virtualization Fully supports Intel VT and AMD-V enabled processors Hardware-assist allows high performance without emulation Future: Paravirtualization with Hardware-Assist Benchmarks vs running on native Linux can expect % overhead Windows can expect 2 - 7% overhead VMware (EULA prevents publishing) 5-10% Overhead seen on internal testing In both cases, the Xen hypervisor removes the need to scan all executing instructions from guest VMs. The result is increased performance over emulation technology.

Lessons Learned: The other DOM - DOMU
We can only push around 350 Mb/s through a single Windows VM. We can 1 to 1.1 Gb/s on a single Linux VM To push more traffic through a single Windows VM: Manually add additional netback processes xen-netback.netback_max_groups=<#Procs> under /boot/extlinux.conf in section labeled xe-serial just after the assignment console=hvc0. Manually pinning vCPUs to specific VMs xe vm-param-set uuid=<vm-uuid> VCPUs-params:mask=<CPUs to pin> We’ve actually had a dom0 network bottleneck of 3 Gb/s in XS 5.x! Today with Boston we can now push somewhere around 6-7 Gb/s through dom0. Not because of changes in Linux networking stack to the Open vSwitch (OVS) networking stack Mainly due to irqbalance and only partly due to OVS and general NIC-bonding improvements. We can only push around 350 Mb/s through a single Windows VM. Compare this to a single Linux VM and it’s more like 1 or 1.1 Gb/s. To push more traffic through a single Windows - manually adding additional netback processes, manually pinning vCPUs to specific VMs Now that I’ve shared the bad news…keep in mind this is all without SR-IOV (Single root IO Virtualization. If you really want to try and achieve true line-speed (i.e. come close to saturating a 10 Gb link), then SR-IOV is the only way to go. This is actually what we use for NetScaler SDX.

XenServer Overview Beyond Single Server

Management Architecture Comparison
VMWare XenServer Traditional Management Architecture Single backend management server Distributed Management Architecture Clustered management layer VMware requires a clustered SQL backend. We have a distributed database that’s built into the product. This is great in theory but does it work in practice? 52

Resource Pools Join multiple physical servers into one logical pool
Enables shared configurations, including storage Required for Automated VM Placement and XenMotion Xen Hypervisor Grouping of XenServers into a single logical unit. 53

High Availability Xen Hypervisor Xen Hypervisor Xen Hypervisor Protected Protected Do not restart Protected VM’s on failed physical servers can automatically be restarted on other servers in the pool. Shared Storage 54

XenMotion - Live Migration
Xen Hypervisor Xen Hypervisor Xen Hypervisor XenMotion allows running Guest VM’s to be migrated without service downtime Zero down-time during planned maintenance Load Balance VMs over different servers Shared Storage 55

XenMotion – Live VM Movement
XenMotion allows minimal downtime movement of VMs between physical systems Generally ms of actual “downtime” Most of the downtime is related to network switch moving IP traffic to new port XenMotion allows you to move a live VM from one physical server to another without any service downtime. The switchover period is normally around ms. In XenEnterprise v4 XenMotion is an operator-initiated operation.

XenMotion CPU Requirements
XenMotion requires systems XS have similar CPUs Must be the same manufacturer Must be the same type Can be different speed Example Xeon 51xx series chips Could have a mix of 5130 and 5120 chips CPUs within a pool need to be of the same Vendor and Type. Without this, XenMotion would not be possible. This is a general rule with all virtualization technologies. Different CPU speeds can be used within a given type (e.g and 5340). The main dependency is down to the processor feature flags. If a VM is using a specific CPU feature and it moves to another server with CPU lacking that feature, the VM will likely crash.

Pre-Copy Migration: Round 1
Systems verify correct storage and network setup on destination server VM Resources Reserved on Destination Server Source Virtual Machine Destination XenMotion Round 1 Does the destination have the right resources? If yes, the destination reserves the relevant resources.

While source VM is still running XenServer copies over memory image to destination server XenServer keeps track of any memory changes during this process XenMotion Round 1 Live Memory is copied from the source to the target, tracking any changes that are happening while this main copy is in progress

XenMotion Round 1 Copy operation is throttled to 10% of network bandwidth and 10% of server CPU so as not to impact other VMs on the systems

After first pass most of the memory image is now copied to the destination server Any memory changes during initial memory copy are tracked Some of the memory has been updated since the first copy started, so these updates are tracked.

XenServer now does another pass at copying over changed memory XenMotion Round 2 Memory active during the last copy is now copied over to the destination

XenMotion Round 2 Memory active during the last copy is now copied over to the destination

Xen still tracks any changes during the second memory copy Second copy moves much less data Also less time for memory changes to occur XenMotion Round 2 Memory active during the last copy is now copied over to the destination

XenMotion Round 2 Memory active during the last copy is now copied over to the destination

Pre-Copy Migration Xen will keep doing successive memory copies until minimal differences between source and destination Xen will keep doing successive memory copies until minimal differences exist between source and destination

XenMotion: Final Source VM is paused and last bit of memory and machine state copied over Master unlocks storage from source system and locks to destination system Destination VM is unpaused and attached to storage and network resources Source VM resources cleared At the final moment before the migration takes place, the last bit of memory is copied over, the VM is paused. The Pool master then unlocks the VM storage that was mounted by the Source server and attaches it to the Destination. The VM is then unpaused and operations continue.

The Concept of MetaData
The metadata contains information about the VMs: Name, description, Universally Unique Identifier (UUID), VM configuration, and information about the use of resources on the host or Resource Pool such as Virtual Networks, Storage Repository, ISO Library. Most metadata is written when the VM, SR and network are created and is updated if you make changes to the VM configuration. If the metadata is corrupted, the resource pool and/or VMs may become unusable. Always backup metadata during: Resource pool and/or host upgrade Before patching Migration from one storage repository to another

Backing up the MetaData
It is not necessary to perform daily exports of all the VM metadata. To export the VM metadata: Select Backup, Restore and Update from the menu. Select Backup Virtual Machine Metadata. If prompted, log on with root credentials. Select the Storage Repository where the VMs you want to back up are stored. After the metadata backup is done, verify the successful completion on the summary screen. In XenCenter, on the Storage tab of the SR selected in step 3, a new VDI should be created named Pool Metadata Backup. XenServer can be configured to automatically backup the metadata on a daily, weekly, or monthly basis. If a Resource Pool is corrupted or a DR situation occurred, a Metadata backup can be used to restore the Resource Pool assuming the shared storage is also backed up or replicated.

Restoring MetaData Prerequisites: From the console menu:
Set up/re-attach the vDisk Storage Repository Virtual Networks are set up correctly by using the same names From the console menu: Select Backup, Restore and Update from the menu. Select Restore Virtual Machine Metadata. If prompted, log on with root credentials. Select the Storage Repository to restore from. Select the Metadata Backup you want to restore. Select if you want to restore only VMs on this SR or all VMs in the pool. After the metadata restore is done, verify the summary screen for errors. Your VMs are now available in XenCenter and can be started at the new site. See attached Package

Data Replication for DR
From an architectural point of view, a XenServer virtual machine (VM) consists of two components: metadata describing the virtual machine environment, and the virtual disk image (VDI) that is being used by the virtual machine. The VM metadata is stored in a small database on the XenServer host, and the virtual disk images are stored on the configured Storage Repository, which in multiple-host deployments will be a NAS or SAN device. To provide effective DR, the metadata and the virtual disk images must be replicated from the production environment to a DR environment. This is easily accomplished by exporting the metadata from the production environment, and importing this data into the DR environment. In XenServer 5.0+, exporting metadata creates a new VDI on the Storage Repository that contains all the information about your virtual machines. Additionally, XenServer 5.0+ allows for automated scheduling of metadata exports, ensuring that the most recent VM metadata is available at your DR site.

Simplifying Disaster Recovery
Xen Hypervisor Xen Hypervisor Automated backup of VM metadata to SR Replication of SR includes Virtual Disks and VM metadata Attach replicated SR Restore of VM metadata will recreate VMs Xen Hypervisor 1 Xen Hypervisor 4 2 1 3 3 Even if you are not using remote storage you can backup VMs and move them around using our import/export functionality. Again since the VMs are isolated from any hardware differences between the underlying servers you remove all of the driver headache found when moving a physical OS instance around to different boxes. Shared Storage Shared Storage 4 2 Production Site DR Site

XenServer Overview Workload Balancing . /etc/xensource-inventory
staticmax=`xe vm-param-get uuid=$CONTROL_DOMAIN_UUID param-name=memory-static-max` echo staticmax=$staticmax xe vm-memory-target-set uuid=$CONTROL_DOMAIN_UUID target=$staticmax You can confirm the modifications to Dom0 memory are working correctly by examining the contents of /proc/xen/balloon. The sum of Current allocation, Low-mem balloon and High-mem balloon should be equal to (or be pretty close to) the value you specified in /boot/extlinux.conf. (Usually the latter two of these parameters are zero.) You only need to execute those commands once. The values you set for the dom0 memory target are saved into the xapi database which persists across reboots. There are some (rare) circumstances in which XenServer adjusts the memory target automatically (for example: adding more memory to the host), but these should not affect us because we’ve set the target as high as static-max. This means that if you want to go back to the old behavior, with a standard-sized dom0, you must reverse your changes – rebooting does not revert your configuration. If you undo the edits to /boot/extlinux.conf and reboot, executing the same commands should have the effect of causing dom0 to revert to 752 megabytes

Workload Balancing Highly granular, continuous guest and host performance profiling Policy-driven workload balancing across XenServer pools and hosts Historical analysis and trending for planning purposes Reports available in XenCenter

Workload Balancing – Critical Thresholds
Components included in WLB evaluation: CPU Memory Network Read Network Write Disc Read Disk Write Optimization recommendation is being triggered if a threshold is reached

Workload Balancing – Weights
Weights are all about the recommended target host Simple interface to determine which factors are most relevant for the VM

Workload Balancing Ideal placement recommendations at VM start-up, resume and live relocation Ideal VM relocation recommendations for host maintenance

Workload Balancing – Placement Strategies
Maximize Performance Default setting Spread workload evenly across all physical hosts in a resource pool The goal is to minimize CPU, memory, and network pressure for all hosts Maximize Density Fit as many virtual machines as possible onto a physical host The goal is to minimize the number of physical hosts that must be online Maximize Performance (default) Workload Balancing attempts to spread workload evenly across all physical hosts in a resource pool. The goal is to minimize CPU, memory, and network pressure for all hosts. When Maximize Performance is your placement strategy, Workload Balancing recommends optimization when a host reaches the High threshold. Maximize Density Workload Balancing attempts to fit as many virtual machines as possible onto a physical host. The goal is to minimize the number of physical hosts that must be online. When you select Maximize Density as your placement strategy, you can specify rules similar to the ones in Maximize Performance. However, Workload Balancing uses these rules to determine how it can pack virtual machines onto a host. When Maximize Density is your placement strategy, Workload Balancing recommends optimization when a host reaches the Critical threshold.

Workload Balancing Server
Analysis Engine service All components can run on one (virtual) machine Multiple server deployment recommended for larger, multi-pool environments Data store can be hosted on existing DB platform Architecture look familiar? Data Store Data Collection Manager service Familiar? This was developed by EdgeSight architecture team… Ouch! Web Service Host

Workload Balancing - Components
Analysis Engine service Workload Balancing Components Data Collection Manager service Analysis Engine service Web Service Host Data Store XenServer XenCenter Data Store XenServer Resource Pool Performance Metrics XenCenter Data Collection Manager service Performance Metrics XenServer Resource Pool Web Service Host Recommendations

Hypervisor Overview Lab 1 – Intro Lab . /etc/xensource-inventory

In this lab… 30 minutes Create a XenServer Resource Pool
XenServer Resource Pool Management (switching pool master, Dom0 tweaks) Upgrading from XenServer to 6.1 Backing up and Restoring Metadata

XenServer Storage Deep Dive

Agenda Introduction to XenServer Storage SRs – Deep Dive
Xen Storage – Command line Storage XenMotion Under The Hood Debugging Storage Issues Key Takeaways and References

Before we get started… Lets familiarize ourselves with the following terms: HBA LUN Fabric Storage Processor SR VDI PBD and VBD Multipathing Fast Clone, Linked Clone XenMotion and Storage XenMotion Spend 10 minutes around the table and discuss definitions for the above terms.

XenServer Storage Introduction to Storage

Disk Types in common use
SATA (Serial ATA) disks Size up to 3 TB (4TB + due for release) Spin speed 7200 RPM Transfer rate ~ MB per second IOPS 70-80 Pros: Inexpensive, large Storage. Cons: Relatively slow, long rebuild time SAS (Serial Attached SCSI) Disks Size up to 600 GB (900+ GB due for release) Spin speed RPM Transfer rate ~ MB per second IOPS Pros faster access time, redundancy Cons Higher cost per GB SSD (Solid state Disk) Size 100GB upwards (enterprise class) Extremely fast access speeds Transfer rate ~ up to full bus speed IOPS read > 40,000 write > 10,0000 Pros Extremely fast access times low power Cons Very expensive, Limited life.

Reported Size in windows
Disk Sizing Disk Size In MB Expected MB Diff Reported Size in windows 100 GB 100,000,000,000 107,374,182,400 7.37% 93.1 GB, 95,367 MB 1 TB 1,000,000,000,000 1,099,511,627,776 9.95% 931 GB, 953,674 MB Latency Time The faster the disk spins, the shorter the wait for the computer. A hard drive spinning at 7,200 rotations per minute has an average latency of 4.2 thousandths of a second. This is approximately one-half the time it takes for the disk to turn one revolution. Latency Consequences While 4.2 thousandths of a second seems inconsequential, a computer might process many thousands of data requests. The computer's CPU is fast enough to keep up with the workload but the hard disk's rotational latency creates a bottleneck.

RAID (redundant array of independent disks)
Way of presenting multiple disks so that they appear to the server as a single larger disk. Can also provide for redundancy in the event of a single/multiple disk failure(s) Has multiple types each with its advantages and disadvantages.

Raid Sets Pros – a failure of any drive will still provide full access to data Only 1 additional write to secure data Con - Double the amount of disks needed Pros – cheapest Con – A failure of any disk will fail all of the data on the disk

Pros – Data is striped across multiple disks with double parity
Can support two disk failures without data loss A single disk failure does not affect performance Cons – Write intensive applications can run slower because of the overhead of calculating and writing two parity blocks Additional 2 disks per raid set required to handle parity requirements. Pros – Data is striped across multiple disks with parity Can support a single disk failure without data loss Cons – when a disk fails, performance is reduced because the controller has to calculate the missing disks data. Full performance is not recovered until the failed disk is replaced and rebuilt. Write intensive applications can run slowly due to the overhead of calculating and writing the parity blocks

Hybrid RAID types RAID 0+1 RAID 10 RAID 50

IOPS (Input/Output Operations per second)
IOPS are primarily a benchmarking figure that is used to find/compare the performance of storage solutions. IOPS are affected by many factors – such as RAID type, stripe size, Read/Write ratios, Random/sequential balance, cache sizes on controller and disk. Quoted maximum IOPS from manufactures seldom translate to real world applications. Measurement Description Total IOPS Total number of I/O operations per second (when performing a mix of read and write tests) Random Read IOPS Average number of random read I/O operations per second Random Write IOPS Average number of random write I/O operations per second Sequential Read IOPS Average number of sequential read I/O operations per second Sequential Write IOPS Average number of sequential write I/O operations per second

What is a SAN? SAN (Storage Area Network)
Dedicated network that provides access to consolidated, block level data storage. Can be a single device or Multiple Devices Does not provide any file level functionality However file systems that make use of storage on a SAN will allow for file locking sharing etc. Provide access to a pool of shared storage using Either ISCSI, Fibre Channel, FCOIP (fibre channel over IP) for connectivity. Is (ideally) on a separate network from other traffic.

NAS (Network Attached Storage)
A NAS differs from a SAN in that it offers remote Access to a filesystem as if it were a fileserver whereas a SAN offers remote access to Block level storage as if it was a hard disk controller. NAS normally allows connection to its storage via NFS (Unix/Linux), SMB/CIFS (Windows), AFP (Apple) Some NAS can be used for other protocols such as FTP, HTTP, HTTPS,NDMP, and various media streaming protocols. A NAS Head (or Filer) can have single or multiple nodes A NAS essential replaces one or more fileservers

What are SAN/NAS used for
Both provide access to SHARED storage. Both can provide high availability/fault tolerance by having multiple Disk Controllers / NAS Heads A SAN is used to provide BLOCK level storage to multiple Hosts. Connection to a SAN is Normally through Fibre channel or ISCSI (1GB/10GB) over Ethernet. A NAS is used to replace FILE servers by providing CIFS / NFS/FTP etc. features. Connection to a NAS is normally via Ethernet.

XenServer Storage SRs, SR types and more…

XenServer Storage Objects
Describes a particular storage target in which Virtual Disk Images (VDIs) are stored. Flexible—supports a wide variety of storage types. Centralized—easier to manage, more reliable with a XenServer pool. Must be accessible to each XenServer host. So, what is an SR? An SR, or Shared Repository, [Read point 1]. It is the fundamental container object that XenServer defines to store virtual disks for use with virtual machines. [Read point 2] [Read point 3] SRs are specific to the type of storage they connect to, like iSCSI or fiber-channel, and an SR must be accessible to each host in a XenServer pool—in the case of iSCSI or NFS this means a common network interface. In the case of Fiber Channel it means a common fabric connection between the storage HBAs and the XenServer host HBAs.

Types of SRs supported in XenServer
Block based – LVM Fiber Channel (FC) iSCSI Local LVM File based Ext (local) NFS CIFS (ISOs only)

Protocols NFS – inexpensive, easiest to implement, 1 or 10 Gb, overhead* iSCSI – fairly cheap and easy, HW and SW flavors, 1 or 10 Gb FCoE – pretty expensive and difficult, requires CNAs, 10 Gb only FC – most expensive and difficult to implement, 1/2/4/8/16** Gb FC is most common in the enterprise, but 10 Gb+ networking is making NFS and iSCSI more and more common In terms of performance, typically you will find that 8 Gb FC will outperform 10 Gb iSCSI or NFS Ask the customer if they have “storage tiers” Ex: 1. EMC 8 Gb FC 2. HP 10 Gb iSCSI 3. NetApp 1 Gb NFS *increased CPU usage on the hypervisor hosts and array controllers is quite common with NFS and software iSCSI **although even if you have 16 Gb FC, ESX and XS hosts have to throttle it down to 8 Gb, so it’s really 8 Gb FC (Taken from Nick’s C-All presentation)

Comparisons to other Hypervisors
Facet XenServer VMware vSphere Microsoft Hyper-V VM Storage LVM, File-level File-level CSV, SMB 3.0 (CIFS) Storage plugins Storagelink (integrated) PSP, NMP, Vendor supported N/A Multipathing Yes NFS Support No

XenServer – under the hood
streaming services XenAPI vm consoles kernel dom0 xapi XML/RPC; HTTP db VM1 Shared memory pages VM1 Storage Network Domains xenstore SM i/f scripts: qemu-dm Linux storage devices Linux Network devices Blktap/ blkback Netback Xen Hypervisor Hardware devices Shared memory pages

XenServer - under the hood
I/O (Network and Storage) depends significantly on Dom0 This changes in the future releases of Xenserver (disaggregation of Dom0) All VMs use para-virtualized drivers (installed with Xen tools) to communicate Blkfront, netfront components in VMs send I/O traffic to Dom0’s Blkback, netback Hence the SLOW performance and scalability limits with XenServer

The competition ESXi - file based architecture
VMFS - clustered filesystem VMFS 5 - can be shared between 32 hosts Custom plugins can be installed based on shared filesystem

SRs, VDIs, PBDs and VBDs Virtual Disk Data Formats File-based VHD, LVM and StorageLink Before we go into detail about monitoring and troubleshooting storage with XenServer, I want to take a few minutes to review some of the fundamental concepts and terminologies that define how XenServer perceives storage. We will talk about the basic container objects and connectors that XenServer uses to interface with storage and provision space for virtual machines. This includes Storage Repositories, Virtual Disk Images, Physical Block Devices and Virtual Block Devices. We will also talk a bit about the different virtual disk data formats which define the various types of containers that XenServer supports for storing virtual machine data. These include file-based VHD, LVM, or logical volume, based VHD and StorageLink.

VDIs, PBDs, VBDs Virtual Disk Images are a storage abstraction that is presented to a VM. Physical Block Devices represent the interface between a physical server and an attached SR. Virtual Block Devices are connector objects that allow mappings between VDIs and VMs. VDIs, PBDs, and VBDs. XenServer is full of acronyms… So, within an SR you will find VDIs, or virtual disk images. These are a storage abstraction that is presented to a VM. They appear to the VM to be locally attached block devices. The PBD, or physical block device, represents the interface between a physical server and an attached SR. The PBD translates Linux kernel block device operations across the XenAPI SM layer to the virtual containers within the SR. A VBD, or Virtual Block Device, is the connector object that allow mappings between VDIs and VMs. The VBD interprets virtual machine I/O requests and translates them into file operations against the VDI.

SR PBD VDI VBD XenServer Host Virtual Machine PBD VDI VBD XenServer Host This diagram illustrates the relationships between the different storage objects we just discussed. Note that an SR maps a single PBD to each XenServer host. An SR can contain multiple VDIs, and each VDI maps a single VBD to its parent virtual machine. A virtual machine with multiple VDIs has an equal number of VBDs (1 VBD per VDI). Virtual Machine VDI VBD PBD XenServer Host

Virtual Disk Data Formats
Logical Volume (LVM)-based VHDs The default XenServer block device-based storage inserts a Logical Volume manager on a disk. VDIs are represented as volumes within the Volume manager. Introduced LVHD in XenServer 5.5 Enhances LVM for SRs Hosts VHD files directly on LVM volumes Adds Advanced Storage features like Fast Cloning and Snapshots Fast and simple upgrade Backwards compatible The most common type of file system you will see in XenServer for storing virtual disks is LVM-based VHD. [Read slide point 1] LVM is actually a Linux native file system type that is designed to provide a storage abstraction to underlying physical block devices. This solution offers greater performance than file-based VHD because it creates a thinner, more direct type of interface to the actual block device, typically a LUN, being presented by a storage array. Prior to XenServer 5.5 this type of SR used raw logical volumes mapped as VDIs that were accessed directly by the attached virtual machine. [Reveal LVHD] As of XenServer 5.5 this model has been improved into LVHD which enhances LVM for SRs. It hosts VHD files directly on the LVM volumes. It also adds features like fast cloning and snapshots. It is a fast and simple upgrade from XenServer 5.0 LVM to 5.5 LVHD and it is also backwards compatible.

File-based VHD VM images are stored as thin-provisioned VHD format files on either a local non-shared file system (EXT type SR) or a shared NFS target (NFS type SR). What is VHD? A Virtual Hard Disk (VHD) is a file formatted to be structurally identical to a physical Hard Disk Drive. Image Format Specification was created by Microsoft in June, 2005. Let’s talk a little bit about the different virtual disk data formats supported in XenServer. The first one we’ll cover is file-based VHD. [Read slide point 1] -Typically used with NFS -Possible to implement file-based VHD manually on an EXT type SR via the host CLI. Each VDI has a corresponding VHD file stored on the native Linux file system. This has some advantages, such as being cheap, easy to implement and is not vendor-specific. It is, however, less ideal as an enterprise solution since it does not support multipathing and, arguably, will not compare in performance to iSCSI or Fiber Channel type storage. I’ve mentioned VHD several times—it is worth defining it and pointing out why XenServer uses this as the default container type for virtual block devices. [Reveal] VHD stands for Virtual Hard Disk—it is, simply, a file formatted to be structurally identical to a physical hard disk. The specification for this format was developed and published by Microsoft in June of 2005. This technology is not proprietary, therefore, third parties like Citrix are free to leverage it and factor the design into their own products. This has allowed cross-product compatibility for solutions that leverage VHD. We can move VHD files seamlessly between products like Provisioning Server, XenServer and Microsoft hyper-V which offers a tremendous amount of flexibility.

StorageLink (LUN per VDI) LUNs are directly mapped to VMs as VDIs by SR types that provide an array-specific plug-in (NetApp, Equallogic or StorageLink type SRs). The array storage abstraction therefore matches the VDI storage abstraction for environments that manage storage provisioning at an array level. The last disk data format we are going to discuss is StorageLink, or more technically LUN-per-VDI. With StorageLink LUNs are directly mapped to VMs as VDIs by SR types that provide an array-specific plugin. The array storage abstraction therefore matches the VDI storage abstraction for environments that manage storage provisioning at an array level. Dell and NetApp have developed plugins which optimize this technology for EqualLogic and NetApp-type storage arrays. These plugins are natively available in XenServer 5.0 or later. StorageLink, part of the Essentials bundle, provides additional adapters for a wider variety of storage arrays.

StorageLink Architecture XenServer calls direct to Array API‘s to provision and adjust storage on demand. Fully leverages array hardware capabilities. Virtual disk drives are individual LUNs. High performance storage model. Only the server running a VM connects to the individual LUN(s) for that VM. Because StorageLink is able to access the array API directly it can manage and provision array storage on demand. It can, therefore, also fully leverage array hardware capabilities. Virtual disk drives are individual LUNs. This removes complexity and improves performance by offloading storage management to the array itself, allowing it to do what it is best at—managing storage! Because of the direct LUN-to-VDI mapping only the server running a VM connects to the individual LUNs for that VM—this reduces load on the I/O path. This is managed by a special master server which coordinates which servers connect to which LUNs.

iSCSI / FC + integrated StorageLink
LVM vs. StorageLink XenServer 6.1 iSCSI / FC + integrated StorageLink XenServer 6.1 iSCSI / FC Storage Repository Storage Repository LUN LUN LUN VHD header VHD header LVM Logical Volume LVM Logical Volume Looking at this diagram it becomes evident how StorageLink simplifies the storage solution. The VM virtual disk maps directly to the LUN on the array removing all of the storage abstractions and associated overhead from the XenServer hosts. LVM Volume Group VM Virtual Disk

Lessons Learned: Which one is a better SR?

XenServer Storage Command line vs. XenCenter

Xen Storage – command line vs. XenCenter
Can you trust XenCenter info? The answer is – NO XenCenter relies on the database integrity of XenServer resource pool Each entity of XenServer (Resource pool data, CPU, NIC and Storage) are stored in XenServer’s built-in database From a Storage perspective: PBD SR VDI VBD All the above entities are stored in each XenServer’s database And now…why should I remember this? If a VM fails to boot – the above information will come in handy during troubleshooting If SR disappears from XenServer (oh yes it happens!), you will need these commands to recreate the storage path

Xen Storage – Command line
All Xen commands start with xe <Object>-<Action> <parameters> For example: # xe vbd-list vm-name-label=WIN7 The above command will list all the disks attached to the VM Object can be SR, VBD, VDI, PBD Action(s) can be create, delete, list, probe, scan etc. Parameters are specific to the command Entire list of Xen Commands can be found here

XenServer Storage Storage XenMotion

Storage XenMotion – under the hood
New with XenServer 6.1 Supported to migrate VMs across multiple resource pools Also supports Storage migration between local storage and shared storage Please keep in mind – its version 1.0

Storage XenMotion and VDI Migration
Live VDI Migration Storage XenMotion: 1.) Upgrade storage array 2.) Upgrade a pool with VMs on local storage 3.) Provide tiered storage arrays 4.) Rebalance VMs between XenServer pools or CloudStack clusters Live VDI Migration: 1.) Move a VM from cheap, local storage to fast, resilient, array-backed storage; 2.) Move a VM from a development to a production environment; 3.) Move between tiers of storage when a VM is limited by storage capacity; 4.) Perform storage array upgrades.

Feature Overview: Use cases
Upgrade a storage array Upgrade a pool with VMs on local storage Provide tiered storage arrays Rebalance VMs between XenServer pools, or CloudStack clusters “The Cloud” was the major use case we had in mind when designing this.

Feature Overview: Bird’s-eye view
Cross-pool migration and VDI migration consist of the following: Synchronously mirror VDIs between source and destination Create new VM object on destination pool (new ref, same uuid) When copy complete, migrate VM as usual Note: VDI migrate implemented with “localhost” cross-pool migrate!

Feature Overview: Limitations
No more than 6 VDIs, and no more than 1 VM snapshot No more than 3 concurrent ops per host No VMs with PCI pass-through enabled The following are untested and unsupported: HA and WLB on source or destination pool DVSC integration VDI-migrate between multiple local SRs

Feature Overview: Caveats
Minimum network or storage throughput requirements? This is currently unknown, but we’ll begin investigations soon. Can’t check whether destination SR has space available for incoming VDIs – if you fill up an SR, your migration will fail to complete. Extra temporary storage is required on the source (and destination) SR, so you must be careful when migrating VMs off of a full SR. IO performance inside guest will be reduced during migration because of synchronous mirroring of VDIs.

Feature Overview: API walkthrough
Host.migrate_receive host:ref network:ref options:map Result = receive_token VDI.pool_migrate VM.migrate_send vdi:ref vm:ref sr:ref receive_token live:bool Result = vdi:ref vdi_sr:map vif_network:map Result = None VM.assert_can_migrate dest:map

Feature Overview: CLI walkthrough
xe vm-migrate New params: remote-address, remote-username, remote-password, remote-network, vif, vdi Extends the original vm-migrate command Bold params are required to enable cross-pool migration vif and vdi map VIFs to target networks and VDIs to target SRs remote-network specifies the network used for data transfer Can use host/host-uuid to specify host on pool to send VM xe vdi-pool-migrate Params: uuid, sr-uuid uuid of target VDI sr-uuid of destination SR

Feature Overview: GUI walkthrough

Architecture: VDI operations
For each VDI: Snapshot VDI and synchronously mirror all subsequent writes to destination SR Copy the snapshot to destination SR Finally, compose those writes onto the snapshot on the destination SR Continue to mirror all new writes Each of these operations occurs sequentially for each VDI, but each VDI mirror continues until the VM migration is complete VM memory is copied only after final VDI compose is complete VDI 1: snapshot & start mirroring VDI 1: copy snapshots VDI 2: snapshot & start mirroring VDI 2: copy snapshots Copy VM memory

VDI mirroring in pictures
no color = empty gradient = live SOURCE DESTINATION VM mirror VM copy root

VMware Storage VMotion In Action
Delete original VM home and disks 5 “Fast suspend/resume” VM to start running on new home and disks 4 Start changed block tracking 2 Copy VM home to new location 1 Pre-copy disk to destination (multiple iterations) 3 Copy all remaining disk blocks 4 Storage VMotion for ESX 5.x (and 4.x)/VC 3.0 uses two new technologies: changed block tracking and fast suspend/resume. Storage VMotion works as follows: First, all files except for the disks (config file, logs, nvram, etc) are copied from the VM’s old directory to the new directory for VM on the destination datastore. Second, changed block tracking is enabled on the VM’s disks. Changed block tracking tracks changes to the disk, so that ESX knows what regions of the VM’s disk have been written to. It stores this data in a bitmap that can either reside in memory or as a file. For Storage VMotion, the change bitmap is generally kept in memory, but for simplicity we’ve shown it next to the disk on the source datastore. Third, ESX “pre-copies” the data (the VM's disk and the VM's swap file) from the disk on the source to the disk on the destination. During this time, the VM is running and thus may be writing to its disk. Thus some regions of the disk will be changed and will need to be resent. This is where changed block tracking comes in. ESX first copies the entire disk contents to the destination; this is the first pre-copy iteration. It then queries the changed block tracking module to determine what regions of the disk were written to during the first iteration. It then performs a second iteration of pre-copy, only copying those regions that were changed during the first iteration. Ideally the number of changed regions is significantly smaller than the total size of the disk, meaning that the second iteration should take significantly less time. After the second iteration, the changed block tracking module is again queried to see what new regions were modified during the second iteration. A third iteration may then need to be performed, and so on. Eventually the amount of modified data will be small enough that it can be copied very quickly. At this point pre-copy is stopped. Fourth, fast suspend/resume is invoked on the VM. Fast suspend/resume does exactly what its name implies: the VM is quickly suspended and resumed, with the new VM process using the destination VM home and disks. Before the new VM is allowed to start running again, the final changed regions of the source disk are copied over to the destination so that the destination disk image is identical to the source. Fifth, now that the VM is running on the destination datastore, the source VM home and disks can be safely deleted. Source Destination

XenServer Storage Debugging Storage Issues

Troubleshooting tips Check logs: /var/log/{xensource.log,SMlog,messages} Note: xensource.log and messages both implemented with syslog, so they now have consistent timestamps! xn command: CLI to xenopsd Try ‘xn help’ for documentation. tap-ctl command Could be useful for diagnosing problems (can’t describe usage here)

Logging/Debugging All backend drivers use the Smlog to record storage events /var/log/Smlog Logs are rotated, same as system message log, xensource.log files In the python module util.py there is a helper log function -> util.SMlog(STRING) On retail edition, all python backend files are editable for logging/debugging purposes Some things to look for in SMLog (tail –f /var/log/SMLog): Slow SR Probing: Check for breaks in the SMLog timestamps or for SR_BACKEND_FAILURE_46, this is likely an issue with multipathing Storage Issue: Look for the word FAILURE within the log, this is likely an issue communicating with the storage or a corrupted SR Metadata Corruption: Look for SR_BACKEND_FAILURE_18, this is likely an issue with metadata being corrupted due to special characters being used in VM names VDI Unavailable: This is typically caused by faulty clones/snapshots but could also point to misbehaving Storage. Slow SR probing Fix Metadata Fix VDI unavailable:

Debugging tips Use tail –f on any log file while actively debugging a system Correlate logs between xensource.log, SMLog and messages Verify IP settings, firewall config for any IP based storage connections Check status of dependent objects, such as PIFs, VIFs, PBDs etc...

XenServer Storage Lab 2 – Storage Lab

In this lab… 90 minutes Create a local SR
Create a shared SR with Integrated Storagelink Storage XenMotion Lab (Advanced Lab) Create shared SR using NetApp SIM

XenServer Networking Foundational Concepts

Agenda Networking Overview XenServer Networking
XenServer 6.1 – What’s New? Link Aggregation (LACP) Debugging Networking Issues Key Takeaways and References

Before we get started… Lets familiarize ourselves with the following terms: Common Enterprise Networking terminologies: NIC Switch – Core Router, Hub Firewall DHCP, PXE, TFTP, TSB XenServer: LACP VLAN VIF PIF Spend 10 minutes around the table and discuss definitions for the above terms.

Example Enterprise Switching Environment
Server Farm Firewall Backbone Switch Internet Distribution Switch Ask questions such as: Typical speeds seen in LAN environments Where does the NetScaler get connected? Discuss enterprise network connectivity and components (switching, routing, firewall) Access Switch

Example WAN Environment
Discuss any WAN related terminology MPLS Point to point networks

Network Vendors Vendor Components Cisco
Switches, ACE (load balancing), ASA (Firewall), Routers Juniper Network Security Solutions Brocade Switches, Routers Citrix NetScaler F5 River Bed Access Gateway, WAN Accelerators,Load Balancing White board this with the team and find out other vendors that they have seen in projects and what they do. The goal is to get familiarized with different vendors and at least be aware of their top tier products.

XenServer Advanced Training
Introduction to Enterprise Networking

XenServer Networking Conceptual Model
Control Domain (Dom 0) Virtual Machine Bridge VIF vNIC PIF Netback Netfront Linux Driver Start with the VM: VM is attached to a vNIC -> this is displayed in XenCenter -> which is in turn connected to vSwitch (can be a bridge or a Open vSwitch) Discuss – default vswitch in XenServer 6.0 and above; Discuss – how the netfront driver (from xentools) communicates with netback driver (and the vif) and how the vif/pif communicate. Discuss – how Dom0 queues and handles the requests to the driver Discuss – how IRQs work – IRQs can cause issues (Pablo’s infamous project in TX) Xen Hypervisor Hardware Network Card

XenServer Networking Configurations
XAPI Command Line XenCenter Linux Config Files XenServer Pool DB Linux NIC Drivers Network Card Shrugs!! XenServer and its infamous XenServer pool db and how it can be inconsistent during projects. Issues due to pool metadata being inconsistently replicated. Explianed in more detail in the metadata section. Discuss the workflow in this diagram

XenServer Network Terminology
Private (xapi1) VIF Virtual Switches Virtual Machine Network 0 (xenbr0) Network Card VIF PIF (eth0) Virtual Machine VIF

(xenbr0) Network Card PIF (eth0) VIF Virtual Switches Virtual Machine Network 1 (xenbr1) Network Card VIF PIF (eth1) Virtual Machine VIF

Bond 0+1 (xapi2) Network Card PIF (eth0) PIF VIF Virtual Machine PIF (bond0) Network Card VIF VIF PIF (eth1) Virtual Machine

VLAN 25 Virtual Switches Network 0 Network Card PIF VIF Virtual Machine Network 1 Network Card VIF D PIF VIF VLAN 55 Virtual Machine

VLAN 25 Network Card Bond 0+1 VIF Virtual Machine Network Card VIF VLAN 55 PIFs VIF Virtual Machine

Bonding Type (Balance SLB)
0:10 SEC 0:30 SEC 0:20 SEC 0:00 SEC Virtual Machine Bond Network Card Virtual Machine Network Card Unsupported Options: How about using an industry standard bonding mode instead like Etherchannel (mode 0, or balance-rr), LACP (mode 4, or 802.3ad) or active/passive (mode 1, or active-backup). Example: LACP: xe pif-param-set uuid= other-config:bond-mode=802.3ad Active-Backup: xe pif-param-set uuid= other-config:bond-mode=1 Etherchannel: xe pif-param-set uuid= other-config:bond-mode=0 More about Bonding Types: Virtual Machine Stacked Switches

What about faster iSCSI/NFS?
0:10 SEC 0:30 SEC 0:00 SEC 0:20 SEC Virtual Machine Bond Network Card Virtual Machine Network Card iSCSI/NFS Dom0 with iSCSI or NFS software iSCSI/NFS

Distributed vSwitch

Open Virtual Switch for XenServer
Visibility· Resource control · Isolation · Security VM VM VM VM VM VM VM VM VM VM VM Hypervisor Hypervisor Hypervisor With virtualization, one challenge is network visibility. The “last hop” to the VM is now a switch living in in the virtualization software on the host, not the top-of-rack switch as network administrators are more accustomed to. With Distributed Virtual Switching, we can enable better visibility into the network and accomplish things such as: Real time network traffic statistics (Rx bytes, packets etc.) that you can easily get on switches in the physical world Enhanced security. Setting of ACLs on virtual interfaces (VIFs) permits you to provide a configurable, XenServer-provided firewall for the VM. Example: block HTTP, enable only HTTP, and various other configurations are now possible. Enhanced monitoring. Through port monitoring, you could for example determine “if the XenDesktop user is running Pandora and causing performance issues” Simpler network isolation and configuration of VLANs which are especially important in service provider environments—leading to much simpler “multi-tenancy” in the future. Open Source Virtual Switch maintained at Rich layer 2 feature set (in contrast to others on the market) Ships with XenServer 5.6 FP1 as a post-install configuration option

Distributed Virtual Switch Controller
VM VM VM VM VM VM VM VM VM VM VM DVS Hypervisor Hypervisor Hypervisor Controller has it’s own web UI. This provides access by network administrators who probably don’t need XenCenter. One controller can serve multiple XenServer pools. Can have cross-host internal networks Support for Jumbo frames if using vSwitch technology When controller fails, it will fail ‘open’, but may change to fail ‘closed’ in the future. Hypervisor DVS Controller is a XenServer Virtual Appliance that controls multiple Open vSwitches

Distributed Virtual Switch
Built-in policy-based ACLs move with VMs VM VM VM VM VM VM VM VM VM VM VM DVS Hypervisor Hypervisor Virtual Interface (VIF) {MAC, IP} ACLs permit tcp eq domain permit tcp eq domain permit tcp eq domain permit udp eq domain permit udp eq domain permit udp eq domain permit tcp eq 123 Virtual Interface (VIF) {MAC, IP} ACLs permit tcp eq domain permit tcp eq domain permit tcp eq domain permit udp eq domain permit udp eq domain permit udp eq domain permit tcp eq 123 Hypervisor VM ACLs move with the VM, even after a live migration, instead of being tied to a specific host.

XenServer Networking LACP Support – Finally!!

Overview Existing bonding modes and related issues:
active-active (balance-slb) ARP table confusion for some switches (due to MAC flapping) active-passive (active-backup) only one link used at a time LACP (802.3ad) not supported working only on Linux Bridge The modern way – LACP bonding mode: Link aggregation on both server and switch side. Sides communicate with LACP protocol. Better load balancing. Link Aggregation Control Protocol (LACP) – Allows for bonding of multiple network connections to increase throughput and also provides redundancy in the event that one of the links fails. Proprietary aggregation control protocols not supported (EtherChannel, Multi-link) Configured in XenCenter or via CLI IP or MAC based bonding CTX LACP Bonding in XenServer - Configuration and Troubleshooting

Architecture For given NICs, we bond the PIFs and set mode to LACP.
Bond object is created on xapi level. Command bond-create executes vSwitch commands. LACP is configured on the switch ports. Switch and server exchange LACP frames. Switch chooses the active members. Switch and server balance their outgoing traffic independently.

Load balancing Hash-based traffic balancing: Two hashing algorithms:
vSwitch computes flow number (0-255) for each packet Each flow (hash) is assigned to an active link Flows can be moved between links (re-balancing every 10s) Two hashing algorithms: tcpudp_ports: based on IP and port of source and destination default algorithm for LACP source MAC also taken into account VM traffic from different applications can use more than one link src_mac: based on source MAC vSwitch uses the same mechanism as for active-active traffic from one VIF will not be split

Configuration Two new bond modes in XenCenter wizard: CLI commands:
LACP with load balancing based on IP and port of source and destination LACP with load balancing based on source MAC address CLI commands: xe bond-create mode=lacp network-uuid=… pif-uuids=… \ properties:hashing-algorithm=src_mac xe bond-param-set uuid=… \ properties:hashing_algorithm=tcpudp_ports Create LACP bond specify hashing algorithm Hashing algorithm can be specified at bond creation or changed later.

Switch configuration Switch must support 802.3ad (LACP)
Choose unused Link Aggregation Group (LAG) For each NIC being bonded: Find the switch port/unit connected to the NIC Set LAG membership for the port Enable LACP for the port If necessary, bring up the LAG interface If required, configure the VLAN settings on the LAG LACP can be called „mode auto” in some switches. LAG interface is called port-channel for Dell switches If you decide to use LAG, but not LACP

Side note: Static LAG Static LAG: Static LAG is not LACP.
Ports are members of a Link Aggregation Group. Ports do not have LACP set. All links within the LAG are active. Static LAG is not LACP. Use balance-slb, not LACP bond mode for static LAG. LACP can be called „mode auto” in some switches. LAG interface is called port-channel for Dell switches If you decide to use LAG, but not LACP

Limitations and changes
LACP for Linux Bridge not supported. If used anyway, note that it was renamed from 802.3ad to lacp. EtherChannel/Cisco Port Aggregation Protocol (PAgP) not supported. Switches must be stacked. Up to four NICs per bond for all bond modes. Only 2 NICs per bond supported for Linux Bridge. Improvements for bonding planned in near future: Switching off dynamic load balancing (preventing MAC flapping) Choice of active link for active-passive bonds

Troubleshooting Frequent surprises: Mismatching settings
Wiring misinformation – wrong ports might be aggregated Switch rules – and it can decide to use just one link as active No HCL – 802.3ad is a standard Longer set-up – more time might be required for LACP and DHCP One ‘fat’ flow still will not be split In ovs commands, ‘bond mode’ means ‘hashing algorithm’ Mismatching settings LACP only on server side – we should have active-active mode LACP only on switch side – you are on the mercy of the switch ‘Fat’ flow – e.g. One VM to one NFS server

Debugging CLI command: xe bond-param-list vSwitch commands:
ovs-vsctl list port ovs-appctl bond/show bond0 (lacp_negotiated: false/true) ovs-appctl lacp/show bond0 (actor and partner information, hashes) xensource.log – network daemon entries, including DHCP /var/log/messages – vSwitch entries, including hash shifting ‘Fat’ flow – e.g. One VM to one NFS server

XenServer Networking SR IOV and more

SR-IOV SR-IOV – VM to talk directly to a NIC rather than having to pass network traffic through Dom0 SR-IOV virtualizes the network in hardware, rather than in software benefits and disadvantages: good: there has generally a significant speed improvement in per-VM bandwidth (and usually in latency) when using SR-IOV instead of Dom0-mediated networking good: partitioning good: avoiding the switch bad: avoiding the switch bad: migration issues Please check the SRIOV supported hardware in

SR-IOV Pre-requisites
SR-IOV capable network device (i.e. Intel Gb-E Controller) The SR-IOV NIC cannot be used as the management. A second physical NIC must be installed on the system iommu must be enabled on the XS host /opt/xensource/libexec/xen-cmdline --set-xen iommu=1 Please check the SRIOV supported hardware in

XenServer Networking – without SR-IOV
Control Domain (Dom 0) Virtual Machine Bridge VIF vNIC PIF Netback Netfront Linux Driver Xen Hypervisor Hardware Network Card

XenServer Networking – with SR-IOV
Control Domain (Dom 0) Virtual Machine Bridge VIF vNIC PIF Netback VF Driver Linux Driver Xen Hypervisor Hardware SR-IOV Aware Network Card Network Card

Using Multi-Queue NICs
Post grant on I/O channel Push into the network stack 9 1 UnMap buffer I/O Channels Driver Domain 7 Guest Domain Backend Driver Frontend Driver gr Advantage of multi-queue Avoid data copy Avoid software bridge 3 Map buffer post buf on dev queue 2 Physical Driver 5 DMA One RX queue per guest 6 8 event IRQ Xen guest MAC addr MQ NIC Hardware Incoming Pkt demux 4

VLAN Overview A VLAN allows a network administrator to create groups of logically networked devices that act as if they are on their own independent network, even if they share a common infrastructure with other VLANs. Using VLANs, you can logically segment switched networks based on functions, departments, or project teams. You can also use a VLAN to geographically structure your network to support the growing reliance of companies on home-based workers. These VLANs allow the network administrator to implement access and security policies to particular groups of users.

VLAN Overview

VLAN in details A VLAN is a logically separate IP subnetwork.
VLANs allow multiple IP networks and subnets to exist on the same switched network. For computers to communicate on the same VLAN, each must have an IP address and a subnet mask that is consistent for that VLAN. The switch has to be configured with the VLAN and each port in the VLAN must be assigned to the VLAN.

VLAN in details A switch port with a singular VLAN configured on it is called an access port. Remember, just because two computers are physically connected to the same switch does not mean that they can communicate. Devices on two separate networks and subnets must communicate via a router (Layer 3), whether or not VLANs are used.

Dom0 – Limitations with VLAN
Dom0 doesn’t support VLANs Management network needs to be on its on NIC XenMotion cannot be separate from management NIC iSCSI, NFS can be on VLANs Dom0 supports NIC bonding

XenServer Networking Lab 3 – Networking Lab . /etc/xensource-inventory

In this lab… 30 minutes Create a LACP bond
Create a distributed virtual switch Explore VIFs, PIFs and eth# interfaces

Resource Links XS Administrator’s Guide – revised section on bonding The standard: Internal wiki page: Feature lead: Kasia Borońska 802.3ad became 802.1AX due to avoiding layering violation

XenServer Health Monitoring & Troubleshooting

Agenda Managing & Monitoring this wonderful piece of software
It’s broken, How do I fix it? Show me Your Commands!

XenServer Monitoring & Troubleshooting
Managing & Monitoring this wonderful piece of Software

Using Console Folder View
Manage XenServer environment in your own way Selection styles: SRs (Remote and Local) Snapshots Networks VMs Pools Servers Custom Folders Simple drag & drop functionality

Delegated Management (Web Self Service)
Delegate VM level access to end-users View consolidated virtual machine guests from multiple resource pools Basic life cycle operations such as Start, Stop, Suspend and Reset on virtual machine Remote login (VNC for Linux Guests and RDP for Windows Guests) to the virtual machine guests Force Shutdown & Force Reboot (Only with XS Tools) Fully AD Integrated Release Notes Installation Guide

Web Self Service Roles XenServer Roles WSS Roles Pool Admin WSS Admin
Pool Operator WSS Operator VM Admin VM Operator VM Power Admin Read Only WSS User No Role (Default) More Detail on Admin Guide

Web Self Service Oddities
Sharing Through Tags? What? The sharing functionality is implemented in Web Self Service through tags in XenCenter. When you share a VM with a user in Web Self Service, a tag with the name of the user is created in the XenCenter.

Web Self Service Oddities
Authentication Can’t be changed – Better Make up your mind Integrated Auth is pretty annoying Not Really AD, XS AD! Process is still manual for AD? Must Add every User unless auto-login is enabled User authentication is configured either to use the built-in database or through XenServer Active Directory. This is done while setting up XenServer Web Self Service and cannot be changed thereafter. If you choose to use built-in database, you will need to manually create username and password for every user. Users can be added in XenServer Web Self Service only if they belong to the Active Directory Users in XenCenter either as a part of a group or an individual user. You can enable auto-login from the Server Settings.

Legacy Performance Monitoring
In Citrix XenServer 5.6+ you are no longer able to collect Host and VM performance data via the “legacy” API calls. Instead they direct you to use the web services via URL

RRDs (Round Robin Databases)
RRDs are the OpenSource industry standard, high performance data logging and graphing system for time series data. XenServer’s Performance monitoring file is actually an RRDTool File, go figure. Download format is: ceJan1,1970> Use &host=true suffix to get RRD for a XS Host XML downloaded contains every VM's data. It is not possible to query a particular VM on its own or a particular parameter. To differentiate, the 'legend' field is prefixed with the VM's uuid. Free RRDTool available at:

TAAS (Previously XenoScope)
XenServer Metrics: Uses RRD Metrics Can be drilled down by domain, time frame, graph size. Host memory usage Piechart Host Hardware Pool hosts Overview Pool Hardware Comparison Installed Hotfixes Comparison Running Virtual Machines Overview CPU Flags Overview and Comparison Recommendations Server Log review tools

TAAS (Previously XenoScope)
A Very Clean Interface (Also for XA/XD/NS) XenApp 6 & XenDesktop 5 Prereqs Microsoft .NET Framework, Version 3.5, with Service Pack 1 Microsoft Windows PowerShell version 2.0 Must be run from the same domain as the Citrix Desktop Delivery Controller machines when capturing XenDesktop information. User must be a local administrator and domain user for each machine being queried. Need to run as administrator XenServer NetScaler

DoubleRev’s Management Pack (http://www.doublerev.com/ )
Pros: Automatic and agent less mass update of XenTools DVD Manager Storage Analyzer Mass VM-Copy Re-convert templates to virtual machines VM-Inventory Export Seamless XS Console Integration

DoubleRev’s Management Pack (http://www.doublerev.com/ )
Cons: CLI Commands Limited Monitoring Most of these features probably will come in Future XS Console Costs $$$ Verdict: Should only be considered if you are extremely lazy as most of these features are nothing terribly difficult to accomplish

It’s Broken, How do I fix it?

Methodology For Recovery Overview

Step 1 - Is the Pool Master down?
Then the Master server cannot be contacted and is likely non-functional. If connection to the XenServer pool is not possible from XenCenter If issuing commands (xe host-list) at CLI of a surviving host returns “Cannot perform operation as the host is running in emergency mode” Under some circumstances (such as power outages) there may have been multiple XenServer host failures in the pool. In this case, the pool master should be attempted to be recovered first. Brief Troubleshooting Guide:

Who’s the Pool Master Anyways?
Determined through pool.conf located under /etc/xensource Master for PoolMaster slave:<pool master IP> for all others Important Commands when in Emergency Mode: xe pool-emergency-reset-master xe pool-emergency-transition-to-master Follow up with: xe pool-recover-slaves Check that default-SR is valid: xe pool-param-list

Step 2 - Recover Pool operations
If recovery of the pool master is not possible within a short timeframe, it will be necessary to promote a member server to a master to recover pool management and restart any failed Virtual Machines. Select any running XenServer within the pool that will be promoted. From the server’s command line, issue the following command: xe pool-emergency-transition-to-master Once the command has completed, recover connections to the other member servers using the following command: xe pool-recover-slaves Verify that pool management has been restored by issuing a test command at the CLI (xe host-list) Each member server has a copy of the management database and can take control of the pool without issue

Step 3 - Verify which XenServer(s) failed
This step verifies which XenServer(s) in the pool have failed Issue the following command at the Command line of a surviving pool member: xe host-list params=uuid,name-label,host-metrics-live Any servers listed as host-metrics-live = false have failed. If it was necessary to recover the Pool Master with Step 2, the Master will now show as true. Note down the first few characters of the UUID of any failed servers (or the UUID of the Master server if it originally failed). This step also gives the chance to note the Universally Unique Identifiers (UUIDs) of the failed servers for use later. Important: Wherever possible, visually verify that the server has failed. If a failure has occurred in another subsystem on the server (for example, the xapi service has failed) and virtual machines are still active, starting the VMs again could in rare cases cause data corruption. It is not necessary to note the whole UUID because tab-completion within the CLI can be used in the next step.

Step 3 - Verify which XenServer(s) failed
Result of xe host-list params=uuid,name-label,host-metrics-live Important: Wherever possible, visually verify that the server has failed. If a failure has occurred in another subsystem on the server (for example, the xapi service has failed) and virtual machines are still active, starting the VMs again could in rare cases cause data corruption.

Step 4 - Verify which Virtual Machines have failed
Regardless of whether a Master or Slave server fails, virtual machines running on other hosts continue to run. Issue command at the CLI of a surviving server. Use the UUIDs from Step 3, type first 3 digits of the UUID, and press [tab] to complete: xe vm-list is- control-domain=false resident-on=UUID_of_failed_server Note that the power state of these VMs is still “running” even though the server has failed. Step 5 will take care of this problem. Repeat this step if necessary using the UUID of any other failed servers (including master). This step verifies which VMs were running on the failed server(s).

Step 4 - Verify which Virtual Machines have failed
Result of xe vm-list is-control-domain=false resident- on=UUID_of_failed_server

Step 5 - Reset power state on failed VMs
To restart VMs after a host failure, it is necessary to reset their power state. Issue the following command at the command line of a surviving server: xe vm-reset-powerstate resident-on= UUID_of_failed_server --force – multiple Alternately, you can reset VMs individually. Verify that there are no VMs still listed as resident on the failed server by repeating step 4. The vm-list command should now return no results. Caution! Incorrectly using the --multiple option could result in ALL virtual machines within the pool being reset. Be careful to use the resident-on parameter as well. Alternately, you can reset VMs individually.

Step 6 - Restart VMs on another XenServer
Load XenCenter and verify that each VM that was originally running on the failed server is now marked as halted (Red icon next to the VM) Note: VMs which have a home server assigned will not appear in XenCenter – Why? xe vm-param-set uuid=<uuid of vm to change> affinity=<uuid of new home server> Restart each VM on a surviving pool member Why?- Because that specific home server host is still down. A workaround for this is changing the home server by changing the affinity parameter from the CLI: xe vm-param-set uuid=<uuid of vm to change> affinity=<uuid of new home server> After this, the VM will appear in XenCenter and the Home Server configuration can be changed. Note: It is only possible to restart virtual machines on a surviving pool member if they are located on a remote shared storage repository. If one or all on the virtual disk files of a VM are located on the local hard disk of the failed server, it will not be possible to start them without recovering the failed server.

Collecting Crash Dumps (Regular Style)
Configure System for Memory Dump: Find the Universally Unique Identifier (UUID) of the VM from which you would like to collect memory dump using: xe vm-list Based on the UUID of the virtual machine, find a domain ID in the current system. The Domain ID changes every time the VM starts: list_domains Use the Domain ID (1st number listed) to crash the host using: /usr/lib/xen/bin/crash_guest <VM Domain> INTERNAL - How to Force a Complete Memory Dump for an Unresponsive Windows Virtual Machine in Both XenServer and VMware ESX Environments MSFT Configuring Crash Dumps

Collecting Crash Dumps (Ninja Style)
Crashdump redirection to dom0 filespace When streaming PVS guests, where you cannot save the crashdump to the Pagefile after a crash due to the network being disabled automatically When the root drive doesn't have enough space to store a Pagefile as large as the guest's memory size Consider mounting either an iscsi lun or nfs, cifs filesystem under /crashdumps to store large dumps since root partition typically has <1.5GB available space

Open /opt/xensource/libexec/qemu-dm-wrapper with vi Search for "SDL_DISABLE_WAITMAPPED“ and modify by adding the following at the line immediately following it: qemu_args.append("-priv") qemu_args.append("-dumpdir") qemu_args.append("/crashdumps") qemu_args.append("-dumpquota") qemu_args.append("1024") Create the directory where the dumps will be stored: mkdir -p /crashdumps

Reboot the VM use the crash_guest util to force the crashdump. If the condition leads to the BSOD, just wait for the streaming to finish and check the file at /crashdumps. Example: Find dom ID: xe vm-list name-label="Windows Server 2003" params=dom-id Result: dom-id ( RO) : 3 To crash the domain id 3: /usr/lib/xen/bin/crash_guest 3 Copy the dumps outside the VM with a tool such as winscp. See doc for more info

VM Needs Imaginary Storage
While working on-site with a client, XenServer decided to subscribe to the cult of Jacob Marley and refused to turn on VM saying ‘This VM needs storage that cannot be seen from that host’

VM Needs Imaginary Storage
Caused by CIFS/DVD drives on XenServer and the machine where the ISOs/DVD resided failed. xe vm-cd-eject –multiple (Gets rid of Faulty CD/DVD media)

File System on Control Domain Full

Result of the Dom0 Storage allocation becoming Full Typically caused by overgrown Log Files XenServer logrotate feature can break - it cannot parse the size parameter. Check Disk Space to validate issue: df –h Check what’s causing the problem: find {/path/to/directory/} -type f -size +{size-in-kb}k -exec ls -lh {} \; | awk ‘{ print $9 “: ” $5 }’ Specify the path and the size in kb In order to determine root cause lower the log rotations and enable compression (/etc/logrotate.conf)

In Case of Total Loss (XAPI died): xe-toolstack-restart Typically this issue is due to TCPOffload freaking out causing unnecessary entries in logfiles. Run Script to disable TCP/Checksum Offload Disabling TCP Offload and checksum means that the main CPU handles all the load which was previously handled directly by the network card The network driver throws exceptions because changes have been made in the main Linux kernel around network drivers Script is provided

XenServer Unable To See NICs
The XenServers are experiencing problems detecting their network cards. The network cards are registered in the XenServer OS, however, they do not display in the XenServer console.

XenServer Unable To See NICs
Issue is caused by disconnecting the pool master while it has attached slave servers. Log into the local XenServer, opening the file pool.conf located under /etc/xensource, deleting the contents of this file and replacing them with the word master. Set the management console’s NIC to a valid IP/subnet Run xe-toolstack-restart Edit the pool.conf again to read slave:<pool master IP>

Frozen VM My VM(s) Froze and now XenCenter refuses to respond because it simply does not care about how troubled my life already is

Frozen VM This is typically caused by conflicting commands being sent through the Xen Center console resulting in erroneous metadata To force reboot VM (Like a Boss) Find out the target VM UUID, type the command line : xe vm-list (Hint: Use | more when there are too many VMs) Find your VM(s) UUID Run the command: xe vm-reset-powerstate uuid=<Machine UUID> force=true Might Require hotfix

XenServer Installation Fails
I started installing XenServer on a bunch of machines, went out to a super relaxing lunch only to come back to my XenServer installation stuck at 49%

XenServer Installation Fails
Some versions of XenServer do not include the firmware for the Qlogic QLA2100, QLA2200, QLA2300 and QLA2322 series NICs Turn on the host, using the XenServer installation media At the boot prompt, type: shell You will now be presented with a command prompt. Type the following: rmmod qla2xxx; rmmod qlge Type: Exit Install XS normally Might Require hotfix

Commands CheatSheet XenServer Host and Dom0: xentop Host Network
xe-toolstack-restart to restart XAPI eject to eject physical CD from server. cat /etc/xensource-inventory to see your host information. Network ifconfig to see what interfaces are up for networking. ethtool –p eth0 60 for make NIC flash for identification. ethtool eth0 to check the status of the interface. ethtool –i eth0 to check the driver type and version of NIC. Full List

Commands CheatSheet Disk Multipath VM Logs
fdisk –l to view local disk information. df –h to see how much space you have left in root disk. Multipath multipath -ll to view the current mulitpath topology as presented by control domain. VM xe vm-reboot vm=<VM Name> force=true to hard-reboot a VM. Logs xen-bugtool –yestoall to get the logs for support. tail –f /var/log/messages to view events in messages log.

Lab 4 -Show Me Your Commands!

In this lab… 60 minutes Install Performance monitoring pack (new with XenServer 6.1) Configure Performance VM and run sample workloads (Discuss results) TAAS.CITRIX.COM – Upload server status reports (Advanced) – Break/Fix lab

Questions, Yes?

VHD or VHDX? New VHDX virtual hard disk format: Larger capacity 64TB
Protection against data corruption during power failures by logging updates to the VHDX metadata Improved alignment of the virtual hard disk format to work well on large sector disks Custom metadata (OS version, patches applied, etc.) A 4-KB logical sector virtual disk that allows for increased performance when used by applications and workloads that are designed for 4-KB sectors. Efficiency in representing data (trim), results in smaller file size and allows the underlying physical storage device to reclaim unused space. Trim requires physical disks directly attached to a virtual machine or SCSI disks, and trim-compatible SSDs. VHDX Now supports up to 64 TB of storage. It helps to provide protection from corruption due to power failures by logging updates to the VHDX metadata structures. It also helps to prevent performance degradation on large-sector physical disks by optimizing structure alignment.

Slide Overflow

Memory Ballooning Hyper-V 2.0 introduced memory over commitment
Memory over commitment increases virtual machine density Minimum memory and startup memory were the same things in Hyper-V 2.0. A VM always started with the minimum memory An idle VM may use less memory than is required at startup Hyper-V 3.0 differentiates between minimum memory and startup memory When I talked about the Hyper-V release history I mentioned that there were two big new features that came out with Hyper-V memory over commitment and live migration. So I want to talk about those two features first, and talk about how Hyper-V 3.0 builds on what was already there with Hyper-V 2.0. Memory over commitment still exist with Hyper-V 3.0 and the reason why it exists is because memory over commitment allows you to allocate more memory to your virtual machines collectively then is physically present within the system. By doing this you are able to increase your virtual machine density and improve your overall return on investment for your server hardware. That's great, but memory over commitment had one fatal flaw within Hyper-V 2.0. The fatal flaw has something to do with the way the Windows works. When Windows starts up it usually consumes more memory a startup that it does when Windows is idle. The problem is that the memory over commitment featuring Hyper-V 2.0 treated startup memory and minimal memory is the same thing. Startup memory was the minimum memory. There was no way that the virtual machine could ever use less memory than what it used to start. This is all change in Hyper-V 3.0. In Hyper-V 3.0 Microsoft introduced a feature called memory ballooning. Memory ballooning is the same as memory over commitment except now the startup memory and minimum memory settings have been broken apart and are treated as two completely separate things. So you can allocate the memory that a VM needs at system startup and then release some of that memory as the virtual machine stabilizes and goes idle. So this could help you to use Hyper-V memory a lot more efficiently.

Memory Ballooning Hyper-V 2.0 Hyper-V 3.0
So to make the concept of memory ballooning make just a bit more sense, take a look at the slide. What I’ve done here in this slide is to take the virtual machine settings screen from Hyper-V 2.0 and put it right next to the virtual machine settings screen from Hyper-V 3.0 and select the memory configuration option. You can see that both Hyper-V 2.0 and 3.0 support the use of static or dynamic memory. If you enable dynamic memory then Hyper-V 2.0 lets you specify startup RAM and maximum RAM, but if you look over Hyper-V 3.0 you’ve got some different options. You can still set the startup RAM and maximum RAM, but there is also an option to set the minimum RAM. You can set the minimum RAM to be lower than the startup RAM so that Hyper-V is able to reclaim some of that memory that gets used when a Windows virtual machine is starting up. So that's what memory ballooning looks like in Hyper-V 3.0. Hyper-V 2.0 Hyper-V 3.0

Live Migrations Beyond the Cluster
Previously it was only possible to live migrate a VM to a cluster node Hyper-V 3.0 makes it possible to live migrate a VM to a host that exists outside of the cluster. This is called Shared Nothing Live Migration Shared Nothing Live Migration should not be used as a long term replacement for traditional failover clustering Microsoft has actually done a tremendous amount of work around live migrations beyond what you've already seen. One of the big changes to live migrations that with previous versions of Hyper-V it was only possible to live migrate a virtual machine that existed within a virtual machine cluster. You could move a virtual machine to another cluster node, but you couldn't move it anywhere else and you couldn't do live migrations from non-cluster members. In Hyper-V 3.0 that limitation goes away. Now it's possible to live migrate a virtual machine to hosts that exists outside of a cluster, or even to live migrate from outside a cluster to inside a cluster. So you have ways of moving virtual machines around with a tremendous amount of flexibility. This is something that Microsoft refers to as Shared Nothing Migrations. It is important to note that shared nothing migrations shouldn't be used as a long-term replacement for a traditional failover clustering solution. It's more a convenience feature that you can use to move virtual machines around and get them where you need them to be.

Shared Nothing Live Migration
Requirements: At least 2 Hyper-V 3.0 hosts Each server requires adequate storage (local or remote SMB) Hosts must have the same family or type of processor if you are using the processor compatibility feature Hosts must exist in the same AD domain The hosts must be connected by a gigabit or faster network link Hosts must have Client for Microsoft Networks and File and Print Sharing for Microsoft Networks enabled Pass through storage cannot be used The same virtual switch configuration must exist on each host I don't want to get too technical because this presentation is designed to be a light overview of the new features in Hyper-V 3.0, but I do at least want to mention that there are some requirements that you have to adhere to in order to be able to do shared nothing live migrations. The first criteria is that you have to have at least two Hyper-V 3.0 host. They can be standalone hosts or they can be clustered. It doesn't really matter. The next requirement is that each server requires adequate storage this can be local storage or it can be remote storage in terms of SMB. The third requirement is that hosts must have the same family or type of processor if you are using the processor compatibility feature. Next, the hosts must exist within the same Active Directory domain. If you've got hosts in different domains and you want to live migrate between those, then that’s not something that you're going to be able to do unfortunately. The next criteria is that the hosts have to be connected to each other via a gigabit or faster network link. Live migrations are bandwidth intensive and you have to have a gigabit or higher network to facilitate shared nothing live migrations. Another requirement - and this is a biggie - is that the host have to have the Client for Microsoft Networks and the File and Print Sharing Service for Microsoft Networks enable. Otherwise shared nothing live migrations won't work. Also, pass-through storage can’t be used because it can’t be live migrated. Finally, both machines that are involved in the shared nothing live migration have to have the same virtual switch configuration. So those are the requirements behind making a shared nothing live migration work

No Shared Storage Requirement
Live Migration in Hyper-V 2.0 required the use of shared storage Shared storage is recommended, but not required in Hyper-V 3.0 Shared Storage VM Host 1 Host 2 Another huge change that Microsoft has made with regard to live migration in Hyper-V 3.0 is that now shared storage isn’t required anymore. It's recommended that you shared storage for performance reasons and for reliability, but you don't have to. In Hyper-V 2.0 you had to have what was known as a Clustered Shared Volume. This was shared storage that could be used by all the host within the virtualization cluster. That requirement no longer exists in Hyper-V 3.0. Now instead of using shared storage you can actually use local storage and perform live migration across an Ethernet cable. So this is a huge new feature for Hyper-V 3.0, and it's one of the centerpieces of the new hypervisor.

Failover Clustering Microsoft has made numerous changes to failover clustering in Hyper-V The most significant change is the ability to create clusters without shared storage Microsoft has also made scalability improvements (which will be discussed later) As you can see a great number of enhancements that Microsoft has made an Hyper-V 3.0 are related to failover clustering. Obviously the ability to do live migrations without the need for shared storage or a clustered shared volume is huge, but there are also other improvements as well. Microsoft has made tremendous scalability improvements, which I will be talking about a little bit later on.

Storage Migrations Live storage migration replaces Hyper-V 2.0 Quick Storage Migration Storage migration is different from live migration. It is designed to move a virtual machine from one storage location to another, not from one host to another In Hyper-V 2.0 there was a brief outage that occurred at the end of a storage migration In Hyper-V 3.0 storage migrations occur without down time Another area where Microsoft has enhanced Hyper-V 3.0 is with regard to storage migration. Storage migrations are different from live migrations. Live migrations involve moving a running virtual machine from one host server to another. Storage migrations involve moving a virtual machine’s components (specifically the virtual hard drive files and the components that make up the virtual machine) from one storage location to another. Storage migrations existed in Hyper-V 2.0, but they were referred to as quick migrations. The downside to quick storage migrations was that there was a brief outage that occurred at the end of the storage migration while Hyper-v switched over to using the new location. In Hyper-V 3.0 that limitation is gone. Storage migrations actually occur without any downtime at all. So storage migrations can now be referred to as storage live migrations because the process really is done live, with no downtime.

Hyper-V Replica Replicate a running virtual machine to an alternate host Replication can occur near real time with an adjustable schedule The Replica feature works well in high latency environments or in environments where connectivity is not always reliable Hyper-V replica is not a fail over clustering solution. It is a disaster recovery solution One of my personal favorite new features in Hyper-V 3.0 is something called Hyper-V replica. Hyper-V replica refers to the ability to replicate a running virtual machine to another host. This host can be an alternate Hyper-V server in a remote datacenter. The nice thing about the Hyper-V Replica feature is that it works really well in high latency environments or in environments where connectivity is not always reliable, such as over an Internet connection. It is worth noting that the replication process isn't real time. You're not going to see change that you make in the primary virtual machine show up instantly in the replica. Instead, replication is near real-time. There's actually an adjustable schedule that controls how frequently changes are replicated. Generally virtual machines are going to be replicated at five-minute intervals, but there are a number of different factors that can affect this. Another thing that you need to know about the Hyper-V replica feature is that this isn't a failover clustering solution. When a failure occurs in the primary data center the replica is not going to automatically take over (although there is a process of manually activating the replica server). Hyper-V replicas are intended to be a disaster recovery solution. In other words if something goes wrong in your primary data center and you lose a virtual machine you can use the replica to get the contents of that virtual machine back. So Hyper-V Replica is a very powerful feature and it brings enterprise class replication to small businesses. That type of feature would have previously been totally out for smaller organizations due to financial reasons.

Virtual Fibre Channel Virtual Fibre Channel allows VMs to connect directly to Fibre Channel storage through a virtual SAN Virtual Fibre Channel is based on N_Port ID Virtualization (NPIV) mapping Physical Host Bus Adapters must support NPIV and must be connected to an NPIV enabled SAN Each Host Bus Adapter must be assigned two World Wide Names in order to facilitate Live Migration Another new feature that hasn't really received as much press as some of the other features, but that I'm excited about nonetheless is something called virtual Fibre Channel. Virtual Fibre Channel will allow virtual machines to connect directly to Fibre Channel storage through a virtual SAN. Virtual Fibre Channel is based on something called N_Port ID virtualization mapping, or NPIV as it is often abbreviated. So because of that physical Fibre Channel host bus adapters must support NPIV and they have to be connected to an NPIV enable SAN, and that SAN has to adhere to an NPIV topology. One of the really cool things about virtual Fibre Channel is it's actually possible to live migrate virtual machine that uses virtual Fibre Channel. But in order to make it happen there are some underlying requirements. Specifically the host from which you're going to live migrate virtual machines to has to have host bus adapters that can be used for Fibre Channel communications. That's a very logical requirement, but also each host bus adapter has to be assigned two separate worldwide names in order to facilitate live migration. The reason for that is because if there weren’t two separate worldwide names for each host bus adapter then connectivity would be momentarily lost to this Fibre Channel storage during the live migration process. Having two separate worldwide names keeps that from happening, so you can truly do a live migration without losing connectivity to Fibre Channel storage.

Hyper-V Extensible Switch
Layer 2 virtual network switch that provides programmatically managed and extensible capabilities to connect virtual machines to the physical network Provides an open platform for multiple vendors to provide extensions written to standard Windows API frameworks Hyper-V Extensible Switch The Hyper-V Extensible Switch in Windows Server 2012 is a layer-2 virtual network switch that provides programmatically managed and extensible capabilities to connect virtual machines to the physical network. The Hyper-V Extensible Switch is an open platform that makes it possible for multiple vendors to provide extensions that are written to standard Windows API frameworks, the reliability of which are strengthened through the Windows standard framework. In addition, along with its extensions, the open platform allows you to manage with Windows PowerShell, programmatically with Windows Management Instrumentation, or the Hyper-V Manager user interface. <

Hyper-V Replica Hyper-V Replica provides asynchronous replication of virtual machines for the purposes of business continuity and disaster recovery Write operations are tracked on the source machine and replicated to the destination VM, so that upon failure, the replica can take over at a consistent point in time Hyper-V Replica Windows Server 2012 introduces Hyper-V Replica, which provides asynchronous replication of virtual machines for the purposes of business continuity and disaster recovery. In the event of failures such as a power outage, fire, or natural disaster at your primary site, you can manually fail over the production virtual machines to the server running Hyper-V at your recovery site. During failover virtual machines are brought back to a consistent point in time, the rest of your network can access the virtual machines within minutes with minimal impact to business activities and you can manually revert the virtual machines to the server running Hyper-V at your primary site, after the primary site recovers from the failure. <

“Share Nothing” Live Migration
“Share Nothing” Live Migration enables VMs to transfer between Hyper-V servers without the need to be on the same cluster or shared storage Hyper-V utilizes SMB 2.2, which uses Remote Direct Memory Access (RDMA), allowing direct access to storage – shared storage no longer required Support for simultaneous live migrations is now provided! "Share Nothing" Live Migration In Hyper-V, “shared nothing” live migration enables the migration of a virtual machine from a server running Hyper-V to another one without the need for both of them to be in the same cluster or to share storage. This capability means you can live migrate a virtual machine from one cluster to a different cluster without setting up complex storage mappings. <

Support for Native NIC Teaming
Built-in support for heterogeneous NIC teaming with LACP support Supports up to 32 network adapters in a team

Hyper-V | XenServer | vSphere
Hyper-V (2012) XenServer (6.1) vSphere vSphere Ent+ Storage Virtual Fibre Channel Yes No 3rd Party Multipathing (MPIO) Yes (VAMP) Native 4-KB Disk Support Undocumented Maximum Virtual Disk Size 64TB VHDX 2TB 2TB VMDK Maximum Pass Through Disk Size Varies 15TB 64TB Offloaded Data Transfer Yes (VAAI) Dynamic Virtual Machine Queue VMq NetQueue IPsec Task Offload SR-IOV DirectPath I/O Storage Encryption Network Extensible Switch Replaceable Confirmed Partner Extensions 4 2 Private Virtual LAN (PVLAN) ARP/ND Spoofing Protection vShield App/Partner DHCP Snooping Protection VShield App/Partner Virtual Port ACLs Trunk Mode to Virtual Machines Port Monitoring Per Port Group Port Mirroring

Hyper-V | XenServer | vSphere
Hyper-V (2012) XenServer (6.1) vSphere Ent+ Live Migration VM Live Migration Yes 1GB Simultaneous Live Migrations Unlimited Undocumented 4 10GB Simultaneous Live Migrations 8 Live Storage Migration Shared Nothing Live Migration No Network Virtualization Partner Other Incremental Backups Inbox VM Replication NIC Teaming Integrated High Availability Guest OS Application Monitoring Failover Prioritization Affinity & Anti-Affinity Rules Cluster-Aware Updating

Hypervisor Overview Real World Applications

Interoperability Overview
Pool Master Slaves Resource Pool Hosting Management Hypervisor Communication Library (HCL) Database (SQLServer) VDA Active Directory Desktop Controller Connection to XAPI on pool master via HTTP port 80 Shared Storage Virtual Desktops running Receiver Windows Communication Foundation (WCF) [shrink speaker notes] First let’s take a couple minutes to review how XenServer and XenDesktop communicate with each other, and how these interactions might present certain bottlenecks that could compromise performance and stability in larger deployments. [reveal] A XenServer resource pool is a group of machines running XenServer on similar hardware that are bound together into a single managed entity which can host virtual machines. One machine within the resource pool is always acting as the pool master, while the rest of the machines are slaves. Though any machine can take the master role, only one host at a time is allowed to assume it. The XenServer host designated as the pool master is responsible for receiving, queuing and processing tasks issued to the XenAPI from the outside world. The XenAPI, or XAPI, is the service stack that runs on the XenServer hosts which is responsible for handling management functions for the hypervisor. A resource pool can be combined with shared storage to facilitate live migration, or XenMotioning, of virtual machines between hosts with minimal downtime. On the XenDesktop side we have a suite of services that handle communications between the hypervisor, in this case XenServer, and also the virtual desktops themselves. Hosting management is responsible for queuing operations to be sent to the hypervisor, such as starting virtual desktops. This communication is facilitated by the hypervisor communication library, or HCL, [reveal] which understands which protocols and communication methods to initiate when logging into and requesting services from the XenServer hosts. XenDesktop communicates directly with the XenAPI on the pool master and thus can queue tasks in the pool automatically. When virtual desktops are booted and running on the XenServer pool, the virtual machines launch receiver, which is an agent that communicates directly back to the VDA management services on the desktop controller and facilitate registration of the virtual desktops to notify the controller that they are ready and waiting for user connections. In large-scale XenDesktop deployments some of these components can present bottlenecks which can degrade performance and cause instability. If enough virtual desktops are started up on a single host we might start to see performance issues on that host and its virtual machines. This can happen gradually over time as additional virtual desktops are booted to accommodate user demand for desktops, or over a short period of time if many virtual desktops are booted or rebooted all at once. We’ll talk more about this in a minute. With technologies like PVS and IntelliCache, the shared storage could pose an issue if many storage operations are occurring at once. We will discuss this in more detail later in the presentation. And lastly, the number and frequency of tasks being sent to the pool master from the desktop controller can also create performance issues in certain circumstances.

Real World Applications: XenDesktop
Deployment Considerations: Number of VMs per cluster/pool Large number of concurrently running VMs-per-host Boot/Reboot Storms Virtual Disks per LUN depending on storage architecture PVS/IntelliCache can add storage management overhead The XAPI task queue Dom0 load Power Management integration and availability – Hyper-V Virtual Machine Manager, vSphere vCenter, XenServer Pool Master Windows 7 vCPU allocation for >2 vCPUs Read 1. This can cause performance degradation, stability issues and lead to problems with the Desktop Delivery Controller managing the desktop groups. Read 2. A common business requirement in environments leveraging XenDesktop is to “reset” or “recycle” desktop sessions, for example in between shift changes at a hospital or between classes at a university. This would normally involve shutting down or rebooting the virtual desktops to revert them to their original state using provisioning services or MCS. If too many desktops are instructed by the XenDesktop Controller to reboot at once it can cause a surge in load on the XenServer pool as the hosts are busy processing all of the shutdown or restart requests. This can result in a cascading failure condition where the desktops take too long to restart and register with the Desktop Controller, so it attempts to start additional VMs to meet the idle pool count requirements for the desktop group. Read 3. In XenDesktop environments leveraging either PVS or IntelliCache it is important to understand how these features make use of XenServer shared storage to avoid possible problems. Typically virtual desktops will create and attach to temporary write cache disks, on-the-fly, and depending how the storage is configured in the XS pool this could produce enough load to compromise the stability of the hosts. I will come back to this a bit more when we discuss how to monitor the XenServer pool for specific issues. Read 4. If not optimized correctly it is possible for the XenDesktop Controller to push too many tasks into the queue, or push them too frequently, and this could cause the pool master to be unable to process the queue efficiently which could lead to performance issues for both the XenServers and the virtual desktops.

Real World Applications: XenDesktop
The control domain (“dom0”) needs some TLC to really scale over 100 VMs/host or 1000 VMs/pool Dom0 Memory -> 2940 MB IRQBALANCE and NUMA enabled (if not on XS 6+) Unplugging and pinning dom0 vCPUs may be required Log tuning, storage buffer size, Receive Side Copy, C-states, checksum offloading and more! Nick Rintalan: XenDesktop and Hypervisors – Best Practices and Lessons Learned from the field. *Taken from Nick‘s ServTech2012 presentation BLOG Andreas Skuban For ~>60 VMs Configure Dom-0 for 8 vCPUs /etc/sysconfig/unplug-vcpus Configure Dom-0 for 2940MB RAM Increase storage buffer size /opt/xensource/libexec/xen-cmdline –set-dom0 blkbk.reqs=256 Make sure c-states are disabled Even MORE tweaks: -Disabling Task/Checksum Offloading on the XS hosts: This increases network throughput. -Disable C-States and turn off Turbo mode in BIOS -Set BIOS to High Performance mode: Whenever this is not done it sometimes causes XS to slow down and not be able to use CPUs fully. -Log Tuning is required to prevent storage filling up endlessly. The two settings are setting the rotation scheduler hourly and enabling compression. The reason behind this is that XenServer logrotate feature can break as it cannot parse the size parameter properly and this results in the dom0 Storage allocation becoming full, caused by overgrown Log Files. (

Real World Applications: PVS
Network (I/O) is the bottleneck at two main areas: vDisk network storage to PVS server PVS server to target device Most organizations are using 1 Gb NICs 10 Gb NICs are becoming the norm LACP finally supported by XenServer 6.1 Without LACP network (I/O) caps out at NIC speed Boot storm – can be mitigated by boot pacing tuning Network I/O Proper storage to allow windows caching on PVS server Provide necessary RAM for PVS servers and target devices Enough RAM so that PVS server does not need to read from vDisk store, and target device does not need to read from PVS server often NIC Try to use 10 Gb NIC when possible Use SR-IOV offloads network traffic to virtual function that operates on the NIC as opposed to the Dom0 Faster performance Less network I/O errors Additional:

Advanced XenServer Training

Similar presentations

Presentation on theme: "Advanced XenServer Training"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Advanced XenServer Training

Similar presentations

Presentation on theme: "Advanced XenServer Training"— Presentation transcript:

Similar presentations

About project

Feedback