XenServer Storage Management and Troubleshooting

XenServer Storage Management and Troubleshooting
Welcome to the “XenServer Storage Management and Troubleshooting” session, my name is Dan Lazar – I am a Lead Escalation Engineer with the North America XenServer Escalation team. I have been working with Citrix Technical Support for over 2 years. This session is intended to provide an overview of XenServer storage concepts, as well as provide some tools and methods to monitor and troubleshoot XenServer problems specifically related to storage. Please hold your questions for the end, or feel free to meet with me after the presentation. Daniel Lazar Lead Escalation Engineer May 11, 2010

Citrix Confidential - Do Not Distribute
Agenda XenServer Storage Overview Management and Monitoring Troubleshooting and Diagnosing Common Storage Issues Q & A Here is an overview of what we will be covering. As I mentioned we will discuss general XenServer Storage concepts as well as management, monitoring and finally troubleshooting some real-world storage problems. We’ll also save some time for questions. Citrix Confidential - Do Not Distribute

XenServer Storage Overview
Let’s begin. XenServer Storage Overview

XenServer Storage Overview
XenServer Storage Objects SRs, VDIs, PBDs and VBDs Virtual Disk Data Formats File-based VHD, LVM and StorageLink Before we go into detail about monitoring and troubleshooting storage with XenServer, I want to take a few minutes to review some of the fundamental concepts and terminologies that define how XenServer perceives storage. We will talk about the basic container objects and connectors that XenServer uses to interface with storage and provision space for virtual machines. This includes Storage Repositories, Virtual Disk Images, Physical Block Devices and Virtual Block Devices. We will also talk a bit about the different virtual disk data formats which define the various types of containers that XenServer supports for storing virtual machine data. These include file-based VHD, LVM, or logical volume, based VHD and StorageLink. Citrix Confidential - Do Not Distribute

XenServer Storage Objects
What is an SR (Shared Repository)? Describes a particular storage target in which Virtual Disk Images (VDIs) are stored. Flexible—supports a wide variety of storage types. Centralized—easier to manage, more reliable with a XenServer pool. Must be accessible to each XenServer host. So, what is an SR? An SR, or Shared Repository, [Read point 1]. It is the fundamental container object that XenServer defines to store virtual disks for use with virtual machines. [Read point 2] [Read point 3] SRs are specific to the type of storage they connect to, like iSCSI or fiber-channel, and an SR must be accessible to each host in a XenServer pool—in the case of iSCSI or NFS this means a common network interface. In the case of Fiber Channel it means a common fabric connection between the storage HBAs and the XenServer host HBAs. Citrix Confidential - Do Not Distribute

VDIs, PBDs, VBDs Virtual Disk Images are a storage abstraction that is presented to a VM. Physical Block Devices represent the interface between a physical server and an attached SR. Virtual Block Devices are connector objects that allow mappings between VDIs and VMs. VDIs, PBDs, and VBDs. XenServer is full of acronyms… So, within an SR you will find VDIs, or virtual disk images. These are a storage abstraction that is presented to a VM. They appear to the VM to be locally attached block devices. The PBD, or physical block device, represents the interface between a physical server and an attached SR. The PBD translates Linux kernel block device operations across the XenAPI SM layer to the virtual containers within the SR. A VBD, or Virtual Block Device, is the connector object that allow mappings between VDIs and VMs. The VBD interprets virtual machine I/O requests and translates them into file operations against the VDI. Citrix Confidential - Do Not Distribute

SR PBD VDI VBD XenServer Host Virtual Machine This diagram illustrates the relationships between the different storage objects we just discussed. Note that an SR maps a single PBD to each XenServer host. An SR can contain multiple VDIs, and each VDI maps a single VBD to its parent virtual machine. A virtual machine with multiple VDIs has an equal number of VBDs (1 VBD per VDI). PBD VDI VBD XenServer Host Virtual Machine VDI VBD PBD XenServer Host

Virtual Disk Data Formats
File-based VHD VM images are stored as thin-provisioned VHD format files on either a local non-shared file system (EXT type SR) or a shared NFS target (NFS type SR). What is VHD? A Virtual Hard Disk (VHD) is a file formatted to be structurally identical to a physical Hard Disk Drive. Image Format Specification was created by Microsoft in June, 2005. Let’s talk a little bit about the different virtual disk data formats supported in XenServer. The first one we’ll cover is file-based VHD. [Read slide point 1] -Typically used with NFS -Possible to implement file-based VHD manually on an EXT type SR via the host CLI. Each VDI has a corresponding VHD file stored on the native Linux file system. This has some advantages, such as being cheap, easy to implement and is not vendor-specific. It is, however, less ideal as an enterprise solution since it does not support multipathing and, arguably, will not compare in performance to iSCSI or Fiber Channel type storage. I’ve mentioned VHD several times—it is worth defining it and pointing out why XenServer uses this as the default container type for virtual block devices. [Reveal] VHD stands for Virtual Hard Disk—it is, simply, a file formatted to be structurally identical to a physical hard disk. The specification for this format was developed and published by Microsoft in June of 2005. This technology is not proprietary, therefore, third parties like Citrix are free to leverage it and factor the design into their own products. This has allowed cross-product compatibility for solutions that leverage VHD. We can move VHD files seamlessly between products like Provisioning Server, XenServer and Microsoft hyper-V which offers a tremendous amount of flexibility. Citrix Confidential - Do Not Distribute

Logical Volume (LVM)-based VHDs The default XenServer block device-based storage inserts a Logical Volume manager on a disk. VDIs are represented as volumes within the Volume manager. Introduced LVHD in XenServer 5.5 Enhances LVM for SRs Hosts VHD files directly on LVM volumes Adds Advanced Storage features like Fast Cloning and Snapshots Fast and simple upgrade Backwards compatible The most common type of file system you will see in XenServer for storing virtual disks is LVM-based VHD. [Read slide point 1] LVM is actually a Linux native file system type that is designed to provide a storage abstraction to underlying physical block devices. This solution offers greater performance than file-based VHD because it creates a thinner, more direct type of interface to the actual block device, typically a LUN, being presented by a storage array. Prior to XenServer 5.5 this type of SR used raw logical volumes mapped as VDIs that were accessed directly by the attached virtual machine. [Reveal LVHD] As of XenServer 5.5 this model has been improved into LVHD which enhances LVM for SRs. It hosts VHD files directly on the LVM volumes. It also adds features like fast cloning and snapshots. It is a fast and simple upgrade from XenServer 5.0 LVM to 5.5 LVHD and it is also backwards compatible. Citrix Confidential - Do Not Distribute

StorageLink (LUN per VDI) LUNs are directly mapped to VMs as VDIs by SR types that provide an array-specific plug-in (NetApp, Equallogic or StorageLink type SRs). The array storage abstraction therefore matches the VDI storage abstraction for environments that manage storage provisioning at an array level. The last disk data format we are going to discuss is StorageLink, or more technically LUN-per-VDI. With StorageLink LUNs are directly mapped to VMs as VDIs by SR types that provide an array-specific plugin. The array storage abstraction therefore matches the VDI storage abstraction for environments that manage storage provisioning at an array level. Dell and NetApp have developed plugins which optimize this technology for EqualLogic and NetApp-type storage arrays. These plugins are natively available in XenServer 5.0 or later. StorageLink, part of the Essentials bundle, provides additional adapters for a wider variety of storage arrays. Citrix Confidential - Do Not Distribute

StorageLink Architecture XenServer calls direct to Array API‘s to provision and adjust storage on demand. Fully leverages array hardware capabilities. Virtual disk drives are individual LUNs. High performance storage model. Only the server running a VM connects to the individual LUN(s) for that VM. A special master server coordinates which servers connect to which LUNs Because StorageLink is able to access the array API directly it can manage and provision array storage on demand. It can, therefore, also fully leverage array hardware capabilities. Virtual disk drives are individual LUNs. This removes complexity and improves performance by offloading storage management to the array itself, allowing it to do what it is best at—managing storage! Because of the direct LUN-to-VDI mapping only the server running a VM connects to the individual LUNs for that VM—this reduces load on the I/O path. This is managed by a special master server which coordinates which servers connect to which LUNs. Citrix Confidential - Do Not Distribute

LVM vs. StorageLink + XenServer 5.5 XenServer 5.5 iSCSI / FC
Storage Repository Storage Repository LUN LUN LUN VHD header VHD header LVM Logical Volume LVM Logical Volume Looking at this diagram it becomes evident how StorageLink simplifies the storage solution. The VM virtual disk maps directly to the LUN on the array removing all of the storage abstractions and associated overhead from the XenServer hosts. LVM Volume Group VM Virtual Disk

Storage Management and Monitoring
Now let’s take a look at some methods for managing and monitoring storage in XenServer. Storage Management and Monitoring

Management and Monitoring Overview
Understanding how XenServer Perceives the Storage Monitoring Storage Protecting Your Data We’re going to discuss commands and utilities to help describe the storage environment from the perspective of XenServer. [Reveal] We are then going to move on and talk about how we can monitor XenServer storage to check performance and identify potential problems. Last we are going to discuss what tools are available for protecting your virtual machine data. Citrix Confidential - Do Not Distribute

Management and Monitoring
Understanding the physical disk layout # fdisk –l # Lists the physical block devices on the host Disk /dev/cciss/c0d0: GB, bytes 255 heads, 32 sectors/track, cylinders Units = cylinders of 8160 * 512 = bytes Device Boot Start End Blocks Id System /dev/cciss/c0d0p1 * Linux /dev/cciss/c0d0p Linux /dev/cciss/c0d0p Linux Denotes a SCSI block device locally attached to the system (HP RAID array in this case) The first partition on the disk contains the boot information for the OS. First we are going to review a few native Linux commands that are useful for describing block devices available to a Linux OS. These commands exist for nearly all distributions of Linux. XenServer is based off CentOS Linux and therefore also includes these commands. Fdisk is a simple command to list all fixed disks that the kernel is aware of. [Reveal Output] Let’s take a look at the output for this command… [Reveal Highlight 1] In this example we see /dev/cciss/c0d0 as the first disk enumerated by the OS. CCISS happens to be the disk driver Linux loads to manage HP SmartArray RAID controllers, so not only can we conclude that this is a physical, locally attached hard disk, we can even deduce the manufacturer of the machine. [Reveal Highlight 2] We can also see that this disk has three partitions on it—all default XenServer installations will create these same three local partitions. Partition ‘p1’, the first partition, is designated as the boot partition as indicated by the asterisk below the boot column. Everything required to boot XenServer is located on this partition. Partition ‘p2’ is not mounted at boot time and is reserved as a backup partition to ‘p1’—note that their size, in blocks, is nearly the same. When upgrading XenServer to a newer version a copy of the original boot partition is always made to the backup partition. The last partition is composed of the remaining space on the hard disk and is automatically allocated as storage for virtual machines. It is worth noting that it is possible to boot XenServer from remote storage, and using fdisk alone would not necessarily indicate that you are doing so. Citrix Confidential - Do Not Distribute

Understanding the physical disk layout (continued) # fdisk –l # Continued output Disk /dev/sda: GB, bytes 255 heads, 63 sectors/track, cylinders Units = cylinders of * 512 = bytes Disk /dev/sda doesn't contain a valid partition table Implies a block device using the SCSI Generic (sg) driver. It is likely attached via a separate interface such as iSCSI or FC HBA This disk is part of a Storage Repository using an LVM file system and therefore does not require a local partition table. Looking at the continued output for fdisk we see there are additional block devices presented to the kernel. [Reveal Highlight 1] In conjunction with the cciss driver the presence of /dev/sd devices, in XenServer, implies shared storage mapped using Linux sg, or scsi generic, drivers. [Reveal Highlight 2] We can also see that the disk does not contain a readable partition table since it is part of an LVM type SR. Citrix Confidential - Do Not Distribute

Understanding the physical disk layout (continued) # sg_map –x # Displays the mapping between Linux sg and regular SCSI devices /dev/sg /dev/sg /dev/sda /dev/sg /dev/sdb /dev/sg /dev/sg /dev/sdc /dev/sg /dev/sdd The next command we will discuss is sg_map. [Reveal Output] This command displays the mapping between Linux sg and regular scsi devices. It has knowledge of the physical presentation of the storage—let’s talk about what it tells us. [Reveal] The LUN number is particularly useful in this command output as you can check this against the LUN assignments configured on the SAN, but this command can also provide some insight when describing disk topology for a host configured for multipathing as it is easy to see the different host numbers that correspond to the different paths to the storage array. [Reveal] I will highlight the remaining fields for your reference. Host Number Bus SCSI ID LUN SCSI Type Citrix Confidential - Do Not Distribute

Understanding the physical disk layout (continued) # ll /dev/disk/by-id # List the attached block devices by SCSI ID. cciss b > ../../cciss/c0d0 cciss b part1 -> ../../cciss/c0d0p1 cciss b part2 -> ../../cciss/c0d0p2 cciss b part3 -> ../../cciss/c0d0p3 scsi-360a f4a b57 -> ../../sda ‘ll’ dev disk by-id simply does a long listing of the contents of this directory. [Reveal Output] [Reveal] Dev disk by-id is populated by udev at boot-time and enumerates all of the scsi block devices by their hard-coded SCSI ids. Not only does it list the SCSIid itself, but it also displays the mapping of that scsi device to the linux device that gets referenced by the kernel for block device operations. Another useful udev listing is /dev/disk/by-scsibus which contains similar information to what we get from sg_map. Unique ID assigned by udev. It corresponds to individual block devices. This SCSI device is mapped to /dev/sda Citrix Confidential - Do Not Distribute

Understanding the physical disk layout (continued) To identify a specific SR based on the SCSI ID, compare /dev/disk/by-id with the SR in XenCenter Looking in XenCenter we can see that the SCSIid acquired from /dev/disk/by-id can be used to identify the specific SR that corresponds to the SCSI device. [Reveal] Citrix Confidential - Do Not Distribute

LVM-related commands # pvs # Lists physical volumes PV VG Fmt Attr PSize PFree /dev/sda VG_XenStorage-40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 lvm2 a G G # vgs # Lists volume groups VG #PV #LV #SN Attr VSize VFree VG_XenStorage-40bbf542-b9d9-ffa1-6efe-aa9c56aadd wz--n G G Linux sg device LVM Volume Group stored on the physical volume. SR UUID Now that we have a handle on the physical block devices presented to our XenServer, let’s move on and learn about some commands that describe the LVM file system on our SR storage. The first command we can issue is pvs which displays information about the physical volumes in the LVM file system. Let’s have a look at the output. [Reveal] Note the command lists both the Linux device, /dev/sda, which we know from the last several slides to be remote shared storage, and the LVM volume group stored on that device. The unique id, or UUID, assigned to the volume group is the same ID XenServer assigns to the SR that represents this storage in its database—we’ll refer back to this UUID later. Vgs, similar to pvs, displays information about the volume groups in the LVM file system. Note that the psize of the physical volume is the same as the vsize of the volume group—this is enforced by XenServer when these containers are created since XenServer automatically allocates all of the space from a physical volume to the volume group it creates to map to an SR. Citrix Confidential - Do Not Distribute

LVM (continued) # lvs # Lists the logical volumes LV VG Attr LSize VHD-c67a887f-3a1a-41f4-8d40-1b21f6307c4a VG_XenStor wi G VHD-c9b919a7-b93b-49ea-abe5-00acb8240cf5 VG_XenStor wi-ao G VHD-f3d26dde-254f-4d80-a3bb-d993e904bd63 VG_XenStor wi G LV-e056f479-b0f3-49f3-bc5d-6c226657ae6c VG_XenStor wi-ao G LV-ebdcad46-66d baa1-0d5b6ac439c7 VG_XenStor wi-ao G The ‘a’ and ‘o’ attributes indicate the LV is ‘active’ and ‘open’ implying it is attached to a running VM Lastly we can issue lvs to describe the logical volumes that exist inside particular volume groups. [Reveal Output] [Reveal] Note that the UUIDs assigned to the logical volumes correspond to the VDI UUIDs in the XenServer database—we’ll revisit these UUIDs, as well. Looking at this output we can see that there are two types of logical volume containers—ones prefixed with ‘LV’ and others with ‘VHD’. LV type containers denote older raw LVM VDIs that were created prior to XenServer 5.5 and coexist with the newer VHD type containers, which were created in XenServer 5.5 or later. We can also tell, looking at the attribute column, what state the VDI is in. For example, the ‘ao’ flags indicate that the VDI is active and open, which means it is attached to a running VM. A final note about LVM--You can issue ‘lvm help’ on the CLI to get a complete list of LVM command options. Tip: Type ‘lvm help’ for a complete list of LVM command options. Represents Logical Volume containers for individual VDIs. Citrix Confidential - Do Not Distribute

Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI # xe sr-list type=lvmoiscsi params=name-label,uuid,VDIs,PBDs # Lists the SRs configured for the pool name-label ( RW) : NetApp - iSCSI uuid ( RO) : 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 VDIs (SRO) : f3d26dde-254f-4d80-a3bb-d993e904bd63; c67a887f-3a1a-41f4... PBDs (SRO) : 27d05ffc-07d3-4f02-d a2179f8f So, now that we’ve described the physical storage and the LVM file system let’s look at some commands we can issue to see how XenServer turns all of this into virtual storage. The ‘xe’ command line utility allows us to query the XenServer configuration for storage objects that it is aware of, and how it believes they are mapped to physical block devices. First, we can issue ‘xe sr-list’ to display the SRs that XenServer has configured. We can further narrow down this output by specifying parameters to customize what we want to look at. In this example I am asking XenServer to give me only SRs of type LVM-over-iSCSI, meaning they are connected using the iSCSI block protocol, and only return the name of the SR, the UUID of the SR, all VDIs contained in the SR, and the PBD the host is using to attach this SR. [Reveal Output] [Reveal] As I had mentioned earlier when we were looking at the lvs output, the VDI UUIDs correspond to the logical volume UUIDs—we will use these later to query the VDI characteristics. In the next slide we will take the PBD UUID and learn more about its configuration. Note that the VDI UUID is the same as the logical volume ID. We will make a note of this UUID to refer back to. Using the PBD UUID from this command output we will query for its characteristics in the next slide… Citrix Confidential - Do Not Distribute

Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI (continued) # xe pbd-list uuid=27d0… params=uuid,sr-uuid,device-config,currently-attached # List PBD params uuid ( RO) : 27d05ffc-07d3-4f02-d a2179f8f sr-uuid ( RO): 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 device-config (MRO): port: 3260; SCSIid: 360a f4a b57; target: ; targetIQN: iqn com.netapp:sn currently-attached ( RO): true We can now issue xe pbd-list and pass the PBD UUID from the sr-list output as a parameter to see its details. [Reveal Output] In this example I have asked only for the PBD UUID, the device-config and the currently-attached parameters. [Reveal] Note that the device config field pretty much describes everything that is required to establish a connection to this block device. The currently attached field is sometimes useful for diagnosing SR issues. Often unattached or incorrectly attached PBDs are cause for SR Not Available error messages when XenServer attempts to access the SR. I will cover some methods for troubleshooting behaviors like this later in the Troubleshooting section of the presentation. ‘device-config’ describes all the physical characteristics of the block device attached to this PBD. Note the SCSIid as referenced earlier from /dev/disk/by-id Citrix Confidential - Do Not Distribute

Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI (continued) # xe vdi-list uuid=f3d26dde-254f-4d80-a3bb-… params=uuid,sr-uuid,vbd-uuids # List VDI params uuid ( RO) : f3d26dde-254f-4d80-a3bb-d993e904bd63 sr-uuid ( RO): 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 vbd-uuids (SRO): 69afb055-3b52-57e3-63fa-d26b82a9b01d Next we can issue xe vdi-list using a VDI gathered earlier to see its characteristics. For this example we are asking only for the VDI UUID, the SR UUID and the VBD UUIDs parameters. [Reveal Output] [Reveal] We will make note of the VBD UUID returned by this command as we will need it to query the VBD characteristics to tell us what VM is attached to this virtual disk. This tells us what VBDs are attached to this VDI. We will use this UUID in the next slide to query for the VBD characteristics and determine which VM this disk is attached to. Citrix Confidential - Do Not Distribute

Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI (continued) # xe vbd-list uuid=69afb055-3b52-… params=uuid,vm-uuid,vm-name-label,vdi- uuid,mode # List VBD params uuid ( RO) : 69afb055-3b52-57e3-63fa-d26b82a9b01d vm-uuid ( RO): 2c3a0e82-3f96-eab db33fdb3bd88 vm-name-label ( RO): Windows 7 Test vdi-uuid ( RO): f3d26dde-254f-4d80-a3bb-d993e904bd63 mode ( RW): RW Now we have finally drilled down to the virtual machine layer in our storage abstraction model. Issuing xe vbd-list using the UUID from the last slide we can now see which virtual machine is using this VBD to attach to this specific virtual disk. [Reveal Output] [Reveal] In addition to giving us details about attached virtual machines note that we can also determine if the VDI is read/write or read-only accessible to the VM since I have specified the mode parameter in the command. Another useful tip: you can issue xe help followed by the command name, such as sr-list, to get a list of parameters and syntax for any xe command. This tells us which VM (name and UUID) this VBD is attached to, and which VDI it is providing to the VM. Tip: You can issue ‘xe help <command>’ to get syntax help for any ‘xe’ commands. Citrix Confidential - Do Not Distribute

Fibre Channel LUN Zoning Since Enterprise SANs consolidate data from multiple servers and operating systems, many types of traffic and data are sent through the interface, whether it is fabric or the network. With Fibre Channel, to ensure security and dedicated resources, an administrator creates zones and zone sets to restrict access to specified areas. A zone divides the fabric into groups of devices. Zone sets are groups of zones. Each zone set represents different configurations that optimize the fabric for certain functions. Now that we have a picture of the storage as XenServer perceives it we will now talk briefly about the storage topology outside of the xenserver pool. In the enterprise this will usually come in the form of shared storage, using typically Fibre Channel or iSCSI as the block transport. [Read point 1] [Read point 2] [Read point 3] To uniquely identify a specific host connecting to a zone, each host bus adapter, or HBA has a unique World Wide Name, or WWN. This is similar to an Ethernet MAC address. There are two types of WWN—the node WWN, which can be shared by some or all ports of a device, and the port WWN, which is unique to each port. WWN - Each HBA has a unique World Wide Name (similar to an Ethernet MAC) node WWN (WWNN) - can be shared by some or all ports of a device port WWN (WWPN) - necessarily unique to each port

Fibre Channel LUN Zoning
Pool1 Pool2 Xen1 Xen2 Xen3 Zone1 Xen1 WWN Xen2 WWN Storage WWN Zone2 Xen3 WWN Storage WWN FC Switch As we can see in the diagram Zone1 is isolated from Zone2—all storage traffic passing through Zone1 to and from hosts Xen1 and Xen2 are completely isolated from traffic passing to and from host Xen3. [Reveal] On the storage array Initiator groups have been created to further isolate access to the actual LUNs. This way we create more secure access to the data and improve efficiency by keeping traffic on assigned paths between the hosts and the storage array. Initiator Group Xen1, Xen2 LUN0 LUN1 LUN2 Initiator Group Xen3 Storage FC Switch example

iSCSI Isolation With iSCSI type storage a similar concept of isolation as fibre-channel zoning can be achieved by using IP subnets and, if required, VLANs. IQN – Each storage interface (NIC or iSCSI HBA) has configured a unique iSCSI Qualified Name Target IQN – Typically associated with the storage provider interface Initiator IQN – Configured on the client side, i.e. the device requesting access to the storage. IQN format is standardized: iqn.yyyy-mm.{reversed domain name} (e.g. iqn com.acme:storage.tape.sys1.xyz) [Read point 1] With iSCSI, similarly to WWN names in the context of Fibre, we have IQNs, or iSCSI qualified names, to uniquely identify different iSCSI interfaces. There are two common types of IQN—target IQNs and initiator IQNs. The Target IQN is typically bound to the storage array and listens for inbound iSCSI connection requests from the iSCSI initiator, ergo the initiator IQN is bound to the client which is requesting access to the storage array.

iSCSI Isolation Pool1 Pool2 Xen1 Xen2 Xen3 VLAN1 / Subnet1
Xen1 Initiator IQN Xen2 Initiator IQN Controller 1 Target IQN VLAN2 / Subnet2 Xen3 Initiator IQN Controller 2 Target IQN Network Switch Looking at this diagram we can see that it bears many similarities to the fibre channel topology—VLANs provide the same traffic isolation that Zones provide to a fibre channel network. [Reveal] On the storage array we can bind individual network controller interfaces to target IQNs which only allow connections from registered initiator IQNs on the XenServer hosts. This topology, just like LUN zoning for fibre, creates a more secure and better performing storage environment. Following these best practices when designing and implementing a storage solution will result in better performance and a higher level of data security across the board with XenServer. Controller Interface 1 LUN0 LUN1 LUN2 Controller Interface 2 Storage iSCSI Example

Monitoring XenServer Storage - Alerts XenServer will generate alerts for certain storage events: Missing or duplicate IQNs configured HA state file lost or inaccessible PBD plug failure on server startup XenServer can be configured to send alert notifications via too. See the XenServer Administrator’s Guide for more information about configuring alerts. Having discussed in good detail storage concepts as they apply to XenServer we will now move on to talk about managing and monitoring the storage. XenServer will generate alerts for some types of storage events. Missing or duplicate IQNs configured, HA statefile lost or inaccessible and PBD plug failures on server startup are a few of these events. [Reveal] It is also possible to configure XenServer to send alert notifications via . See the XenServer administrator’s guide for more information about configuring alerts. Citrix Confidential - Do Not Distribute

Monitoring XenServer Storage – CLI Commands # iostat –k # Reports basic I/O stats for devices and partitions avg-cpu: %user %nice %system %iowait %steal %idle Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d sda We can use native Linux utilities to help monitor storage throughput. As with describing storage layout, these commands are common to nearly any Linux distribution. Iostat is a basic utility that reports I/O statistics for connected devices and partitions. [Reveal output] It is a good indicator of disk performance for locally attached block devices, but since it is unaware of the storage abstractions in XenServer, such as connections to shared storage over iSCSI or fiber channel, it might not be an accurate performance indicator for these types of storage. Note: iostat is not a great performance indicator for shared storage devices because it is unaware of external bottlenecks, for example the network in the case of iSCSI. Citrix Confidential - Do Not Distribute

Monitoring XenServer Storage – CLI Commands # hdparm –t /dev/<device> # Performs timed sequential reads /dev/cciss/c0d0: Timing buffered disk reads: 286 MB in seconds = MB/sec Has some limitations: Does not measure non-sequential disk reads. Does not measure disk write speed May not be accurate with non-local storage devices since it is unaware of underlying bus architecture (iSCSI, FC, etc.) Must be sampled repeatedly over time to get an accurate picture of I/O read performance. Hdparm does active benchmarking of storage throughput which can make it particularly useful for identifying storage bottlenecks as they are occurring. [Reveal Output] It does have some limitations, however. It does not measure non-sequential disk reads. It does not measure disk write speed. Similarly to iostat it also might not be accurate with non-local storage since it is not aware of the underlying bus architecture. Because it does active benchmarking it should be run repeatedly to get an accurate picture of I/O read performance over time. Citrix Confidential - Do Not Distribute

Monitoring XenServer Storage – CLI Commands # dd if=<infile> of=<outfile> # Simple, common block device copy utility # dd if=/dev/<device> of=/dev/null records in records out bytes (1.0 GB) copied, seconds, 73.9 MB/s if = ‘infile’, the source dd reads from. of = ‘outfile’, the target dd writes to. The last native linux command we will discuss is dd. Dd is probably the most basic disk monitoring tool of the lot, but in many respects it can be the most useful. Quite simply all it does is copy raw, block-level data from one place to another and tells you how long it takes. [Reveal Output] Since it operates at such a low level on the storage device it provides a very accurate representation of sequential read/write throughput regardless of the type of storage. Dd, in the hands of a savvy administrator, can even be used to perform low-level data recovery if an LVM container is unmountable but might contain valid VHD data. [Reveal] A word of warning, if you tell dd to write data somewhere it will do so indiscriminately—it is very quick way to break your XenServer, or perhaps worse, overwrite valid VHD files if you specifiy one as the outfile. WARNING: NEVER run dd specifying an active, running VHD as the outfile—it WILL destroy the VM container making it unreadable!! Citrix Confidential - Do Not Distribute

Monitoring XenServer Storage – Additional Tips iSCSI storage throughput can usually be tied directly to network performance. If there is slow throughput for an iSCSI storage array, perform network diagnostics first!! Many SAN arrays have native logging and monitoring tools that can identify bottlenecks affecting storage performance. Refer to the Citrix Knowledge Base for best practices and known issues relating to storage performance. A few additional tips regarding storage monitoring. [Read point ] [Reveal] [Read point 2] [Read point 3] Here are a few links to some relevant articles. Citrix Confidential - Do Not Distribute

Protecting Your Data – Backup VM Metadata Can use xsconsole or the CLI. Makes the SR “portable”. Can be used as part of a Disaster Recovery solution, or, as part of regular maintenance of the environment. Can be scheduled within xsconsole. For more information relating to using XenServer as a Disaster Recovery solution, refer to the Citrix Knowledge Center: Now we will talk a little bit about protecting your data in XenServer. We all know as IT professionals that backing up data early and often goes without saying. In the world of virtualization, where everything is abstracted into something else “how” to back up this data can become a tricky question. Thankfully, XenServer has some capabilities to make this process easier and safer. The first one we’ll talk about is metadata backups. By default, XenServer stores all the information it uses to locate and bind a VDI to a virtual machine locally in its database. If there is a failure of a host or pool, even though the actual VDI data residing on the shared storage might be unaffected, this VDI metadata might be lost making it much more difficult to recover this information. XenServer has included a means to archive this metadata directly on the SR where those VDIs are stored. Metadata backups can be initiated via the XenServer cli, or through the xsconsole utilitity. The process creates a small VDI directly on the SR that contains all the metadata for the virtual machines that refer to that SR. This effectively makes the SR portable, meaning it can detach from a failed XenServer and re-attach to a completely different XenServer host with everything it needs to restore the VMs and their associations with VDIs on that SR. This is useful for XenServer pools that are part of a DR solution, or just for regular maintenance. It is also possible to schedule metadata backups daily, weekly or monthly within xsconsole. [Reveal] You can refer to these CTX articles in the Citrix Knowledge Center for detailed information about configuring XenServer as a DR solution. Citrix Confidential - Do Not Distribute

Protecting Your Data – Exporting VMs Virtual machines can be exported directly out of XenServer into XVA files that contain a complete clone of the VM and all of its attached VDIs. Can be initiated via XenCenter or from the XenServer CLI. VM must be offline (shutdown) during export process. Since it backs up all the VM data it can take a very long time depending on the size of the VM! Backing up metadata is important, but it doesn’t protect the data in the actual VDIs. To help with this XenServer has the capability to export virtual machines out of the XenServer pool. [Read point 1] [Read point 2] [Read point 3] and, [Read point 4] Citrix Confidential - Do Not Distribute

Protecting Your Data – Creating VM Snapshots Snapshots create VDI clones of a VM that can be used for backup or quickly provisioned into new VMs or templates. XenServer supports two types in version 5.5 Regular – Supports all guest environments, including Linux Quiesced – Takes advantage of Windows Volume Shadow Copy Service (VSS). It requires the manual installation of in-guest components to enable. XenServer can also create virtual machine snapshots. Snapshotting is arguably more of a management feature but it can serve as a great tool for backing up VMs as well. [Read point 1] [Reveal] In XenServer 5.5 there were two types. Regular which supports all guest environments, including Linux, and quiesced which takes advantage of Windows Volume Shadow Copy Service, VSS. This type requires manual installation of in-guest components to enable. Citrix Confidential - Do Not Distribute

Protecting Your Data – Creating VM Snapshots (continued) New in XenServer 5.6! Introduces snapshot “Revert”, a.k.a. “Checkpoint”. Introduces a new snapshot mode: “Snapshot with disk and memory” XenCenter GUI enhanced for easier management of VM snapshots and to support Checkpoint feature. In XenServer 5.6 we have introduced a whole new way to manage snapshots. [Reveal] We now have the ability to revert to older snapshots through the use of the checkpoint feature. We have also introduced a new snapshot mode called snapshot with disk and memory. This type of snapshot captures not only the disk state but also the memory state at the time of the snapshot. The major use cases would be to create a disposable backup of a VM that contains applications running. It could also be for testing purposes, say, to install patches and then revert if anything goes wrong. It can also be a quick way to create different configurations from a template. Finally, the XenCenter GUI has been enhanced for easier management of snapshots and to support the checkpoint feature. Citrix Confidential - Do Not Distribute

Protecting Your Data – Third-Party Solutions There are also Third-Party backup options: In-guest backups can be performed using any guest-supported solution (backup agents running in Windows or Linux, for example). Volume snapshots performed directly on the storage via StorageLink plugins (for Dell and NetApp). Backup solutions that plug into the XenAPI to capture VM data, or clone the LVM data directly. In addition to native backup options within XenServer there are also third-party solutions that can ensure your data is protected. In-guest backups can be performed using any guest-supported solution. Any backup solution with compatible agents for Windows and Linux should work fine inside a XenServer virtual machine. Many storage arrays leveraging StorageLink or the StorageLink plugins can perform back-end volume-level snapshots to automatically archive VDIs from the storage side. Lastly, there are some third-parties who have developed software that plugs directly into the XenServer host API to perform LVM-based replication of virtual machine containers. Citrix Confidential - Do Not Distribute

Troubleshooting and Diagnosing Common Storage Issues
Okay, let’s move on to some troubleshooting tips. Troubleshooting and Diagnosing Common Storage Issues

Troubleshooting XenServer Storage
Native Troubleshooting Tools – XenServer Logs Always check the logs first! XenServer creates several logs that are useful for diagnosing storage problems /var/log/messages # General messages and system related stuff /var/log/xensource.log # Logging specific to XenAPI /var/log/SMlog # Logging specific to XenServer storage manager Often errors logged in any of these files can be searched for in the Citrix Knowledge Center for a solution. See When troubleshooting any XenServer issue, storage-related or not, the first place to look is always the system logs. All system logs are located, by default, under slash var slash log. [Reveal] Var log messages contains general messages and system related stuff. This log, if it doesn’t point directly to root cause, will often give you an idea of where to look next. Var log xensource.log records messages specific to the XenAPI and will contain detailed information relating to VM operations, like starting and stopping VMs. For storage, var log Smlog is particularly useful. This log is written to by the storage manager service of the xapi toolstack and will contain important information relating to mounting, accessing and monitoring of XenServer storage objects. Often you can get additional information, or even resolution, for many of the errors written to these log files by searching for them in the Citrix Knowledge Center. Citrix Confidential - Do Not Distribute

Native Troubleshooting Tools – XenAPI commands The XenAPI (xe) can be used to troubleshoot storage issues too # xe sr-scan # Force XAPI to sync the database with local VDIs present in the underlying substrate. # xe sr-probe # Using device-config parameters you can probe a block device for its characteristics, such as existing VM metadata and SR uuid. # xe pbd-plug/unplug # Manually plug or unplug a PBD for an SR. This can be useful when repairing an SR in XenCenter fails. The XenAPI can be used to troubleshoot many storage issues in XenServer. [Reveal] Issuing xe sr-scan can restore access to an SR under some circumstances by forcing xapi to synchronize the database with local VDIs present in the underlying substrate. Xe sr-probe can be useful for determining if there is any viable metadata on a storage device to refer to when introducing an SR to the pool. For SRs that are unable to attach or repair in XenCenter often issuing xe pbd-plug or unplug on the CLI can resolve this behavior. Citrix Confidential - Do Not Distribute

Native Troubleshooting Tools – VHD commands See and verify mount point of VHD SR # /var/run/sr-mount/<SR UUID> “full provision” VHD SR vhd-util See Check VHD architecture # hexdump -vC <VDI-UUID>.vhd | less There are some commands that can be used to check the health of VHD files. For file-based VHD you can confirm the mount point of the SR by looking under /var/run/sr-mount. You can then pass the VHD file as a parameter to vhd-util to check and confirm the consistency of the VHD file. If you want to manually validate the VHD architecture you can hexdump the VHD and have a look at it. It is also possible to manually activate an LVHD container using LVM to access the underlying VHDs—this is an advanced topic, however, and we won’t cover it in detail in this session. Citrix Confidential - Do Not Distribute

Storage Multipathing Ensure that multipathing is enabled if you have multiple paths zoned to the XenServer Use ‘sg_map –x’ and check the host and bus IDs Problems if you do not enable multipath I/O Errors Decrease in performance Introduce errors with SR.create What is multipath.conf vs multipath-enabled.conf multipath.conf is symlink to multipath-enable.conf or multipath-disabled.conf DMP vs. MPP multipathing Storage multipathing is a topic we have not covered in detail in this session since it really does require a session of its own to do it justice. Still, it is worth talking briefly about some troubleshooting practices relating to multipath issues. First, if there are multiple storage paths zoned to a XenServer always ensure that multipath is enabled so that the path handler can negotiate IO down the available paths. [Reveal] If you do not enable multipath you will see I/O errors resulting in decreased performance and potentially problems introducing SRs into the pool. When going to check the multipath configuration, remember that multipath.conf is actually a symlink to either multipath-enabled.conf (if multipath is turned on) or multipath-disabled.conf (if multipath is turned off). Lastly, there are two types of path handlers that XenServer can be configured to use. DMP is the default, natively supported path handler within XenServer, and MPP, if required, can only be configured using the underlying Linux OS. Check CTX on the Citrix Knowledge Base for more information. DMP = Dynamic Multipathing; MPP = Multipathing Proxy Citrix Confidential - Do Not Distribute

SAN Debugging Always start at the hardware adapter, use the Qlogic or Emulex CLI tools to verify the LUNs known to the adapter For QLogic, run ‘scli’ For Emulex, run ‘hbanywhere’ Use ‘xe sr-probe type=lvmohba’ to trigger a bus refresh A couple other general SAN debugging tips. [Read point 1] It isn’t hard to imagine spending hours combing through a XenServer config trying to figure out why an SR isn’t mapping only to realize that the LUN was zoned incorrectly and is not even visible to the host adapter. Troubleshooting the hardware adapters first can avoid a wild goose chase! Also, using xe-probe and specifying the storage type is a quick way to trigger a bus refresh. This is a convenient way to get XenServer to recognize changes to the storage without rebooting the host or restarting any services. Citrix Confidential - Do Not Distribute

Additional Scenarios Unable to create SRs Verify that XenServer can see the storage/LUN Use fdisk and /dev/disk/xxx Verify that HBA can see the LUN Use the HBA CLI tools Verify that iSCSI can login: # iscsiadm –m node –L all # Will force iscsid service to log into the storage array. Clearing the device mappings via CMD line # echo 1 > /sys/class/scsi_devices/x:x:x:x/device/delete Be extremely careful what device is being deleted! Clean up of orphaned VDIs, XC not displaying the right amount of free storage If a logical volume has no corresponding VDI it can be deleted. Be extremely careful with this because if you delete a parent disk, then you lost all differentiated disks. Here are some additional troubleshooting scenarios. What do we do if we can’t create an SR? [Reveal] Well, we can verify that XenServer can see the storage, for one. As described in the last slide, also verify that the HBA can see the LUN. And, in the case of iSCSI, make sure the initiator can successfully log into the target. Using iscsiadm can help diagnose these issues. Sometimes XenServer retains references to device mappings for storage that is no longer connected to the system. To manually clear these mappings without rebooting the host, in the CLI, issue echo 1 redirected to the device under slash sys slash class. Be very careful which device you delete as this is destructive and irreversible. Finally, if you notice that XenCenter is not displaying the right amount of free space for an SR it might mean there are orphaned VDIs within the LVM file system. If a logical volume has no corresponding VDI it can be deleted, using the LVM utility, to correct this behavior. Be extremely careful with this because if you delete a parent disk you lose all differentiated disks which might contain valid virtual disk information.

Alright, well that is my time for this session
Alright, well that is my time for this session. If you have any questions for me, or tips for everyone, please feel free to speak up… Questions? Comments?

TechEdge Survey, Video Postings & PPTs
The TechEdge survey will be ed out to end-user customers If you complete the survey, you will be entered to win a $250 Amazon gift card. The winner will be announced June 1st. View TechEdge videos & PPTs on the Knowledge Center by Monday, May17th

XenServer Storage Management and Troubleshooting

Similar presentations

Presentation on theme: "XenServer Storage Management and Troubleshooting"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

XenServer Storage Management and Troubleshooting

Similar presentations

Presentation on theme: "XenServer Storage Management and Troubleshooting"— Presentation transcript:

Similar presentations

About project

Feedback