Presentation is loading. Please wait.

Presentation is loading. Please wait.

Optimizing XenServer Deployments to Best Support XenDesktop Daniel Lazar Senior Escalation Engineer May 8, 2012.

Similar presentations


Presentation on theme: "Optimizing XenServer Deployments to Best Support XenDesktop Daniel Lazar Senior Escalation Engineer May 8, 2012."— Presentation transcript:

1 Optimizing XenServer Deployments to Best Support XenDesktop Daniel Lazar Senior Escalation Engineer May 8, 2012

2 #CitrixSummit Tweet about this session with hashtag #SUM302 and #CitrixSummit

3 #CitrixSummit XenServer and XenDesktop Interoperability Overview Monitoring XenServer to Identify Problems Best Practices and Troubleshooting Resources Questions Agenda 3

4 XenServer and XenDesktop Interoperability Overview

5 #CitrixSummit Interoperability Overview Pool Master Slaves Resource Pool Shared Storage Hosting Management Hypervisor Communication Library (HCL) Database (SQLServer) VDA Management Active Directory Desktop Controller Connection to XAPI on pool master via HTTP port Virtual Desktops running Receiver Windows Communication Foundation (WCF)

6 #CitrixSummit Interoperability Overview How do large-scale XenDesktop implementations impact XenServer? Large number of concurrently running VMs-per-host. Boot/Reboot Storms PVS/IntelliCache can add storage management overhead The XAPI task queue 6

7 Monitoring XenServer to Identify Problems

8 #CitrixSummit Monitoring XenServer to Identify Problems Online Monitoring Load Average # top XAPI Task Queue # xe task-list General storage and network monitoring # iostat, hdparm, dd # tcpdump, netstat, ifconfig 8

9 #CitrixSummit Monitoring XenServer to Identify Problems Online Monitoring – Load Average Use top to get real-time information relating to load top - 13:35:11 up 2 days, 19:02, 4 users, load average: 36.27, 23.64, Tasks: 435 total, 27 running, 408 sleeping, 0 stopped, 0 zombie Cpu0 : 13.6%us, 60.5%sy, 0.0%ni, 4.7%id, 18.4%wa, 0.0%hi, 0.0%si, 2.5%st Cpu1 : 14.2%us, 60.4%sy, 0.0%ni, 2.8%id, 19.9%wa, 0.0%hi, 0.0%si, 2.4%st Cpu2 : 13.0%us, 60.7%sy, 0.0%ni, 4.6%id, 18.9%wa, 0.0%hi, 0.0%si, 2.5%st Cpu3 : 13.3%us, 60.4%sy, 0.0%ni, 6.1%id, 17.5%wa, 0.0%hi, 0.0%si, 2.5%st Mem: k total, k used, 22260k free, 20388k buffers Swap: k total, 85720k used, k free, k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8722 root m 46m 4320 S :09.34 xapi R :12.56 qemu-dm R :48.05 qemu-dm R :38.39 qemu-dm R :12.09 qemu-dm R :21.79 qemu-dm 1-Minute Avg 5-Minute Avg 15-Minute Avg 9

10 #CitrixSummit Monitoring XenServer to Identify Problems Online Monitoring – Task Queue You can count the number of running tasks: # xe task-list | grep VM. | wc –l Example task types could be start, shutdown or migrate. Try to tune the XenDesktop Controller to minimize the number of tasks concurrently being processed on the pool master. 10

11 #CitrixSummit Monitoring XenServer to Identify Problems Online Monitoring – Storage # iostat# Reports basic I/O stats for devices and partitions # hdparm# Performs timed sequential reads # dd# Simple, common block device copy utility See CTX for more information on how to monitor storage in XenServer. 11

12 #CitrixSummit Monitoring XenServer to Identify Problems Online Monitoring – Network # tcpdump # Dumps traffic on a network # netstat # Display network interface statistics # ifconfig # Display and configure network interfaces See CTX for more information on monitoring the network in XenServer TIP: You can always type man followed by a Linux command name (i.e., man netstat) to get detailed help for the command. 12

13 #CitrixSummit Monitoring XenServer to Identify Problems Offline Monitoring – System Logs System Status Reports XAPI Connection Limit Exceeded Tools as a Service (TaaS) System Activity Reporting (SAR) – Part of the CentOS base 13

14 #CitrixSummit Monitoring XenServer to Identify Problems Status reports available via XenCenter, or... the command line by running # xen-bugtool –yestoall 14 See CTX for detailed instructions.

15 #CitrixSummit Monitoring XenServer to Identify Problems Offline Monitoring – XAPI Connection Limit XAPI and the control domain (Dom0) can only maintain 200 concurrent connections (per- host). Limit can be reached more easily in XenDesktop environments because the number of tasks being queued on the pool master will often be high. Can parse /var/log/xensource.log* for "db_gc] Session.destroy to give an indication of whether connection limits are being met. A simple bash script can do this quickly: #!/bin/bash LIST="$(find -name xensource.lo*)" for i in "$LIST"; do grep -h "db_gc] Session.destroy" $i done 15

16 #CitrixSummit

17 Monitoring XenServer to Identify Problems Offline Monitoring – SAR Logs Located under /var/log/sa They are NOT included in the host system status report and therefore need to be collected manually for analysis: # tar -cvzf /tmp/$HOSTNAME-$(date +%F-%H-%M-%S)-SARlogs.tar.gz /var/log/sa/* Can give a historical picture of average load on the host and indicate when and if there are periods of high load in the environment. 17

18 #CitrixSummit Monitoring XenServer to Identify Problems Offline Monitoring – SAR Logs 00:00:01 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 13:10: :20: :30: :40: :50: :00: :10: :20: TIP: Third-party tools are available to graph and analyze SAR data files! 18

19 #CitrixSummit Monitoring XenServer to Identify Problems VHD link dependencies with PVS amd IntelliCache Provisioning Services (PVS) and IntelliCache leverage disk fast-cloning to quickly provision many virtual desktops. Fast-clones create new VDIs which are linked in parent-child relationships. Large XenDesktop environments can create many of these links, and this can cause issues…. 19

20 #CitrixSummit PBD Storage Repository (SR) VDI VBD XenServer HostVirtual Machine Parent VDI (Base Copy) Child VDI (fast-clone/diff-disk) VDA001-diff VDA002-diff VDA003-diff. VDAxxx-diff

21 #CitrixSummit Monitoring XenServer to Identify Problems MCS/PVS – Why might this be a problem? MCS profile creation and management can take a very long time, or fail completely. Host storage operations can be affected. Can cause poor performance, or even instability and XenServer host crashes! 21

22 #CitrixSummit Monitoring XenServer to Identify Problems Uploading a status report from the pool to TaaS can give a good graphical representation of the VDI link dependencies taas.citrix.com 22

23 #CitrixSummit Monitoring XenServer to Identify Problems MCS/PVS – How to Monitor For LVM-based storage: # vhd-util scan -f -c -p -m 'VHD-*' -l VG_XenStorage- For NFS-based storage: # vhd-util scan -f -c -p -m /var/run/sr-mount/ /*.vhd # vhd-util scan -f -c -p -m -v 'VHD-*' -l VG_XenStorage-8021d7b1-0b4d-03ff-d ef6eaf01 vhd=VHD-759d484d-2bc9-44c2-8d40-c84a b hidden=1 parent=none vhd=VHD a5-bd42-4d89-a172-a31387ed1c7a hidden=0 parent=VHD-759d484d-2bc9-44c2-... vhd=VHD-1daf350c fb d hidden=0 parent=VHD-759d484d-2bc9-44c2-... vhd=VHD c7d-4b03-a dfb2e5f67 hidden=0 parent=VHD-759d484d-2bc9-44c2-... vhd=... Parent VDI (base copy) Child VDIs (diff disks) 23

24 #CitrixSummit Monitoring XenServer to Identify Problems MCS/PVS – How to Monitor Check tapdisk2 process utilization top - 13:35:11 up 2 days, 19:02, 4 users, load average: 36.27, 23.64, Tasks: 435 total, 27 running, 408 sleeping, 0 stopped, 0 zombie Cpu(s): 15.2%us, 67.2%sy, 0.0%ni, 3.1%id, 10.1%wa, 0.0%hi, 1.8%si, 2.7%st Mem: k total, k used, 22260k free, 20388k buffers Swap: k total, 85720k used, k free, k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8722 root m 46m 4320 S :09.34 xapi root S :00.24 tapdisk root S :02.19 tapdisk

25 #CitrixSummit Monitoring XenServer to Identify Problems MCS/PVS – How to Monitor 00:00:01 CPU %user %nice %system %iowait %steal %idle 00:10:01 all :10: :10: :10:01 all :10: :10: :10: :10:

26 #CitrixSummit Monitoring XenServer to Identify Problems MCS/PVS – How to Monitor For extended monitoring, performance capture scripts can also be deployed on the XenServers to collect process and memory utilization statistics over time. For more information see CTX and CTX

27 Best Practices and Troubleshooting

28 #CitrixSummit Best Practices and Troubleshooting Tweaking XenServer Increase Dom0 memory allocation See CTX for instructions. Limit the number of hosts in the pool, or create multiple pools Optimize shared storage for PVS or IntelliCache See CTX for more information. Ensure XenServer and XenDesktop are at compatible versions and have all critical hotfixes and updates applied. 28

29 #CitrixSummit Best Practices and Troubleshooting Tweaking XenDesktop Max active actions Max new actions per minute Max power actions as percentage of desktops 29

30 #CitrixSummit Best Practices and Troubleshooting Tweaking XenDesktop (continued) Power Management and idle pool configuration. 30

31 #CitrixSummit Best Practices and Troubleshooting Decoupling VDIs When the VHD chains get too long, the VDIs need to be decoupled, i.e., fully copied to remove the parent-child links. Can be tedious and time-consuming. Contact Citrix Technical Support for expert assistance! 31

32 #CitrixSummit Best Practices and Troubleshooting General Storage and Network Troubleshooting When using NFS monitor for connection drops with the storage: # grep 'kernel: nfs: server [0-9.]* not responding, timed out' /var/log/messages Try to isolate storage, VM and host management network traffic. Ensure all the virtual desktops can communicate directly with the XenDesktop Controller. 32

33 #CitrixSummit Best Practices and Troubleshooting Other Common Issues Virtual desktops disconnect or hang when connecting Check the virtual desktop to see if 3 rd -party apps are interfering with logins Confirm the virtual desktop is not having issues communicating with the XenDesktop Controller. Ensure there are no GPOs or other Active Directory policies enabled on the virtual desktops that would interfere with logon/logoff behavior. 33

34 #CitrixSummit Best Practices and Troubleshooting Other Common Issues VMs fail to register in Desktop Studio Make sure the VMs are booting up OK and the XenServer hosts are not under heavy load preventing the VMs from operating normally. Also might be due to communication issues between the virtual desktops and the XenDesktop Controllerensure there is network connectivity between them. Make sure DNS is configured correctly in the environment. 34

35 #CitrixSummit Best Practices and Troubleshooting Other Common Issues XenServer pool master crashes or becomes unresponsive Ensure there are not too many VMs running on the master. Offloading VMs to the slaves and/or placing desktop groups into maintenance mode can mitigate this in the short-term. Monitor the load average in the pool and confirm that the master is not overburdened with specific tasks, such as storage management or XAPI task management. Check for long VHD chains. In large pools this could indicate the need to split the pool to decrease load on the pool master. 35

36 #CitrixSummit Best Practices and Troubleshooting Other Common Issues Takes a long time for all the virtual desktops to boot and register with the Desktop Controller Check the Advanced Host Configuration in the Desktop Studio and compare to the XAPI task queue and master load average to determine if the XenDesktop Controller is sending too many tasks at once. Check the idle pool settings to determine if the pool is trying to maintain too high of an idle pool count, and if so reduce the idle pool count to bring the pool into balance. Issues with registration which relate to load in large pools might also indicate the need to split the pool into multiple smaller pools. 36

37 #CitrixSummit Resources Monitoring and Troubleshooting 37 CTX XenServer performance: reality and myths CTX Memco.sh - Memory Data Collection Script for XenServer Dom0 or Linux SystemsCTX Memco.sh - Memory Data Collection Script for XenServer Dom0 or Linux Systems CTX Perfco.sh - Performance Data Collection Script for XenServer Dom0CTX Perfco.sh - Performance Data Collection Script for XenServer Dom0 CTX Troubleshooting XenServer Deployments CTX Troubleshooting XenDesktop, Provisioning Services & XenServer IntegrationCTX Troubleshooting XenDesktop, Provisioning Services & XenServer Integration

38 #CitrixSummit Resources Storage and Networking 38 CTX XenServer Storage Management and Troubleshooting CTX Introduction to Storage Technologies CTX Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network IssuesCTX Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues CTX Introduction to XenServer Networking CTX XenDesktop Planning Guide - Storage Best Practices

39 #CitrixSummit Resources Configuration and Reference 39 CTX XenServer 6.0 Administrator's Guide CTX XenDesktop Planning Guide – XenServer Integration CTX How to Collect Diagnostic Information for Citrix XenServer CTX How to Configure Dom0 Memory in XenServer 5.6 or later

40 #CitrixSummit 40 Tools as a Service checkered racing shoes Find out how to rev up environment maintenance See your Citrix pit crew in the expo hall with the

41 Questions

42 #CitrixSummit We value your feedback! Take a survey of this session now in the mobile app Click 'Sessions' button Click on today's tab Find this session Click 'Surveys'

43 #CitrixSummit Before you leave… Conference surveys are available online at starting Thursday, May 10www.citrixsummit.com Provide your feedback and pick up a complimentary gift at the registration desk Download presentations starting Monday, May 21, from your My Organizer tool located in your My Account

44


Download ppt "Optimizing XenServer Deployments to Best Support XenDesktop Daniel Lazar Senior Escalation Engineer May 8, 2012."

Similar presentations


Ads by Google