1. CCAT Troubleshooting Training – Provisioning Services April 2012 Citrix Consulting Architecture Team

2. PVS Troubleshooting – Key Areas WireShark Analysis System Memory Dumps Key Takeaways Agenda

3. Troubleshooting PVS Issues

4. PVS Ports used - Recap ServiceProtocolPortComments DHCPUDP67DHCP Server port UDP68DHCP Client port PXEUDP67If PXE separate from DHCP server UDP4011If PXE on DHCP server TFTPUDP69Trivial File Transfer Protocol TSBUDP6969Two-Stage Boot Service ConsoleTCP54321-54322SOAP Service DatabaseTCP1433Microsoft SQL Server Stream Service UDP6890-6909Inter-server communications UDP6910-6930Streaming, initial login on port 6910

5. Troubleshooting Narrowing down scope of issue by determining where in the boot cycle the issue is occurring. Packet Traces Capturing Kernel Dump Files

6. Boot Stages DHCP – PXE boot downloading of bootstrap file Login to the PVS Server Streaming of the I/O Pre-BNIStack (Citrix Target Device driver) BNIStack up burst traffic

7. Using Wireshark – Packet Trace Analysis

8. Analyzing Packet Captures

9. Packet Capture PXE TFTP download TFTP Provisioning Server 10.1.1.1 DHCP Target 10.1.1.100 PXE

10. Common Issues During DHCP TFTP Download Target does not receive IP address Target gets IP but times out to TFTP PXE Errors No boostrap found TFTP Provisioning Server 10.1.1.1 DHCP Target 10.1.1.100 PXE

11. Login Process Traffic TFTP Provisioning Server 10.1.1.1 Target 10.1.1.100 Stream Process TFTP - Login

12. Common issues during login process Incorrect Bootstrap IP information Login port blocked Target not entered in Database Stream Service failed to start TFTP Provisioning Server 10.1.1.1 DHCP Target 10.1.1.100 PXE

13. Streaming Pre-BniStack TFTP Stream Process OS Booting UNDI Driver Target 10.1.1.100

14. Common issues pre BNIStack Large Send Offload Network instability Conflicts with UNDI Driver

15. Determining if BNIStack Driver is up TFTP Stream Process OS Booting BNISTACK Driver Target 10.1.1.100 Provisioning Server 10.1.1.1

16. Common issues post BNIStack Filter Driver conflicts Firewall/Antivirus Services kickoff Antivirus Updates being loaded Domain Profile creation

17. Capturing Windows System Dumps

18. Windows Kernel Dump files

19. Capturing Kernel Dump Files Capturing Kernel Dump files using Win2k3 and XP Capturing Kernel Dump files using Win2k8 Vista and Windows 7 Capturing Kernel Dump files in XenServer and ESX

20. Capturing Kernel Dump Files Win2k3 and XP Limitations Crash dumps to be on same drive as pagefile Page file to be on the system drive Redirection of dump not supported Live debug can be accomplished via serial cable.

21. Capturing Kernel Dump Files Win2k8 Vista and Windows 7 Enhancements allow kernel dumps to be directed to a location other then a system drive Requires a local hard drive with enough free space for the dump file. (Pagefile Size)

22. Capturing Kernel Dump Files Win2k8 Vista and Windows 7 Create a new string value under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl named DedicatedDumpFile Create Value data box, type :\ Create DWORD Value DumpFileSize use decimal under base and type the name of dump file in megabytes More detailed information in this process can be found: http://support.microsoft.com/kb/969028 http://support.citrix.com/article/ctx123642

23. Capturing a Kernel dump via XenServer Vista, Windows 7, Win2k8 redirection only possible if XenTools are not installed Live Debug via WinDBG possible by redirecting serial to DOM 0

24. Capturing a Kernel Dump Via Xenserver DOM 0 XEN SERVER WinDBG SockPipe.EXE Serial Redirect

25. http://support.citrix.com/article/ctx124979

26. Capturing Kernel dump via ESX Workstation 7 utility vmss2core Suspend hung VM Download suspended VM session (.vmss) Retrieve downloaded file and run command vmss2core –W filename.vmss

27. http://support.citrix.com/article/ctx125086

28. Booting Target Devices – TFTP vs. BDM Whiteboard Discussion

29. PVS Retries – Potential Causes Network Traffic High bandwidth utilization Congested pipe Application layer filtering Physical network problems UDP (LSO, Checksum Offload) Issues Check the vDisk Properties (System Tray) How to fix these issues? (Discussion)

30. Key Takeaways

31. Stream Process Workflow How to identify where in the boot process a PVS Target is failing Steps to capture a kernel level dump file from a provisioned target