Implementing & Troubleshooting PVS Gareth O’Brien October 2012

Introduction Gareth O’Brien, escalation engineer Intermediate to advanced PVS knowledge expected

Agenda Planning Installing Configuring Troubleshooting Database, storage, network, farm layout Installing Configuring Files needed for booting, boot process Troubleshooting Stream Service Console, MAPI and IPC

Implementing PVS

Planning Planning the database configuration Hardware requirements Client & Server considerations Network Farm layout considerations

Database Size of farm determines database server choice Fault tolerance and failover Mirroring dbOffline – if all else fails SQL native client needed for mirroring Permissions Dbcreator & securityadmin for Configuration Wizard Db_datareader, db_datawriter & execute permissions on stored procedures

Hardware Requirements Supported Operating systems Windows Server 2003 R2, 2008, 2008 R2 Minimum spec of the PVS server Known hardware issues http://support.citrix.com/article/CTX131611 NIC Teaming Make sure hardware is not on above list If streaming doesn’t work with team, try break the team Processor: 3GHz RAM: 4GB NIC: 1GB per 250 targets

Network Configure UDP & TCP Ports Server to server ports must be the same on all servers Target device communication ports 6910-6930 SOAP Server 54321 & 54322 TFTP & TSB 69 & 6969 Firewall must be configured to allow all these ports If in doubt, allow ANY/ANY to test PortFast must be enabled Blocked Listen Learn Forward Disabled From blocking to listening: 20 seconds From listening to learning: 15 seconds From learning to forwarding: 15 seconds PortFast dramatically reduces this time. needed for DHCP, DNS & PxE (among others)

Plan the farm Site 1 Site 2 Provisioning Services Farm License Server MSSQL Server Database Shared Storage Site 1 Device Collection Provisioning Servers Provisioning Servers Device Collection Site 2

Installing PVS

Installing PVS Server installation Console installation If installing a new version of .NET, reboot when prompted Specify the mirror failover if applicable Console installation Needs PowerShell 2.0 If upgrading, make sure you remove the old version Target installation Make sure you set the target to boot from network first If imaging fails Start with a vanilla install & try to image. Remove any teams XenConvert & BNImage are other options

Configuring PVS

Configuring PVS PXE Boot Two Stage Boot (TSB) Boot Process Stores IPC Key

PXE Boot Must configure DHCP Uses TFTP Protocol Option 66: IP of PVS Server Option 67: Name of bootstrap (ardbp32.bin) Uses TFTP Protocol TFTP listens on one NIC only Configure using c:\program Files\citrix\provisioning services\tftpcpl.cpl TFTP Downloads bootstrap, ARDBP32.BIN Other TFTP servers can be used

Two Stage Boot (TSB) Used in situations where DHCP or PXE cannot be used Uses ISO burned from PVS Server Uses a boot strap called tsbbdm.bin Listens on NIC configured for streaming in configuration wizard Uses port 6969 Uses Two Stage Boot service on PVS server

Boot Process Target device either uses DHCP to obtain an IP or loads static IP (for TSB) Once it has an IP, it will contact the 1st logon server specified in the bootstrap The logon server will then create a context for the device, determine which vDisk the target is assigned and the least loaded server This calculation is based on both RIMS information and the load information in the database RIMS uses notifier and inventory service to update load information If subnet affinity is enabled, this will alter the calculation Once a server has been selected, a streaming IP and port will be provided IP information will be in HEX, not decimal 192.168.0 = C0A801 172.1.2 = AC12 10.0. = A0

Boot Process (cont) Excerpt from log of a target logging on 172.1.2.107

Boot Process (cont) As streaming begins, the boot control device (BCD) is downloaded This phase is referred to as single I/O Once Single I/O has finished, we enter the admin phase BNIStack handshake Vdisk configuration (name, mode, write cache type) is requested and provided Target device requests personality string HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BNIStack\Parameters DWORD EnableGetComputerName set to 1 AD information requested Multi I/O phase entered

Stores Configure vDisk stores Centralised Distributed Centralised vs Distributed stores Centralised Works well in HA environment  Low storage overhead  High cost  Potential SAN/Storage issues  Distributed Low implementation cost  High storage overhead  Configuration  IPC is the method for communication between servers for stores

IPC Key Defines which NIC to use for IPC communication in a multi NIC environment HKEY_LOCAL_MACHINE\Software\Citrix\ProvisioningServices\IPC\ Reg_sz called IPv4Address with the IP of the NIC for IPC Without it, stores, replication, load balancing etc won’t work Affects stream service Manager key for MAPI works the same way HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices\Manager RegSZ called GeneralInetAddr with the IP of the NIC and port Eg 10.1.1.2:6909

Troubleshooting PVS

A more structured method of troubleshooting PVS

Troubleshooting PVS Stream service troubleshooting SOAP/Console The basics Tracing Dumps Log Analysis Common Issues SOAP/Console Unable to validate Console timeouts

Stream Service Basics The devil is in the detail. Bluescreens Network Topology Hardware of target, server & database Storage type & location Write cache type Database location, version, HA MSInfo32 Bluescreens Common Image 7B What does it mean Common causes 7B = Inaccessable Boot Device

Stream Service Tracing Network traces Wireshark trace on the server Tcpdump on the target Make sure you always use –s 0 to not limit capture size PVS Logs Set to trace level C:\programdata\Citrix\Provisioning Services\Logs Target side logging also All logs and traces should be capture simultaneously, while the issue is happening.

Stream Service Tracing (cont)

Stream Service Tracing (cont)

Stream Service Tracing (cont)

Crash Dumps If possible, take it in private mode In 2008, it’s possible to redirect to a non system drive http://support.citrix.com/article/CTX123642 Not possible on 2003 Configure citrix symbols Live debugging can be done, but difficult without symbols Use live debugger to help set it up http://support.citrix.com/article/CTX131945 Check for BNIstack being loaded !devnode 0 1 On 5.6 check for bnns !ndiskd.miniports

Common issues Imaging Computer Account Let the wizard do the work for you Always revert to a clean install as a first step If the wizard doesn’t work, pre-create the disk and use xenconvert BNImage can work well if all else fails Computer Account Is AD Management configured on the vDisk? Is Active directory correctly configured? GPO disabling password update, and password age Reset the password within the PVS console Check all relevant Microsoft hotfixes are installed – Google! BNDevice 1015 in the event viewer – unable to negotiate a new password Trace logs will show you the requests

Performance Follow the best practice guide Intermediate buffering http://support.citrix.com/article/CTX127549 http://support.citrix.com/article/CTX117374 Intermediate buffering Generally enabled if free space on target device disk is greater than vDisk, otherwise disabled Toggling the state is always worth trying to see the performance impact Check interrupt safe mode is not enabled Check the network with a network monitor

SOAP & Console Console timeout Can’t communicate with other servers The console timeout can be set in the registry Use DWORD ConnectionTimeout in HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices. Default value is 120s Can’t communicate with other servers Check IPC and manager keys HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices\IPC\IPv4Address HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices\Manager\GeneralInetAddr Mentioned above Load Balancing issues Check the StreamProcess & Inventory logs to make sure there are no issues around RIMS

Questions? Remove clustering mention