1. Virtualizing 3D Professional Graphics Apps
Yvonne Dresser, Sr. Product Marketing Manager, Citrix Desktops and Apps
Thomas Poppelgaard, Solution Architect, Commaxx
March 21, 2013
Copyright Citrix Systems, Inc.

2. Agenda XenDesktop & XenApp HDX 3D technologies Tips and Tricks
Audience Q & A

3. Industries that use 3D professional graphics
Aerospace
Automotive
Construction
Energy
Engineering Services
Hi Tech - Electronics
Industrial Equipment
Medical Equipment
If you are attending today’s webinar it’s likely because you work in one of these industries and support 3D professional graphics apps or will eventually need to support 3D apps.

4. Enterprise challenges and opportunities
Desire to leverage a global talent base and collaborate across the supply chain
Corporate requirements to safeguard product design IP
Need to extend access to design models on mobile devices
Economic demands for cost control and faster time to market
Business continuity and disaster-recovery plans
Intro:
In an increasingly global economy, companies are looking to improve time to market by managing design lifecycles in collaboration with offshore, mobile and remote employees while maintaining security and control over intellectual property (IP). Many organizations see desktop virtualization as an enabling technology that can play a key role in accomplishing these dual goals.
Until recently, companies did not seriously consider virtualizing graphics-intensive applications, such as CAD/CAM programs, due to technology and cost limitations. With today’s advancements in deep compression for improving network utilization and new GPU acceleration technology for virtualized environments, many organizations have begun to successfully virtualize their high-end graphics apps.
Talking Points:
A number of business drivers are accelerating this trend:
The desire to leverage a global talent base – Organizations recognize that to be competitive, they need to leverage technical talent wherever it’s located. Reasons include cost control and the ability to provide support close to the end customer.
Corporate requirements to safeguard product design IP – The need to share information with contractors, business partners and employees of outsourcing services providers is driving organizations to improve protection of their IP.
Workers’ need to view or present design models on mobile devices – The ability to pull up design documents and sophisticated models on the shop floor or at a customer location is becoming essential.
Economic demands for cost control and faster time to market – Follow-the-sun (24 x 7) development cycles and dispersed development teams require real-time, remote collaboration on design data.

5. Classification of 3D professional graphics users
Designers & Engineers
Power Users
Knowledge & Task users
Create & manipulate large 3D models
View & edit 3D data
Business graphics apps
Dedicated GPU Shared GPU Shared GPU
3D professional graphics users are classified based on GPU requirements. The table below shows this classification.
Designers and engineers are the most demanding user group. They create and manipulate large, complex, 3D models and require a dedicated GPU for graphics acceleration.
Hardware GPU acceleration is also required for power users. Users are classified in this segment when they need to view or edit graphics intensive 3D files, but do not perform complex actions.

6. Solutions for GPU-accelerated graphics
XenDesktop HDX 3D Pro
XenApp HDX 3D
GPU acceleration of DirectX and OpenGL ~1.5 Mbps
One user per GPU
H.264-based Deep Compression
3D SpaceMouse support
TOP PERFORMANCE SOLUTION
GPU acceleration of DirectX & OpenGL ~3 Mbps
GPU sharing
Lower cost Microsoft licensing
Apps must be compatible with RDSH (Terminal Services)
MOST COST-EFFECTIVE SOLUTION
When choosing between XenApp and XenDesktop for 3D professional graphics, these key decision factors should be considered.
Note that XenApp HDX 3D isn’t just for professional graphics; it is also a great fit for 3D business graphics apps such as WPF apps used in Healthcare for data visualization. With NVIDIA Q6000 or M2070Q cards, XenApp HDX 3D can support 8 to 24 users per card (limited primarily by video RAM and secondarily by the processing power of the GPU), and XenServer 6.0 can support four GPUs per server.
Citrix Confidential - Do Not Distribute

7. XenApp (Multi Users : 1 GPU) XenDesktop (1 User : 1 GPU)
Pod
XenServer with GPU cards
XenApp (Multi Users : 1 GPU)
Technical Overview
NetScaler Gateway
StoreFront Services
Receiver
HDX 3D Graphics
Primary
Secondary
Pod
XenServer with GPU cards
Delivery Controller
XenDesktop (1 User : 1 GPU)
Solution Architecture
The architecture shown in Figure is a visual representation of the complete virtual 3D professional graphics solution using that can support both designers and engineers as well as power user viewer and editors of 3D data. As you can see, no additional infrastructure is required to support 3D graphics users. You can easily leverage your existing infrastructure.
Internet
DMZ
LAN

8. with HDX 3D Pro Heavy 3D graphics app usage
Exceptional graphics performance and response optimizations
Multiple users per server
Very usable experience ~1.5 Mbps
OpenGL and DirectX
with HDX 3D Pro
Intro: The solution begins with XenDesktop which is our apps and desktop virtualization software. HDX 3D Pro is a feature of the XenDesktop VDI delivery method that supports high-end 3D graphics applications. With XenDesktop you can deliver a persistent desktop user experience, and leverage other virtualization benefits such as single image management and improved user density. With HDX 3D Pro and XenDesktop, you can deliver graphics applications as part of a complete virtual workstation or as only a VM-hosted app.
Key Points:
Provides multiple VMs on a workstation or blade host (up to 4 VMs or users each with a dedicated GPU) leveraging XenServer hardware direct GPU acceleration for optimizing graphics performance and response.
Delivers fantastic performance over a LAN and great performance over WANs with as little as 1.5 – 2 MB of bandwidth and RTL as high as 200 ms.
XenDesktop with HDX 3D Pro has been internally tested on all of the major CAD/CAM, GIS, and imaging applications and because no special hooks are required to run these apps, OpenGL and DirectX based apps are easily supported.
Transition: For light 3D graphics workloads and 2D business graphics apps there’s another option…

9. HDX 3D Pro Additional Features Pixel-perfect lossless compression
Can combine with lossy compression during motion, for responsive remote access
Dual monitor support
Receiver for Windows and Linux
Connexion SpaceMouse
Seamless app delivery
plug-in released May 2010 providing all capabilities of standard 12.0 online plug-in for Windows plus decoder for the HDX 3D Pro deep compression codec.
13.0 plug-in released August 2011 with Citrix Receiver 3.0.
TOC

10. Compression Options CPU-based, deep compression codec
min. 1.5 Mbps bandwidth
GPU-based implementation,
min. 2 Mbps
HDX 3D Pro provides two Deep Compression options – One is CPU based, the other GPU.
CPU-based deep compression:
In June 2012, we released the latest HDX 3D Pro VDA which includes advancements in CPU compression and is designed to offload compression processing from the GPU with the following benefits: One you can use a less expensive GPU card if you choose and the GPU processing can focus on graphics command delivery for better graphics performance. With our CPU deep compression codec, it is possible to achieve good performance with as little as 1.5 Mbps of bandwidth.
GPU-based deep compression:
GPU-based compression is advantageous on servers with limited CPU resources. To leverage our GPU-based deep compression codec, the host workstation in the data center must be equipped with an NVIDIA graphics card with at least 96 CUDA cores (128 or more recommended).
Fine drawing codec: HDX 3D Pro also includes a fine drawing codec for rendering fine detail graphics and images.
Lossless compression. HDX 3D Pro supports lossless compression, which enables you to
deliver pixel-perfect images for applications such as medical imaging.

11. XenDesktop with HDX 3D Pro
Desktop Virtualization for High-end 3D Graphics Users
Client options
Optimized with special codecs:
Std. ICA codec:
Minimum versions: Mac Receiver 11.4, Linux Receiver 12.0, Windows Receiver 3.0 (online plug-in for Windows 12.1), Android Receiver 3.0, iOS Receiver 5.0.
HDX 3D Pro in XenDesktop 5.6 FP1 includes Progressive Display (but not Adaptive Display) for the standard ICA codec.

12. with HDX 3D Light 3D apps usage Many users per VM
Very usable experience at 3 Mbps
Graphics performance and response optimizations
DirectX and OpenGL
with HDX 3D
Intro: you may also leverage a hosted-shared delivery model provided by our XenApp product (included with XenDesktop license) to host and share a full desktop or just apps on a Windows Server 2008.
Key Points:
XenApp hosted-shared delivery method is designed primary for running business graphics and light 3D professional graphics apps usage.
XenApp is limited to supporting only Direct X apps today
XenApp provides direct hardware GPU acceleration on a shared model vs. a 1: 1 user per GPU model with XenDesktop HDX 3D Pro combined with XenServer
Transition:

13. XenApp with HDX 3D additional features
Multi-monitor support
Lossless compression
When first introduced, HDX 3D Pro was focused on remote access over long-haul networks. Now, with dual-monitor support, this solution is also valid as part of a desktop replacement strategy. Virtual monitors are configured in the NVIDIA control panel using Forced EDID. Host must be configured with at least as many virtual monitors as the user device.

14. XenApp with HDX 3D Std. ICA codec only
Apps & Desktop Virtualization for 3D graphics users
Std. ICA codec only
Minimum versions: Mac Receiver 11.4, Linux Receiver 12.0, Windows Receiver 3.0 (online plug-in for Windows 12.1), Android Receiver 3.0, iOS Receiver 5.0.
HDX 3D Pro in XenDesktop 5.6 FP1 includes Progressive Display (but not Adaptive Display) for the standard ICA codec.

15. GPU Options
Any graphics card (NVIDIA, AMD/ATI) supported by the application vendor
NVIDIA cards deliver highest frame rate with VGX™ API (“Monterey”) and support full-screen applications
CPU-based deep compression is recommended, but if server CPU is limited then deep compression can be offloaded to the NVIDIA GPU
Generally speaking, this solution works with any graphics card that is supported by the ISV. However, our recommendation is an Nvidia CUDA-enabled graphics card. Citrix and Nvidia have partnered to deliver a graphics optimized solution that’s more scalable and performs better. With our latest HDX 3D Pro release, we began integrating with Nvidia’s VGX API to improve the overall frame rate resulting in much smoother visual performance.
If GPU-based compression is required, we recommend 128 CUDA cores for single monitor configurations and 256 CUDA cores for dual monitor configurations.

16. GPU Passthrough introduced in XenServer 6
XenDesktop HDX 3D Pro and XenApp HDX 3D
Windows 7 (32/64-bit), Windows XP (32-bit), Windows Server 2008 R2 (64-bit)
Multiple GPUs per host
Servers with 4 GPUs currently on HCL; testing underway with 8
One graphics-accelerated VM (single-user or multi-user) per GPU
Depending on CPU power, same host may also support non-graphics-accelerated users (i.e. regular office workers)
Leverages NVIDIA high-performance Quadro and Grid GPUs
e.g. Quadro 1000M/2000/3000M/4000/5000/6000, Tesla M2070Q, GRID K1/K2
In Q4 2011, Citrix introduced GPU passthrough on XenServer 6.0. The XenServer Multi-GPU Passthrough feature enables you to create VMs with exclusive access to dedicated graphics processing hardware. You can install multiple GPUs on the hypervisor and assign VMs to each of these GPUs on a one-to-one basis.
Use latest NVIDIA Quadro driver (min ).
This is typically a single or dual monitor solution since in the XS6 release you can have one GPU per VM and a single GPU generally supports two monitors.
Citrix Confidential - Do Not Distribute

17. XenServer GPU Passthrough with XenDesktop
XenDesktop Windows VMs
non-3D VM
non-3D VM
non-3D VM
non-3D VM
non-3D VM
non-3D VM
3D Pro VM
3D Pro VM
3D Pro VM
3D Pro VM
non-3D VM
non-3D VM
XenServer hypervisor
Hardware platform
GPU
GPU
GPU
GPU

18. XenServer GPU Passthrough with XenApp
Session 1
Session 2
Session 3
Session 4
Session 5
Session N-1
Session N
. . .
XenApp Windows Server VMs
XenApp VM
XenApp VM
XenApp VM
XenApp VM
XenServer hypervisor
Hardware platform
GPU
GPU
GPU
GPU

19. Multi-GPU Hardware Platforms
ws460c G6 blade (Q4000/5000/6000)  XenServer certified
Servers: DL980 G7, DL370 G6, SL390S G7 and others
R720 rack workstation (two dual-slot cards, e.g. M2070Q or GRID)  tested
R5500 rack workstation (3x Q4000 or 4x Q2000)  XenServer certified R5400 rack workstation (2 PCIe x16 GPUs, e.g. Q4000)
PowerEdge T620 (4x Q2000)
PowerEdge M610x with M2070Q plus C410x PCIe expansion chassis (4 GPU devices per server, 16 total GPUs per chassis)  tested w/ XenApp
HS22 with GPU expansion blade  tested with 2x M2070Q
System x iDataPlex dx360 M3/M4 (supports Q4000/Q5000/M2070Q)  tested
System x iDataPlex x3650 M3/M4 (2 dual-slot cards, e.g. GRID)  certified
Others
Cisco C240 M3; Fujitsu TX300; SGI Asterism ID112  XenServer certified
Citrix has been working closely with the system vendors to support GPU passthrough on XenServer. And, most system vendors on this list have at least one hardware platform tested and validated.
The HP ws460c G6 was the first hardware platform to appear on the HCL for XenServer 6 GPU Passthrough, but testing has now been done on additional platforms and they are being added to the list at:
The HP ws460c will support VGX cards.
To support 3x Q4000 on the Dell R5500 you require a power splitter to take the 8 pin to 2x6 pin, per Gary Radburn.
HS22 is expected to also support the Q2000 and Q HS22 will not support VGX cards.
R5500 can support 3 x Q4000 or 4 x Q2000.
PowerEdge M610x supports XenApp only (i.e. Windows Server 2008 R2).
iDataPlex dx360 M3 tested with 2 x M2070Q.

20. XenDesktop/XenApp on vSphere 5.1
GPU passthrough and GPU sharing on vSphere 5.1
vSphere 5.1 has introduced “virtual Direct Graphics Acceleration” (vDGA) and “virtual Shared Graphics Acceleration” (vSGA)
Testing for XenDesktop HDX 3D Pro compatibility with vDGA has been successfully completed (
Testing with vSGA successful (single monitor) for “tier 3” use cases
vSGA uses API Intercept method of sharing the GPU so performance may be an issue with large 3D models, and graphics API compatibility is limited to DirectX 9c and OpenGL 2.1

21. Looking ahead…
3D graphics enhancements in Excalibur (first release from project Avalon)
XenDesktop HDX 3D Pro
Multi-monitor support (i.e. more than two)
Dynamic resizing without disconnect
Auto-config / auto-adapt (no config tool / user intervention required)
XenApp HDX 3D
Deep Compression
Citrix Receiver (enhancements targeted for Excalibur timeframe)
Deep Compression support on Mac, iPad, Android
HDX SoC support for low-cost Linux thin clients
Multi-monitor support: More than 1 GPU supported, for example let's say there are 2 GPU cards and each card supports 2 monitor, then new release supports 4 monitors with 2 GPUs.

22. Looking ahead…
Hardware vGPU
At the 2012 GPU Technology Conference, NVIDIA and Citrix announced our collaboration on GPU Hardware Virtualization
Enhancements to the VGX™ API and XenServer along with new GRID graphics cards will extend Citrix’s GPU sharing capability from Windows Server RDSH to Windows XP/7/8 VDI
Higher performance with large 3D models compared to API Intercept method used by VMware vSphere/ESX and Microsoft Hyper-V
Ideal for second tier users of 3D professional graphics
Beta trials with selected partners starting Q2 2013

23. . . . . . . . . . GPU Virtualization GPU XenDesktop Windows VMs1 2 3 4 N
. . .
3D Pro VM
3D Pro VM
3D Pro VM
3D Pro VM
3D Pro VM
XenServer hypervisor
. . .
vGPU
vGPU
vGPU
vGPU
vGPU
Hardware platform
GPU
. . .

24. Tips and Tricks
Now, I’d like to bring Thomas Poppelgaard in to have a discussion about 3D graphics implementations and best practices….

25. Tips and Tricks XenApp 6.x HDX 3D
Optimize 3D application
HDX Policies (i recommend)
CTX Best Practices and Recommendations for Citrix Receiver 3 and HDX Technology with XenDesktop 5.x CTX132799
XenDesktop and XenApp Best Practice

26. Tips and Tricks XenApp 6.x HDX 3D
Citrix Receiver ADM.file

27. Tips and Tricks XenDesktop HDX 3D
VDA Agent (VDA 4.x, 5.5, 5.6, 5.6FP1)
Nvidia drivers (5.6FP1)
Multi-monitor support (VDA 4.x, 5.5, 5.6,)
3D connexion - 3D Space Mouse (VDA 4.x, 5.5, 5.6, 5.6 FP1)
Codec Compression (VDA 4, 5.5, 5.6)
ClearType (VDA 4, 5.5, 5.6)
Assign CPU cores to VM HDX 3D Pro

28. Upgrade existing VDA to HDX 3D Pro
Tips and Tricks XenDesktop HDX 3D
Upgrade existing VDA to HDX 3D Pro
If you have standard installed VDA and you want to enable HDX 3D Pro, you have to uninstall and install with HDX 3D Pro parameters.

29. Installing or upgrading Nvidia drivers
Tips and Tricks XenDesktop HDX 3D
Installing or upgrading Nvidia drivers
If you install NVIDIA drivers before you install XenDesktop Virtual Desktop Agent with HDX 3D Pro, NVIDIA VGX is enabled by default If you install NVIDIA drivers after you install XenDesktop Virtual Desktop Agent with HDX 3D Pro, NVIDIA VGX is disabled. Enable NVIDIA VGX by using the Montereryenable tool provided by NVIDIA To enable NVIDIA VGX, run the following command, then restart the virtual desktop agent: Montereyenable.exe –enable –noreset To disable NVIDIA VGX, run the following command, then restart the virtual desktop agent: Montereyenable.exe –disable –noreset

30. Tips and Tricks XenDesktop HDX 3D Pro - Multi-monitor support VDA 4-5

31. Tips and Tricks XenDesktop HDX 3D Pro
Enable USB redirection to enable 3D spacemouse
We need to enable USB direction as this is disabled by default and below is an example of the syntax for the rules
Examples of administrator-defined USB policy rules:
Allow: VID=1230 PID=0007 # ANOther Industries, ANOther Flash Drive
Deny: Class=08 subclass=05 # Mass Storage
To create a rule that denies all USB devices, use “DENY:” with no other tags.
Related Policies: Client USB device redirection

32. Tips and Tricks XenDesktop HDX 3D Pro
Enable USB redirection to enable 3D spacemouse
Change Virtual Channel Priority CTX Create a key named “WDSettings” under:
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\GroupPolicy\Defaults\] in VDA registry.
Create a MULTI_SZ named “VirtualChannelPriority” with the following content from CTX128190:
increasing the priority of the Generic USB Redirection virtual channel to 0 default value CTXGUSB,1 = configure this to CTXGUSB,0
We need to enable USB direction as this is disabled by default and below is an example of the syntax for the rules
Examples of administrator-defined USB policy rules:
Allow: VID=1230 PID=0007 # ANOther Industries, ANOther Flash Drive
Deny: Class=08 subclass=05 # Mass Storage
To create a rule that denies all USB devices, use “DENY:” with no other tags.
Related Policies: Client USB device redirection

33. Sizing GPU solutions XenApp HDX 3D - XenDesktop HDX 3D Pro
NVIDIA GRID (Worlds first GPU virtualization card)
Vendor
Card type
CUDA
GB/S
Power
Frame buffer MI
Slot
PCIe
Nvidia
GRID K1
768
150 W
16GB DDR3 2
PCI e 3.0
GRID K2
3072
225 W
8GB DDR5
Vendor
Card type
CUDA
GB/S
Power
Frame buffer MI
Slot
PCIe
Nvidia
Quadro 2000
192
41.6
62 W
1GB DDR5
128 bit 1
PCI e 2.0
Quadro 4000
256
89.6
142 W
2GB DDR5
256 bit
Quadro 5000
352
120
152 W
2.5GB DDR5
320 bit 2
Quadro 6000
448
144
204 W
6GB DDR5
384 bit
Tesla M2070Q
148
225 W
Quadro K5000
1536
173
122 W
4GB DDR5
PCI e 3.0

34. XenServer 6.0 Multi-GPU Passthrough for XenDesktop HDX 3D Pro Graphics
XenServer Host
The Server CPU should be IOMMU enabled for AMD processors or VT-d enabled for Intel processors.
Virtual CPUs
HDX 3D Pro Graphics utilizes one virtual CPU completely for extracting desktop surfaces, typically 12 to 30 times per second. If Aero is enabled on Windows this requires a second virtual CPU. In addition, at least two vCPU is required for the graphics applications. Graphics applications are often CPU intensive, even with hardware acceleration of graphics rendering provided by the GPU. Hence it is important to configure the virtual machine running HDX 3D Pro Graphics with at least four virtual CPUs. Following are the instructions for increasing the number of virtual CPUs on XenServer.

35. XenServer 6.0 Multi-GPU Passthrough for XenDesktop HDX 3D Pro Graphics
Warning:
Snapshot does not work with GPU Cards. It is highly recommended to create/delete/revert snapshot of VM when GPU card is not assigned to VM or else you may have garbage entries in XenServer and VM will not start properly.
When you assign virtual CPUs to a virtual machine, by default one virtual CPU is equal to one core for a virtual machine. Some virtual machines have higher core limitations. By setting the cores-per-socket parameter, it is possible to present a virtual machine with certain amount of cores. Note: This feature is only available in the Advanced, Enterprise, and Platinum editions and is not available in the free version of XenServer.

36. XenServer 6.0 Multi-GPU Passthrough for XenDesktop HDX 3D Pro Graphics
Complete the following procedure to set the cores-per-socket parameter for Windows virtual machine:
Get the Universally Unique Identifier (UUID) of the Virtual Machine for which the parameter needs to be set: xe vm-list name-label=<Name of the VM>
Set the cores-per-socket parameter:  xe vm-param-set platform:cores-per-socket=’X’ uuid=<VM UUID from Step1> Here, X equals 2 for dual core and 4 for quad core. For example: To have a CPU with four cores:  xe vm-param-set platform:cores-per-socket=4 uuid=<VM UUID>
Set the virtual CPUs startup parameter equal to the number of virtual CPUs (number of cores) to be assigned to the virtual machine and set the VCPUs-max parameter to the total number of cores. xe vm-param-set VCPUs-at-startup=<Number of VCPUs> uuid=<VM-UUID> xe vm-param-set VCPUs-max=<Maximum number of cores>.  For a dual quad core CPUs, VCPUs-max=8 The number of virtual CPUs at startup is the number of cores to assign to the virtual machine. 

37. HDX 3D and HDX 3D Pro Tools for troubleshooting/monitoring
Microsoft Process Explorer
Microsoft GPUView
Nvidia CUDA SDK
GPUz
HDX Monitor for XenApp

38. HDX 3D and HDX 3D Pro Tools for troubleshooting/monitoring
HDX Monitor for XenDesktop
HDX 3D Pro monitor
eG Innovations
Lakeside Software
Login VSI

39. Additional Information
Resources:
Blog: