Blaine A. Anaya XenServer Escalation Engineer 05/24/2011

Blaine A. Anaya XenServer Escalation Engineer 05/24/2011
Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues Blaine A. Anaya XenServer Escalation Engineer 05/24/2011

Agenda Overview XenServer Networking Architecture / vSwitch Architecture Troubleshooting the Network Introduction to XenServer Networking/ vSwitch Overview Terminology Architecture (Linux Stack\ vswitch) vSwitch Configuration DVSC import/configuration Troubleshooting the network On a live system Off-line using system status report

Agenda Overview XenServer Networking Architecture / vSwitch Architecture Troubleshooting the Network Performance Testing Introduction to XenServer Networking/ vSwitch Overview Terminology Architecture (Linux Stack\ vswitch) vSwitch Configuration DVSC import/configuration Troubleshooting the network On a live system Off-line using system status report

Networking Terminology
Switch Bridge Trunk Router Bond We are using terms that seem common but maybe understood in different ways. It is important to be on the same page and have a clear understanding of the terms we are using.

XenServer Networking Terminology
PIF- Physical Interface Object – directly correlates to a physical interface VIF- Virtual Interface Object- directly correlates to a virtual interface in a VM Bridge- Represents a network and is where PIFs and VIFs are plugged in Trunk – a switch port designated to carry traffic for more than one VLAN PIF – eth0 eth2 VIF- vif5.0 Identified by Domain ID and Device # Bridge- XAPI1,XENBR0 Trunk- Used to allow multiple networks on a given bond/network card Bond- Used to provide redundancy not aggregate traffic Bond- is the association of two network interface cards to make them appear as one Dom0- Short form of Domain 0 the control domain in XenServer that manages network and storage connections for virtual machines

Host Machine (Hardware)
XenServer Networking Dom0 DomU DomU Toolstack App App App App netback/0 netback/1 netback/2 netback/3 Guest OS Guest OS netfront/ VIF netfront/VIF Native Driver / PIF Bridge Looking at this diagram, we can see we have a physical machine capable of running XenServer. [Show Xen layer] -Add the Xen hypervisor when we install XenServer on the machine. -Runs on top of CentOS Linux -Augmented with processes and services to allow it to host virtual machines. [Reveal Next] Uses the concept of a ‘control domain’, or Dom0 Is itself a virtual machine that is “privileged” and brokers resources to virtual machines. Automatically starts when the XenServer machine boots. Has direct access to the host hardware using native Linux drivers installed with XenServer. - DomUs, or “unprivileged” guest domains are the user-created virtual machines that request resources from Dom0. - Gets created when we start a new virtual machine and runs alongside Dom0. To allow the guest VM to access hardware resources, Dom0 has a netback interface that communicates to a netfront interface within the guest OS. As we spin up VMs, additional DomUs get created and connect their netfront interfaces to Dom0 netback allowing them access to hardware resources. Finally, we have the xen toolstack, or XenAPI The group of services and processes that provide a means for managing Dom0 and making it aware of other XenServer objects such as storage repositories and network configurations. Prior to version 5.6 Feature Pack1 netback was more likely to be a bottleneck in virtual machine performance since all the VM’s were connecting to a single process running on a single vCPU but now with 5.6 FP1 we have a multi-vCPU Dom0 which by default allocates 4 vCPUs and 4 Netback processes that are each assigned an individual vCPU. Hardware can present a bottle-neck as it might not have enough memory or CPU resources to sustain all of the DomUs. Control domain—if it runs out of resources you can begin to imagine how that might affect the performance of the DomUs. Xen Hypervisor Host Machine (Hardware)

XenServer Networking Configurations- Linux Stack
XAPI Command Line XenCenter xsconsole Linux Config Files XenServer Pool DB Linux NIC Drivers Network Card This is the Linux Stack example showing how the XenServer configuration integrates with the Linux configuration files. Configuration changes are made through the XAPI interface via CLI xe commands, XenCenter, xsconsole. Those changes are written to the pool database The changes in the database also trigger a script to write out the Linux config files

XenServer Network Terminology
Private (xapi1) VIF Internal Switches Virtual Machine Network 0 (xenbr0) Network Card VIF Private networks appear as virtual switches that vifs and pifs plug into. The Network0 is connected to the physical network through the pif. The Private network is internal to the host since it is not connected to the physical network. PIF (eth0) Virtual Machine VIF

(xenbr0) Network Card PIF (eth0) VIF Internal Switches Virtual Machine Network 1 (xenbr1) Network Card VIF In this case we have XENBR0 network identifiers indicating that there is a 1-to-1 relationship to a PIF assigned to that bridge. If we saw something like XAPI1 then you would know that the bridge is associated with a Bond or is a Private network. PIF (eth1) Virtual Machine VIF

Bond 0+1 (xapi2) Network Card PIF (eth0) PIF VIF Virtual Machine PIF (bond0) Network Card VIF In this case Eth0 and Eth1 have been bonded so they are no longer individually connected to their own bridge. The xenbr0/xenbr1 configuration will be removed. A bond0 interface is created using the linux bonding driver and it is plugged into the bridge XAPI2. VIF PIF (eth1) Virtual Machine

Bonding Type (Balance SLB)
0:10 SEC 0:00 SEC 0:30 SEC 0:20 SEC Virtual Machine Bond Network Card Virtual Machine Network Card Supported Options: Balance SLB: xe pif-param-set uuid= other-config:bond-mode=7 *Active-Backup: xe pif-param-set uuid= other-config:bond-mode=1 *Supported with XenServer 5.6 FP1 Unsupported Options: Example: LACP: xe pif-param-set uuid= other-config:bond-mode=4 More about Bonding Types: - Understanding Network Interface Card Bonds in XenServer Virtual Machine Stacked Switches

Distributed vSwitch

Open Virtual Switch for XenServer
Visibility· Resource control · Isolation · Security VM VM VM VM VM VM VM VM VM VM VM Hypervisor Hypervisor Hypervisor With virtualization, one challenge is network visibility. The “last hop” to the VM is now a switch living in in the virtualization software on the host, not the top-of-rack switch as network administrators are more accustomed to. [Visibility]- Real time network traffic statistics (Rx bytes, packets etc.) that you can easily get on switches in the physical world [Resource control]- QoS rules enforcement. Enhanced monitoring. Through port monitoring, you could for example determine “if the XenDesktop user is running Pandora and causing performance issues” [Isolation]-Simpler network isolation and configuration of VLANs which are especially important in service provider environments—leading to much simpler “multi-tenancy” using Cross Host Private Networks. [Security] Setting of ACLs on virtual interfaces (VIFs) permits you to provide a configurable, XenServer-provided firewall for the VM. Example: block HTTP, enable only HTTP, and various other configurations are now possible. Note: In this slide the vSwitch are represented individually on each host showing that they can function as stand-alone switches but you do not get the benefit of central management as you will see in the next slide. Open Source Virtual Switch maintained at Rich layer 2 feature set (in contrast to others on the market) Ships with XenServer 5.6 FP1 as a post-install configuration option

Distributed Virtual Switch Controller
VM VM VM VM VM VM VM VM VM VM VM DVS Hypervisor Hypervisor Hypervisor Note: See that the DVSC merges the individual vswitches into a single logical switch that is managed centrally through the DVSC. Controller has it’s own web UI. This provides access to network administrators who probably don’t need XenCenter. One controller can serve multiple XenServer pools. Cross-host internal networks CHIN or Cross Server Private Networks CSPN. This feature supports muti-tenancy service providers. Support for Jumbo frames if using vSwitch technology When controller fails, it will fail ‘open’ allowing traffic to pass unhindered. This means that no ACLs or QoS rules will be applied. Hypervisor DVS Controller is a XenServer Virtual Appliance that controls multiple Open vSwitches

Distributed Virtual Switch
Built-in policy-based ACLs move with VMs VM VM VM VM VM VM VM VM VM VM VM DVS Hypervisor Hypervisor Virtual Interface (VIF) {MAC, IP} ACLs permit tcp eq domain permit tcp eq domain permit tcp eq domain permit udp eq domain permit udp eq domain permit udp eq domain permit tcp eq 123 Virtual Interface (VIF) {MAC, IP} ACLs permit tcp eq domain permit tcp eq domain permit tcp eq domain permit udp eq domain permit udp eq domain permit udp eq domain permit tcp eq 123 Hypervisor VM ACLs move with the VM, even after a live migration, instead of being tied to a specific host. This addresses the dynamic nature of a virtual environment allowing the security applied to a VM to follow it from host to host. This provides independence from physical network switch ACLs that would require manual intervention in some cases.

Enabling the vSwitch #xe-switch-network-backend openvswitch (Command must be ran on each individual host) ~]# xe-switch-network-backend openvswitch Cleaning up old ifcfg files Remove... ifcfg-bond0 Remove... ifcfg-bond1 Remove... ifcfg-eth0 Remove... ifcfg-eth1 Remove... ifcfg-eth2 Remove... ifcfg-eth3 Remove... ifcfg-eth4 Remove... ifcfg-eth5 Remove... ifcfg-xapi2 Remove... ifcfg-xapi4 Remove... ifcfg-xenbr0 Remove... ifcfg-xenbr3 Enabling openvswitch daemon Configure system for openvswitch networking You *MUST* now reboot your system Import Distributed vSwitch Controller (DVSC) Appliance Perform configuration at CLI on first boot Point web browser at DVSC and begin configuration. Select All Resource Pools => Add Resource Pool to add additional pools.

DVS Controller vSwitch Architecture – Process Level View JSON-RPC OpenFlow OVS ovsdb-server vswitchd Flow Table Cache vSwitch Network A Flow Table Cache vSwitch Network B ovsdb-server- Contains the network state database.This database holds information such as virtual ports, interface bonding specifications and per-port QoS policies. It can be manipulated remotely through a JSON-RPC interface, allowing an external controller to monitor various aspects of the network and respond when they change. As an example, this would allow the controller to respond to VM migration events, moving the network configuration of the VM‟s vSwitch port together with the VM. While traditional physical and virtual network switches use MAC forwarding tables for processing packets, OVS treats packet forwarding as a matching process, based on flow entries. Flow entries are stored in a flow table that is centrally managed by an external entity, generically known as the OVS Controller, which implements the packet forwarding logic. vswitchd- manages flow-table entries. ACL’s from the DVSC are translated into flow table entries. The flow table on managed vSwitch modules is updated by the vSwitch Controller using the OpenFlow protocol . When a packet does not match any flow table entry in the local vSwitch module, the packet is sent to the controller for evaluation. The controller then decides how the packet should be forwarded and makes this decision permanent in the OVS flow table by adding an entry for it. Unlike traditional MAC based forwarding, flow-based forwarding enhances flexibility and granularity in switching removing the restriction of simple layer 2 processing and making it is possible to specify how packets are handled based on their layer 2, layer 3, and layer 4 headers. Source: Virtual Networking for XenServer Technology Briefing Authors:Salvatore Orlando/Simon Farrugia VIF VIF VIF VIF PIF PIF

XenServer Networking Configurations- vSwitch
XAPI Command Line XenCenter xsconsole vSwitch Config XenServer Pool DB Linux NIC Drivers Network Card This is the vSwitch example showing how the XenServer configuration integrates with the vSwitch configuration. Configuration changes are made through the XAPI interface via CLI xe commands, XenCenter, xsconsole. Those changes are written to the pool database The changes in the database are written to the vSwitch configuration.

DVSC Web Interface Video Goes Here-
Demonstrate how to add additional pools for the DVSC to manage. Demonstrate setting up ACL Demonstrate how to view flow/network statistics Demonstrate how to change the netflow collector Give overview of DVSC server settings –configuration backup,ntp time sync,syslog

Agenda Overview XenServer Networking Architecture / vSwitch Architecture Troubleshooting the Network Introduction to XenServer Networking/ vSwitch Overview Terminology Architecture (Linux Stack\ vswitch) vSwitch Configuration DVSC import/configuration Troubleshooting the network On a live system Off-line using system status report

Troubleshooting The Network
Symptoms Issue Intermittent Packet Loss/ Dropped Connections Physical Connection/Switch Configuration, Bonding Network Appears Disconnected Physical Connection/Switch Configuration, Change in Hardware, Configuration Conflict. Bond Fails To Pass Traffic When One Leg is Disconnected 1.Intermittent Packet Loss/Dropped Connections – Note: Usually this is related to switch configuration when using bonds. Actions: Check physical connections and switch port configuration. Check network configuration on each host comparing the configuration in the XAPI database as well as the Linux config files /etc/sysconfig/network-scripts 2. Network appearing disconnected Actions: Check physical connections and switch port configuration. Check NIC status using ifconfig. If it shows as down use ifup to bring online. If NIC or motherboard was replaced then a pif-forget and pif-scan may need to be ran to make the new NIC available in XenCenter. 3. Bond failover does not appear to work Actions: Check physical connections and switch port configuration. If there are 2 switches and they are not stacked or are in an active passive configuration the bond mode should be set to active-backup. Physical Connection/Switch Configuration, Bond Mode

Using Command Line Interface (CLI) Off-line using a system status report BareGrep Pro Xenoscope

1.Check switch port configuration – Physical – Layers1-3 (Cables,NICs,Switch/Router connections) 2.Verify enabled network backend (Linux Bridge/vSwitch) 3.Use ifconfig –a to see bonds, physical interface statistics, bridges. 4.Use “brctl show” to see bridge/bond association. 5.Verify bonding configuration 6.Use ethtool for NIC settings, driver and firmware versions. [Step1] Physical-Layers 1-3 Use CTX and CTX for examples of network configuration. Also see page 20 of CTX127885 [Step2] Check Backend - Cat /etc/xensource/network.conf [Step 3] ifconfig –a Checking specifically for increasing errors and drops [Step 4] Brctl shows where everything is plugged in and how it is associated. [Step 5] Verify Bond configuration-Bridge mode: Cat /proc/net/bonding/bondX vswitch: ovs-appctl bond/show bondX also ovs-appctl bond/list [Step 6] Ethtool –k / ethtool -i [Step 7] Using the xe commands with params=all will give the most information about each objects configuration. 7.Use xe network-list, xe pif-list, to check XAPI configuration.

Troubleshooting the Network

Common Configuration Items to Check /etc/xensource/network.conf /etc/sysconfig/network-scripts /proc/net/bonding/bond0 /etc/sysconfig/iptables /etc/xensource/network.conf – If it contains bridge the Linux Stack is in use if it contains openvswitch then the vSwitch is in use and the Linux Stack has been disabled /etc/sysconfig/network-scripts – Network Configuration files when in bridge mode (Linux Stack Activated) /proc/net/bonding/bond0 – Check your active bond mode /etc/sysconfig/iptables –Check the firewall rules

Linux Bridge/vSwitch Enabled # brctl show # Shows the bridges and the interfaces plugged into them ~]# brctl show bridge name bridge id STP enabled interfaces xapi b8f no bond1 eth5 eth4 xapi d09699d no bond0 eth1 eth2 vif5.0 vif6.0 xenbr d09699d no eth0 xenbr b8c no eth3 brctl is used to set up, maintain, and inspect the ethernet bridge configuration in the linux kernel. An ethernet bridge is a software device commonly used to connect different networks of ethernets together, so that these ethernets will appear as one ethernet to the participants. Each of the ethernets being connected corresponds to one physical interface in the bridge. These individual ethernets are bundled into one bigger (logical) ethernet, this bigger ethernet corresponds to the bridge network interface. The command brctl show shows all current instances of the ethernet bridge.

Linux Bridge Enabled #brctl showmacs <brname> #Shows a list of learned MAC addresses for this bridge. ~]# brctl showmacs xenbr0 port no mac addr is local? ageing timer :00:0c:07:ac:3c no :0c:29:3a:12: no :0c:29:fa:8e:e no brctl is used to set up, maintain, and inspect the ethernet bridge configuration in the linux kernel. brctl showmacs <brname> shows a list of learned MAC addresses for this bridge. This will not work with vSwitch enabled.

Linux Bridge/vSwitch Enabled # netstat -np # Provides information on connections and processes. ~]# netstat -np Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp : : ESTABLISHED 2645/stunnel tcp : : ESTABLISHED 6280/stunnel tcp : : ESTABLISHED 31145/5 tcp : : ESTABLISHED 6280/stunnel tcp : : ESTABLISHED 6280/stunnel tcp : : ESTABLISHED 5304/ovs-vswitchd Netstat - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships -a -Show both listening and non-listening sockets. -n - Show numerical addresses instead of trying to determine symbolic host, port or user names. -p - Show the PID and name of the program to which each socket belongs. -s - Display summary statistics for each protocol. Windows Specific -b Displays the executable involved in creating each connection or listening port. In some cases well-known executables host multiple independent components, and in these cases the sequence of components involved in creating the connection or listening port is displayed. In this case the executable name is in [] at the bottom, on top is the component it called, and so forth until TCP/IP was reached. Note that this option can be time-consuming and will fail unless you have sufficient permissions.

Linux Bridge/vSwitch Enabled # netstat -s # Provides summary statistics for each protocol. ~]# netstat -s Ip: total packets received 9190 with invalid addresses 0 forwarded 0 incoming packets discarded incoming packets delivered requests sent out 8 dropped because of missing route Tcp: 69504 active connections openings passive connection openings 0 failed connection attempts 229 connection resets received 17 connections established segments received segments send out 3144 segments retransmited 0 bad segments received. 416 resets sent Netstat - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships -a -Show both listening and non-listening sockets. -n - Show numerical addresses instead of trying to determine symbolic host, port or user names. -p - Show the PID and name of the program to which each socket belongs. -s - Display summary statistics for each protocol. Windows Specific -b Displays the executable involved in creating each connection or listening port. In some cases well-known executables host multiple independent components, and in these cases the sequence of components involved in creating the connection or listening port is displayed. In this case the executable name is in [] at the bottom, on top is the component it called, and so forth until TCP/IP was reached. Note that this option can be time-consuming and will fail unless you have sufficient permissions.

Linux Bridge/vSwitch Enabled #ethtool –k <interface> #Provides information on current offload settings ~]# ethtool -k eth0 Offload parameters for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off ethtool - Display or change ethernet card settings xe pif-param-set uuid=<PIF UUID> other-config:ethtool-tx="on" -k - Queries the specified ethernet device for offload information. -K - Changes the offload parameters of the specified ethernet device. -i - Queries the specified ethernet device for associated driver information.

Linux Bridge/vSwitch Enabled #ethtool –i <interface> #Provides information on driver/firmware versions for network cards ~]# ethtool -i eth0 driver: bnx2 version: 2.0.8e firmware-version: bc 2.9.1 bus-info: 0000:04:00.0 ethtool - Display or change ethernet card settings xe pif-param-set uuid=<PIF UUID> other-config:ethtool-tx="on" -k - Queries the specified ethernet device for offload information. -K - Changes the offload parameters of the specified ethernet device. -i - Queries the specified ethernet device for associated driver information.

vSwitch Enabled #ovs-appctl bond/list #Shows Bridge, Bond, Slave Association ~]# ovs-appctl bond/list bridge bond slaves Xapi2 bond1 eth4, eth5 Xapi4 bond0 eth2, eth1 Disclaimer: Using OVS command line options for configuration purposes is not supported. The vSwitch should only be configured using XenCenter, xe CLI, xsconsole, and the Distributed vSwitch Controller.The commands shared here are for data collection and diagnostic purposes only.

vSwitch Enabled #ovs-appctl bond/show bond0 #Shows bond members, up/down delay, and next rebalance time. ~]# ovs-appctl bond/show bond0 updelay: ms downdelay: 200 ms next rebalance: 4314 ms slave eth2: enabled active slave hash 123: 1 kB load 86:43:b2:1a:f2:d0 slave eth1: enabled

vSwitch Enabled #ovs-appctl fdb/show <bridge_name> #Shows MAC Table/VLAN information for the bridge ~]# ovs-appctl fdb/show xapi4 port VLAN MAC Age :1d:09:2c:c4:c9 58 a:34:ee:08:53: a:e8:14:89:5c:af 42 ba:89:bf:f5:b8:ab 35 :16:c8:d8:f1:

vSwitch Enabled #ovs-ofctl dump-flows <bridge_name> #Shows FlowTable – (ACLs applied from controller) ~]# ovs-ofctl dump-flows xapi4 | grep drop May 02 15:49:07|00001|ofctl|INFO|connecting to unix:/var/run/openvswitch/xapi4.mgmt cookie=0x0, duration_sec=171s, duration_nsec= ns, table_id=1, priority=32763, n_packets=0, n_bytes=0, tcp,dl_dst=86:43:b2:1a:f2:d0,nw_dst= ,tp_src=80,actions=drop cookie=0x0, duration_sec=171s, duration_nsec= ns, table_id=1, priority=65529, n_packets=15, n_bytes=930, tcp,in_port=4,dl_src=86:43:b2:1a:f2:d0,nw_src= ,tp_dst=80,actions=drop FlowTable- Resides in Userspace module vswitchd entries are longer term.

vSwitch Enabled #ovs-dpctl dump-flows <bridge_name> #Shows FlowCache – (ACLs applied from controller) ~]# ovs-dpctl dump-flows xapi4 | grep mac86:43 tunnel :in_port0004:vlan65535:pcp0 mac86:43:b2:1a:f2:d0->00:00:0c:07:ac:3c type0800 proto6 tos0 ip > port4284->80, packets:1, bytes:62, used:2.160s, actions:drop FlowCache- Resides in Kernel module (FastPath) entries are temporary and can be cycled out quickly

vSwitch Enabled #ovs-appctl vlog/list #Show current logging levels ~]# ovs-appctl vlog/list console syslog file bridge EMER ERR INFO vswitchd EMER ERR INFO xenserver EMER ERR INFO ofproto EMER ERR INFO sflow EMER ERR INFO jsonrpc EMER ERR INFO fail_open EMER ERR INFO netflow EMER ERR INFO ovsdb_error EMER ERR INFO Make note of the modules, facility and log level. Modules: bridge, vswitchd,sflow,openflow, etc. Facility: Console,Syslog,File Level: Emer, Err, Info

vSwitch Enabled #vlog/set module[:facility[:level]] #Modify vswitch logging level Sets the logging level for module in facility to level: Module may be any valid module name (as displayed by the --list action on ovs-appctl(8)), or the special name ANY to set the logging levels for all modules. Facility may be syslog, console, or file to set the levels for logging to the system log, the console, or a file respectively, or ANY to set the logging levels for both facilities. If it is omitted, facility defaults to ANY. Note: The log level for the file facility has no effect unless ovs-vswitchd was invoked with the --log-file option. Level must be one of emer, err, warn, info, or dbg, designating the minimum severity of a message for it to be logged. If it is omitted, level defaults to dbg. Note: some important troubleshooting commands not covered are ping,tracepath and route and are instrumental in finding routing issues

Troubleshooting The Network – Off-Line
Status Report and BareGrepPro Perform Demo using saved status report- Look at the *.out files for ifconfig, ethtool, ovs-ofctl,ovs-dpctl. Review *.conf file for the network backend configuration. Use xml parser (CTX129070) to parse XAPI DB and compare configuration in XAPI to the running configuration. BareGrep can be downloaded at

Status Report and Xenoscope Show network specific sections of Xenoscope Step 1- Upload status report

Status Report and Xenoscope Show network specific sections of Xenoscope Step 2- View Host Overview Note: The networking backend field

Status Report and Xenoscope Show network specific sections of Xenoscope Step 3- Troubleshooting reports tab -> Networking Overview Shows NIC statistics, Driver/Firmware Versions, Bonding Mode, Bond Members.

Useful Networking CTX Articles
CTX Introduction to XenServer Networking CTX XenServer VLAN Networking CTX Understanding Network Interface Card Bonds in XenServer CTX Distributing Guest Traffic Over Physical CPUs in XenServer CTX XenServer Virtual Machine Performance Utility CTX How to Move a XenServer Pool to a Different IP Subnet CTX How to Identify the Network Adapters on XenServer CTX Communication Ports Used By Citrix Technologies

Q & A Source: Adams, Scott. Build a better life by stealing office supplies: Dogbert's big book of business. Andrews McMeel Pub, p64. Print. Continue the conversation:

Before you leave… Recommended related breakout sessions:
Session: YN203: Managing VM networking across the datacenter with XenServer distributed virtual switching Date: Wednesday May 25th Time: 4:30-5:15 Room: Moscone Session surveys are available online at starting Thursday, May 26 Provide your feedback and pick up a complimentary gift at the registration desk Download presentations starting Friday, June 3, from your My Organizer Tool located in your My Synergy Microsite event account

Blaine A. Anaya XenServer Escalation Engineer 05/24/2011

Similar presentations

Presentation on theme: "Blaine A. Anaya XenServer Escalation Engineer 05/24/2011"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Blaine A. Anaya XenServer Escalation Engineer 05/24/2011

Similar presentations

Presentation on theme: "Blaine A. Anaya XenServer Escalation Engineer 05/24/2011"— Presentation transcript:

Similar presentations

About project

Feedback