VSphere 4.0 Module 4 – Networking Emiliano Turra Product Support Engineering VMware Confidential Rev. G.

vSphere 4.0 Module 4 – Networking Emiliano Turra Product Support Engineering VMware Confidential Rev. G

2vSphere 4- Mod 4 - Slide Agenda  Module 0 - Product Overview  Module 1 - VI Installation-Upgrade  Module 2 - VirtualCenter  Module 3 - Storage  Module 4 - Networking

3vSphere 4- Mod 4 - Slide Agenda – Lessons for Module 4  Module 4 - Networking  Lesson 1: vNetwork Distributed Switch  Lesson 2: Private VLAN  Lesson 3: IPv6  Lesson 4: VMXNET Generation 3  Lesson 5: VMDirectPath I/O  Lesson 6: Virtual Machine Communication Interface (VMCI)  Lesson 7: Basic Troubleshooting Tips

4vSphere 4- Mod 4 - Slide vNetwork Distributed Switch vCenter Standard Switch Distributed Switch

5vSphere 4- Mod 4 - Slide Distributed Switch Terminology  Terminology (in Red the official names)  DVN - vNetwork  Distributed Virtual Network, is the umbrella name under which the new network infrastructure components are grouped. The official name that customers will hear is vNetwork  dvSwitch, DVS or Distributed Virtual Switch - vNetwork Distributed Switch  Abstraction of multiple hosts sharing the same configuration for vSwitches and portgroups.  vSwitch - vNetwork Standard Switch  The “standard” virtual switch that is available in ESX 3.x and 4.x without vNetwork  dvPort  Port in a dvSwitch that allows VMs, vnics, VMKernel or Service Console nics.  dvPort status is stored in VC Database, so it is persistent across hosts  dvPortgroup  Collection of DVPorts that share the same configuration.

6vSphere 4- Mod 4 - Slide Distributed Switch  Distributed Switch: this means that the configuration is centralised to vCenter.  All the hosts that belong to a dvSwitch will not need further configuration to be compliant  Distributed Switch: the behaviour will still be the same (or consistent) with the vSwitch we are used to deal with:  dvPortgroups, as a set of dvPorts (the dv equivalent of Portgroups as a set of ports in a vSwitch)  Configuration is inherited from dvSwitch to dvPortgroup (the equivalent of what happens for vSwitch/Portgroup)  VMs, Service Console interface (vswif) and VMKernel interfaces can be connected to dvPortgroups as they could be connected to Portgroups in vSwitches  Hosts still own 2 configuration contexts, which are therefore not administered centrally via vNetwork:  Service Console and VMKernel interfaces  Physical NICs and their assignment to dvSwitch Uplink groups

7vSphere 4- Mod 4 - Slide Distributed Virtual Switch Architecture vCenter ESX 4 Distributed vSwitch vSwitch Distributed vSwitch vSwitch Control Plane Data Plane  Control Plane (CP) and Data Plane, or I/O Plane are separated.  CP, responsible for configuring dvSwitches,dvPortgroups, dvPorts, Uplinks, NICTeaming and so on, and for coordinating the migration of the ports, runs on vCenter  DP, responsible for performing the forwarding, runs inside the VMKernel of the ESX (Default VMware implementation of CP is via hidden vSwitch).

8vSphere 4- Mod 4 - Slide Distributed Virtual Switch Architecture – Data Plane vSwitch Data Plane Port IO Filter Forwarding Engine Teaming Engine Port IO Filter  Filters (DVN Switch API, or dvFilter)  Forwarding (DVN Appliance API, or VSafe-net)

9vSphere 4- Mod 4 - Slide Uplink Abstraction UPLINK groups allow for abstraction from the physical implementation of each server.  Each Physical host can contribute with up to 1 NIC to each Uplink group  vCenter will only see the uplink groups when configuring the Distributed Switch, because each host can contribute in a different way (vmnic0,1,2,3,…) vCenter vmnic0,1,2,3,…?

10vSphere 4- Mod 4 - Slide Comparing Standard and Distributed Switch  Both  can forward L2 frames  can segment traffic into VLANs  can use and understand 802.1q VLAN encapsulation  can have more than one uplink (Nic Teaming)  can have traffic shaping for the outbound (TX) traffic  Only Distributed Switch  can shape inbound (RX) traffic  has a central unified management interface through VC  supports Private VLANs (PVLANs)  provides potential customisation of Data and Control Planes StandardDistributed L2 SwitchYES VLAN SegmentationYES 802.1Q TaggingYES NIC TeamingYES TX Rate LimitingYES RX Rate LimitingNoYES Unified management interface NoYES PVLANNoYES 3 rd Party Virtual Switch SupportNoYES

11vSphere 4- Mod 4 - Slide Distributed Switch does/does not’s  DS is/does  Simplify datacentre setup by centralising network configuration  Will make it easier for VI Admins to add hosts to the cluster and have them immediately VMotion compatible  Each dvPort is unique across the dvSwitch, and therefore across the cluster, and will follow the “client” if it is moved around, for example VMotion of a VM.  DS is NOT:  A single and whole Standard Switch across hosts, because:  It behaves roughly as if you had Standard Switches configured consistently across the hosts  The traffic between two VMs on the same dvPortgroup but on different hosts will still go through the physical network via the Distributed Switch Uplinks  PVLANs require physical configuration or VMotion will break connectivity.

12vSphere 4- Mod 4 - Slide Standard Switch + Host Profiles = DS ?  Standard Switch + Host Profiles = Distributed Switch ?  You get all the Standard Switch Features plus the ability to re-create them on new hosts  No DS features  Manual process of applying new modifications to all the hosts  There is no Uplink group, so when vmnic names differ across hosts, configuring nicteaming might be impossible via one single profile  Changes are applied in maintenance mode

13vSphere 4- Mod 4 - Slide Custom Distributed Switch vCenter ESX 4 Distributed vSwitch vSwitch Control Plane  I/O Plane (Data Plane) and Control Plane can be replaced with 3 rd party versions  Custom Data Plane implements Forwarding/Filtering/Teaming, basically replacing the vSwitch  Custom Control Plane is implemented as an appliance, and will be responsible for handling the configuration of the ports (storing, changing and migrating), and coordinating the configuration across DPs (across hosts)  Data Plane Agents (DPAs) will run as VMKernel Worlds and will be responsible of communication between CP and DPs vC Extension vSphere Client Plugin Control Plane Appliance Data Plane DataPlane Agent

14vSphere 4- Mod 4 - Slide Creating Distributed Virtual Switch - 1  Go to Home > Inventory > Networking  If you are in other locations, the “New DVS” button is disabled  Create a new Distributed Switch  Specify:  Name of the Distributed Switch  Number of Uplink Ports  Uplinks can be renamed/added afterwards

15vSphere 4- Mod 4 - Slide Creating Distributed Virtual Switch - 2  Add hosts and Uplinks (vmnic groups) from Cluster  An Uplink is to a Distributed Switch what a vmnic is to a Standard Switch  Due to the fact that the Distributed Switch is a “logical/abstract” entity that exists across hosts, the association between a Distributed Switch and each host’s vmnic is done via this further abstraction called Uplink.  What is called Uplink here is a group of vmnics, grouped by the VI Administrator when adding hosts/vmnics to the Distributed Switch

16vSphere 4- Mod 4 - Slide Creating Distributed Virtual Switch – 3  Select whether to create a default Portgroup or not  The Distributed Switch is ready Uplinks

17vSphere 4- Mod 4 - Slide Assigning Uplinks to a Distributed Switch  Uplinks are associated automatically at Distributed Switch creation time  If changes need to be applied, they have to be applied from the host  Therefore in vCenter, go to Host > Configuration > Networking  Select DVS view  Click on “Manage Physical Adapters”  If you click on the first “ ”, the NIC will be added to the “Pending Uplink Assignment” group and assigned automatically when you press “Ok”  Click on “ ” below the Uplink group you wish to assign the vmnic to

18vSphere 4- Mod 4 - Slide Managing Distributed Switch Distributed Switch properties are grouped in 3 tabs:  Properties  General  Advanced  Network Adapters  View Physical adapter contributed by each member (ESX). No modification allowed from this screen, you need to go to the specific host configuration for managing Uplinks  Private VLAN  Where you can associate/edit Primary and Secondary PVLANs.  Changes might not take place if you try to edit PVLANs that are in use, disconnect the VMs first. We will see PVLANs later

19vSphere 4- Mod 4 - Slide Managing Distributed Switch - General  General  Allows you to define (Prompted also at DVS Creation time)  the DVS name,  the number of UPLINK ports,  Additionally, allows you to define  notes  It allows also to edit the Uplink names.

20vSphere 4- Mod 4 - Slide Managing Distributed Switch - Advanced  Advanced  Allows to define:  Max value for Maximum Transmission Unit (Useful for enabling Jumbo Frame) For the Standard vSwitch, the only options are: esxcfg-vswitch –m and -l  Cisco Discovery Protocol Status For the Standard vSwitch, the only options are: esxcfg-vswitch –B and -b  Administrator’s details

21vSphere 4- Mod 4 - Slide Distributed Switch Portgroups  Similarly to what happens with the standard vSwitch, also in a Distributed Switch Portgroup:  represents a group of Ports that share the same configuration template.  does not constitute the means to segregate traffic  Settings divided into 3 categories :  General  Policies  Security  Traffic Shaping  VLAN  Teaming and Failover  Miscellaneous  Advanced

22vSphere 4- Mod 4 - Slide Distributed Switch Portgroups - General  General  Allows you to define  The name of the portgroup  A description  The number of ports available  The type of Port Binding, which can be  Static  Dynamic  None (Ephemeral ports)

23vSphere 4- Mod 4 - Slide Port Binding  Static Binding (Default): means that the dvPort will be assigned to the VM at configuration time. Once all the ports are “booked” by VMs, it will not be possible to connect any more VM, independently from the fact that the connected VMs are powered up or not, and an error message will be displayed  Dynamic Binding: means that the dvPort will be assigned at the moment of powering the VM up. This option allows for over committing the number of dvPorts.  Ephemeral Ports or No Binding: this behaviour has been introduced to resemble the behaviour in the standard vSwitch. If you select this option, the number of ports will be automatically set to 0, and the Portgroup will allocate one port for each connected VM, up to the maximum number of ports available in the Switch.

24vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – Security  Policies (shows all the options below together)  Security  Similar to what we have already seen in the vSwitch, this section allows you to define security policies for:  Promiscuous mode  Allowing machines to see the traffic of all the other machines in the DVS  Mac address changes  Allows VMs to receive frames with a Mac Address that is different from the one configured in the VMX  Forged Transmits  Allows VMs to send frames with a Mac Address that is different from the one specified in the VMX

25vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – Traffic Shaping - 1  Policies (shows all the options below together)  Traffic Shaping  Allows you to define ingress and egress traffic shaping.  Ingress shaping is a new feature, and available only with DVS (not on vSwitch)

26vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – Traffic Shaping – 2  Traffic Shaping concepts:  Average Bandwidth  Target traffic rate cap that the switch will try to enforce. Every time a client uses less than the defined Average Bandwidth builds up credit.  Peak Bandwidth  Extra bandwidth available, above the Average Bandwidth specified above, for a short burst. The availability of the burst depends on credit accumulated so far  Burst Size  Amount of traffic that can be transmitted or received at Peak speed (Combining Peak Bandwidth and Burst Size you can calculate the maximum allowed time for the burst)

27vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – VLAN - None  Policies (shows all the options below together)  VLAN (Allows you to specify the VLAN behaviour of the dvSwitch, VDS Only): NONE Physical equivalent to: No VLAN Tagging Standard vSwitch equivalent to: VLAN ID option set to 0 EST – External Switch Tagging

28vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – VLAN – Single VLAN  Policies (shows all the options below together)  VLAN (Allows you to specify the VLAN behaviour of the dvSwitch, DVS Only): VLAN Physical equivalent to: VLAN in Access/Untagged mode Standard vSwitch equivalent to: VLAN ID option VLAN ID 4095 is not allowed here VST – Virtual Switch Tagging

29vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – VLAN - Trunk  Policies (shows all the options below together)  VLAN (Allows you to specify the VLAN behaviour of the dvSwitch, VDS Only): VLAN Trunking Physical equivalent to: VLAN in Trunk/Tagged mode Standard vSwitch equivalent to: VLAN ID set to 4095 VGT – VLAN Guest Tagging VDS Only: option to specify the range of VLANs to trunk, to improve security.

30vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – VLAN - PVLAN  Policies (shows all the options below together)  VLAN (Allows you to specify the VLAN behaviour of the dvSwitch, DVS Only): PVLAN Physical equivalent to: PVLAN Standard vSwitch equivalent to: Does not exist PVLAN option to specify which Primary and Secondary VLAN to use (Selecting from the list defined in the Switch)

31vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – Teaming & Failover  Policies (shows all the options below together)  Teaming and Failover  Allows policies to be defined for:  Load Balancing  Failover detection  Notify Switches  Failback  Failover order  Specific Uplink usage From the screenshot on the right, you can see how the Active/Standby status is applied to each uplink group (dvUplink1 and 2 in this case), and not to the vmnics directly, as it used to be with standard vSwitches

32vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – Misc.  Policies (shows all the options below together)  Miscellaneous  Allows you to block all the dvPorts of the dvPortgroup, DVS Only

33vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – Advanced - 1  The dvPortgroup Advanced subcategory is different from dvSwitch:  It allow each single dvPort to override the settings of the dvPortgroup. clicking on “Edit Override Setting” the VI Admin can also specify which properties to allow/not allow to be overridden at lower levels.

34vSphere 4- Mod 4 - Slide Distributed Switch Portgroups – Advanced - 2  The dvPortgroup Advanced subcategory is different from dvSwitch:  It allow each single dvPort to override the settings of the dvPortgroup. clicking on “Edit Override Setting” the VI Admin can also specify which properties to allow/not allow to be overridden at lower levels.

35vSphere 4- Mod 4 - Slide Configuring Distributed Switch Virtual Adapters -1  Two types of Virtual Adapters:  Service console vswif  VMKernel vmknic  To use Virtual Adapters inside a dvSwitch, you need to configure them via Host > Configuration > Networking, as this is not a cluster-wide option.  Select Distributed Virtual Switch view and click on “Manage Virtual Adapters”

36vSphere 4- Mod 4 - Slide Configuring Distributed Switch Virtual Adapters -2  You’ll be prompted with the “Manage Virtual Adapters” dialog, where you can:  Add a new adapter  If you already have DVS virtual Adapters, you’ll be able to:  Edit the adapter (IP address/netmask, default gateway, DNS servers)  Migrate it back to a vSwitch  Delete it (Deleting the last vswif is not allowed)

37vSphere 4- Mod 4 - Slide Configuring Distributed Switch Virtual Adapters -3  If you click on “Add”, for each Virtual Adapter type, there will be 2 options:  Create a new Adapter  Migrate the existing from vSwitch to dvSwitch  Either way, you’ll be prompted to specify an existing dvPortgroup to be connected to

38vSphere 4- Mod 4 - Slide Migrating from Standard Switches  If after selecting “Add”, you chose to “Migrate existing virtual network adapters”, you’ll be prompted with the form below  Select which adapters you wish to migrate  For each selected adapter, specify which dvPortgroup you want to connect it to.  The migration will take care of not interrupting the traffic, so for example vCenter won’t show the ESX as disconnected even if you migrate its only vswif interface

39vSphere 4- Mod 4 - Slide Migrating from vSwitches - logs Example: migrating vswif2 with IP address 192.168.9.1 (Hex 0x109a8c0) from vSwitch0 to dvSwitch: cpu1:4175)DVSDev: DVSDevDataSet: setting data com.vmware.common.port.connectid on port 97 cpu3:4177)DVSDev: DVSDevDataSet: setting data com.vmware.common.port.portgroupid on port 97 cpu4:4179)DVSDev: DVSDevDataSet: setting data com.vmware.common.port.block on port 97 cpu4:4170)DVSDev: DVSDevDataSet: clearing data com.vmware.common.port.shaper.input on port 97 cpu4:4168)DVSDev: DVSDevDataSet: clearing data com.vmware.common.port.shaper.output on port 97 cpu1:4167)DVSDev: DVSDevDataSet: setting data com.vmware.etherswitch.port.teaming on port 97 cpu3:4178)DVSDev: DVSDevDataSet: setting data com.vmware.etherswitch.port.security on port 97 cpu3:4169)DVSDev: DVSDevDataSet: setting data com.vmware.etherswitch.port.vlan on port 97 cpu1:4175)DVSDev: DVSDevDataSet: clearing data com.vmware.etherswitch.port.ipfix on port 97 cpu3:4177)DVSDev: DVSDevDataSet: setting data com.vmware.common.port.statistics on port 97 cpu0:4096)Tcpip_Socket: vmk_set_ip_address:968: index = 145660792, ip_addr = 0x109a8c0, netmask = 0xffffff cpu1:4109)Mirror: Mirror_PortDisable: removing wildcard INPUT match port vswif2(0x8) from session legacy_promiscuous cpu0:4096)Net: NetDisconnect:1250: disconnected from net vSwitch0, PortID = 0x8 Preparing dvPort 97 to receive vswif2

40vSphere 4- Mod 4 - Slide Migrating from vSwitches - logs cpu0:4096)NetDVS: DVS_PortAssociate:413: Connecting to DVS 49 89 34 50 eb b6 a0 ae-d9 d3 3e e1 68 b4 5d 45 port 97 cpu0:4096)NetDVS: DVSPortAssociate:1155: port 0x410004256ce0 (type 1) cpu0:4096)NetDVS: DVS_PortAssociate:438: Connected to DVS port 97 (type 1), dvs 49 89 34 50 eb b6 a0 ae-d9 d3 3e e1 68 b4 5d 45 cpu0:4096)NetPortset: Portset_ConnectPort:1251: newID 0x300000c, newIDIdx 0xc, psMask 0x1ff, newPort 0x41000412db80, portsInUse 6, portCfgName cpu0:4096)Net: NetConnectCommon:1054: connected to net (null), portset 0x410004004428, PortID = 0x300000c, status 0x0 cpu6:4111)Net: COSVMKDev_Enable:1419: port = 0x300000c, cosStateVA = 0x41007cb88000, cosStateVP = 0x41007cb88000, cosStateLen=0x649c cpu6:4111)Net: COSVMKDev_Enable:1444: txRing = 0x41007cb8949c, rxRing = 0x41007cb8809c, numRxBufs = 0x80, numTxBufs = 0x80 cpu6:4111)Net: COSVMKDev_Enable:1468: COS VMK gen count = 11 cpu6:4111)Net: COSVMKDev_Enable:1481: Enabling NIC in the shadow vmkernel tcpip stack cpu6:4111)Tcpip_Interface: vmk_nic_attach:893: ether attach complete cpu6:4111)NetDVS: DVS_PortLinkUp:501: DVS_PortLinkUp portID 0x300000c DVS port 97 cpu6:4111)NetPort: PortBlockSet:2040: resuming traffic on DV port 97 cpu6:4111)VLAN: VLAN_UpdateDVSPortCfg: VLAN 64 configured for DVPort 50331660 cpu6:4111)etherswitch: NCP_AddBeaconVID: 64 cpu6:4111)Mirror: MirrorSessionWildcardAddPort: adding wildcard match port vswif2(0x300000c) for INPUT to session legacy_promiscuous

41vSphere 4- Mod 4 - Slide Migrating VMs Between dvPortgroups  VI4 introduces a new feature that allows you to mass-move VMs from one dvPortgroup to another  To initiate a Migration, go to the Summary page of the dvSwitch (from Host > Inventory > Networking)  Click on “Migrate Virtual Machine Networking”  Select Source and Destination dvPortgroup  Click on “Show Virtual Machines”  Select the VMs you want to Migrate

42vSphere 4- Mod 4 - Slide Migrating to DS Step by Step vswif0 vmk0 vm1 vm2 vSwitch0 vSwitch1 dvPG0 dvPG1 DS Uplink1 Uplink2 1 Uplink3 Uplink4 Uplink1 Uplink2 Uplink3 Uplink4 0 Steps: 1.Create a DS with as many Uplink groups as Physical NICs connected to the Standard Switches 2.Create in the DS as many Portgroups as you already have in the SS 3.Assign Uplinks to each Portgroup in the DS 4.Break each teaming and transfer one NIC from each vSwitch to a corresponding Uplink group 5.Migrate the Virtual Adapters and the Virtual Machines to the appropriate Portgroups 6.Transfer the remaining uplinks to the Uplink groups associated with the appropriate Portgroups 7.Remove the Standard Switches and their Portgroups 2 3

43vSphere 4- Mod 4 - Slide Migrating to DVS Step by Step vswif0 vmk0 vm1 vm2 vSwitch0 vSwitch1 vSwitch0 vSwitch1 DVS Uplink1 Uplink2 vSwitch0 vSwitch1 DVS Uplink1 Uplink2 vswif0 vmk0 vm1 vm2 DVS Uplink1 Uplink2 Uplink3 Uplink4 dvPG1 dvPG2 0 1 2 3 0 1 2 3 0 1 2 3 0 2 1 3 1234 vswif0 vmk0 vm1 vm2 vswif0 vmk0 vm1 vm2

44vSphere 4- Mod 4 - Slide Lab Exercise Lab 1: vNetwork Distributed Switch

45vSphere 4- Mod 4 - Slide Agenda – Lessons for Module 4  Module 4 - Networking  Lesson 1: vNetwork (Distributed Virtual Networks)  Lesson 2: Private VLAN  Lesson 3: IPv6  Lesson 4: VMXNET Generation 3  Lesson 5: VMDirectPath I/O  Lesson 6: Virtual Machine Communication Interface (VMCI)  Lesson 7: Basic Troubleshooting Tips

46vSphere 4- Mod 4 - Slide What are Private VLANs ? What is a Private VLAN?  VLAN is a mechanism to divide a broadcast domain into several logical broadcast domains  Private VLAN is an extension to the VLAN standard, already available in several (most recent) physical switches. What it does is add a further segmentation of the logical broadcast domain, to create “Private” groups  Furthermore, because it divides a VLAN (which will be called “Primary” PVLAN) into one or more “groups” (called “Secondary” PVLANs), this means that all the Secondary PVLANs exist only within the Primary VLAN.  Private because, depending upon the type of the “groups” involved, hosts will not be able to communicate each other, even if they belong to the same group.  Each Secondary PVLAN has an associated VLAN ID, and the physical switch will associate the behaviour (Isolated, Community or Promiscuous) depending on the VLAN ID found in each packet. Do not Disturb

47vSphere 4- Mod 4 - Slide Secondary Private VLAN Types PrimarySecondaryType 5Promiscuous 5155Isolated 517Community Host 1 Host 2 Host 3 Host 4 Host 5 Host 6 155 5 5 17  Three types of Secondary PVLANs:  Promiscuous  A node attached to a port in a promiscuous secondary PVLAN may send and receive packets to any node in any others secondary VLAN associated to the same primary. Routers are typically attached to promiscuous ports.  Isolated  A node attached to a port in an isolated secondary PVLAN may only send to and receive packets from the promiscuous PVLAN.  Community  A node attached to a port in a community secondary PVLAN may send to and receive packets from other ports in the same secondary PVLAN, as well as send to and receive packets from the promiscuous PVLAN.

48vSphere 4- Mod 4 - Slide Private VLAN Implementation  Standard 802.1Q Tagging  No Double Encapsulation  Switch software decides which ports to forward the frame, based on the tag and the PVLAN tables PrimarySecondaryType 55Promiscuous 5155Isolated 517Community VLAN 5 PVLAN 5 (Promiscuous ) PVLAN 5 (Promiscuous ) PVLAN 155 (Isolated) PVLAN 155 (Isolated) PVLAN 17 (Community) PVLAN 17 (Community) 1555175

49vSphere 4- Mod 4 - Slide Why Private VLANs ? Problem  Why PVLANs? (examples)  Machines can be violated/infected, and can be used as a bridge for violating/infecting other machines in the same network segment  Attacks like ARP poisoning are still a danger, and port-security type of defence does not work well with ESX (For example in case of VMotion, if you set port-security to allow a maximum of X different MAC addresses, when you VMotion a VM that happens to be the X+1th, you’ll lose connectivity)  Segmentation of each and every host in the network is required Internet Infected Machine, acting as a bridge to infect others Machine that would not be reachable from Internet Gateway/Serer Rogue machine performing ARP Poisoning impersonating the gateway Victim sends traffic to the rogue instead of the gateway

50vSphere 4- Mod 4 - Slide Why Private VLANs ? Solutions Solutions:  One VLAN per host or group of hosts  CONS:  A a lot of subnets of the /30 type, with waste of IP addresses (50%)  Consequently, lot of routes, which are difficult to maintain and change  Complex and expensive gateway (firewall) rules  Available VLANs are 4095*, but switches allow much less, about 1000  Too complex/expensive to maintain  One VLAN per VM, with one VM acting as transparent/software bridge with firewall, thus on the same subnet  Can be implemented inside ESX 3.x  Even more complexity/cost  PVLAN

51vSphere 4- Mod 4 - Slide Private VLANs: Example without PVLANs Internet Gateway ISP For bigger Customers, /29 or /28 Subnets Several /30 Subnets Example: Hosting company:  Many different customers that should not be able to “see” each other  Possible solution:  One VLAN per customer, but:  Creating a VLAN for each customer is expensive:  One subnet per customer is required, gateway maintenance is a nightmare  If a customer grows in size, subnets might have to be changed (for example /30 to /29)  Physical switches can handle a limited amount of VLANs per switch (less than 4000)

52vSphere 4- Mod 4 - Slide Private VLANs: Example with PVLANs  PVLANs  Single Subnet  Gateway in the promisc PVLAN  Each Customer in Isolated PVLAN  Community PVLAN if Customer expands Internet Gateway ISP Community for big customer Isolated for small customers Promisc for the gateway

53vSphere 4- Mod 4 - Slide Private VLANs & vNetwork - 1 vSphere 4 supports PVLANs if you are using vNetwork (DS)  PVLAN in dvSwitch works like PVLAN in Physical Switches:  Primary VLAN is associated with one or more secondary VLANs  Secondary PVLANs have an additional attribute, which is one of the 3:  Promiscuous  All the machines connected to a Promiscuous PVLAN portgroup will be able to send to and receive from any other portgroup that is an Isolated or Community PVLAN associated to the same Primary VLAN  Community  All the machines connected to a Community PVLAN Portgroup can send to and receive from any other machine on the same Community or Promiscuos PVLAN associated with the same primary VLAN  Isolated  Each machine connected to an Isolated PVLAN Portgroup can send to or receive from only machines on the Promiscuous PVLAN associated to the same primary VLAN

54vSphere 4- Mod 4 - Slide Private VLANs & vNetwork - 2  Promiscuous PVLANs will have the same VLAN ID both for Primary and Secondary VLAN  Community and Isolated PVLANs traffic will travel tagged as the associated Secondary PVLAN  Traffic inside PVLANs will not be encapsulated (NO Secondary PVLAN encapsulated inside a Primary PVLAN Packet)  Traffic between VMs on the same PVLAN but on different ESX will go through the Physical Switch  Therefore the Physical Switch must be PVLAN aware and configured appropriately, in order to allow the secondary PVLANs to reach destination. PrimarySecondaryType 55Promiscuous 5155Isolated 517Community

55vSphere 4- Mod 4 - Slide PVLAN and Physical Switch  Because of the PVLAN implementation, packets travel tagged with the secondary ID, and each VM can receive and send to different secondary PVLANs (For example Community and Promiscuous)  Physical Switch can be confused by the fact that each mac address is visible in more than 1 VLAN tag  Physical switch is REQUIRED to be PVLAN aware, and to have the same PVLAN mapping as the vDS  Still, the physical switch must trunk to the ESX, and NOT be in a secondary PVLAN!  PVLAN in the vDS will work even with non PVLAN aware physical switches if these are not discovering mac addresses per VLAN  Because this way the mac address is associated to the single port.

56vSphere 4- Mod 4 - Slide PVLAN and Physical Switch - Example Example: a VM in a Promiscuous PVLAN tries to do an ARP request for a VM in an Isolated PVLAN, on a different ESX, and the Physical Switch is not PVLAN aware. dvSwitch Isolated Promisc Arp request Tag: none PrimarySecondaryType 55Promisc 5155Isolated 517Comm Arp request Tag: 5 PVLAN logic detects that the destination is Isolated so act as if the tag were 155 Arp request Tag: none Arp Reply Tag: none Arp Reply Tag: 155 Arp Reply Tag: 155 Switch ports that see the same mac address through different VLAN tags Arp Reply Tag: none Arp request Tag: 5

57vSphere 4- Mod 4 - Slide Private VLANs – Isolated PrimarySecondaryType 55Promisc 5155Isolated 517Comm VM 1 VM 5 VM 4 155 5 5 17 155 5 5 5 5 Physical dvSwitch VM 3 VM 2 VM 6  VM 1 can’t talk to any VM  in PVLAN 155  in PVLAN 17  VM 1 can talk to VMs  in PVLAN 5 in  Virtual Switches  Physical Switch  VM 1 can talk to VM 2 and 3 only if the physical switch is configured to handle PVLAN 155.  If the Physical switch allows VLAN 155, the isolation might be compromised.

58vSphere 4- Mod 4 - Slide Private VLANs – Community PrimarySecondaryType 55Promisc 5155Isolated 517Comm VM 7 VM 5 VM 4 155 5 5 17 5 5 5 5 Physical dvSwitch VM 3 VM 2 VM 6  VM 7 can’t talk to any VM  in PVLAN 155  VM 7 can talk to VMs  in PVLAN 17  in PVLAN 5 in  Virtual Switches  Physical Switch  VM 7 can talk to VM 2 and 3 only if the physical switch is configured to handle PVLAN 17.  If the Physical switch allows VLAN 17, the isolation might be compromised. 17

59vSphere 4- Mod 4 - Slide Creating Private VLANs  Create the PVLAN table in the dvSwitch  Edit Properties fo the dvSwitch, and select the PVLAN Tab  On the Primary Tab, add the VLAN that will be used outside the PVLAN domain, and select it  On the Secondary Tab, create the PVLANs of the desired type. There can be only one Promiscuous PVLAN and is created automatically for you.  Beware: before deleting any primary/secondary PVLAN, make sure that they are not in use, or the operation will not be performed.

60vSphere 4- Mod 4 - Slide Lab Exercise Lab 2: Using PVLANs

61vSphere 4- Mod 4 - Slide Break

63vSphere 4- Mod 4 - Slide IPv6  IPv6 Concepts  VI4 and IPv6  New TCP/IP Stack  GuestOS and IPv6

64vSphere 4- Mod 4 - Slide IPv6 Concepts - 1  IP Next Generation (v4 was officialised in 1981)  Addresses are 128-bits long  Example: localhost (127.0.0.1) now is:  0000:0000:0000:0000:0000:0000:0000:0001  or ::1 for short (:: means pad with zeros)  fe8x: fe9x: feax: febx: are Link-local addresses (will never be routed), similar to RFC 3927 defined 169.254/16 range  fecx: fedx: feex: fefx: are Site-Local addresses (similar to private IPs in IPv4, such as 10.0.0.0/8). The Site-Local addresses are deprecated by RFC 3879 in production but still valid for labs, for example

65vSphere 4- Mod 4 - Slide IPv6 Concepts - 2  No more IP broadcasts, but advanced multicast  IPv6 has autoconf capabilities, and via multicast can discover routers and receive the configuration from them.  There is also an IPv6 version of DHCP.  DNS can serve IPv6 entries, even over IPv4 connections (or vice versa).  IPv6 can be tunnelled over IPv4, but they can’t be mixed (you can’t access an IPv6 host via an IPv4 network, only across an IPv4 network via tunnels.

66vSphere 4- Mod 4 - Slide IPv6 Concepts: DNS and IPv6  DNS records can be IPv4 (A) or IPv6 (AAAA) $ dig www.ipv6.org AAAA ; > DiG 9.5.0-P2 > www.ipv6.org AAAA ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57681 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.ipv6.org. IN AAAA ;; ANSWER SECTION: www.ipv6.org. 3600 IN CNAME shake.stacken.kth.se. shake.stacken.kth.se. 3600 IN AAAA 2001:6b0:1:ea:202:a5ff:fecd:13a6 ;; AUTHORITY SECTION: stacken.kth.se. 3600 IN NS primary.se. stacken.kth.se. 3600 IN NS secondary.se. stacken.kth.se. 3600 IN NS b.ns.kth.se. stacken.kth.se. 3600 IN NS ns.stacken.kth.se. ;; Query time: 671 msec ;; SERVER: 10.21.64.212#53(10.21.64.212) ;; WHEN: Tue Nov 4 16:21:06 2008 ;; MSG SIZE rcvd: 174

67vSphere 4- Mod 4 - Slide VI4 and IPv6 - 1  ESX 3.5 added support for IPv6 for VMs  NO TSO (TCP Segmentation Offload) with IPv6  VI 4 adds full VI IPv6 support:  Service Console  VMWare Tools (to display the ipv6 address in vCenter)  VMKernel (and therefore VMotion)  IPv6 Storage (software iSCSI and NFS) is experimental  vCenter will display correctly IPv6 addresses for Service Console, VMKernel and VMs as reported by the tools

68vSphere 4- Mod 4 - Slide VI4 and IPv6 - 2  What is still not supported in IPv6 in VI4  VI CLI (previously known as RCLI). Configuring IPv6 parameters works, connecting does not.  CIM  Disabled by default,  Enable via GUI: Host > Configuration > Networking > Properties  Enable for VMKernel (also in VI CLI) esxcfg-vmknic -6 true  Enable for Service Console esxcfg-vswif -6 true  Enabling IPv6 on the ESX does not disable IPv4

69vSphere 4- Mod 4 - Slide VI4 and IPv6 – 3  To edit IPv6 addresses assigned to Service Console or VMKernel adapters,  Go under Host > Configuration > Networking  Select “Virtual Switch” or “Distributed Virtual Switch” as appropriate  Edit the vswif interface IPv6 Address Dialog box: The box where you can enter the IPv6 address is free-form. There is no more the concept of subnet mask, but subnet prefix, which is the number of bits that constitute the prefix (Similar to CIDR notation for IPv4)

70vSphere 4- Mod 4 - Slide Verifying IPv6 Activation – ESX Classic  New VMKernel module: tcpip2  IPv4 module is loaded by default  Based on FreeBSD 6.1  Improved performance and scalability due to locking and threading improvements (more CPUs can be used)  If IPv6 is enabled for the VMKernel, it will look like this:  For the Service Console, lsmod will contain ipv6 if enabled: # vmkload_mod -l Name R/O Addr Length R/W Addr Length ID Loaded tcpip2v6 0x4180225fd000 0xbd000 0x417fe3676f80 0x37000 47 Yes # vmkload_mod -l Name R/O Addr Length R/W Addr Length ID Loaded tcpip2 0x4180157ed000 0x63000 0x417fd687ac80 0x26000 46 Yes # lsmod Module Size Used by ipv6 259232 18 Note: esxcfg-module -l is equivalent to vmkload_mod -l, and is available also in the vi-cli.

71vSphere 4- Mod 4 - Slide Verifying IPv6 Activation - ESXi With ESXi you have two possible ways for checking IPv6 activation:  By logging into the ESX itself, either via the “unsupported” mode, or the unsupported ssh connection, and using the same command as per the ESX Classic:  vmkload_mod -l  By using the vi-cli (also available in the vMA), with the command:  esxcfg-module –l  Since there is no service console here, the lsmod part is not necessary. $ esxcfg-module -l --server esxi.vmware.com --username root --password secret Name ID Loaded tcpip2 45 Yes $ esxcfg-module -l --server esxi.vmware.com --username root --password secret Name ID Loaded tcpip2v6 45 Yes

72vSphere 4- Mod 4 - Slide GuestOS and IPv6  IPv6 support does not require just OS support, applications need to be made compatible as well!  In Linux, IPv6 is supported since 2.4 but the implementation is not fully compliant until 2.6 versions  In Windows,  2003 SP1 and XP SP2 have the infrastructure for IPv6, even though some components of the system and applications are not IPv6-ready. (For 2003 check http://technet.microsoft.com/en- us/library/cc776103.aspx)http://technet.microsoft.com/en- us/library/cc776103.aspx  Vista and 2008 fully support IPv6

73vSphere 4- Mod 4 - Slide Windows and IPv6 addresses: ipv6-literal.net  Most versions of Internet and Windows Explorer do not support literal IPv6 addresses as described in RFC 2732 (because the colon : is a reserved character), so DNS AAAA records must be used (for example for IPv6 web-access to the ESX).  Microsoft has registered ipv6-literal.net as a workaround. The builtin resolver in windows will intercept this domain and resolve it automatically, giving access to the corresponding IPv6 address. For example, the ip address 2001:db8:28:3:f98a:5b31:67b7:67ef would be accessible as 2001-db8-28-3-f98a-5b31-67b7-67ef.ipv6-literal.net

74vSphere 4- Mod 4 - Slide GuestOS and IPv6 – Linux -1  Make sure IPv6 is enabled by checking whether the ipv6 module is loaded using the lsmod command. If it is not, you might have it disabled in /etc/modprobe.conf, with a line such as: alias net-pf-10 off that should be removed (A reboot is required)  In RedHat based distributions, including the Service Console:  /etc/sysconfig/network contains the general information regarding network, including default gateway: NETWORKING=yes HOSTNAME=phobos.vmware.com GATEWAY=10.21.67.254 GATEWAYDEV=eth0 IPV6_AUTOCONF=no NETWORKING_IPV6=yes IPV6_DEFAULTGW=fec0::1 IPV6_DEFAULTDEV=eth0  /etc/sysconfig/network-scripts/ifcfg-eth0 contains the information to configure both IPv4 and IPv6, for example: DEVICE=eth0 ONBOOT=yes BOOTPROTO=static BROADCAST=172.16.5.255 NETMASK=255.255.255.0 DHCPV6C=no IPADDR=172.16.5.99 IPV6ADDR=fec0::d/112 IPV6INIT=yes IPV6_AUTOCONF=no IPV6_DEFAULTGW can have a %eth0 appended at the end, thus overriding IPV6_DEFAULTDEV IPV6_AUTOCONF specifies whether IPV6 advertising should be used to configure NICs IPV6_ADDR contains also the prefix size (similar to IPv4 Netmask, in CIDR format)

75vSphere 4- Mod 4 - Slide GuestOS and IPv6 – Linux - 2  In Debian based distributions, such as Ubuntu:  The file /etc/network/interfaces contains IPv4 and IPv6 for each interface, for example: iface eth0 inet6 static address fec0::d netmask 112 gateway fec0::1  IPv6 commands will generally have a -6 option or a 6 at the end to distinguish from the IPv4 equivalents  ip ip -6 address add fec0::5/112 dev eth0 ip -6 route add default via fec0::1  ping ping6 fec0::1  tracepath tracepath6 fec0::1  traceroute traceroute6 fec0::1  iptables ip6tables

76vSphere 4- Mod 4 - Slide GuestOS and IPv6 – Windows  In Windows, IPv6 is not enabled by default. You will need netsh to configure it, so we will see how to enable IPv6 with it as well.  Enable IPv6 netsh interface ipv6 install  Identify the vNIC name, for example in the Network Connections (where you can also rename it), or with the netsh command netsh interface show interface In this example, we will imagine it is “Local Area Connection” (the default name)  Add an IPv6 address to the selected interface netsh interface ipv6 add address "Local Area Connection" fec0::1  Add a route for the newly added IP address netsh interface ipv6 add route fec0::/112 " Local Area Connection “  netsh has several “dump” commands you can use to get information netsh interface ipv6 dump netsh interface dump

77vSphere 4- Mod 4 - Slide Lab Exercise Lab 4: IPv6

79vSphere 4- Mod 4 - Slide VMXNET Generation 3  New “state of the art” Virtual Network Adapter  Also known as Advanced VMXNET  Based on Enhanced VMXNET introduced in ESX 3.5  Introduces new features:  IEEE 802.1Q VLAN Tagging.  No more need for e1000 in such a case  VLAN Tagging and Tag removal offloading  Only one VLAN per NIC for Windows  TCP Segmentation Offloading for IPv4 and IPv6  TCP and UPD Checksum Offloading for IPv4 and IPv6  MSI (Messaged Signalled Interrupt) and MSI-X support (subject to guest kernel support)  Receive Side Scaling (supported in Windows Vista, 2008 and any other system using NDIS 6.x)

80vSphere 4- Mod 4 - Slide VMXNET Generation3  No Record/Replay support  Supported Guest OSes (both 32-bit and 64-bit versions):  All Windows 2003 variants  Windows 2008 variants  Vista and Vista SP1  Windows XP Professional  RHEL 5.x  SLES 10  Ubuntu 7.04+ 8.04, 8.10  Solaris 10 U4 and later

81vSphere 4- Mod 4 - Slide Lab Exercise Lab 5: VMXNET Generation 3

83vSphere 4- Mod 4 - Slide VMDirectPath I/O - 1  VMDirectPath I/O is a mechanism by which VMs are allowed to directly access a physical device using the native driver in the GuestOS. Each Device will be accessible by one single VM.  Main use cases for this feature are I/O devices that may have high performance/low-latency/CPU efficiency requirements  VMDirectPath I/O (Also known as Fixed Passthrough) is  fully supported for networking I/O devices with the Intel 82598 10 Gigabit Ethernet Controller and Broadcom 57710 10 Gigabit Ethernet Controller  experimentally supported for storage I/O devices with the QLogic QLA25xx 8Gb Fibre Channel and the LSI 3442e-R and 3801e (1068 chip based) 3Gb SAS adapters.

84vSphere 4- Mod 4 - Slide VMDirectPath I/O - 2  Support will be limited to Intel and AMD CPUs with EPT/NPT/RVI and IOMMU (VT-d for Intel) support  The following features are unavailable:  VM can’t be VMotion-ed (Uniform Pass Through will allow VMotion, but it is not available in vSphere 4.0)  Therefore, DRS (limited availability – The virtual machine can be part of a cluster, but cannot migrate across hosts)  Hot add/remove of virtual devices  Suspend and Resume  Record and Replay  Fault Tolerance  High Availability  Memory Overcommitment and Page Sharing

85vSphere 4- Mod 4 - Slide VMDirectPath I/O : Configuring Devices - 1 ESX supports direct PCI device connection for virtual machines running on Intel Weybridge and Stoakley platforms. Each virtual machine can connect to up to two pass-through devices. To configure pass-through devices on an ESX host: 1.Select an ESX host from the inventory panel of the VI Client. 2.On the Configuration tab, click Advanced Settings. The Pass-through Configuration page appears, listing all available pass-through devices. A green icon indicated that a device is enabled and active. An orange icon indicates that the state of the device has changed and the host must be rebooted before the device can be used. 3.Click Edit. 4.Select the devices and click OK.

86vSphere 4- Mod 4 - Slide VMDirectPath I/O : Configuring Devices - 2 Once you click “Edit”, Select the devices you want to use for VMDirectPath I/O and Click Ok. All the dependent devices will be also configured the same way (wether used by the VMKernel or used for VMDirectPath). These devices will be automatically selected for you.

87vSphere 4- Mod 4 - Slide VMDirectPath I/O : Configuring Devices - 3 The configured devices become Orange You will need to reboot for the devices to become ready (Green)

88vSphere 4- Mod 4 - Slide VMDirectPath I/O : Configuring Devices - 4 After the reboot, the devices are green, and ready to be used in a VM Note: the configuration changes will go into /etc/vmware/esx.conf. In the case above, the PCI slot where the device was connected is 00:0b:0, so it will be: /device/000:11.0/owner = "passthru“ (0b is 11 in decimal)

89vSphere 4- Mod 4 - Slide VMDirectPath I/O : Configuring VM - 1  To configure a PCI device on a virtual machine  Select a virtual machine from the inventory panel of the VI Client.  From the Inventory menu, select Virtual Machine > Edit Settings.  Select the Hardware tab  click Add  Select PCI Device  click Next.

90vSphere 4- Mod 4 - Slide VMDirectPath I/O : Configuring VM - 2 From the list, select the pass- through device you wish to assign to the VM. Once the device is assigned, the VM must have a memory reservation for the full configured memory size.

91vSphere 4- Mod 4 - Slide VMDirectPath I/O : Logs - 1 VMDirectPath I/O requires IOMMU feature in the host’s chipset  Check that the vtd module is loaded, using vmkload_mod –l (for ESXi available only on the console) or esxcfg-module –l (Available in VI CLI)  If the module is not loaded, you either do not have the correct/supported chipset, or there was an issue when loading the module. To find more information on what happened, you can either attempt to load the module or check the boot logs:  Check /var/log/boot-logs/sysboot.log ( /var/log/messages for ESXi)  Locate the “ sysboot: iommu... ” section: The log example below was taken from a machine using AMDIommu, the experimental AMD based IOMMU chipset, the module will be vtd at GA time (as with Intel chipsets already): vmkernel: 0:00:00:51.143 cpu2:4875)ForkExec: UWVMKSyscall: ForkExec:2936: /sbin/vmkload_mod vmkernel: 0:00:00:51.178 cpu0:4876)Loading module AMDIommu... vmkernel: 0:00:00:51.205 cpu0:4876)AMDIOMMU: ule:428: Loading AMD IOMMU driver... vmkernel: 0:00:00:51.212 cpu0:4876)AMDIOMMU: ule:438: AMD IOMMU driver version 1.22, built on: Oct 27 2008

92vSphere 4- Mod 4 - Slide VMDirectPath I/O : Logs - 2 vmkernel logs showing the device being assigned to VMDirectPath I/O: vmkernel: 0:00:09:16.642 cpu0:7662)PCI: ChangeDevOwnership:1336: 004:00.0 to passthru vmkernel: 0:00:09:16.649 cpu0:7662)VMK_PCI: vmkpci_PCIDeviceCallback:285: device 004:00.0 event: Device changed ownership: new owner vm vmkernel: 0:00:09:16.661 cpu0:7662)VMK_PCI: vmk_PCIGetDeviceName:625: Device 004:00.0 name: vmnic0 vmkernel: 0:00:09:16.669 cpu0:7662)LinPCI: LinuxPCIDeviceRemoved: Remove 004:00.0 vmnic0 vmkernel: 0:00:09:16.676 cpu0:7662)WARNING: LinPCI: LinuxPCIDeviceRemoved: no driver (or not hotplug compatible) vmkernel: 0:00:09:16.687 cpu0:7662)LinPCI: LinuxPCIDeviceRemoved: Removed device 004:00.0 at event ownership-changed. VM’s vmware.log showing the VM is correctly configured to access the device: vmx| DICT pciPassthru0.present = TRUE vmx| DICT pciPassthru0.deviceId = 1639 vmx| DICT pciPassthru0.vendorId = 14e4 vmx| DICT pciPassthru0.systemId = 4872045d-4d63-ad8e-7fbd-0010182a0a6c vmx| DICT pciPassthru0.id = 04:00.1 […] vmx| Registering device pciPassthru0 (A6F3488)

93vSphere 4- Mod 4 - Slide VMDirectPath I/O : Troubleshooting  If there is no device available  Check the SysBoot logs ( /var/log/boot-logs/sysboot.log ) to see if the IOMMU driver (vtd) loaded successfully  Check if the vmkernel module vtd is loaded, with one of:  vmkload_mod -l  esxcfg-module -l (Available also on VI CLI for ESXi)

94vSphere 4- Mod 4 - Slide Lab Exercise Lab 6: VMDirectPath I/O

96vSphere 4- Mod 4 - Slide Virtual Machine Communication Interface - 1  The Virtual Machine Communication Interface (VMCI) is an infrastructure that provides fast and efficient communication between a virtual machine and the host operating system and between two or more virtual machines on the same host.  The VMCI SDK facilitates development of applications that use the VMCI infrastructure.  Without VMCI, virtual machines communicate with the host using the network layer.  Using the network layer adds overhead to the communication. With VMCI communication overhead is minimal and different tasks that require that communication can be optimized.

97vSphere 4- Mod 4 - Slide Virtual Machine Communication Interface - 2  To enable VMCI on your virtual machine, add the following two lines to the virtual machine configuration file (.vmx file): # The following line is REQUIRED. vmci0.present = "TRUE" # The following line is OPTIONAL. vmci0.id = "num"  num is a positive integer that is unique for each virtual machine on your host. That is, for any virtual machine, you can choose a number (1, 2, 3, etc.) but two virtual machines must not have the same number as their vmci0.id.  You also need the VMCI component of the VMware Tools to be installed inside the VM

98vSphere 4- Mod 4 - Slide VMCI – What is it?  Two types of communication  Datagrams  connectionless – Similar to UDP  Queue Pairs  Connection oriented – Similar to TCP  VMCI provides Socket APIs, which is extremely similar to what is already used for TCP/UDP applications  IP addresses are replaced with VMCI ID numbers  For example, it has been possible to port netperf to use VMCI sockets instead of TCP/UDP

99vSphere 4- Mod 4 - Slide VMCI: Use Case  Application server VM connected to a Database server VM.  Internal network can transmit an average of slightly over 2Gbit/s using vmxnet3  VMCI can go up to nearly 10Gbit/s with 128k sized Queue pairs

100vSphere 4- Mod 4 - Slide Break

102vSphere 4- Mod 4 - Slide Basic Troubleshooting Tips  VMX Changes  net-dvs, a tool to work with dvSwitch (Beware: not supported)  How to find out about dvPortgroups  esxcfg-vswitch  esxcfg-vswif  esxcfg-vmknic  esxcfg-route  Private VLANs  Cisco Nexus 1000V  esxcfg-firewall  Maximums  Known Issues

103vSphere 4- Mod 4 - Slide VMX changes DVS ethernet1.dvs.switchId = "7a f2 34 50 21 55 6c 70-a4 b1 10 f1 3f 9d 2c c1" ethernet1.dvs.portId = "1423" ethernet1.dvs.connectionId = "419447540" ethernet1.dvs.portgroupId = "dvportgroup-302“ VMXNET3 ethernet0.virtualDev = "vmxnet3”

104vSphere 4- Mod 4 - Slide net-dvs output switch 7a f2 34 50 21 55 6c 70-a4 b1 10 f1 3f 9d 2c c1 (etherswitch) Global properties: com.vmware.common.alias = dvSwitch com.vmware.common.uplinkPorts = Uplink1,Uplink2,Uplink3,Uplink4 com.vmware.common.host.uplinkPorts = 5,6,7,8 com.vmware.etherswitch.pvlanMap = (11, 11) - Promiscuous (11, 12) - Community (11, 13) - Isolated (68, 68) - Promiscuous (68, 681) - Isolated (68, 682) - Community com.vmware.etherswitch.mtu = 0xdc. 5. 0. 0 com.vmware.etherswitch.cdp = 0x 0. 1 com.vmware.common.pgmap =vSwitch-DVUplinks-211:dvportgroup- 212,PVLAN-11-I:dvportgroup-239,PVLAN-11-C:dvportgroup-240,VGT:dvportgroup- 241,PVLAN-11-P:dvportgroup-242,VLAN68:dvportgroup-243,PVLAN-68-I:dvportgroup- 244,PVLAN-86-C:dvportgroup-245,PVLAN-68-P:dvportgroup -246,Ghost:dvportgroup-299,dvPortGroup:dvportgroup-300,VLAN64:dvportgroup-302 Host properties: com.vmware.common.host.portset = DvsPortset-1 dvSwitch identifier Uplink Identifiers DVPorts used for Uplink PVLAN map MTU 1500 = 0x5DC (beware of endian-ness) CDP Enabled 0/1 dvPortgroup Map, associating vCenter dvPortgroup names and dvPortgroup labesl dvSwitch Name

105vSphere 4- Mod 4 - Slide net-dvs output switch 7a f2 34 50 21 55 6c 70-a4 b1 10 f1 3f 9d 2c c1 (etherswitch) Global properties: com.vmware.common.alias = dvSwitch com.vmware.common.uplinkPorts = Uplink1,Uplink2,Uplink3,Uplink4 com.vmware.common.host.uplinkPorts = 5,6,7,8 com.vmware.etherswitch.pvlanMap = (11, 11) - Promiscuous (11, 12) - Community (11, 13) - Isolated (68, 68) - Promiscuous (68, 681) - Isolated (68, 682) - Community com.vmware.etherswitch.mtu = 0xdc. 5. 0. 0 com.vmware.etherswitch.cdp = 0x 0. 1 com.vmware.common.pgmap =vSwitch-DVUplinks-211:dvportgroup- 212,PVLAN-11-I:dvportgroup-239,PVLAN-11-C:dvportgroup-240,VGT:dvportgroup- 241,PVLAN-11-P:dvportgroup-242,VLAN68:dvportgroup-243,PVLAN-68-I:dvportgroup- 244,PVLAN-86-C:dvportgroup-245,PVLAN-68-P:dvportgroup -246,Ghost:dvportgroup-299,dvPortGroup:dvportgroup-300,VLAN64:dvportgroup-302 Host properties: com.vmware.common.host.portset = DvsPortset-1 dvSwitch identifier Uplink Identifiers DVPorts used for Uplink PVLAN map MTU 1500 = 0x5DC (beware of endian-ness) CDP Enabled 0/1 dvPortgroup Map, associating vCenter dvPortgroup names and dvPortgroup labesl dvSwitch Name

106vSphere 4- Mod 4 - Slide net-dvs output port 5 com.vmware.common.port.alias = Uplink1 com.vmware.common.port.connectid = 1912494964 com.vmware.common.port.portgroupid = dvportgroup-212 com.vmware.common.port.block = false com.vmware.etherswitch.port.teaming = load balance = source virtual port id link selection: link state up; link speed>=10Mbps; link behavior: notify switch; reverse filter; best effort on failure; shotgun on failure; active: standby: com.vmware.etherswitch.port.security = 0x 1. 0. 0. 0 com.vmware.etherswitch.port.vlan = Guest VLAN tagging ranges: 1-4094 com.vmware.common.port.statistics: pktsInUnicast = 1699111 bytesInUnicast = 865718684 pktsInMulticast = 2204789 bytesInMulticast = 580474616 pktsInBroadcast = 7441346 bytesInBroadcast = 623725320 pktsOutUnicast = 1091384 bytesOutUnicast = 783242007 pktsOutMulticast = 34 bytesOutMulticast = 2744 pktsOutBroadcast = 2069749 bytesOutBroadcast = 179071956 pktsInDropped = 159 pktsOutDropped = 0 pktsInException = 1285 pktsOutException = 0 com.vmware.common.port.volatile.vlan = VLAN 0 ranges: 1-4094 com.vmware.common.port.volatile.status:inUse linkUp portID = 0x2000002

107vSphere 4- Mod 4 - Slide net-dvs output port 519 com.vmware.common.port.alias = com.vmware.common.port.connectid = 1502730467 com.vmware.common.port.portgroupid = dvportgroup-241 com.vmware.common.port.block = false com.vmware.etherswitch.port.teaming = load balance = source virtual port id link selection: link state up; link speed>=10Mbps; link behavior: notify switch; reverse filter; best effort on failure; shotgun on failure; active: Uplink1 Uplink2 Uplink3 Uplink4 standby: com.vmware.etherswitch.port.security = 0x 0. 0. 0. 0 com.vmware.etherswitch.port.vlan = Guest VLAN tagging ranges: 11-14 64-72 com.vmware.common.port.volatile.persist = /vmfs/volumes/f1c540c6-3bd757e8/.dvsData/7a f2 34 50 21 55 6c 70-a4 b1 10 f1 3f 9d 2c c1/519 com.vmware.common.port.volatile.vlan = VLAN 0 ranges: 11-14 64-72 com.vmware.common.port.statistics: pktsInUnicast = 3972 bytesInUnicast = 571094 pktsInMulticast = 27 bytesInMulticast = 2166 pktsInBroadcast = 17 bytesInBroadcast = 2712 pktsOutUnicast = 6499 bytesOutUnicast = 7405784 pktsOutMulticast = 2488 bytesOutMulticast = 664816 pktsOutBroadcast = 1103380 bytesOutBroadcast = 95151238 pktsInDropped = 0 pktsOutDropped = 0 pktsInException = 503 pktsOutException = 0 com.vmware.common.port.volatile.status:inUse linkUp portID = 0x200000d

108vSphere 4- Mod 4 - Slide net-dvs notes  Launch with /usr/lib/vmware/bin/net-dvs  Output collected by vm-support  Not Available for ESXi unless you connect directly via SSH (Not supported)  DVS information is cached in /etc/vmware/dvsdata.db  Binary file  Collected by vm-support  Can be used to produce net-dvs output from any linux host (for example scripts server) with the net-dvs –f [FILE] command  DVS Port information is stored in a shared VMFS volume root, under.dvsData/, net-dvs output will indicate the exact location. This can be useful to quickly locate which ports are still accessing a given DSwitch  References to the DVS are also on /etc/vmware/esx.conf  VMKernel ports

109vSphere 4- Mod 4 - Slide DVS Information in vCenter’s DB DvPortgroups are defined at vCenter level, there is no way to gather information about them from the host. In vCenter’s database, you can find out about dvPortgroups with: select * from VPX_DVPORTGROUP Do not alter the contents of the table in any way! If you remove anything, you might not be able to clean up the “ghost” ports anymore.

110vSphere 4- Mod 4 - Slide esxcfg-vswitch #esxcfg-vswitch -l Switch Name Num Ports Used Ports Configured Ports MTU Uplinks vSwitch0 32 2 32 1500 vmnic0 PortGroup Name VLAN ID Used Ports Uplinks Switch Name Num Ports Used Ports Configured Ports MTU Uplinks vSwitch1 64 3 64 1500 vmnic1 PortGroup Name VLAN ID Used Ports Uplinks VM Network 0 1 vmnic1 DVS Name Num Ports Used Ports Configured Ports Uplinks dvSwitch 64 6 512 vmnic3,vmnic2 DVPort ID In Use Client 5 1 vmnic2 6 1 vmnic3 7 0 8 0 391 0 390 0 1422 1 vmk0 1419 1 vswif1 1423 1 519 0 1420 0 Applies also for ESXi via VI CLI

111vSphere 4- Mod 4 - Slide esxcfg-vswif  Create a new vswif  Same syntax as ESX 3.x esxcfg-vswif -a vswif1 -i 10.21.64.25 -n 255.255.252.0 -p “Service Console”  For DVS you’ll need to specify dvSwitch name and dvPort: esxcfg-vswif -a vswif0 -i 10.21.64.125 -n 255.255.252.0 -P 1421 -V dvSwitch  IPv6 (supposing IPv4 already configured) esxcfg-vswif -i fec0::4/112 vswif1  IPv6 with DHCP (supposing IPv4 already configured) esxcfg-vswif -i DHCP6 vswif1  Output of esxcfg-vswif –l NamePort Group/DVPortIP FamilyIP AddressNetmaskBroadcastEnabledTYPE vswif11419IPv410.21.64.25255.255.252.010.21.67.255trueSTATIC vswif11419IPv6fec0::4112trueSTATIC vswif11419IPv6fe80::250:56ff:fe4f:cba64trueSTATIC Does not apply for ESXi via VI CLI

112vSphere 4- Mod 4 - Slide esxcfg-vmknic  Add a vmknic on a vSwitch esxcfg-vmknic –a -i 10.21.66.25 -n 255.255.252.0 –p “VMKernel Network”  Add a vmknic on a DVS (dvPort 1422) esxcfg-vmknic –a -i 10.21.66.25 -n 255.255.252.0 -s dvSwitch -v 1422  Add an IPv6 address to the newly created vmknic esxcfg-vmknic -i fec0::5/112 -s dvSwitch -v 1422  Add an IPv6 DHCP address to the newly created vmknic esxcfg-vmknic -i DHCP6 -s dvSwitch -v 1422  Output of esxcfg-vmknic -l Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type vmk1 1421 IPv4 10.21.66.25 255.255.252.0 10.21.67.255 00:50:56:75:79:ae 1500 65536 true STATIC vmk1 1421 IPv6 fe80::250:56ff:fe75:79ae 64 00:50:56:75:79:ae 1500 65536 true STATIC vmk1 1421 IPv6 fec0::5 112 00:50:56:75:79:ae 1500 65536 true STATIC Applies also for ESXi via VI CLI

113vSphere 4- Mod 4 - Slide esxcfg-route  Add an IPv6 default gateway (all the other operations are the same as 3.5) esxcfg-route -f V6 -a default fec0::1  Display IPv6 routes for VMKernel esxcfg-route -f V6 -l VMkernel Routes: Network Netmask Gateway default 0 fec0::1 fe80:: 64 Local Subnet fec0:: 112 Local Subnet ff01:: 32 Local Subnet ff02:: 32 Local Subnet Applies also for ESXi via VI CLI

114vSphere 4- Mod 4 - Slide Troubleshooting PVLANs  Key concepts to keep in mind when troubleshooting PVLANs:  Packets in PVLANs travel tagged as if they were in a VLAN with ID as the Secondary ID, there is no encapsulation. This is valid for both virtual and physical switches  Physical switches need to be configured to forward packets in such VLAN IDs between source and destination  Consider PVLAN as a particular case of VST, so:  Physical switch to ESX should be “trunking”  Physical switches should be connected via trunks  Unless they are not PVLAN aware, in which case the trunk should be a PVLAN trunk if you are using Isolated PVLANs  Physical hosts should be connected to a PVLAN port  VTP (Vlan Trunking Protocol) has to be in transparent mode in the physical switch, because PVLANs are defined locally on the single physical switch

115vSphere 4- Mod 4 - Slide Troubleshooting PVLANs  Troubleshooting hints  Make sure that the physical and virtual switch configuration matches:  Physical switch port is trunking for all the primary and secondary PVLAN IDs  Compare the PVLAN maps in physical and virtual switch  In Cisco switches, you can use the commands:  show running-configuration  show interface private-vlan mapping  show interface [interface-id] switchport

116vSphere 4- Mod 4 - Slide PVLAN in Physical Switches  CISCO IOS  Create the primary PVLAN (in this example VLAN 11) (config)# vlan 11 (vlan-config)# private-vlan primary  Similarly, create the secondary PVLAN (ex. VLAN 13, Isolated, 12, Community) (config)# vlan 13 (vlan-config)# private-vlan isolated (config)# vlan 12 (vlan-config)# private-vlan community  Bind Primary and Secondary PVLANs (config)# vlan 11 (vlan-config)# private-vlan association 12,13  Bind switch ports to the PVLANs (1/10 Isolated, 1/11 Community and 1/1 promisc): (config)# interface Fastethernet 1/10 (config-if)# switchport mode private-vlan host (config-if)# switchport private-vlan host-association 11 12 (config)# interface Fastethernet 1/11 (config-if)# switchport mode private-vlan host (config-if)# switchport private-vlan host-association 11 13 (config)# interface Fastethernet 1/1 (config-if)# switchport mode private-vlan promiscuous (config-if)# switchport private-vlan mapping 11 12,13

117vSphere 4- Mod 4 - Slide Cisco Nexus 1000 SVS—Troubleshooting If traffic doesn’t work, try the following: On the CP, check that the DP module is visible. # show module (Should show the UUID of the ESX 4.0 host.) # show server_info (Should show the hostname of the ESX 4.0 host). Ensure that your uplinkportprofile1 includes the VLAN that is configured on your VMs’ port profile. # show port-profile name uplinkportprofile1 To isolate how far the traffic gets, do tcpdump inside the VMs, ‘ cb print ingress ’ on the DP, and ‘ debug ip packets detail ’ on the upstream Cisco switch

118vSphere 4- Mod 4 - Slide esxcfg-firewall - 1 New feature (soon available also in 3.5) of filtering connections per host/port, with the option: --ipruleAdd As you might already know from ESX 3.x, list firewall rules with esxcfg- firewall –q and be careful, because -l will reload the firewall rules instead, overwriting the possible root cause of your investigation. There is no mechanism to temporarily stop the firewall like in ESX 3.5 using service firewall stop|start because the service firewall stop will not do anything but print the following: “firewall can't be stopped. To disable the firewall run, esxcfg-firewall --allowIncoming –allowOutgoing ”

119vSphere 4- Mod 4 - Slide esxcfg-firewall - 2 But keep in mind that:  esxcfg-firewall --allowIncoming –allowOutgoing modifies the firewall configuration, so to return to the previous configuration you need to use esxcfg-firewall --blockIncoming --blockOutgoing, because esxcfg-firewall -l won’t.  If you use allowIncoming and allowOutgoing, previously defined IP Rules will still be applied

120vSphere 4- Mod 4 - Slide esxcfg-firewall - 3 So what can we do for temporarily disabling the firewall for troubleshooting?  Remember to save the actual configuration before doing anything else! Otherwise you might not be able to identify the root cause.  Save the output of iptables -L or better of iptables-save to a file.  You can use iptables -F or iptables-save, and then reload the firewall with esxcfg-firewall –l, when the troubleshooting is done.

121vSphere 4- Mod 4 - Slide esxcfg-firewall - 3  With iptables –F you’ll flush all the rules. Keep in mind that usually the default policy is to drop connections, and the rules are allowing you in. This means that before flushing the rules, you should make sure that at least the INPUT chain has default set to ALLOW, with iptables –P INPUT ALLOW, or you’ll lock yourself out.  With iptables-save>file you can save to a file the rules, then edit the files so that you remove all the rules and the chains, edit the policy to be ALLOW, review what you’ve done, and then apply your changes with iptables- restore<file. For example: *filter :INPUT ACCEPT [4495370:1545008248] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [3029364:951838897] COMMIT

122vSphere 4- Mod 4 - Slide Maximums - 1 Maximums VI3 Standard Switch vNetwork Standard Switch vNetwork Distributed Switch Switches per VC 4096 16 Switches per ESX host 248 16 Port groups per ESX host 512 Port groups per switch 512 Ports per host 4096 Uplinks per host 32 Ports per switch 1016 8000 Uplinks per virtual switch 32 Max number of hosts per switch NA 300 VLANs/Private VLANsLimited by Max # of Portgroups

123vSphere 4- Mod 4 - Slide Maximums - 2 Physical NIC Type Max Number of ports per ESX Host tg3 (Broadcom 1GigE)32 bnx2 (Broadcom 1GigE)16 e1000e (Intel 1GigE PCIe)32* s2io (Neterion 10GigE)4 e1000 (Intel PCIx)32* nx_nic (Netxen 10GigE)4 Igb (Intel Zoar)16 bnx2x (10GigE Broadcom)4 igbe (Intel 10GigE Oplin)4 (*) If the Hardware supports them. Hardware Version Max Virtual NICs 44 710 Type Max Virtual Adapters VMKernel32 Service Console 32

124vSphere 4- Mod 4 - Slide Known Issues  VMDirectPathI/O requires GuestOS support. For example, Oplin NIC in passthrough mode does not perform well with SLES10 in VGT mode.  IPv6 default gateway might not be effective: you might want to use static routes for the specific destination  Removing IPv4 default gateway might cause IPv6 default gateway to fail, especially if the gateway does not do IPv6 advertisment.  Configuring a NIC with neither ant static IP (v4 or v6) nor any dynamic configuration (no DHCP not IPv6 autoconf), after reboot you will have to remove it and add it again to be able to reconfigure it.

125vSphere 4- Mod 4 - Slide Recovering  Find out the uplink port for the NIC you want to use esxcfg-vswitch -l  Remove the Uplink from the DVS esxcfg-vswitch -Q vmnic1 -V 5 dvSwitch  Create a new LifeSaver Standard vSwitch esxcfg-vswitch –a LifeSaver  Give the LifeSaver Standard vSwitch a portgroup and the uplink esxcfg-vswitch –A SOSC LifeSaver esxcfg-vswitch –L vmnic1 LifeSaver  Move the vSwif 0 to the LifeSaver vSwtich esxcfg-vswif –d vswif0 esxcfg-vswif –a –i DHCP –p SOSC vswif0  Use vCenter to fix all via GUI and then cleanup, otherwise: esxcfg-vswitch -P vmnic1 -V 5 dvSwitch

Questions?

127vSphere 4- Mod 4 - Slide Discarded slides

128vSphere 4- Mod 4 - Slide Private VLANs PrimarySecondaryType 55Promisc 5155Isolated 517Comm VM 1 VM 2 VM 3 VM 4 VM 5 VM 6 Physical VLAN 5 155 5 5 VM 7 VM 8 Physical Switch needs to be PVLAN Aware and allow Secondary PVLANs 155 17 dvSwitch Physical

129vSphere 4- Mod 4 - Slide NetQueue NetQueue and Networking Performance NetQueue in ESX takes advantage of the capability of some network adapters to deliver network traffic to the system in multiple receive queues that can be processed separately. This allows processing to be scaled to multiple CPUs, improving receive size networking performance. To enable NetQueue on an ESX host: 1.Log in to the VI Client and select the host from the inventory panel. 2.Click the Configuration tab, and click Advanced Settings. 3.Select VMkernel. 4.Select VMkernel.Boot.netNetQueueEnable and click OK. 5.Log in directly to your host’s console to configure your NIC driver to use NetNetQueue. If you are using the s2io NIC driver, use the esxcfg-module -s "intr_type=2 rx_ring_num=8“ s2io command to set the appropriate parameters on the s2io module. If you are using the ixgbe NIC driver, use the esxcfg-module –s “InterruptType=2 MQ=1 VMDQ=16”ixgbe command to set the appropriate parameters on the ixgbe module. For third party vendor for appropriate configurations. 6.Reboot the ESX host

130vSphere 4- Mod 4 - Slide esxcfg-vmknic with IPv6 Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type vmk0 1422 IPv4 10.21.66.25 255.255.252.0 10.21.67.255 00:50:56:76:06:ff 1500 65536 true STATIC vmk0 1422 IPv6 fe80::250:56ff:fe76:6ff 64 00:50:56:76:06:ff 1500 65536 true STATIC vmk0 1422 IPv6 fec0::5 112 00:50:56:76:06:ff 1500 65536 true STATIC

131vSphere 4- Mod 4 - Slide esxcfg-vswif with IPv6 Name Port Group/DVPort IP Family IP Address Netmask Broadcast Enabled TYPE vswif0 Service Console IPv4 172.16.5.99 255.255.255.0 172.16.5.255 true STATIC vswif0 Service Console IPv6 fec0::d 112 true STATIC vswif0 Service Console IPv6 fe80::250:56ff:fe4b:f6e0 64 true STATIC vswif1 1419 IPv4 10.21.64.25 255.255.252.0 10.21.67.255 true STATIC vswif1 1419 IPv6 fec0::4 112 true STATIC vswif1 1419 IPv6 fe80::250:56ff:fe4f:cba 64 true STATIC

132vSphere 4- Mod 4 - Slide Distributed Virtual Networks (vNetwork) vCenter ESX 4 Virtual Center 2 ESX 3 vswifs vSwitches Portgroups VLANs vmnics vswifs vSwitches Portgroups VLANs vmnics VIClient 2 vswifs vSwitches Portgroups VLANs vmnics vswifs vSwitches Portgroups VLANs vmnics vNetwork Distributed vSwitch VIClient 4

133vSphere 4- Mod 4 - Slide VMDirectPath I/O  Fixed passthrough is a mechanism by which VMs are allowed to directly access a physical device using the native driver in the VM  Main use cases for this feature are I/O devices that may have high performance/low-latency/CPU efficiency requirements  IOMMU converts guest physical addresses that appear in the PCI DMA requests to host physical addresses, thereby by-passing vmkernel  Vmkernel has to be made aware of the existence of these pass-through devices. This will be made possible by changing VC to support these type of devices  VMs using FPT feature will not have VMotion, Suspend/Resume, Snapshots and memory over commitment  FPT support will be limited to Intel and AMD CPUs with EPT/NPT support

134vSphere 4- Mod 4 - Slide VMDirectPath  VMDirectPath I/O device access is primarily targeted to those applications that can benefit from direct access by the guest operating system to the I/O devices.  Other virtualization features, such as VMotion, hardware independence and sharing of physical I/O devices will not be available to the virtual machines using this feature.  VMDirectPath I/O for networking I/O devices is fully supported with the Intel 82598 10 Gigabit Ethernet Controller and experimentally supported for storage I/O devices with the QLogic QLA25xx 8Gb Fibre Channel and the LSI 3442e-R and 3801e (1068 chip based) 3Gb SAS adapters

135vSphere 4- Mod 4 - Slide VMDirectPath  Fixed Passthru (FPT):  Feature: Allows assignment of physical PCI device(s) to VMs  Limited to only PCI End Points (non Bridge devices)  For K/L we may limit this to network and storage class devices and have ability to add device classes to a white-list for subsequent releases.  Features that needs to be turned off or not available for this VM  VM can't VMotion (VC and vmx code will have logic to disallow VMotion)  snapshot/suspend/resume in UI turned off  Page sharing turned off (needs.vmx options)  ESX K/L will have three distinct ownerships maintained for PCI devices  Host(COS), VMkernel/Vmklinux, PCIPassthru  FPT can use both non-SR IOV and SR IOV device  Assignment of a device requires PCI location as a primary identifier (Bus/Dev/Fun)  UI will display list of passthru devices that are assignable to a VM  SR IOV devices will show up in the list of available passthru devices. For SR IOV device, PF/VF0 is always owned by vmkernel/vmklinux. Only the VFs (1 through n-1) can be assigned to VMs. Configuring a VM to use SR IOV device for FPT does not have any impact on the ownership of the device nor does it require reboot of the ESX host.  Non-SR IOV device by default is assigned to vmkernel/vmklinux. In such cases, Host management needs to be done first if user wants to use it for a VM. This requires changing the ownership from vmkernel to PCIPassthru for that device. Movement of device ownership between vmkernel to PCIPassthru (VM) requires reboot of host. This is to simplify the configuration as checking for storage volumes gets tricky for dynamic movement of HBAs.  VMkernel will have a built in PCI Manager to configure and bring up SR IOV devices. In addition vmkernel maintains the ownership of PCI devices and manages any VFs if the underlying device is a SR IOV device. It will export APIs to that others parts of the system like FPT, vmxnet3/UPT. In addition vmkctl needs this during host management for FPT.  vmk_IsPCIDevVFCapable  vmk_GetDeviceOwnership  vmk_GetMaxVFsForPF  vmk_GetNumVFsForPF  vmk_GetActiveVFsForPF  vmk_AllocateVFFromPF  vmk_DeallocateVF .vmx file for the VM will have 3 identifiers for any FPT entry: VMK-UUID, PCI information, PCI location (BDF). This will be sufficient to trigger proper checks on cold VM migration to warn users since PCI location on one host may not be same on another host.

136vSphere 4- Mod 4 - Slide VMDirectPath  Vmxnet3/UPT  Feature: VMotionable Passthru with single driver inside the guest  UI will be somewhat similar to Enhanced vmxnet2 – it is a new vNIC and needs to be shown only for selected guest OSes.  For Vmxnet3 vNIC, there will be a radio button for requesting UPT if available (Note that UPT is a best effort). There will be a LED to indicate if a vNIC is in passthru mode or not (guest driver is completely unaware of this).  There is one master control per host to turn off UPT. This is mostly for trouble shooting if UPT has some trouble  UPT requires SR IOV device, so whatever we do for FPT/SR-IOV from configuration and management within vmkernel applies here. But VC/UI will not need to know anything.  UPT helper will interface with vmkernel PCI Manager for allocation of VFs and deallocation of VFs.  Logic for figuring out which VF is used for a given vNIC, when to trigger switching out of UPT to vNIC/vSwitch mode will be kept outside of PCI Manager and the NIC teaming code. There is will be some callbacks for getting link state notifications to enable dynamic assignment and movements between UPT and vNIC modes.  Reference to DVSUPT: https://wiki.eng.vmware.com/DVSUPcom/DVSUPT

137vSphere 4- Mod 4 - Slide net-dvs notes  DVS information is cached in /etc/vmware/dvsdata.db  Binary file  Collected by vm-support  Might be useful for ENG  References to the DVS are also on /etc/vmware/esx.conf  VMKernel ports  net-dvs  Launch with /usr/lib/vmware/bin/net-dvs  Re-initialise the database with /usr/lib/vmware/bin/net-dvs -i  VSPAN (Not Supported): net-dvs -m "sid;dname;snaplen;[oiveld];encapvlan;wildcardsIn,wildcardsOut;dstPort1,dstPort2,...;srcInPort1,srcInport2,... ;srcOutPort1,srcOutPort2,...;:sid2;dname2...“  Define PVLANs: net-dvs -V "primaryVID,secondaryVID,i|c|p;primaryVID,secondaryVID,i|c|p..."

138vSphere 4- Mod 4 - Slide Logs  ESX Classic  /var/log/boot- logs/sysboot.log  /var/log/weasel.log  ESXi  /var/log/messages

VSphere 4.0 Module 4 – Networking Emiliano Turra Product Support Engineering VMware Confidential Rev. G.

Similar presentations

Presentation on theme: "VSphere 4.0 Module 4 – Networking Emiliano Turra Product Support Engineering VMware Confidential Rev. G."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

VSphere 4.0 Module 4 – Networking Emiliano Turra Product Support Engineering VMware Confidential Rev. G.

Similar presentations

Presentation on theme: "VSphere 4.0 Module 4 – Networking Emiliano Turra Product Support Engineering VMware Confidential Rev. G."— Presentation transcript:

Similar presentations

About project

Feedback