Presentation on theme: "Cisco Unified Computing System Nuts & Bolts"— Presentation transcript:
1Cisco Unified Computing System Nuts & Bolts Sean HicksGeneral DataTech, L.P.Managing Principal - Compute & Storage
2What the heck is a UCS?Cisco UCS is a next-generation compute platform designed to address the following concerns:Data Center SprawlPoints of ManagementCablingPower & CoolingVirtualizationChanging Admin BoundariesResource BottlenecksSoftware-based L2 OverheadReliability & ScalabilityHigh AvailabilityMinimal Effort Rapid DeploymentStateless Hardware
3Fabric Interconnects – G1 Cisco UCS 6120XP1 RU, Redundant Power, Front-to-Back Air Flow20 x 10 Gbps Ethernet, L1, L2, Mgmt0, Mgmt1 (unused) & Console1 Expansion BayCisco UCS 6140XP2 RU, Redundant Power, Front-to-Back Air Flow40 x 10 Gbps Ethernet, L1, L2, Mgmt0, Mgmt1 (unused) & Console2 Expansion BaysExpansion ModulesN10-E0080: 8 x 4 Gbps FCN10-E0060: 6 x 8 Gbps FCN10-E0440: 4 x 10 Gbps Ethernet + 4 x 4 Gbps FCN10-E0600: 6 x 10 Gbps Ethernet
4Fabric Interconnects – G2 Cisco UCS 6248UP1 RU, Redundant Power, Front-to-Back Air Flow32 x Unified Fabric Ports, L1, L2, Mgmt0 & Console1 Expansion BayCisco UCS 6296UP2 RU, Redundant Power, Front-to-Back Air Flow48 x Unified Fabric Ports, L1, L2, Mgmt0 & Console3 Expansion BaysExpansion ModulesUCS E16UP: 16 x Unified Fabric Ports
5End-Host Virtualization In its default configuration (End-Host mode), a Fabric Interconnect appears to the network as a server with a bunch of NICs, much like a Hypervisor. It is not a true Ethernet switch!Does not participate in Spanning Tree elections.Allows for rapid L2 link establishment and multiple active uplinks.Does not learn northbound MAC addresses.Makes intelligent forwarding decisions based on learned (internal server) versus unlearned (external network) MAC addresses.Traversing frames are forwarded/dropped based on pinning traffic to interfaces.Déjà vu check, reverse path forwarding check, inter-uplink port blocking and bcast/mcast link election defeat the purpose of Spanning Tree between the FI and the northbound switch.Server-to-server in same VLAN is the only locally switched traffic.Ethernet SwitchVisio StencilFabric InterconnectVisio StencilYou see, Kyle?! Totally different!!!
6Historical Need For Spanning Tree BroadcastBroadcast frame enters Switch 1.Switch 1 learns Src MAC, floods to every port except the originator.Switches 2 and 3 do the same, creating a loop. Broadcast storm begins.Trogdor strikes again! And again, and again, and again…
7Spanning Tree Behavior BroadcastSwitches hold Spanning Tree election, Switch 1 blocks port to Switch 3.Broadcast frame enters Switch 1. Learns Src MAC then floods.Switch 2 does the same. No loop, but… Trogdor strikes again!Half of Switch 3’s cabled bandwidth is unused.Redundant link convergence is very slow.
8EHV Bcast/Mcast Behavior BroadcastBcast/McastLAN UplinkBroadcast frame enters Switch 1. Learns Src MAC and floods.Broadcast frame enters Switch 3. Learns Src MAC and floods.FI sees unlearned Bcast on Bcast/Mcast LAN Uplink, forwards to Server Ports.FI sees unlearned Bcast on any other LAN Uplink, drops the frame.All LAN Uplinks on FI are active and utilized for pinned MAC addresses.MAC’s dynamically and rapidly re-pin for LAN Uplink state change.
9EHV Bcast/Mcast Behavior BroadcastBcast/McastLAN UplinkLearned Bcast enters FI on Server Port. FI forwards out pinned LAN Uplink.Broadcast frame enters Switch 1. Switch 1 learns Src MAC and floods.Broadcast frame enters Switch 3. Switch 3 learns Src MAC and floods.If Bcast gets back to FI it will drop since Src isn’t pinned to that LAN Uplink.All LAN Uplinks on FI are still active and utilized for pinned MAC addresses.MAC’s still dynamically and rapidly re-pin for LAN Uplink state change.
10N-Port Virtualization In its default configuration (NPV mode), a Fabric Interconnect is also not a true Fibre Channel switch!Does not have a Fibre Channel domain ID.Conserves FC domain ID’s, but unable to assign FCID’s to connected N-ports.Keeps SAN Uplinks from becoming E-ports. No ISL means no FSPF negotiation.Acts as a “N-port Proxy” for connected N-ports.Requires “N-Port ID Virtualization” (NPIV) be enabled on an upstream Fibre Channel switch.Do not confuse the two!A Cisco MDS or Nexus switch can be enabled for NPV mode, too, but this setting erases the config and reboots the switch. An additional FC switch is then needed farther upstream to be enabled for NPIV.Cisco MDS SwitchVisio StencilFabric InterconnectVisio StencilYou see, Kyle?! Still different!!!
11Historical Fibre Channel N-portF-portFLOGIFCID 0x010001WWPNADomain ID 0x01Server with WWPNA needs an FCID so its HBA can begin forwarding FC frames. Sends FLOGI to FC switch with FC Domain ID 0x01.FC switch responds with an FCID built off its FC Domain ID. The server’s HBA is now logged into the fabric and can begin forwarding FC frames.The same process happens between the FC switch and the FC storage array. It is often likened to DHCP in IP networks.In traditional FC design, there are dual fabrics such as this one for path redundancy.Storage Array
12Blade Server Considerations FLOGIFLOGIFLOGIFLOGIWTF?! LOL!!!WWPNAWWPNBDomain ID 0x01WWPNCWWPNDServers need FCIDs so HBAs can begin forwarding FC frames.The problem is there is traditionally a 1:1 relationship between N-port and F-port.Storage Array
13NPV/NPIV Behavior NPV NP-port NPIV FLOGI FCID 0x010001 FLOGI 0x010002 FDISCDomain ID 0x01FCID 0x010002Storage ArrayOnly one FLOGI is received on the switchport for the FI’s WWPN.FI Looks like F-port to Blade’s N-port, but acts as an N-port Proxy.FLOGI received by FI from Blade, FI kicks off FDISC to switch.Switch issues FCID for the FDISC, FI forwards on to Blade.
14IO Modules (Fabric Extenders) Cisco UCS 2104XP (G1)External: 4 x 10 Gbps EthernetInternal: 8 x 10 Gbps Ethernet1 x 10 Gbps to each blade slot2:1 Oversubscription, Static PinningCisco UCS 2204XP (G2)Internal: 16 x 10 Gbps Ethernet2 x 10 Gbps to each blade slot2:1 Oversubscription, Static Pinning or Port ChannelCisco UCS 2208XP (G2)External: 8 x 10 Gbps EthernetInternal: 32 x 10 Gbps Ethernet4 x 10 Gbps to each blade slot1:1 Oversubscription, Static Pinning or Port Channel
15What is a Fabric Extender Extends the presence of the Fabric Interconnect into the blade server chassis.Works exactly like Nexus Fabric Extenders. In fact… C-series integration!Unable to make any L2 decisions on its own. Must rely on parent switch or FI.Looks exactly like line cards in a modular switch. Single point of management!
16VIFs and VNTagsEach upstream port is a FEX port (external ports on IOM).Each downstream port is a Virtual Interface or VIF (internal ports on IOM).VIFs are identified at the parent switch using VNTags (Virtual Networking Tags).A VNTag is sort of like a virtual cable inside an actual cable.vEthVNTag 1FI PortVNTag 2VNTag 3vFC
17Fibre Channel Over Ethernet Creates a virtual SAN fabric over physical LAN links.Encapsulates FC frames inside Ethernet frames.Ethertype is FCoE (0x8906)Uses Priority Flow Control (PFC) to match the lossless behavior of Fibre Channel.Uses Jumbo Frames. SCSI data cannot be fragmented. CoS is on by default in UCS.Requires FCoE VLAN ID’s. Cannot overlap with existing VLANs.FI requires VSANs to pin FC traffic to SAN Uplinks. Needs a VLAN ID to reference for each VSAN.Allows cabling consolidation between FI’s and chassis.Inherits SAN fabric separation, which is why each IOM is connected to only one FI.Fibre Channel PayloadMaximum 2112 BytesEthernetHeaderFCoE HeaderFC HeaderCRCEOFFCS
18UCS 5108 Blade Server Chassis 6 RU, Front-to-Back Air Flow2 x IO Module Slots8 x Half-width Blade SlotsMaximum 8 Half-width BladesMaximum 4 Full-width BladesCan combine Half- and Full-width Blades4 x 2500 Watt Power Supplies2 Required for Full Population (Non-redundant)3 for N+1 Redundancy4 for Full Grid RedundancyUp to 20 per FI ClusterUp to 160 Half-width Blade ServersCurrent Supported Limit, Not Hard Limit“Rolled Steel”
19UCS B-Series Blade Servers Sub-light Speed: UCS B22 M32 x Intel Xeon E (16 Cores Max)192 GB MaximumHalf-widthLight Speed: UCS B200 M32 x Intel Xeon E (16 Cores Max)768 GB MaximumRidiculous Speed: UCS B230 M22 x Intel Xeon E (20 Cores Max)512 GB MaximumLudicrous Speed: UCS B420 M34 x Intel Xeon E (32 Cores Max)1.5 TB MaximumFull-widthPlaid: UCS B440 M24 x Intel Xeon E (40 Cores Max)1 TB Maximum
20Stateless HardwareA Service Profile is a logical representation of a server.Contains unique identifiers (UUID, MACs & WWNs). Makes them portable!Also contains policies related to network and storage connectivity, management.Can be rapidly deployed from a Service Profile Template.
21Virtual Interface Cards (VICs) M81KR “Palo” (G1)2 x 10 Gbps, 1 to each IOMMaximum 128 Virtual Interfaces*Mezzanine Adaptor (PCIe x16)VIC 1240 (G2)4 x 10 Gbps, 2 to each IOMMaximum 256 Virtual Interfaces*mLOM (M3 blades only)Expansion ModuleVIC 1280 (G2)8 x 10 Gbps, 4 to each IOMArchitectural limit. Real limit based on the number of server ports cabled between FI and IOM.
22Single Root IO Virtualization SR-IOV allows a PCIe device to appear as multiple devices of the same type.Made possible by distinction between Physical Functions and Virtual Functions.Physical Functions (PF’s) can control the device and move data in or out of the device.Virtual Functions (VF’s) can only move data in or out of the device.Requires Hardware, Firmware and Software (OS) support to implement.Cisco VICs have the intelligence to handle all of it on board.Transparent to the OS (no support needed).Multiple adaptors for different CoS/QoS markings.Allows “Shadow NIC” capability (aka Fabric Failover).Allows for VM-FEX capability…Replaces Hypervisor soft-switch, off-loading L2 overhead to real, dedicated hardware.Returns control of the access layer switching environment to the network administrators.Does not break vMotion!
24Single Point of Management The pair of Fabric Interconnects represents a single UCS Manager domain. Every connected physical blade server chassis is actually a module in a single logical blade server chassis. Every component in this larger, logical chassis is configured, operated and monitored via the pair of Fabric Interconnects across their Mgmt0 ports.Mgmt0 ports and Cluster IP address must be in the same broadcast domain.
25High AvailabilityThe Fabric Interconnects negotiate high availability for the cluster IP address of the UCS Manager domain through a primary-subordinate election over their L1 and L2 ports (known as cluster links) and through the connected chassis (failsafe mechanism to prevent management split brain in the event of cluster links loss). The dual fabric, active-active design provides LAN and SAN connectivity to every blade in every chassis even if one of the Fabric Interconnects malfunctions.HA for UCS Manager is unavailable until the first connected chassis is acknowledged.