1. 5.3 Ethernet connectivity and options

2. Unit objectives After completing this unit, you should be able to: List the BladeCenter components required for Ethernet connectivity Name the management components supporting Ethernet switch modules Select the Ethernet I/O switch module to meet a specific requirement List the components required to provide additional Ethernet connectivity to a blade server Identify the method used to provide blade server Ethernet connection redundancy 2

3. BladeCenter chassis I/O overview Internal chassis traffic flow Switch I/O module management connectivity BNT (Nortel) Ethernet switch I/O modules Server connectivity module Pass-thru module High speed I/O modules overview Ethernet expansion cards 3

4. BladeCenter chassis overview: Switching path BladeCenter Chassis Physical Components 4

5. Chassis overview: Gigabit Ethernet routing Midplane (Upper Section) Midplane (Lower Section) SW Module 1 1………..14 SW Module 3 1………..14 SW Module 2 1………..14 SW Module 4 Network Interface Expansion Card SERDES Ethernet Line 1 Line 3 Line 4 Line 2 Processor blade #1 5

6. Chassis overview: Ethernet port assignments (1 of 2) 1 B S 1 2 1 B S 1 4 2 1 B S 2 2 1 B S 3 2 1 B S 4 2 1 B S 5 2 1 B S 6 2 1 B S 7 2 1 B S 8 2 1 B S 9 2 1 B S 1 0 2 1 B S 1 1 2 1 B S 1 2 2 1 B S 1 3 2 Bottom ESM(Bay2) MM1 2 Top ESM(Bay1) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mgt 1 Mgt 2 ESM External uplinks ESM External uplinks MM1 Uplink MM2 Uplink BladeCenter Chassis Eth1 1 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mgt 1 Mgt 2 6

7. Chassis overview: Ethernet port assignments (2 of 2) 1 B S 1 2 1 B S 2 2 1 B S 3 2 1 B S 4 2 1 B S 5 2 1 B S 6 2 MM1 Top ESM(Bay1) 1 2 3 4 5 6 8 9 10 11 12 13 Mgt 1 ESM External uplinks MM1 Uplink BladeCenter S Chassis Eth1 0 7

8. Internal chassis traffic flow BladeCenter chassis Ethernet overview Internal chassis traffic flow Switch I/O module management connectivity BNT (Nortel) Ethernet switch I/O modules Server connectivity module Pass-Thru module High speed I/O modules overview Ethernet expansion cards 8

9. Ethernet connectivity: Layer 2 traffic flow 14 Internal Blade ports Management Module Ports CPU Hidden internal filter that blocks traffic flow between the External ports and Mgmt Mod ports. Prevents STP loops between ESM uplinks and MM uplinks (as well as between ESMs) Ethernet Switch Module (ESM) MM1 MM2 BladeCenter Chassis Upstream connections Red = Ethernet External ports 9

10. Traffic flow: NIC teaming and trunk failover 1 2 3 4 5 6 ESM NIC 1 NIC 2 XX Teamed Active/Standby VLAN X x Blade Server Logical NIC Interface If failure anywhere on the link toward the upstream switch, the NIC on blade server does not know about the failure and may continue to send traffic toward the top switch, which will discard the traffic. The Trunk Failover feature addresses this issue. If the switch fails in such a way that the link toward the Blade server goes down, or NIC fails, Blade server can sense this and redirect traffic out the other NIC toward the bottom switch. NIC Teaming can take care of this without the need for trunk failover. 10

11. Traffic flow: Trunk failover example 1 If Po1 goes down, trunk failover takes over and shuts down internal downstream defined port(s) –This alerts NIC Teaming to an upstream failure at which point Teaming switches to the other path out of the server –This example shows a single VLAN to Teamed NICs, It is possible to also carry multiple VLANs to the Teamed NICs 1 2 3 4 5 6 ESM NIC 1 NIC 2 Teamed Active/Standby VLAN X Blade Server Logical NIC Interface VLAN X Must be carried between ESMs via an external path Link state group1 Upstream Link state group Downstream VLAN X Must be carried on all uplinks from ESM NIC Teaming Po1 Po2 11

12. Traffic flow: Trunk failover example 2 If only one port channel goes down, spanning tree allows backup path to other port channel If both port channels go down, trunk failover takes over and shuts down internal downstream defined port(s) –This alerts NIC Teaming to an upstream failure at which point it switches to the other CIGESM –This example shows a single VLAN to Teamed NICs, It is possible to also carry multiple VLANs to the Teamed NICs 1 2 3 4 5 6 ESM NIC 1 NIC 2 Teamed Active/Standby VLAN X Blade Server Logical NIC Interface Link state group1 Downstream VLAN X Must be carried on all uplinks from ESM NIC Teaming Po2Po1 Po2 Link state group1 Upstream Po1Po2 Po3 12

13. Switch I/O module management connectivity BladeCenter chassis Ethernet overview Internal chassis traffic flow Switch I/O module management connectivity BNT (Nortel) Ethernet switch I/O modules Server connectivity module Pass-Thru module High speed I/O modules overview Ethernet expansion cards 13

14. Management connectivity for the ESM Management Module Ethernet Switch Module Management Network Routed Production Network 2 13 Web Interface 1B 1A 2 34 4 Legend Ethernet I2C path Serial Serial path 5 Management Workstation Blade Server I2C interface Ethernet path 14

15. BNT (Nortel) Ethernet switch I/O modules BladeCenter chassis Ethernet overview Internal chassis traffic flow Switch I/O module management connectivity BNT (Nortel) Ethernet switch I/O modules Server connectivity module Pass-Thru module High speed I/O modules overview Ethernet expansion cards 15

16. BNT GbESM switch I/O modules: Layer 2-3 Layer 2/3 copper GbESM –14 internal 1 Gb FDX connections –6 external 1 Gb FDX connections –2 10/100 Mb FDX management ports Layer 2/3 Fibre GbESM –14 internal 1 Gb FDX connections –6 external 1 Gb FDX fibre (SX) connections –2 10/100 FDX management ports BNT Copper and Fibre Layer 2-3 Switch I/O Modules 16

17. BNT L2/3 GbESM: Simplified block diagram Gigabit Ethernet Accelerator Application Switch connector Management Processor Memory Control Data 6 external 10/100/1000 Mbps Copper Ports (Auto-negotiation enabled) Or 6 external 10/100/1000 Mbps Copper Ports (Auto-negotiation enabled) And 14 internal 1000 Mbps server links 2 internal 100 Mbps management links 17

18. BNT L2/3 GbESM: Network interfaces Blade 1 Blade 2 Blade 3 Blade 12 Blade 13 Blade 14 Int1 (1) Int2 (2) Int3 (3) Int12 (12) Int13 (13) Int14 (14) Ext1 (17) Ext2 (18) Ext5 (21) Ext6 (22) Mgt1 (15) Mgt2 (16) Management Module 1 Management Module 2 Customer Network Default VLAN VLAN 1 Management VLAN VLAN 4095 IP Interface 128 18

19. BNT Nortel layer 2-7 GbESM The Layer 2-7 GbESM features: Support for up to 300,000 simultaneous Layer 2-7 sessions with full wire-speed packet forwarding for all connections Link aggregation on the four external ports Layer 4 load balancing for computing optimization Virtual router redundancy to facilitate high application availability in the event of application, operating system or hardware failures Layer 2/3Layer 4Layer 5-7 19

20. BNT (Nortel) Layer 2-7 GbESM: Additional function 20

21. BNT 1/10 Gb uplink Ethernet switch module Combined 1 Gb and 10 Gb external ports Three slots for 10 Gb Ethernet SFP+ modules Six 1 Gb RJ45 ports Fourteen internal 1 Gb ports 21

22. Server connectivity module BladeCenter chassis Ethernet overview Internal chassis traffic flow Switch I/O module management connectivity BNT (Nortel) Ethernet switch I/O modules Server connectivity module Pass-Thru module High speed I/O modules overview Expansion cards 22

23. Server connectivity module: Product highlights Low-cost Layer 2 device Simplified user interface to enable server administrator control of device Ideal for SMB customers and applications Function summary –Basic Layer 2 connectivity within chassis –“Non-networking”, server admin setup and control –Simple, GUI/CLI interface Create blade groups (simple VLANs) Create external port groups (auto Etherchannel) Enable trunk failover (redundancy option) Set external port speed Assign tagged VLANs to blade groups 23

24. Pass-Thru module BladeCenter chassis Ethernet overview Internal chassis traffic flow Switch I/O module management connectivity BNT (Nortel) Ethernet switch I/O modules Server connectivity module Pass-Thru module High speed I/O modules overview Expansion cards 24

25. Intelligent copper pass-thru module: Product highlights 14 external RJ-45 copper ports for making connections to the external network. –The external ports can operate at 100 or 1000 Mbps. 14 internal bi-directional ports, connected to each of the blade servers One-to-one mapping of internal Ethernet ports on the blade to external copper RJ-45 GUI based operator interface 25

26. High speed I/O modules overview BladeCenter chassis Ethernet overview Internal chassis traffic flow Switch I/O module management connectivity BNT (Nortel) Ethernet switch I/O modules Server connectivity module Pass-Thru module High speed I/O modules overview Expansion cards 26

27. High speed I/O modules: Overview BNT 10Gb Ethernet Switch I/O Module – BladeCenter H and HT High Speed Module –10 External Ports –14 Internal Ports BNT 10Gb Ethernet Pass-Thru I/O Module – BladeCenter H and HT High Speed Module – 14 External Ports – 14 Internal Ports BNT 10Gb Ethernet Switch I/O Module BNT 10Gb Ethernet Pass-Thru I/O Module 27

28. Expansion cards BladeCenter chassis Ethernet overview Internal chassis traffic flow Switch I/O module management connectivity BNT (Nortel) Ethernet switch I/O modules Server connectivity module Pass-Thru module High speed I/O modules overview Expansion cards 28

29. Ethernet expansion cards: Combination I/O vertical (CIOv) PCI Express host interface Broadcom BCM5709S communication module BladeCenter Open Fabric Manager (BOFM) support Connection to 1000BASE-X environments using BladeCenter Ethernet switches IBM HS22 Blade Server with Ethernet CIOv card installed CIOv Ethernet Expansion Card 29

30. Ethernet expansion cards: Combination form factor vertical (CFFv) Based on Broadcom 5704 chipset Two 1 Gb ports per blade server IBM HS22 Blade Server with Ethernet CFFv card installed Ethernet CFFv Expansion Card 30

31. Ethernet expansion cards: QLogic iSCSI Dual ports TCP/IP Offload Engine – TC/IP stack implemented in firmware Provides iSCSI diskless boot capability QLogic iSCSI StFF Expansion Card QLogic iSCSI Management Utility 31

32. Ethernet expansion cards: High speed overview High Speed Form horizontal (CFFh) expansion cards – Interface to Horizontally mounted I/O modules – BladeCenter H and HT chassis PCI-E bus connections QLogic 2-Port 10Gb Converged Network Adapter CFFh Broadcom 10Gb 4-Port CFFh card IBM BladeCenter 2/4-Port CFFh card NetXen 10Gb 2-Port CFFh card 32

33. Key words Device Manager Failover Fibre Ethernet Graphical user interface (GUI) Local Area Network (LAN) Layer 2 – 7 Media Access Control (MAC) Address Power On Self Test (POST) Spanning Tree Serializer/deserializer (SERDES) Server Connectivity Module (SCM) Open Shortest Path First (OSPF) Uplink Virtual Local Area Network (VLAN) tagging Virtual Router Redundancy Protocol (VRRP) Trunking 33

34. Checkpoint (1 of 2) 1.True/False: In order to provide Ethernet connectivity, each blade server requires an Ethernet expansion card. 2.A customer requires policy management capability in the application through Transport layers of the OSI model. Select the switch module appropriate for these requirements. a.BNT 10Gb Pass-thru Module b.BNT L2/3 Copper GbESM c.IBM Server Connectivity Module d.BNT L2-7 GbESM 3.True/False: The Ethernet switch I/O modules can be managed through the AMM management network, or through the switch I/O module external ports 34

35. Checkpoint solutions (1 of 2) 1.True/False: In order to provide Ethernet connectivity, each blade server requires an Ethernet expansion card. Answer: False 2.A customer requires policy management capability in the application through Transport layers of the OSI model. Select the switch module appropriate for these requirements. a.BNT 10Gb Pass-thru Module b.BNT L2/3 Copper GbESM c.IBM Server Connectivity Module d.BNT L2-7 GbESM Answer: d 3.True/False: The Ethernet switch I/O modules can be managed through the AMM management network, or through the switch I/O module external ports Answer: True 35

36. Checkpoint (2 of 2) 4.In order to support 4 Ethernet connections from a Blade server, which of the following is correct? a.An expansion blade is required b.Add an Ethernet expansion card and switch module in Bay 2 c.Add an Ethernet expansion card and switch modules in Bays 3 and 4 d.Add an IBM Server connectivity module 5.True/False: NIC Teaming can be used to provide redundancy, utilizing the paths from the blade server onboard NICs through the midplane to Ethernet switch I/O modules in Bays 1 and 2 of the chassis. 36

37. Checkpoint solutions (2 of 2) 4.In order to support 4 Ethernet connections from a blade server, which of the following is correct? a.An expansion blade is required b.Add an Ethernet expansion card and switch module in Bay 2 c.Add an Ethernet expansion card and switch module in Bays 3 and 4 d.Add an IBM Server connectivity module Answer: c 5.True/False: NIC Teaming can be used to provide redundancy, utilizing the paths from the blade server onboard NICs through the midplane to Ethernet switch I/O modules in Bays 1 and 2 of the chassis. Answer: True 37

38. Unit summary Having completing this unit, you should be able to: List the BladeCenter components required for Ethernet connectivity Name the management components supporting Ethernet switch modules Select the Ethernet I/O switch module to meet a specific requirement List the components required to provide additional Ethernet connectivity to a blade server Identify the method used to provide blade server Ethernet connection redundancy 38