Unified Fabric aka FCOE

Unified Fabric aka FCOE
Dave Gibson Senior Systems Engineer Cisco Systems 1 1

Legal Disclaimer Many of the products and features described herein remain in varying stages of development and will be offered on a when-and-if-available basis. This roadmap is subject to change at the sole discretion of Cisco, and Cisco will have no liability for delay in the delivery or failure to deliver any of the products or features set forth in this document.

Agenda The Evolution of the Data Center Introduction to FCoE
Standards Defined Nexus and the Unified Fabric Nexus 5000

The Evolution of the Data Center

Data Center Access Layer Trends
Multi-Core CPU architectures allowing bigger and multiple workloads on the same machine Server virtualization driving the need for more I/O bandwidth per server Growing need for network storage driving the demand for higher network bandwidth to the server Increasing adoption of Blades in data centers. 10G LOM on server Motherboard

Next-Gen Switch Design Goals
Consolidate LAN & SAN infrasctucture Standards based solution Reduce total cost of ownership End-to-end data center architecture Operational consistency across platforms Nexus 5000 10G to Server Nexus Family Unified I/O Low Latency Scalable Infra-structure Build with superior performance in mind Support low latency applications (e.g. HPC, clustered app’s) Enable Virtualization Address increase in server processing power Scale to 40G and 100G in future Increase feature velocity

(Modular Switch Platform)
Cisco Nexus Family Complete data center class switching portfolio Consistent data center operating system across all platforms Infrastructure scalability, transport flexibility and operational manageability Nexus 7000 (Modular Switch Platform) Nexus 1000V (Virtual Switch) Nexus 4000 (Blade Switch) 1K Cisco Nexus 1000V 2008 x86 Nexus 2000 (Fabric Extender) Nexus 5000 (Fixed Config Switch) NX-OS Data Center Operating System Data Center Network Manager

Nexus 5000 Platform 8

Before I/O Consolidation
Parallel LAN/SAN Infrastructure Inefficient use of Network Infrastructure 5+ connections per server – higher adapter and cabling costs Adds downstream port costs; cap-ex and op-ex Each connection adds additional points of failure in the fabric Multiple switching modules in Blade Chassis Longer lead time for server provisioning Multiple fault domains – complex diagnostics Management complexity LAN SAN A SAN B Server with NICs and HBAs Blade Chassis with I/O Modules Ethernet FC

I/O Consolidation Reduction of server adapters
Simplification of access layer and cabling Gateway free implementation – fits in installed base of existing LAN and SAN Lower Total Cost of Ownership Fewer Cables Investment Protection (LANs and SANs) Consistent Operational Model LAN SAN A SAN B Nexus 5000 Nexus 5000 Blade Chassis with Nexus 4000 Server with CNAs Data Center Bridging and FCoE Ethernet Fibre Channel (FC)

Adapter Evolution: Consolidation Network Adapter

Operating System View

Transceiver Latency (link)
Evolution of 10G Ethernet Physical Media Role of Transport in Enabling these Technologies! Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s 10Mb 100Mb 1Gb 10Gb UTP Cat 5 SFP Fiber UTP Cat 3 UTP Cat 5 X2 SFP+ Cu (BER better than ) SFP+ Fiber Cat 6/7 -18 Technology Cable Distance Power (each side) Transceiver Latency (link) SFP+ CU Copper Twinax 7m ~0.1W ~0.1μs SFP+ USR Ultra short reach MM OM2 MM OM3 10m 100m 1W SFP+ SR Short reach MM 62.5 μm MM 50 μm 26-33m 66-300m SFP+ LR Long range SMF G.652 10km 0.5W 10GBASE-T Cat6 Cat6a/7 55m 30m ~8W ~4W 2.5μs 1.5μs Server virtualization created the need to look into the network virtualization

Introduction to FCoE

What is Fibre Channel over Ethernet?
From a Fibre Channel standpoint it’s FC connectivity over a new type of cable called… an Ethernet cloud From an Ethernet standpoints it’s Yet another ULP (Upper Layer Protocol) to be transported Two views – must look from the SAN and LAN perspective. Consolidation is not a new concept – iSCSI, etc – enhancing the ethernet is the new FCoE is an extension of Fibre Channel onto a Lossless Ethernet fabric

Unified Fabric Overview Fibre Channel over Ethernet (FCoE)
Benefits Mapping of FC Frames over Ethernet Enables FC to Run on a Lossless Ethernet Network Fewer Cables Both block I/O & Ethernet traffic co-exist on same cable Fewer adapters needed Overall less power Interoperates with existing SAN’s Management SAN’s remains constant No Gateway Ethernet All FC services are still available – zoning, security, separation of SAN A/B Fibre Channel Traffic 4/19/2017 16

FCoE Enablers 10Gbps Ethernet Lossless Ethernet Ethernet jumbo frames
Matches the lossless behavior guaranteed in FC by B2B credits Ethernet jumbo frames Normal ethernet frame, ethertype = FCoE Same as a physical FC frame Ethernet Header FCoE FC FC Payload CRC EOF FCS Control information: version, ordered sets (SOF, EOF)

Unified I/O Fibre Channel over Ethernet (FCoE)
FCoE is managed like FC at initiator, target, and switch level FCoE is Fibre Channel Easy to Understand Completely based on the FC model Same Operational Model Same host-to-switch and switch-to-switch behavior as FC Same Techniques of Traffic Management e.g. in order delivery, FSPF load balancing There are several advantages to FCoE foremost of which is that it is managed as Fibre Channel at the host and switch level. FCoE is: Easy to understand since it is completely based on the FC model Since FCoE uses the same operational model administrators can expect the same host-to-switch and switch-to-switch behavior of FC FCoE uses the same techniques of traffic management such as in order delivery or FSPF load balancing FCoE also implements the same management and security models as FC: WWNs, FC-IDs, hard/soft zoning, DNS, RSCN Essentially FCoE is aligned with the FC-BB-4 model Note: the alignment described here guarantees transparent support for all Fibre Channel advanced features (VSANs, IVR/IFR, security) Same Management and Security Models WWNs, FC-IDs, hard/soft zoning, DNS, RSCN

Network Stack Comparison
SCSI SCSI SCSI SCSI SCSI iSCSI FCP FCP FCP FC FC FC FCIP Less Overhead than FCIP, iSCSI TCP TCP IP IP FCoE Ethernet Ethernet Ethernet PHYSICAL WIRE SCSI iSCSI FCIP FCoE FC

FCoE Frame Format Destination MAC Address Source MAC Address
(IEEE 802.1Q Tag) ET = FCoE Ver Reserved SOF Encapsulated FC Frame (with CRC) EOF FCS

FCoE Logical End Point FC-4 ULP Mapping FC-4 ULP Mapping
Ethernet Media Access Control Ethernet Physical Layer FC-2 Framing & Flow Control FC-3 Generic Services FC-4 ULP Mapping FCoE Logical End Point FC-0 Physical Interface FC-1 Encoding FC-2 Framing & Flow Control FC-3 Generic Services FC-4 ULP Mapping

FCoE Standards Defined

A larger picture IEEE 802 INCITS/T11 IETF
Evolution of Ethernet (10 GE, 40 GE, 100 GE, copper and fiber) Evolution of switching (Priority Flow Control, Enhanced Transmission, Congestion Management, Data Center Bridging eXchange) INCITS/T11 Evolution of Fibre Channel (FC-BB-5) FCoE (Fibre Channel over Ethernet) IETF Layer 2 Multi-Path TRILL (Transparent Interconnection of Lots of Links)

DCE versus DCB DCE is an old Cisco marketing term
Cisco is now using the term DCB The term IEEE uses Cisco supports the DCB standard activity By implementing products that are DCB compliant CIN-DCBX – Cisco, Intel, Nuova Data Center Bridging Exchange protocol, pre-standard CEE-DCBX – Converged Enhanced Ethernet Data Center Bridging Exchange protocol, which is standards base

What’s FC-BB-5 FC-BB-5 covers the majority of the FC features, using Ethernet From an Ethernet perspective, FC-BB-5 is Ethernet control plane referred to as FIP (Fibre Channel over Ethernet Initiation Protocol) discover and build virtual paths between end points Ethernet data plane providing FCoE forwarding including both FC control plane and FC data plane (FCF) FIP – vlan discovery, login/logout of FCoE hosts FCF definition

FC-BB-6 It is an active working group of T11 that will discuss the future of FCoE or FCoE v2.0 It is just started, 18 months to have a standard Approximate target spring 2011 You can track it on

Protocol Organization
FCoE is really two different protocols: FCoE itself … Is the data plane protocol It is used to carry most of the FC frames and all the SCSI traffic FIP (FCoE initiation protocol) It is the control plane protocol It is used to discover the FC entities connected to an Ethernet cloud It is used to login to and logout from the FC fabric FIP – necessary for multi-hop environments because FIP frames will be passed through the lossless Ethernet cloud to discover the FCoE capable devices and points of Fibre Channel switching Must discover places to log in FCoE hosts – must be VLAN specific The two protocols have: Two different Ethertypes Two different frame formats 27 27

What’s NOT FC-BB-5 FC-BB-5 doesn’t deal with how lossless is realized in Ethernet no Priority Flow Control, Bandwidth Management, etc. FC-BB-5 doesn’t deal with management functions

IEEE DCB standards status
DCB technologies allow Ethernet to be lossless and to manage bandwidth allocation of SAN and LAN flows Feature / Standard Standards Status IEEE 802.1Qbb Priority Flow Control (PFC) Enable multiple traffic types to share a common Ethernet link without interfering with each other PAR approved 1.0 published IEEE 802.1Qaz Bandwidth Management (ETS) Enable consistent management of QoS at the network level by providing consistent scheduling Data Center Bridging Exchange Protocol (DCBX) Management protocol for enhanced Ethernet capabilities This is part of IEEE 802.1Qaz PFC - based the pause mechanism found in Ethernet today ETS – avoids the possibility for one type of traffic to starve another Based on CoS – 802.1Q tagging 29 29 29

Data Center Ethernet: PFC & Bandwidth Management
Priority Flow Control Enables lossless behavior for each class of service PAUSE sent per virtual lane when buffers limit exceeded Transmit Queues Ethernet Link Receive Buffers Zero One Two Five Four Six Seven Three STOP PAUSE Eight Virtual Lanes CoS based Bandwidth Management Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth 802.1Qaz Enhanced Transmission 10 GE Realized Traffic Utilization 3G/s HPC Traffic 3G/s 2G/s Storage Traffic 3G/s LAN Traffic 4G/s 5G/s t1 t2 t3 Offered Traffic 6G/s PAUSE functionality per Ethernet priority Use IEEE 802.1Q defines 8 priorities Traffic classes are mapped to different priorities -- no traffic interference Storage traffic may be paused while IP traffic is being forwarded Or, vice versa Requires independent resources per priority (buffers) COS – must be end to end (best practice) 4/19/2017 Nuova Systems Inc.

DCBX Overview Auto-negotiation of capability and configuration
Priority Flow Control capability and associated CoS values Allows one link peer to push config to other link peer Link partners can choose supported features and willingness to accept Discovers FCoE Capabilities Responsible for Logical Link Up/Down signaling of Ethernet and FC DCBX negotiation failures will result in: vfc not coming up Per-priority-pause not enabled on CoS values with PFC configuration Management protocol used to decide on the Lossless Ethernet characteristics between two endpoints Which CoS value will be mapped to “no drop” services

FCoE control plane

FIP: FCoE Initialization Protocol
FCoE VLAN discovery Automatic discovery of FCoE VLANs Device discovery ENodes discover VF_Port capable FCF-MACs for VN_Port to VF_Port Virtual Links VE_Port capable FCF-MACs discover other VE_Port capable FCF-MACs for VE_Port to VE_Port Virtual Links The protocol verifies the Lossless Ethernet network supports the required Max FCoE Size Virtual Link instantiation Builds on the existing Fibre Channel Login process, adding the Negotiation of the MAC address to use Fabric Provided MAC Address (FPMA), or Server Provided MAC Address (SPMA) Virtual Links maintenance Timer based

Fabric Provided MAC Addresses
Server Provided MAC Addresses MAC address assigned for each FC_ID: Consistent with the Fibre Channel model Multiple FC-MAPs may be supported One per SAN No table needed for Encapsulation Multiple MACs may be needed for NPIV Adapter uses burned-in or configured MAC address: Consistent with the Ethernet model FCF needs a table to map between MAC addresses and FC_IDs 48 bits FC-MAP (0E-FC-00) FC-ID 7.8.9 24 bits 48 bits Burned in or Configured MAC Address MAC addresses of the form FC-MAP and FC-ID FC-MAP is Fiber Channel mac address mapped to that specific fabric FC-ID is given by the fibre channel switch at time of FLOGI/FDISC ****NOTE THAT WHEN A FCoE logs into an FCF, all traffic must pass through that FCF when going from host to target**** Assigned only to VN_Ports Not assigned to VF_Ports, VE_Ports, and FCFs Cisco Nexus 5000 uses FPMA

Initial Login Flow ladder
ENode FCoE Switch VLAN Discovery VLAN Discovery FIP: FCoE Initialization Protocol Solicitation FCF Discovery FCF Discovery Advertisement FLOGI/FDISC FLOGI/FDISC Accept FC Command responses FCOE Protocol FC Command 35 35

VLAN Discovery: Multicast Request
MAC (H1) FCF-MAC (A) FCF A H1 Lossless Ethernet Bridge FC Fabric H2 FCF B MAC (H2) FCF-MAC (B) All-FCF-MACs MAC(H2) VLAN Request [MAC(H2)] Request to the FCFs reachable on the default port VLAN which VLANs are available for FCoE

VLAN Discovery: Unicast Notifications
FCF A FCF B Lossless Ethernet Bridge FC Fabric FCF-MAC (A) FCF-MAC (B) H1 H2 MAC (H2) MAC (H1) MAC(H2) FCF-MAC(A) VLAN Notification [F=1, FCF-MAC(A), FCoE VLAN List] MAC(H2) FCF-MAC(B) VLAN Notification [F=1, FCF-MAC(B), FCoE VLAN List] One or more FCoE VLANs are provided

Discovery: Multicast Solicitation
MAC (H1) FCF-MAC (A) FCF A H1 Lossless Ethernet Bridge FC Fabric H2 FCF B MAC (H2) FCF-MAC (B) All-FCF-MACs MAC(H2) Solicitation [F=0, MAC(H2), Capabilities, Other] Solicitation identifies VF_Port capable FCF-MACs with compatible addressing capabilities

Discovery: Unicast Advertisements
FCF A FCF B Lossless Ethernet Bridge FC Fabric FCF-MAC (A) FCF-MAC (B) H1 H2 MAC (H2) MAC (H1) MAC(H2) FCF-MAC(A) Mini-jumbo Advertisement [S=1, F=1, Priority, Fabric, FCF-MAC(A), Switch_Name, Capabilities, Other] MAC(H2) FCF-MAC(B) Mini-jumbo Advertisement [S=1, F=1, Priority, Fabric, FCF-MAC(B), Switch_Name, Capabilities, Other] H2’s FCF list: FCF-MAC(A) [J] FCF-MAC(B) [J] FCF not meeting capabilities of ENode does not reply

FIP FLOGI Request FCF A H1 Lossless Ethernet Bridge FC Fabric H2 FCF B
FCF-MAC (A) FCF-MAC (B) H1 H2 MAC (H2) MAC (H1) FCF-MAC(A) MAC (H2) FIP FLOGI Request [FC Header, FLOGI data, Proposed MAC’(H2)] FCF-MAC(B) MAC(H2) FIP FLOGI Request [FC Header, FLOGI data, Proposed MAC’’(H2)] Capability agreed during discovery

FIP FLOGI LS_ACC FCF A H1 Lossless Ethernet Bridge FC Fabric H2 FCF B
FCF-MAC (A) FCF-MAC (B) H1 H2 MAC (H2) MAC (H1) MAC (H2) FCF-MAC(A) FIP FLOGI LS_ACC [FC Header, LS_ACC data, Approved MAC(H2)’] MAC(H2) FCF-MAC(B) FIP FLOGI LS_ACC [FC Header, LS_ACC data, Approved MAC(H2)’’] ENode uses MAC address in FIP FLOGI LS_ACC as the VN_Port MAC address for the FC-ID contained in the FLOGI data for subsequent FCoE frames

FCoE Virtual Links FCF A H1 Lossless Ethernet Bridge FC Fabric H2
FCF B Lossless Ethernet Bridge FC Fabric FCF-MAC (A) FCF-MAC (B) H1 H2 MAC (H2) MAC (H2)’ MAC (H2)’’ MAC (H1) FCF-MAC(A) MAC(H2)’ Fibre Channel Frame (FCoE) [FC SOF, FC Header, FC Data, FC CRC FC EOF] FCF-MAC(B) MAC(H2)’’ Fibre Channel Frame (FCoE) [FC SOF, FC Header, FC Data, FC CRC, FC EOF] Subsequent FCoE frames use granted MAC address and assigned FC-ID

Virtual Links Maintenance
FCFs generate periodic Advertisements ENodes monitor Advertisements to verify the Virtual Link is alive ENodes generate periodic FIP Keep Alive messages FCFs monitor FIP Keep Alive messages to verify the Virtual Link is alive After two missed messages the Virtual Link is not alive and the associated VN_Port/VF_Port de-instantiated

FCoE data plane

ENode: Simplified Model
ENode (FCoE Node): a Fibre Channel HBA implemented within an Ethernet NIC aka CNA (Converged Network Adapter) FCoE LEP : The data forwarding component that handles FC frame encapsulation/decapsulation FCoE Controller is the functional entity that performs the FIP and instantiates VN_Port/FCoE_LEP pairs. FC Node From the host perspective E node is the actual FCoE host LEP: FCoE Logical End Point used for encapsulation and decapsulation of FC into Ethernet frames Draw connection from FC host to fabric – N-port to F-port In FCoE the way we emulate the N to F port connection in fibre-channel through VN-port to VF-port FCoE_Controller FCoE_Controller FCoE_LEP FCoE_LEP Enet port Enet port 45 45

FCoE Switch: Simplified Model
FCF (Fibre Channel Forwarder), the forwarding entity inside an FCoE switch FC port FCoE Switch FCF FCoE_Controller FCoE_LEP From the Switch perspective: FCF is the where the Fibre Channel Switching is performed. FCoE hosts perform the Login/Logout to the FC fabric at an FCF and establish the N to F port connection Walk through receiving an FCoE frame on the Ethernet port  notice Ethertype and forward to the Logical End Point  decapsulated and sent to the FCF for Fibre Channel look up based on FC_ID  if that FCID resides off of an Ethernet port, back to LEP for encapsulation. If out a fibre channel port, forwards. Ethernet Bridge Eth port 46 46

FCoE Network Topology

FCoE: Initial Deployment
SAN A SAN B 10GE Backbone VF_Ports Nexus 5000 (FCF) Here is FCoE initial deployment. We basically see drawn here in this slide two servers that have two CNAs or one CNA with two ports, that connect to two switches that in this case are two Nexus 5000s. And basically the hosts, the CNA on the host implements the VN port and the switch, in this case the Nexus 5000, implements the VF port. Then the Nexus 5000 implements E port over Fibre Channel towards SAN-A and towards SAN-B and implements regular Ethernet ports towards the 10 Gigabit Ethernet backbone. This is the initial deployment. It's where there is most of the cost savings because basically you rationalize the connection of a server to the first level of switch that we also call the access switch. Here is where most of the savings in terms of cabling, power, cooling, etc, happens. VN_Ports 10GE 4/8 Gbps FC 48

FCoE: Adding Blade Servers
SAN A SAN B 10GE Backbone VF_Ports As the deployments evolve, we will also support the blade server. And in blade server you will basically have, inside the blade server, switches that are lossless Ethernet switches but they don't have an FCF. So you see, the VN ports are on the blade are on the CNA that is mounted on the mezzanine card of a blade. And the VF ports are still on the Nexus 5000, but there is this intermediate lossless Ethernet switch that sits on the blade server and that provides some sort of cabling consolidation, which is very important. And to support this configuration definitely you require the implementation of FIP, the Fibre Channel initialization protocol, because basically that Ethernet switch that is inside the blade server is a small cloud which is now present between the CNA and the VF port, and without FIP they will not discover each other. 10GE VN_Ports 4/8 Gbps FC

FCoE: Adding Native FCoE Storage
SAN A SAN B 10GE Backbone VN_Ports VF_Ports The next step will be adding native FCoE storage array. Companies such as Network Appliance have already announced the availability of native FCoE storage array. So basically you will have direct 10 Gigabit Ethernet connection that goes to storage array, and the storage array of course will implement VN port exactly as blades or server implements VN port. They are endpoints of an FC network and endpoints are basically VN ports because they run over Ethernet. 10GE VN_Ports 4/8 Gbps FC

FCoE: Adding VE_ports SAN B SAN A 10GE Backbone VE_Ports VF_Ports 10GE
The last step will be basically to implement VE port or some of the classical Fibre Channel equipment. And this will happen with a FCoE line card in the MDS and the introduction of FCoE on the Nexus 7000 and so on 10GE VN_Ports 4/8 Gbps FC

Nexus Topologies 52

The Unified Data Center Architecture
Core: L3 boundary to the DC network. Functional point for route summarization, the injection of default routes and termination of segmented virtual transport networks L3 NEXUS 7000 Aggregation: Typical L3/L2 boundary. DC aggregation point for uplink and DC services offering key features: VPC, VDC, 10GE density and 1st point of migration to 40GE and 100GE L3 Service Appliances Service Modules NEXUS VPC L2 Catalyst 6500 Access: Classic network layer providing non-blocking paths to servers & IP storage devices through VPC. It leverages Distributed Access Fabric Model (DAF) to centralize config & mgmt and ease horizontal cabling demands related to 1G and 10GE server environments Unified Compute System L2 NEXUS 5000 NEXUS VPC A B NEXUS 2000 Virtual Access: A virtual layer of network intelligence offering access layer-like controls to extend traditional visibility, flexibility and mgmt into virtual server environments. Virtual network switches bring access layer switching capabilities to virtual servers without burden of topology control plane protocols. Virtual Adapters provide granular control over virtual and physical server IO resources vL2 NEXUS 1000v VM VM VM VM VM VM VM VM VM VM VM VM POD Rack 1 Rack 2 Rack 3 Rack 1 Rack x

Fitting the pieces together…
DC Core Nexus GbE Core WAN Gigabit Ethernet 10 Gigabit Ethernet IP+MPLS WAN Agg Router 4, 8Gb Fibre Channel 10 Gigabit FCoE/DCE DC Aggregation SAN A/B Nexus GbE Agg Catalyst 6500 DC Services Catalyst GbE VSS Agg DC Services MDS 9500 Storage Services DC Access Catalyst 6500 End-of-Row Catalyst 49xx Rack CBS 3100 | MDS Blade Nexus 7000 End-of-Row Nexus 5K|2K Top of Rack UCS blade or Nexus 4K MDS 9500 Storage Nexus 1000V VN-Link 1GbE Server Access 1GbE,10GbE Server Access Storage 54 54

Ethernet Basics 55

Introduction LAN Switching deals primarily with switching of Ethernet frames. So it makes sense to start off with a thorough understanding of how Ethernet works…

Key Functionality Four major functions: Learning of MAC addresses
Segment Port E0 Segment Port E1 Four major functions: Learning of MAC addresses Forwarding/filtering frames Forwarding Broadcasts Loop avoidance- Spanning Tree Switches are data communications devices that operate principally at Layer 2 of the OSI reference model. As such, they are widely referred to as data link layer devices. Today, switching technology has emerged as the evolutionary heir to bridging based internetworking solutions. Switching implementations now dominate applications in which bridging technologies were implemented in prior network designs. Superior throughput performance, higher port density, lower per-port cost, and greater flexibility have contributed to the emergence of switches as replacement technology for bridges and as complements to routing technology. LAN switches are used to interconnect multiple LAN segments. LAN switching provides dedicated, collision-free communication between network devices, with support for multiple simultaneous conversations. LAN switches are designed to switch data frames at high speeds A switch has three basic functions: Address Learning: A switch will read the layer 2 header and read the source address of a device when it transmits. It will learn where that device resides in relation to other devices. Forwarding/Filtering: A switch will forward out information to a specific address, if it has been learned, and filter messages to end users that do not need to hear it. Loop Avoidance: The Spanning Tree Protocol is used to prevent switching loops in a redundant topology 3

Layer 2 Switching Purposes for using switching
Breaks up collision domains Security through VLAN implementation Cost-effective, resilient internetwork Purpose for Spanning-Tree Protocol (STP) Stops loops in layer 2 switched networks

Address Learning A1 B1 A2 B2 C1 C2
MAC Address Table Pre DA SA Type Data FCS A1 B1 0260.8c Port Port 0260.8c E0 E1 A2 Port B2 0260.8c E2 0260.8c C1 C2 0260.8c 0260.8c Source MAC address is associated with NIC Addresses are learned from SA field of ethernet frame. Independent of Destination- Unicast or Broadcast

Address Learning A1 B1 A2 B2 C1 C2 Port Port E0 E1 Port E2
MAC Address Table E0: c Pre DA SA Type Data FCS A1 B1 0260.8c Port Port 0260.8c E0 E1 A2 Port B2 0260.8c E2 0260.8c C1 C2 0260.8c 0260.8c

Frame Forwarding A1 sends a frame to B2. Forwarded
MAC Address Table E0: c E0: c E1: c B1 A1 E1: c E2: c E2: c 0260.8c 0260.8c E0 E1 E2 B2 A2 0260.8c 0260.8c C1 C2 0260.8c 0260.8c A1 sends a frame to B2. Forwarded Occurs when destination is known

Handling Broadcasts A1 sends out a broadcast. Forwarded
MAC Address Table E0: c E0: c E1: c B1 A1 E1: c E2: c E2: c 0260.8c 0260.8c E0 E1 E2 B2 A2 0260.8c 0260.8c C1 C2 0260.8c 0260.8c A1 sends out a broadcast. Forwarded Thus, it is a single broadcast Domain

VLANs 63

Hub to Bridge to Switch Floor #3 6 – Collision Domains
Rpt Hub 6 – Collision Domains 3- Broadcast Domains Floor #3 Floor #2 Floor #1

Hub to Bridge to Switch Floor #3 8 – Collision Domains
Rpt Hub 8 – Collision Domains 1- Broadcast Domain Floor #3 Floor #2 Floor #1

Hub to Bridge to Switch Floor #3
Rpt Hub What if I wanted each floor to be in its own, unique broadcast domain? Floor #3 Floor #2 Floor #1

Hub to Bridge to Router Floor #3
A1 A2 A3 A4 A1, A2 A3, A4 Floor #3 Floor #2 Floor #1 Bridge A B1 B2 B3 B4 One way to do it…separate each floor using a router. B1, B2 B3, B4 Bridge B C3 C2 C4 C1 C1, C2 C3, C4 Bridge C

Hub to Bridge to Router Floor #3 But what’s the downside? Floor #2
A1, A2 A3, A4 Floor #3 Floor #2 Floor #1 Bridge A B1 B2 B3 B4 But what’s the downside? B1, B2 B3, B4 Bridge B C3 C2 C4 C1 C1, C2 C3, C4 Bridge C

Hub to Bridge to Router Floor #3 Each floor needs its own switch
A1, A2 A3, A4 Floor #3 Floor #2 Floor #1 Bridge A Each floor needs its own switch Router interfaces are expensive B1 B2 B3 B4 B1, B2 B3, B4 Bridge B C3 C2 C4 C1 C1, C2 C3, C4 Bridge C

The solution…VLANs!! Method of micro-segmenting a L2 / L3 topology
Each VLAN is a separate broadcast domain Any port on a Catalyst/Nexus switch can be in any VLAN Inter-VLAN communication requires a L3 routing device VLANs may span multiple switches

Linking Different Switches
VLAN10 VLAN30 VLAN20 Building 1 Building 2

Trunking to the Rescue Problem: How do you identify which frame
Carries all VLANs Trunk Problem: How do you identify which frame belongs to which VLAN if all VLANs are carried in single link?

Trunking to the Rescue Carries all VLANs Trunk Answer: We’ll tag each frame placed on the trunk with the VLAN it belongs to. Trunking encapsulation will do this for us.

Trunking Methods Two trunking protocols ISL = Cisco proprietary
802.1q = IEEE specification Trunking status can be negotiated on a link Trunking is also supported on some routers Some NIC vendors support trunking

IEEE 802.1q Overview All frames are encapsulated except the native VLAN (covered in a moment) Inserts TAG into frame, which extends maximum frame size to 1522 bytes from 1518 bytes Frame Check Sequence (FCS) is recomputed for the entire frame after the tag is inserted Assumes that there is only one instance of spanning tree

What is the “Native VLAN”?
VLAN Trunks (either ISL or 802.1Q) carry traffic from ALL VLANs by default. Switches and Routers need to send certain “management” frames to each other such as: Cisco Discovery Protocol (CDP) VLAN Trunking Protocol (VTP) Dynamic Trunking Protocol (DTP) VLAN-1 is the Native VLAN (by default) This can be changed via configuration Native VLAN must always match on both sides of the trunk.

Port type basics How to configure an Access L2 Port? (single vlan)
n5k# config terminal n5k(config)# interface ethernet 1/3 n5k(config-if)# switchport n5k(config-if)# switchport mode access n5k(config-if)# switchport access vlan 2 n5k(config-if)# end n5k#

Port-Channels 78

Port Channel Aggregates multiple physical Ethernet links into a single logical Ethernet link Used to increase availability and bandwidth Similar benefits to a SAN port-channel today

Link Aggregation Control Protocol (LACP)
Method to control the bundling of several physical ports together to form a single logical channel Turn on LACP globally first switch(config)# feature lacp Channel mode needs to be either “active” or “passive” and one side has to be “active” Switch 1 mode Switch 2 mode Port added to EtherChannel active passive Yes No active or passive on Yes but no LACP negotiation

CLI Example switch(config)# feature lacp
switch(config)# int ethernet 1/15 switch(config-if)# channel-group 1 Ethernet1/15 added to port-channel1 switch(config-if)# exit switch(config)# int ethernet 1/16 switch(config-if)# channel-group 1 mode on Ethernet1/16: added to port-channel1 switch(config-if)#

Verification Commands
Cisco NX-OS Port-Channels Command Description show interface Displays statistics all interfaces or a specific interface show interface port-channel <#> Displays statistics for a specific port-channel show port-channel capacity Displays port-channel resources (total, used, free) show port-channel compatibility-parameters Displays the compatibility-parameters (IE: speed, duplex, etc) show port-channel database Displays the aggregation state for one or more port-channels show port-channel load-balance Displays the load-balancing algorithm (hash) configured show port-channel summary Displays a summarized list of all port-channels show port-channel traffic Displays the load per link in a port-channel (Based in interface counters) show port-channel usage Displays the range of used and unused port-channel numbers show lacp counters Displays the LaCP PDU and error counters show lacp interface Displays detailed LaCP information per interface show lacp neighbors Displays detailed LaCP information per neighbor show lacp port-channel Displays the port-channel LaCP configuration show lacp system-identifier Displays the LaCP system ID (Priority / MAC address)

Creating EtherChannel
Three channel group modes: active , passive and on. Switch(conf)#interface e1/1 switch(config-if)# channel-group 1 mode ? active Set channeling mode to ACTIVE on Set channeling mode to ON passive Set channeling mode to PASSIVE Channel mode Description active Initiates LACP negotiation passive Responds to LACP negotiation on No LACP. Adds port to EtherChannel

Hash algorithm CLI CLI to select the fields of frame into the hash calculation n5k(config)# port-channel load-balance ethernet ? destination-ip Destination IP address destination-mac Destination MAC address destination-port Destination TCP/UDP port source-destination-ip Source & Destination IP address source-destination-mac Source & Destination MAC address source-destination-port Source & Destination TCP/UDP port source-ip Source IP address source-mac Source MAC address source-port Source TCP/UDP port Check the hash algorithm n5k# sh port-channel load-balance Port Channel Load-Balancing Configuration: System: destination-mac Port Channel Load-Balancing Addresses Used Per-Protocol: Non-IP: destination-mac IPv4: destination-mac IPv6: destination-mac

Spanning Tree 85

Spanning Tree Protocol (STP)
A link management protocol that provides path redundancy while preventing undesirable loops in the network STP’s main task is to stop network loops from occurring in layer-2 network (bridges or switches). Uses Spanning Tree Algorithm create a topology database search out & destroy redundant links.

Switching Loops Broadcast Storms Multiple Copies of a Frame
Database Instability – thrashing the MAC table Multiple Loops Example: PC B sends a Broadcast A redundant topology is used when an administrator requires that a redundant path be available in the event of a switch outage. If switch A should stop functioning, a redundant path will be available through switch B. A very common occurrence in networking is, while fixing one problem, several new problems are introduced. The redundant solution is not an exception to this rule. Switch functions can cause the following problems: Broadcast Storms- Host A sends a packet with the intention of reaching host C. Switch A does see Host C in the Mac Address Table yet and reliably floods the packet out all interfaces. Switch B does not find an entry for Host C either and also floods the packet. Host A receives the packet for the second time and repeats the process. Switch B will repeat the process again as well. A Broadcast Storm has been created and can eventually shut down a switch. Multiple Copies of a Frame- The same scenario for broadcast storms is used, except the packet reaches Host C directly across the Lan segment immediately. Switch A floods the packet and B does the same. Host C will receive Multiple Copies of the same frame. Database Instability- Host A sends a packet to Host C. The source address of Host A is learned on the e0/0 port of both switches. When frame is reliably flooded by both switches the same Mac address will be learned on the e0/1 port on both switches again. Switch A and B would have learned the same Mac Address on two different ports. Database Instability can cause the following problems: 1) Switch A or B will not forward any frames that are sourced from Host A 2) Switch A or B will not forward any frames from anyone A B C D

Rapid PVST+ Rapid per VLAN Spanning Tree (Rapid PVST+) is an updated implementation of Spanning Tree Protocol (STP). One spanning tree topology for each VLAN. IEEE 802.1w standard. An improvement on the original STP standard, 802.1D, which allows faster convergence. Interoperates with the IEEE 802.1D standard, which mandates a single STP instance for all VLANs. Default STP mode on the switch.

Multiple Spanning Tree (MST)
IEEE 802.1s standard Allows you to assign two or more VLANs to a spanning tree instance. Not the default spanning tree mode on Nexus 5000. MST maps multiple VLANs into a spanning tree instance, with each instance having a spanning tree topology independent of other spanning tree instances. Provides multiple forwarding paths for data traffic, enables load balancing, and reduces the number of STP instances required to support a large number of VLANs. Improves the fault tolerance of the network because a failure in one instance (forwarding path) does not affect other instances (forwarding paths).

Cisco Nexus 5000 Architecture
90

Hardware Architecture
The Cisco Nexus 5000 Series uses a scalable cut-through input queuing switching architecture. The architecture is implemented primarily by two ASICs developed by Cisco: A set of unified port controllers (UPCs) that perform data plane processing -- Each UPC manages 4 ports of 10 Gigabit Ethernet and makes forwarding decisions for the packets received on those ports. After a forwarding decision is made, the packets are queued in VOQs, waiting to be granted access to the UCF. The UPC handles all packet processing operations within the Cisco Nexus 5000 Series server switch. It is an L2MP device with the capability to operate simultaneously and at wire speed with the following protocols: Classical Ethernet, Fibre Channel, FCoE. A unified crossbar fabric (UCF) that cross-connects the UPCs

Virtual Output Queues

Policy Enforcement Switch Port Analyzer (SPAN) and Diagnostic Sampling Control Plane Redirect/Snooping VLAN Membership Check Frames evaluated by multistage engine searches occur in parallel results, and are evaluated in pipeline diagnostics, and control plane tap pipelines. pass fail Interface, VLAN, and MAC Binding pass fail MAC and L3 Binding (IP & Fibre Channel) pass fail Fibre Channel Zone Membership Check pass fail Port ACLs permit deny VLAN ACLs (ingress) Cisco Nexus 5000 Series UPC follows a strict set of comprehensive rules to help ensure that packets get forwarded or dropped based on the intended configuration. The multistage policy engine is responsible for this step and manipulates the forwarding results with a combination of parallel searches in memory arrays, hash tables, and ternary content-addressable memory (TCAM). The parallel search results are then evaluated and prioritized in a pipeline to create a final policy decision of ACL permit, ACL deny, QoS policing, redirect, or Switched Port Analyzer (SPAN) replication. Specifically, ACLs are implemented in 1-Mb TCAM located on each UPC, offering 2048 match access control entries, each 432 bits wide. permit deny Role Based ACLs (egress) permit deny QoS ACLs (ingress) permit policer drop Multipath Expansion To SPAN session To Sup

Quality of Service on the Nexus 5000
94

Hardware QoS Processing Flow
If buffer usage crosses threshold: Tail drop for drop class Assert pause signal to MAC for no-drop system class VoQs for unicast (8 per egress port) Central Scheduler Trust CoS L2/L3/L4 info with ACL Traffic Classification STOP Per-class Buffer usage Monitoring Ingress policing MTU checking MAC 8 muticast queues Truncate or drop packets if MTU is violated PAUSE ON/OFF signal Egress Queues Unicast and multicast follow the same QoS processing flow. The difference is that unicast and multicast have separate queues at ingress. Marking Egress Scheduler Mark packet with CoS value Strict priority + DWRR scheduling

Default QoS Configuration
Default Policy-Map Default Class-Map Qos is always on. Four default class of services defined when system boots up Two for control traffic. One for FCoE traffic and another one for Ethernet traffic Match CoS 3 for class-fcoe. Class-fcoe is no-drop with MTU 2240. Match any for class-default Class-fcoe and class-default get 50% of guaranteed bandwidth by default switch1# sh policy-map Type qos policy-maps ==================== policy-map type qos default-in-policy class type qos class-fcoe set qos-group 1 class type qos class-default set qos-group 0 Type queuing policy-maps ======================== policy-map type queuing default-in-policy class type queuing class-fcoe bandwidth percent 50 class type queuing class-default policy-map type queuing default-out-policy Type network-qos policy-maps =============================== policy-map type network-qos default-uf-policy class type network-qos class-fcoe pause no-drop mtu 2240 class type network-qos class-default mtu 1538 switch2# show class-map Type qos class-maps =================== class-map type qos class-fcoe match cos 3 class-map type qos class-default match any Type queuing class-maps ======================= class-map type queuing class-fcoe match qos-group 1 class-map type queuing class-default match qos-group 0 Type network-qos class-maps ============================== class-map type network-qos class-fcoe class-map type network-qos class-default

QoS Configuration — MQC
MQC(Modular QoS CLI) defines three-step configuration model Define matching criteria class-map Associate action with each defined class policy-map Apply policy to entire system or an interface service-policy NX-OS supports Cisco Modular QoS CLI.

QoS Configuration Steps
1 Configure and apply qos policy-map Define packet classification rules MQC three steps configuration: class-map policy-map and then activate policy-map 2 Configure and apply network-qos policy-map Define various QoS parameters: class type(drop, no-drop), MTU, buffer size, CoS marking 3 Configure and apply queuing policy-map Define how the bandwidth are shared among different classes of service: strict priority and DWRR Output queuing policy for egress interface configures local bandwidth sharing policy Input queuing policy for ingress interface is the policy for network device connected to Nexus 5000

Nexus 5000 Software Features Set
802.1w (Rapid Spanning Tree), 802.1s (Multiple Spanning Tree), RPVST+, Root Guard, Uplink Guard, Bridge Assurance, PortFast, CDP, PVLANs, UDLD, LACP, IGMP Snooping, 802.1Q trunks, Port-Channel, SVI, SPAN, Jumbo Frames, NTP, Link State Tracking (LST) Layer 2 Radius, Tacacs+, AAA, CallHome, SSHv1/V2, telnet, IPv4 & IPv6 mgmt, SNMP MiBs, Traps, EthAnalyzer (wireshark), RBAC, DCNM, RME support via Cisco Works, syslog, coredump, RMON, first-setup script, accounting log, checkpoint and configuration rollback Management/ Security PACLs, VACLs, Session based ACLs, ACL based QOS, egress Bandwidth Limiting, 802.1p priority, strict priority scheduling, WRED, Tail Drop, Storm Control (broadcast, multicast), Egress Shaper ACL/QOS FIP Snooping Bridge, DCBXP, PFC (Priority Flow Control), 8 Virtual Lanes, ETS (Enhance Transmission Selection) FCOE

Nexus 5000 and FC Connectivity
100

Switch Mode Nexus 5000 FC module can be ISL’ed to another FC switch (E_port) Zoning, DPVM, etc. are enforced on the Nexus 5000 Domain manager, FSPF, zone server, fabric login server, name server run on Nexus 5000 Require a domain ID for every VSAN Interop mode considerations when connecting to non-Cisco FC switches Note: Nexus 5000 supports direct connectivity to FC initiator (server HBAs) and targets (storage arrays) DPVM = Dynamic Port VSAN Membership

N-Port Virtualization (NPV) mode
Nexus 5000 FC module can work in NPV mode Server-facing ports are regular F ports Uplinks toward SAN core fabric are NP ports Nexus 5000 switches assign FCIDs to attached devices First byte in FCID received from core SAN switch One VSAN per uplink on Nexus 5000 (will change in future) No trunking or channelling of NP ports Zoning, DPVM, etc. are not enforced on the Nexus 5000 Domain manager, FSPF, zone server, fabric login server, name server They do not run on Nexus 5000 No local switching All traffic routed via the core SAN switches

N-Port Virtualization (NPV): An Overview
NPV-Core Switch (MDS or 3rd party switch with NPIV support) FC F-port VSAN 5 NP-port VSAN 10 Can have multiple uplinks – one VSAN per uplink Two uplinks can be in the same VSAN No port channel or trunking F-ports Host Host N-ports Host Nexus 5000 to SAN Fabric A & B Assign FCIDs to servers – no domain to configure! Servers log in (FLOGI) locally

NX-OS CLI 104 104

CLI Two main command modes:
Description How to Access Prompt EXEC Enables you to temporarily change terminal settings, perform basic tests, and display system information At the switch prompt, enter the required EXEC mode command. switch# Config mode Enables you to configure features that affect the system as a whole. From EXEC mode, enter the configure terminal command. switch(config)# display the commands available in any command mode by typing a question mark (?)

Config Mode switch# configure terminal switch(config)# ?
aaa Configure aaa functions banner Configure banner message boot Configure boot variables callhome Enter the callhome configuration mode cdp Configure CDP parameters cfs CFS configuration commands class-map Configure class-map . switch(config)# interface ethernet 1/1 switch(config-if)# ? bandwidth Set bandwidth informational parameter cdp Configure CDP interface parameters channel-group Add to/remove from a port-channel delay Specify interface throughput delay

Nexus 5000 Configuration 107 107

Basic System Configuration (setup)
---- System Admin Account Setup ---- Enter the password for "admin”: <Type in password> Confirm the password for "admin": <retype password> ---- Basic System Configuration Dialog ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Would you like to enter the basic configuration dialog (yes/no): yes

Basic System Configuration Continued…
Create another login account (yes/no) [n]: n Configure read-only SNMP community string (yes/no) [n]: n Configure read-write SNMP community string (yes/no) [n]: n Enter the switch name : bch1-n4k-b7 Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: y Mgmt0 IPv4 address : Mgmt0 IPv4 netmask : Configure the default gateway? (yes/no) [y]: y IPv4 address of the default gateway : Enable the telnet service? (yes/no) [y]: y Enable the ssh service? (yes/no) [n]: y Type of ssh key you would like to generate (dsa/rsa) : rsa Number of key bits < > : 1024 Configure the ntp server? (yes/no) [n]: n Enable FCOE service? (yes/no) [y]: y Enable FCoE FIP Snooping service (Recommended for FCoE)? (yes/no) [y]: y INFO: Please enable FIP Snooping and fc_map on each FCoE Vlan individually Optional

Basic System Configuration Continued…
The following configuration will be applied: switchname bch1-n4k-b7 interface mgmt0 ip address no shutdown exit vrf context management ip route / telnet server enable feature fip-snooping ssh key rsa 1024 force ssh server enable Would you like to edit the configuration? (yes/no) [n]: n Use this configuration and save it? (yes/no) [y]: y [########################################] 100% Nexus 3000 Switch bch1-n4k-b7 login:

Verifying Boot Variables
Validate “boot variables” are set correctly bch1-n4k-b7# show boot Current Boot Variables: kickstart variable = bootflash:/taishan_kickstart E bin system variable = bootflash:/taishan_system E bin Boot Variables on next reload: Note: Final release image will have different naming convention

Verifying Management Ports
Validate management port configuration bch1-n4k-b7# show interface mgmt 0

Working with Nexus 2148 (Optional)
113 113

Nexus 2000 Fabric Extender Virtual Chassis
114

Nexus 2000 Fabric Extender 1GE Connectivity

Fabric Extender Uplink Modes
Static Pinning Fabric Extender associates (pins) a server side (1GE) port with an uplink (10GE) port Server ports are either individually pinned to specific uplinks (static pinning) or all interfaces pinned to a single logical port channel Behaviour on FEX uplink failure depends on the configuration Static Pinning – Server ports pinned to the specific uplink are brought down with the failure of the pinned uplink Port Channel – Server traffic is shifted to remaining uplinks based on port channel hash Server Interface goes down Port Channel Server Interface stays active 116

How to Configure FCoE on Nexus 5000
117 117

Remotely Connected CNA Designs

Best Practices Topology
A unique dedicated VLAN must every VSAN If MSTP is enabled, a separate MST instance must be used for FCoE VLANs Unified links must be configured as trunk ports. FCoE VLAN must not be configured as a native VLANs Unified links between the CNAs and the blade switches must be configured as spanning-tree edge ports. A blade switch must connect to exactly one converged access switch, preferably over an Ethernet port-channel to avoid disruption due to STP re-convergence on events like provisioning of new links or blade switches.

FC-MAP identifier in a Nexus 5000
It is recommended each fabric be configured with a unique FC-MAP identifier. This is required to prevent data corruption due to cross-fabric talk. 5020-1(config)# fcoe fcmap <0xefc00-0xefcff> Enter FCMAP 5020-1# show fcoe Global FCF details FCF-MAC is 00:0d:ec:a4:02:80 FC-MAP is 0e:fc:00 FCF Priority is 128 FKA Advertisement period for FCF is 8 seconds FC-MAP (FCoE MAC Address Prefix) is a configurable parameter on the FCF, provided to Enode in the FIP advertisement.

FCF priority Setting MN-5020-1(config)# fcoe ?
fcf-priority FCF priority specification fcmap FC MAP specification fka-adv-period FCoE Keepalive Advertisement Period MN # show fcoe Global FCF details FCF-MAC is 00:0d:ec:a4:02:80 FC-MAP is 0e:fc:00 FCF Priority is 128 FKA Advertisement period for FCF is 8 seconds

Show fcoe Qlogic output

Switch Upgrade 124 124

Switch Configuration Relevant Config Files: Two important images:
startup-config running-config Two important images: kickstart system Use “install all” to upgrade operating system image Storing images: bootflash: -- nonvolatile storage, i.e. directory which provides permanent storage. The files in bootflash are preserved through reboots and power outages. volatile: -- directory which provides temporary storage, and is also the default. Files in temporary storage (volatile:) are erased when the switch reboots

Role Base Access Control (RBAC)
126 126

RBAC Allows one to create and manage user accounts and assign roles to limit access Default Users: network-admin (superuser)—Complete read and write access to the entire Cisco Nexus 5000 Series switch. network-operator—Complete read access to the Cisco Nexus 5000 Series switch If you belong to multiple roles, you can execute a combination of all the commands permitted by these role. One can define user role policies to limit the switch resources that the user can access Can limit access to interfaces, VLANs and VSANs

Quality of Service (QoS)
128 128

Default QoS Configuration
Default Policy-Map Default Class-Map Qos is always on. Four default class of services defined when system boots up Two for control traffic. One for FCoE traffic and another one for Ethernet traffic Match CoS 3 for class-fcoe. Class-fcoe is no-drop with MTU 2240. Match any for class-default Class-fcoe and class-default get 50% of guaranteed bandwidth by default switch1# sh policy-map Type qos policy-maps ==================== policy-map type qos default-in-policy class type qos class-fcoe set qos-group 1 class type qos class-default set qos-group 0 Type queuing policy-maps ======================== policy-map type queuing default-in-policy class type queuing class-fcoe bandwidth percent 50 class type queuing class-default policy-map type queuing default-out-policy Type network-qos policy-maps =============================== policy-map type network-qos default-uf-policy class type network-qos class-fcoe pause no-drop mtu 2240 class type network-qos class-default mtu 1538 switch2# show class-map Type qos class-maps =================== class-map type qos class-fcoe match cos 3 class-map type qos class-default match any Type queuing class-maps ======================= class-map type queuing class-fcoe match qos-group 1 class-map type queuing class-default match qos-group 0 Type network-qos class-maps ============================== class-map type network-qos class-fcoe class-map type network-qos class-default

Other: Nexus 5000 Troubleshooting

Nexus 5000 Supported FC Port Types
E Port - In expansion port (E port) mode, an interface functions as a fabric expansion port. This port may be connected to another E port to create an Inter-Switch Link (ISL) between two switches. TE Port - In trunking E port (TE port) mode, an interface functions as a trunking expansion port. It may be connected to another TE port to create an Enhanced ISL (EISL) between two switches. TE ports are specific to Cisco MDS 9000 and Nexus 5020 Family switches F Port - In fabric port (F port) mode, an interface functions as a fabric port. This port may be connected to a peripheral device (host or disk) operating as an N port. An F port can be attached to only one N port. F ports support class 2 and class 3 service. Auto Mode - Interfaces configured in auto mode can operate in one of the following modes: F port, E port, or TE port. The port mode is determined during interface initialization. SD Port- In SPAN destination port (SD port) mode, an interface functions as a switched port analyzer (SPAN). NP Port - An NP port is a port on a device that is in NPV mode and connected to the core NPV switch through an F port. NP ports operate like N ports that function as proxies for multiple physical N ports.

Identifying the Optics – SFP+
n5k# show interface ethernet 1/15 transceiver details Ethernet1/15 sfp is present name is CISCO-EXCELIGHT part number is SPP5101SR-C1 revision is A serial number is ECL123506A7 nominal bitrate is MBits/sec Link length supported for 50/125um fiber is 82 m(s) Link length supported for 62.5/125um fiber is 26 m(s) cisco id is -- cisco extended id number is 4 SFP Detail Diagnostics Information (external calibration) Alarms Warnings High Low High Low Temperature C C C C C Voltage V V V V V Current mA mA mA mA mA Tx Power dBm dBm dBm dBm dBm Rx Power dBm dBm dBm dBm dBm Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

Identifying the Optics – CX1
Need to show CX1 output Invalid calibration bch1-n4k-b7# show int eth1/19 transceiver details Ethernet1/19 sfp is present name is Molex Inc. part number is revision is serial number is nominal bitrate is MBits/sec cisco id is -- cisco extended id number is 4 SFP Detail Diagnostics Information (external calibration) Alarms Warnings High Low High Low Temperature C C C C C Voltage V V V V V Current mA mA mA mA mA Tx Power N/A N/A N/A N/A N/A Rx Power N/A N/A N/A N/A N/A Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning Transceiver Details not available with copper

Displaying Ethernet Interface Counters
n5k# show interface ethernet 1/15 Ethernet1/15 is up Hardware: 1000/10000 Ethernet, address: 0005.ad00.3e7e (bia 0005.ad00.3e7e) MTU 1500 bytes, BW Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA Port mode is trunk full-duplex, 10 Gb/s, media type is 10g Input flow-control is off, output flow-control is off Rate mode is dedicated Switchport monitor is off Last link flapped 11:59:10 Last clearing of "show interface" counters never 1 minute input rate 0 bits/sec, 0 packets/sec 1 minute output rate 0 bits/sec, 0 packets/sec Rx 7516 input packets 0 unicast packets 7516 multicast packets 0 broadcast packets 0 jumbo packets 0 storm suppression packets bytes Tx 2164 output packets 2164 multicast packets 0 broadcast packets 0 jumbo packets bytes 0 input error 0 short frame 0 watchdog 0 no buffer 0 runt 0 CRC 0 ecc 0 overrun 0 underrun 0 ignored 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble 0 input discard 0 output error 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 Rx pause 0 Tx pause 1 interface resets 134

FIP-Snooping Information
n5k# show fip-snooping ? enode Configure ENode interface fcf Configure FCF interface internal Show internal fip-snooping information sessions FIP snooped sessions betweeen ENode and FCF stats Display fip-snooping statistics 135

FIP-Snooping Information
bch1-n4k-b7# show fip-snooping enode Legend: Interface VLAN NODE NAME FIP MAC FCOE MAC FC_MAP Eth1/ :00:00:c0:d 00:c0:dd:04:0d:11 0e:fc:00:16:00:01 0x0efc00 d:04:0d:11 bch1-n4k-b7# show fip-snooping fcf Interface VLAN No of FPMA/ FCMAP FCF-MAC Switch WWN Fabric Name Enodes SPMA por FPMA 0x0efc00 00:0d:ec:b1:1f:c0 20:01:00:0d: 20:01:00:0d: ec:b1:1f:c1 ec:5f:d0:81 136

N5K FCoE Troubleshooting

Fabric Login Server The destination address for fabric login (FLOGI) requests Located at address 0x’FF FF FE’ Can also be used for the Discover Fabric Service Parameters (FDISC), Read Link Error Status Block (LESB), and Read Timeout Values (RTV) commands.

Basic Troubleshooting - FLOGI
Is the device logged into the fabric? pod # show flogi database INTERFACE VSAN FCID PORT NAME NODE NAME vfc xeb :00:00:c0:dd:12:04:f3 20:00:00:c0:dd:12:04:f3 Total number of flogi = 1

What’s on the wire? Tracing the FLOGI
Fibre Channel R_CTL: 0x22 Dest Addr: ff.ff.fe CS_CTL: 0x00 Src Addr: Type: Ext Link Svc (0x01) F_CTL: 0x (Exchange Originator, Seq Initiator, Exchg First, Seq Last, Transfer Seq Initiative, Last Data Frame - No Info, ABTS - Abort/MS, ) SEQ_ID: 0x00 DF_CTL: 0x00 SEQ_CNT: 0 OX_ID: 0x008e RX_ID: 0xffff Parameter: 0x FC ELS Cmd Code: FLOGI (0x04) Common Svc Parameters B2B Credit: 4 Common Svc Parameters: 0x0 (Alt B2B Credit Mgmt) = BB_SC Number: 0 = Receive Size: 2048 Max Concurrent Seq: 18 Relative Offset By Info Cat: 20352 E_D_TOV: 0 N_Port Port_Name: 10:00:00:01:73:00:94:84 (00:01:73) Fabric/Node Name: 10:00:00:01:73:00:94:84 (00:01:73) Class 1 Svc Parameters Service Options: 0x0(Class Not Supported) Class 2 Svc Parameters Class 3 Svc Parameters Service Options: 0x8800(Seq Delivery Requested) Initiator Control: 0x0(Seq Delivery Requested) Recipient Control: 0x0(Seq Delivery Requested) Class Recv Size: 0 Total Concurrent Seq: 0 End2End Credit: 0 Open Seq Per Exchg: 0 Class 4 Svc Parameters Vendor Version: Notice SID F_CTL bits Class 3 only

Response back (ACC) from FLOGI
Fibre Channel R_CTL: 0x23 Dest Addr: 7e.00.00 CS_CTL: 0x00 Src Addr: ff.ff.fe Type: Ext Link Svc (0x01) F_CTL: 0x (Exchange Responder, Seq Initiator, Exchg Last, Seq Last, CS_CTL, Last Data Frame - No Info, ABTS - Abort/MS, ) SEQ_ID: 0x00 DF_CTL: 0x00 SEQ_CNT: 0 OX_ID: 0x008e RX_ID: 0x3d32 Parameter: 0x FC ELS Cmd Code: ACC (0x02) Common Svc Parameters B2B Credit: 12 Common Svc Parameters: 0x1300 (Alt B2B Credit Mgmt) = BB_SC Number: 0 = Receive Size: 2112 Max Concurrent Seq: 0 Relative Offset By Info Cat: 10000 E_D_TOV: 2000 N_Port Port_Name: 20:41:00:05:30:00:51:1e (00:05:30) Fabric/Node Name: 20:01:00:05:30:00:51:1f (00:05:30) Class 1 Svc Parameters Service Options: 0x0(Class Not Supported) Class 2 Svc Parameters Service Options: 0x8800(Seq Delivery Requested) Initiator Control: 0x0(Seq Delivery Requested) Recipient Control: 0x0(Seq Delivery Requested) Class Recv Size: 0 Total Concurrent Seq: 0 End2End Credit: 0 Open Seq Per Exchg: 0 Class 3 Svc Parameters Class 4 Svc Parameters Vendor Version: Notice DID F_CTL bits Class 2 & 3

Sample N5K FLOGI Action Debug
N5K# debug flogi action interface vfc23/1 Shut here 2008 Aug 12 17:26: flogi: fs_add_del_flogi_in_vsan_fcid_hash: Fcid 0xea0000 delete from vsan 5 .. Successful. Has 0 more fcids 2008 Aug 12 17:26: flogi: fs_should_free_fcid: if[0x1a000b00] vsan[0x5] fcid[ea00] fcidrcvd[1] 2008 Aug 12 17:26: flogi: fs_should_free_fcid: if[0x1a000b00] vsan[0x5] num_alloc[1] 2008 Aug 12 17:26: flogi: fs_should_free_fcid: if[0x1a000b00] vsan[0x5] num_alloc[0] decrement 2008 Aug 12 17:26: flogi: FCID free req to DM mask: SINGLE. 2008 Aug 12 17:26: flogi: FCID free to DM for flogi(0x812d79c) 2008 Aug 12 17:26: flogi: fs_set_nx_port_login_status: Fcid 0xea0000 logged out of vsan 5. 2008 Aug 12 17:26: flogi: All ports logged out of port 0x1a000b00. 2008 Aug 12 17:26: flogi: fs_update_port_vsan_flogi_info: Update vsan[0x5] flogi info(1) 2008 Aug 12 17:26: flogi: Cleared fcid 0x info in fc2. 2008 Aug 12 17:26: flogi: flogi_ac_group_cleanup_flogi_done: if[0x5] vsan[0x1a000b00] 2008 Aug 12 17:26: flogi: Group cleanup for 1 flogis 2008 Aug 12 17:26: flogi: Entry count 0 with ifindex 0x1a000b00(0x1a000b00) and vsan 5(5). 2008 Aug 12 17:26: flogi: Bulk cleanup for 1 entries in vsan 5 ifindex vfc23/1 2008 Aug 12 17:26: flogi: fs_update_fcredirect: fcredirect conf not needed if[1a000b00] 2008 Aug 12 17:26: flogi: fs_flogi_send_sync_flogi_message: if[0x1a000b00] vsan[0x5] fc2_req[1] fcid[0] 2008 Aug 12 17:26: flogi: Dropping MTS wrap MTS_OPC_PM_CLEANUP_PORT_STATE(333) id 2008 Aug 12 17:26: flogi: Last ref of MTS_OPC_PM_CLEANUP_PORT_STATE(333) id

Sample N5K FLOGI Action Debug continued
switch# debug flogi action int fc1/1 No shut here: 2008 Aug 12 17:29: flogi: fs_set_flogi_for_wwn_port: table not pres if[1a000b00] vsan[5] 2008 Aug 12 17:29: flogi: fs_set_flogi_for_wwn_port: table not pres if[1a000b00] vsan[5] 2008 Aug 12 17:29: flogi: fs_set_port_vsan_ht: if[1a000b00] vsan[5] type[0] key[3F F1F0 01A 0 B 0] 2008 Aug 12 17:29: flogi: Flogi allowed for port. FLOGI_EV_VALID_FLOGI 2008 Aug 12 17:29: flogi: flogi_ac_copy_flogi_and_generate_event: new_event[0xf] if[0x1a000b00] vsan[0x5] 2008 Aug 12 17:29: flogi: (Error) Port-Security service not running sending port binding request 2008 Aug 12 17:29: flogi: setting first_flogi_timer_started to TRUE 2008 Aug 12 17:29: flogi: flogi_ac_pre_config_check_seq_start: timer_rcode[0] 2008 Aug 12 17:29: flogi: Notify PM of first flogi. 2008 Aug 12 17:29: flogi: FCID 0x allocate req to DM mask: SINGLE. 2008 Aug 12 17:29: flogi: fs_flogi_ac_request_fcid: FCID req to DM for flogi(0x812fb54) 2008 Aug 12 17:29: flogi: fs_assign_fcid_for_flogi: if[0x1a000b00] vsan[0x5] num_alloc[1] increment 2008 Aug 12 17:29: flogi: fs_flogi_send_sync_flogi_message: if[0x1a000b00] vsan[0x5] fc2_req[1] fcid[0] 2008 Aug 12 17:29: flogi: fs_add_del_flogi_in_vsan_fcid_hash: Fcid 0xea0000 add from vsan 5 .. Successful. Has 1 more fcids 2008 Aug 12 17:29: flogi: TLV to LCP sent successfully. 2008 Aug 12 17:29: flogi: fs_flogi_perform_ns_register: sym_name[] if[1a000b00] ns_flags[8a0f] class[c] 2008 Aug 12 17:29: flogi: Init RIB entry successful. 2008 Aug 12 17:29: flogi: fs_update_fcredirect: fcredirect conf not needed if[1a000b00] 2008 Aug 12 17:29: flogi: Added fcid 0xea0000 info in fc2. 2008 Aug 12 17:29: flogi: flogi_ac_flogi_done: timer - rid(0x ), event_id(82), timer_id(1), 2008 Aug 12 17:29: flogi: Sending flogi acc for 0xea0000 2008 Aug 12 17:29: flogi: FC2 flogi resp for vsan 5 ifindex 0x1a000b00 for exchange 0x32a1e66 fcid 0xea0000. 2008 Aug 12 17:29: flogi: Sending fc2 accept. [snip]

N5K FLOGI Debugs - Summary
N5K# debug flogi ? action Configure debugging of flogi actions all Configure all debug flags of flogii demux Configure debugging of flogi message demux error Configure debugging of flogi error event Configure debugging of flogi FSM and Events ha Configure debugging of flogi HA init Configure debugging of flogi adds, deletes and inits timers Configure debugging of flogi message timers trace Configure debugging of flogi trace warning Configure debugging of flogi warning Not enough to just run ethanalyzer Need to run debugs as well to verify FLOGI-ACC exchange

Viewing Fabric Login Service Parameters on N5K
FLOGI Service Parameters Where in CLI?

Basic Troubleshooting – FC Name Server
Is the device logged in and registered with the Fibre Channel Name Server (FCNS)? Other devices in The fabric. Different FCID means they are on a different FC switch pod # show fcns database VSAN 100: FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE 0xe901e2 NL 21:00:00:04:cf:81:c2:f0 (Seagate) scsi-fcp:target 0xe901e4 NL 21:00:00:04:cf:aa:62:bc (Seagate) scsi-fcp:target 0xe901e8 NL 21:00:00:04:cf:81:c8:5f (Seagate) scsi-fcp:target 0xe901ef NL 21:00:00:04:cf:1f:45:49 (Seagate) scsi-fcp:target 0xeb N :00:00:c0:dd:12:04:f3 (Qlogic) scsi-fcp:init Registered that it was an initiator Device shown previously in FLOGI database display

N5K FCNS Debugs - Summary
N5K# debug fcns ? all Debug name server errors Debug name server errors events Debug name server events N5K2# debug fcns events ? mts Debug name server mts events query Debug name server query events register Debug name server registration events

Troubleshooting Displays from a Working Topology

Nexus 5000 Port-Channel Configurations
n5k-2# configure terminal Enter configuration commands, one per line. End with CNTL/Z. n5k-2(config)# feature lacp n5k-2(config)# interface port-channel 1 n5k-2(config-if)#switchport mode trunk n5k-2(config-if)#switchport trunk allow vlan except 30-31 n5k-2(config-if)# interface ethernet 1/ n5k-2(config-if-range)#switchport mode trunk n5k-2(config-if-range)#switchport trunk allow vlan except 30-31 n5k-2(config-if-range)# channel group 1 mode active n5k-2(config)# interface port-channel 300 n5k-2(config)# interface ethernet 1/9 - 10 n5k-2(config-if-range)# channel group 300 mode active n5k-2(config-if-range)# show port-channel summary Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) Group Port Type Protocol Member Ports Channel 1 Po1(SU) Eth LACP Eth1/17(P) Eth1/18(P) Po300(SU) Eth LACP Eth1/9(I) Eth1/10(I)

Nexus 5000 FCoE Configurations: N5K-1 & N5K-2
n5k-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. n5k-1(config)# feature fcoe n5k-1(config)# vlan 30 n5k-1(config-vlan)#fcoe vsan 1 n5k-1(config-vlan)# show vlan fcoe VLAN VSAN Status Operational n5k-2# configure terminal Enter configuration commands, one per line. End with CNTL/Z. n5k-2(config)# feature fcoe n5k-2(config)# vlan 31 n5k-2(config-vlan)#fcoe vsan 1 n5k-2(config-vlan)# show vlan fcoe VLAN VSAN Status Operational

Binding of “vfc” to CNA device on Nexus 5000
n5k-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. n5k-1(config)# interface vfc 104 n5k-1(config-if)# bind mac-address 00:c0:dd:04:0d:11 n5k-1(config-if)# no shut n5k-1(config-if)# show interface vfc104 vfc104 is up Bound MAC is 00:c0:dd:04:0d:11 FCF priority is 128 Hardware is Virtual Fibre Channel Port WWN is 20:67:00:0d:ec:b1:1f:ff Admin port mode is F, trunk mode is on snmp link state traps are enabled Port mode is F, FCID is 0x160001 Port vsan is 1 1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 0 frames input, 0 bytes 0 discards, 0 errors 0 frames output, 0 bytes Interface last changed at Mon Sep 7 05:06: This vfc number is arbitrary This MAC-Address was shown from the Nexus 5000 command

Binding “vfc” from Device Manager

Fabric Manager View

Troubleshooting Tools

What tool captures what?
Ethanalyzer is a tool that will collect frames that are destined to, or originate from the N5K control plane. Node to switch, or switch to switch traffic can be seen with this tool. SPAN (Port Monitor) is a feature whereby frames that are transient to the switch, will be copied to a second port for analysis. Node to switch or node to node traffic can be seen via this method.

Ethanalyzer

Ethanalyzer Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark (formerly Ethereal) open source code. Ethanalyzer is a command-line version of Wireshark that captures and decodes packets. You can use Ethanalyzer to troubleshoot your network and analyze the control-plane traffic.

Ethanalyzer Ethanalyzer does not capture data traffic that Cisco NX-OS forwards in the hardware. Ethanalyzer uses the same capture filter syntax as tcpdump. For more information, see the following For information on the syntax of the display filter, see the following

Ethanalyzer Configuration Examples
N4K(config)# ethanalyzer local interface ? inband Inband/Outband interface mgmt Management interface mgmt-backplane Backplane Management interface N4K(config)# ethanalyzer local interface mgmt brief Capturing on eth2 :41: :50:56:47:50:aa -> ff:ff:ff:ff:ff:ff ARP Who has 17 ? Tell :41: > UDP Source port: 32768 Destination port: 18000 :41: :23:5d:25:bb:0b -> 01:80:c2:00:00:00 STP Conf. Root = 32768/00:23:ab:ad:78:b6 Cost = 4 Port = 0x800b :41: :23:5d:25:bb:0b -> 01:80:c2:00:00:00 STP Conf. Root [snip]

Ethanalyzer Configuration Example - brief
N4K(config)# ethanalyzer local interface inbound brief limit-captured-frames 0 Capturing on eth4 Capturing on inb0 :47: :05:ad:00:3e:7e -> 01:80:c2:00:00:00 STP RST. Root = 32769/00:05:ad:00:3e:86 Cost = 0 Port = 0x9009 :47: :05:ad:00:3e:7e -> 01:00:0c:cc:cc:cd STP RST. Root :47: :05:ad:00:3e:83 -> 01:80:c2:00:00:00 STP RST. Root = 32769/00:05:ad:00:3e:86 Cost = 0 Port = 0x8094 :47: :05:ad:00:3e:74 -> 01:80:c2:00:00:00 STP RST. Root = 32769/00:05:ad:00:3e:86 Cost = 0 Port = 0x8085 :47: :05:ad:00:3e:74 -> 01:00:0c:cc:cc:cd STP RST. Root :47: :05:ad:00:3e:73 -> 01:80:c2:00:00:00 STP RST. Root = 32769/00:05:ad:00:3e:86 Cost = 0 Port = 0x8084 [snip] Spanning Tree Frames FC Frames

SPAN (Port Monitor)

SPAN Sources The Switched Port Analyzer (SPAN) feature selects network traffic for analysis SOURCES SPAN sources refer to the interfaces from which traffic can be monitored Can be Ethernet, vEthernet or Port-Channel Interfaces. Can also span VLANs Ingress source (Rx)—Traffic entering the switch through this source port is copied to the SPAN destination port. Egress source (Tx)—Traffic exiting the switch through this source port is copied to the SPAN destination port. DESTINATIONS SPAN destinations refer to the interfaces that monitors source ports. The Cisco Nexus 4000 Series switch supports Ethernet interfaces as SPAN destinations.

Characteristics of Destination Ports
Each local SPAN session must have a destination port (also called a monitoring port) that receives a copy of traffic from the source ports or VLANs. A destination port has these characteristics: Can be any physical Ethernet port Cannot be a port channel. Does not participate in spanning tree while the SPAN session is active. Is excluded from the source list and is not monitored if it belongs to a source VLAN of any SPAN session. Receives copies of sent and received traffic for all monitored source ports. If a destination port is oversubscribed, it can become congested. This congestion can affect traffic forwarding on one or more of the source ports.

Monitor (SPAN) Caveats
Idiosyncrasies of Nexus5K SPAN: COS values will not be preserved at monitor (span) destination Packets coming in on monitor source with unknown vlan tag will be spanned out with 0 vlan tag (priority tag) For Ethernet destination, monitor session will be up only if the destination port is configured as switchport monitor Out of 18 configurable sessions, only two will be active (up state). The rest will be in down state (hardware resource unavailable). Configuration limitations: Vlan or port-channel cannot be configured as egress source Vlan or port-channel cannot be a monitor destination Only two egress sources supported Only one destination port can be configured for a session

SPAN Configuration You create a SPAN session by assigning a session number using the monitor command. If the session already exists, any additional configuration is added to that session.

SPAN Configuration – Ethernet destination interface
The SPAN destination port can only be a physical port on the switch.

Verifying the SPAN Session
N4K-1(config-if)# show monitor session 1 session 1 type : local state : up source intf : rx : Eth1/4 tx : Eth1/4 both : Eth1/4 source VLANs : rx : destination ports : Eth1/17 Legend: f = forwarding enabled, l = learning enabled source VSANs : rx : rx :

Shutting down a SPAN Session
N4K-1(config)# monitor session 1 shut N5K(config)# show monitor session 1 session 1 type : local state : down (Session admin shut) source intf : rx : Eth1/4 tx : Eth1/4 both : Eth1/4 source VLANs : rx : destination ports : Eth1/17 Legend: f = forwarding enabled, l = learning enabled source VSANs : rx : rx :

Unified Fabric aka FCOE

Similar presentations

Presentation on theme: "Unified Fabric aka FCOE"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Unified Fabric aka FCOE

Similar presentations

Presentation on theme: "Unified Fabric aka FCOE"— Presentation transcript:

Similar presentations

About project

Feedback