Objectives Understand troubleshooting on Nexus 5600 and 6000 platforms

Objectives Understand troubleshooting on Nexus 5600 and 6000 platforms
Provide the tools to effectively resolve any issues © 2011, Cisco Systems, Inc. All rights reserved. BRKCRS-3145

Related sessions BRKARC In-depth and personal with the Cisco Nexus 2000 Fabric Extender Architectures, Features, and Topologies(6/9 8am, 6/10 1pm) BRKARC Cisco Nexus 5600/6000 Switch Architecture(6/11 10am) BRKDCT Nexus 9000/7000/6000/5000 Operations and Maintenance Best Practices(6/9 8am) BRKDCT End-to-End QoS Implementation and Operation with Cisco Nexus Switches(6/9 1pm) BRKDCT Network visibility using advanced Analytics in Nexus switches(6/9 3:30pm) BRKDCT VPC Best Practices and Design on NX OS(6/8 10am) BRKDCT FabricPath Operation and Troubleshooting(6/10 8am) © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Troubleshooting Nexus 5600/6000
Problem Isolation Network Diagrams Types of logging Outputs When to call TAC Platform Overview and troubleshooting © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Troubleshooting Tool #1
RSTP Root A current, accurate diagram vPC peer-link e1/2, 2/2 Po100 Domain 100 N7k-1 N7k-2 vPC peer-keep e1/1 - e1/1 Physical ports Logical ports Spanning-tree root and blocked ports Fabric path switch IDs Helpful to use standard formats .jpg, .bmp, .pdf e3/1 e4/1 e3/1 e4/1 e3/2 e4/2 e3/2 e4/2 vPC po1 vPC Po2 e1/30 e1/31 e1/30 e1/31 e1/30 e1/31 e1/30 e1/31 N5k-1 N5k-2 N5k-3 N5k-4 N5k-5 vPC peer-link e1/1, 1/2 Po101 Domain 101 vPC peer-link e1/1, 1/2 Po102 Domain 102 e1/10 - e1/10 e1/12 - e1/12 STP BLK If you cannot describe how your network operates, time may be wasted © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

“Just grab a show-tech”
Or not… Helpful, but can be too general Large file, time consuming collection and analysis If time permits, also collect targeted outputs or a specific show tech If there is little time, use tac-pac and copy off Much quicker than transmitting to terminal Zips entire output to file in volatile: by default Copy file off of switch for analysis esc-5672-left# tac-pac bootflash esc-5672-left# dir bootflash: | inc show_tech Apr 16 13:18: show_tech_out.gz © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Which Show Tech? As of 7.1(1)N1(1), there are over 110 flavors, more with features esc-5672-left# show tech-support ? <CR> > Redirect it to a file >> Redirect it to a file in append mode aaa Display aaa information aclmgr ACL commands adjmgr Display Adjmgr information arp Display ARP information ascii-cfg Show ascii-cfg information for technical support personnel assoc_mgr Gather detailed information for assoc_mgr troubleshooting auto-config Detailed information related to auto configuration bcm-usd Gather detailed information for BCM USD troubleshooting bgp Display BGP status and configuration <snip> vvlan Gather detailed information for Voice VLAN troubleshooting xml Gather information for xml trouble shooting xos Cross-OS Library Information and Traces zone Show information for zoneserver technical support © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Logging Have clocks synchronized to common source (NTP)
show logging logfile Basis for tracing events chronologically Try using start-time or last show accounting log Basis for tracing configuration changes All commands end with (SUCCESS) or (FAILURE) esc-5672-left# show logging logfile start-time 2015 Apr 16 09:30:00 2015 Apr 16 09:39:57 esc-5672-left %ETHPORT-5-SPEED: Interface Ethernet2/2, operational speed changed to 40 Gbps esc-5672-left# show logging last ? <1-9999> Enter number of lines to display © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Redirect and Append esc-5672-left# show clock > bootflash:debug-file.txt esc-5672-left# show mac address-table >> bootflash:debug-file.txt esc-5672-left# show running-config | count >> bootflash:debug-file.txt esc-5672-left# show file bootflash:debug-file.txt 13:29: EDT Thu Apr Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports ad8.c dynamic F F Po500 * ad8.c dynamic F F Eth101/1/2 845 © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Other System Logs show logging nvram
Persistent logging survives reloads – helpful for crash or reload issues. esc-5672-left# show logging nvram 2015 Apr 13 10:42:41 esc-5672-left %$ VDC-1 %$ %SATCTRL-FEX199-2-SOHMS_ENV_ERROR: FEX-199 Module 1: Check environment alarms. 2015 Apr 13 10:43:23 esc-5672-left %$ VDC-1 %$ last message repeated 1 time 2015 Apr 13 10:43:34 esc-5672-left %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 199 is online 2015 Apr 13 10:43:34 esc-5672-left %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-199 On-line 2015 Apr 13 10:43:35 esc-5672-left %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 199 is online 2015 Apr 13 10:43:38 esc-5672-left %$ VDC-1 %$ %ZONE-2-ZS_MERGE_FAILED: %$VSAN 100%$ Zone merge failure, isolating interface fc1/47 reason: Zoning modes does not match:[reason:0] 2015 Apr 13 10:56:12 esc-5672-left %$ VDC-1 %$ %ZONE-2-ZS_MERGE_FAILED: %$VSAN 100%$ Zone merge failure, isolating interface san-port-channel 100 reason: Zoning modes does not match:[reason:0] 2015 Apr 13 13:25:41 esc-5672-left %$ VDC-1 %$ %SATCTRL-FEX199-2-SATCTRL_IMAGE: FEX199 Image update in progress. 2015 Apr 13 13:31:07 esc-5672-left %$ VDC-1 %$ %SATCTRL-FEX199-2-SATCTRL_IMAGE: FEX199 Image update complete. Install pending ----- Meeting Notes (7/6/11 18:24) ----- logging level on nvram logs, lvl 2 and above © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

When to call TAC Most efficient if you have the following:
A description of the problem observed, with evidence / clues, along with time and scope A current network diagram Any targeted outputs, near the time of the problem Earlier is better than later! Don’t forget release notes and bug toolkit Cisco Support Community sooner than later © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Troubleshooting Nexus 5600 / 6000
Problem Isolation Platform Overview and Troubleshooting NX-OS tips FSM MTS Crashes CPU/Ethanlyzer CRC errors Forwarding Buffering/Queuing Troubleshooting tools ELAM PACL Software upgrades © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

NX-OS operation tips Support for tab auto-complete within current context, but commands will execute at higher levels if available. Filesystems dynamically auto-complete esc-5672-left# (config-if)# switch? switchport Configure switchport parameters <=== matching in config-if mode esc-5672-left# (config-if)# switchn? *** No matching command found in current mode, matching in (config) mode *** WORD Enter switchname (Max Size 32)<=== matching in config mode esc-5672-left(config)# show file bootflash:s? bootflash:stp.log.1 esc-5672-left# install all system bootflash:n6000-uk9.<tab> bootflash:n6000-uk N1.1.bin bootflash:n6000-uk N1.1a.bin bootflash:n6000-uk N1.1.bin bootflash:n6000-uk N1.1b.bin bootflash:n6000-uk N1.1.bin esc-5672-left# install all system bootflash:n6000-uk9.7. © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

NX-OS operation tips CLI list and grep ctrl-c terminates output
esc-5672-left# show cli list | grep switchport show system default switchport san show interface switchport show interface <if-mr> switchport esc-5672-left# show tech-support ---- show tech-support ---- ctrl-c esc-5672-left# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

NX-OS operation tips Mounts could fill and can cause upgrade errors, unexpected logs esc-5672-left# show system internal flash Mount-on K-blocks Used Available Use% Filesystem / /dev/root /proc proc /post none /var none /sys none /isan none /var/tmp none esc-5672-left# install all kickstart bootflash:n6000-uk9-kickstart N1.1.bin system bootflash:n6000-uk N1.1.bin Pre-upgrade check failed. Return code 0x (free space in the filesystem is below threshold). esc-5596-left# Many of these can be deleted safely but contact TAC © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

NX-OS operation tips volatile: filesystem is virtual, use as scratch if needed Will not survive a reload log: filesystem is in root / esc-5672-left# debug logfile CiscoLive_debugs esc-5672-left# debug spanning-tree bpdu_rx tree 10 esc-5672-left# dir log: Apr 16 14:08: CiscoLive_debugs 31 Apr 14 14:38: dmesg 0 Apr 14 14:39: libfipf.3842 esc-5672-left# undebug all esc-5672-left# show debug logfile CiscoLive_debugs 2015 Apr 16 14:08: stp: BPDU RX: vb 1 vlan 10, ifi 0x b (port-channel572) 2015 Apr 16 14:08: stp: BPDU Rx: Received BPDU on vb 1 vlan 10 port port-channel572 pkt_len 64 bpdu_len 42 netstack flags 0x80000ed enc_type sstp 2015 Apr 16 14:08: stp: RSTP(10): msg on port-channel572 esc-5672-left# copy log:CiscoLive_debugs tftp: © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

FSM NX-OS records the finite state machine for many important processes Using this event-history of FSM states and triggers, debugging can be done after a problem has occurred. Important to compare timestamps and watch for inter-process communication. Some common processes: ethpc – ethernet port client: responsible for talking to the mac and phy; ethpm – ethernet port manager: responsible for translating between configuration and ethpc. ethpc would inform ethpm that link is up, and then ethpm will proceed to give instructions on what the configuration is for the port port-channel – port-channeling process responsible for aggregating physical links into logical channels lacp – 802.3ad standard for aggregating links fwm – forwarding manager; responsible for programming hardware according to the software configuration © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

FSM Sometimes it is enough to look at one process FSM, other times you are looking for related events. Timestamps should line up when there is correlation. Example: An LACP Po12 flapped and we are tasked to find out why 2015 Apr 18 08:06:03 esc-5672-left %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel12: first operational port changed from Ethernet1/12 to none 2015 Apr 18 08:06:03 esc-5672-left %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel12 is down (No operational members) <snip> 2015 Apr 18 08:06:18 esc-5672-left %ETH_PORT_CHANNEL-5-PORT_UP: port-channel12: Ethernet1/12 is up 2015 Apr 18 08:06:18 esc-5672-left %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel12: first operational port changed from none to Ethernet1/12 © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

FSM Po12 flapped due to lack of LACP PDU timeout
show system internal ethpm event-history interface ethernet 1/5 369) FSM:<Ethernet1/12> Transition at usecs after Sat Apr 18 08:06: Previous state: [ETH_PORT_FSM_ST_BUNDLE_MEMBER_UP] Triggered event: [ETH_PORT_FSM_EV_EXTERNAL_REINIT_NO_FLAP_REQ] Next state: [FSM_ST_NO_CHANGE] show lacp internal event-history interface ethernet 1/5 64) FSM:<Ethernet1/12> Transition at usecs after Sat Apr 18 08:06: Previous state: [LACP_ST_PORT_MEMBER_COLLECTING_AND_DISTRIBUTING_ENABLED] Triggered event: [LACP_EV_RECEIVE_PARTNER_PDU_TIMED_OUT] Next state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED] Po12 flapped due to lack of LACP PDU timeout © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

MTS NX-OS uses Message and Transaction Service(MTS) to communicate between processes. Useful to check when troubleshooting high CPU unresponsive CLI / timeout control-plane disruption When troubleshooting a process, we may look for specific MTS messages queued. MTS messages may be coming in too fast, or there could be a message stuck at the top of the queue © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

MTS Observed impact is various show commands timing out such as show log and show run Interface commands not accepted Leaf-1(config)# int ethernet 126/1/47 Leaf-1(config-if)# shutdown Please check if command was successful using appropriate show commands Leaf-1# show system internal mts buffers MTS buffers in use = 81 Leaf-1# show system internal mts buffers summary node sapno recv_q pers_q npers_q log_q sup sup sup sup sup Leaf-1# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

MTS persistant queue is generally seen growing old
Ethpm is stuck on Ethpc during link down event but not released Root cause in this case was a bug triggered during link flap event Leaf-1# show system internal mts buffers details Node/Sap/queue Age(ms) SrcNode SrcSAP DstNode DstSAP OPC MsgId MsgSize RRToken Offset sup/175/pers x3B x x3fda x2beee04 sup/175/pers x3D x x x2beec04 sup/175/pers x x x6e2dc x6e2dc x2aad004 sup/175/pers x x x6e2dc x6e2dc x2bef004 sup/175/pers x x x6e5e x6e5e x2aa6004 sup/175/pers x x x6e5e x6e5e x2aa6104 <snip> Leaf-1# show system internal mts sup sap 175 description Ethpm SAP Leaf-1# show system internal mts sup sap 181 description Ethpc SAP Leaf-1# show system internal mts opcodes | grep 8182 MTS_OPC_LINK_EVENT_DOWN: SYNC SEQNO Leaf-1# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Problem Isolation Platform Overview and Troubleshooting NX-OS tips FSM MTS Crashes High CPU/Ethanalyzer Forwarding Buffering/Queuing © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Crashes Some processes in NX-OS are able to be restarted in a stateful manner. Nexus 5600/6000 is a single-supervisor platform; critical processes require a system restart upon a crash. NX-OS attempts to create a core file with information helpful to aid in finding and fixing the problem stack trace memory contents A syslog message sent just before crash and system restart(NVRAM logs saves this) 2015 Apr 16 10:38:48 esc-6004EF %$ VDC-1 %$ %SYSMGR-2-SERVICE_CRASHED: Service "Century" (PID 3751) hasn't caught signal 6 (core will be saved). © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Crashes show process log View status of all processes, including if a core was created esc-6004EF# show processes log Process PID Normal-exit Stack Core Log-create-time Century N N N Thu Apr 16 10:38: ... esc-6004EF# show processes log pid 3751 Service: Century Description: Century USD Executable: /isan/bin/century Started at Thu Apr 16 10:20: ( us) Stopped at Thu Apr 16 10:38: (87531 us) Uptime: 18 minutes 46 seconds Start type: SRV_OPTION_RESTART_STATELESS (23) Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2) ... © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Crashes When NX-OS system manager “sysmanager” resets the switch, a core file for the offending process is often generated. Core in volatile memory. Copy off core file for analysis before further reload/reboots esc-6004EF# show cores VDC Module Instance Process-name PID Date(Year-Month-Day Time) Century :46:09 esc-6004EF# copy core://18/3751/1 ? bootflash: Select destination filesystem ftp: Select destination filesystem scp: Select destination filesystem sftp: Select destination filesystem tftp: Select destination filesystem © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Crashes Sometimes a core file does not exist
not enough room in the file system Kernel panic crashes third-party processes; ntpd, telnetd, others... show logging onboard obfl-logs show logging onboard exception log show logging onboard kernel-trace OBFL is used to capture information related to hardware, bootup, and environmental conditions. Onboard failure logging is non-volatile. obfl-logs – per module; tracks environmental logs, bootup-records, uptime at bootup, version at each boot, stack trace if applicable exception log – crash/exception history and details kernel-trace – display stack of last kernel exception © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Crashes In addition to the core file, these details are essential:
Was there a configuration change? Was there a physical topology change? Can this be reproduced? Was there a recent upgrade? Are you using an uncommon configuration? The more details pointing to a root cause, the more feasible it is to find the problem, provide a workaround, and a fix. © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

High CPU Hardware accelerated switches do not rely on the CPU for frame forwarding and processing. *Some L3 paths do require CPU path if hw entries are missing – “punt” CPU is critical for control-plane activities: LACP – without keeping up with LACPDUs, 802.3ad portchannels would go down STP and STP Bridge Assurance – A downstream switch missing BPDUs will go forwarding on a blocked port. If the CPU cannot keep up with sending BPDUs, loops can form. vPC programming – mac addresses learned on vPC interfaces must be installed on both switches in order to prevent flooding as well as deliver frames to their destination Redundancy – in the event of a switch outage, the CPU needs to reprogram state information for all processes configuration and management – An unresponsive switch is not useful as a troubleshooting tool, and you are blind without a reliable interface with the network © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

High CPU Hopefully you have a baseline to compare the current CPU trends with a known nominal state Always gather 3 commands repeating frequently show processes cpu sort | exclude 0.0 show system resources show processes cpu history © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

High CPU Note the difference between *, maximum CPU and #, average CPU
This is a completely normal looking graph, try to focus on extended high average CPU periods esc-5672-left# show processes cpu history ** * * * * * * * * * * ** 90 ** ** * * * * * ** * * * ** * * * * *** * * ** 80 *** ** * * * *** **** * * * *** * **** * ** *** * ** * ** 70 *** ** **** * *** **** *** *** *** ****** **** *** * ** * ** 60 *** ****************** *** ******* *********** ***** ** **** 50 ************************** ******* ************************* 40 ************************************************************ 30 ***********************************************************# 20 *##**#*******#***********#*#*#**#**##*###*###**##****#****## 10 ############################################################ CPU% per minute (last 60 minutes) * = maximum CPU% # = average CPU% © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Ethanalyzer and high CPU
Use ethanalyzer to aid in identifying external causes of high CPU utilization esc-5672-left# show system resources Load average: 1 minute: minutes: minutes: 1.46 Processes : total, 1 running CPU states : % user, % kernel, % idle <snip> Memory usage: K total, K used, K free esc-5672-left# show processes cpu sort | exclude 0.0 PID Runtime(ms) Invoked uSecs 1Sec Process % snmpd % fcpc % fcp_thread_sm/0 % bigsurusd © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Ethanalyzer NIC UPC CPU NIC MGMT0
Displaying and capturing control-plane frames with built-in Ethanalyzer utility based on wireshark project, NX-OS command frontend Can display like tshark, or capture to .pcap file to analyze elsewhere Can be used on mgmt0 as well as eth3 or eth4, the low and high priority CPU queues UPC low eth3 CDP ICMP NIC ARP CFS eth4 CPU hi South Bridge LACPDU BPDU eth0 NIC DCBX MGMT0 © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Ethanalyzer options esc-5672-left# ethanalyzer local interface ?
inbound-hi Inbound(high priority) interface inbound-low Inbound(low priority) interface mgmt Management interface esc-5672-left# ethanalyzer local interface inbound-hi ? <snip> autostop Capture autostop condition capture-filter Filter on ethanalyzer capture capture-ring-buffer Capture ring buffer option decode-internal Include internal system header decoding detail Display detailed protocol information display-filter Display filter on frames captured limit-captured-frames Maximum number of frames to be captured (default is 10) limit-frame-size Capture only a subset of a frame raw Hex/Ascii dump the packet with possibly one line summary write Filename to save capture to © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Ethanalyzer Example capture mgmt0 traffic and save to a file on bootflash view capture files copy off for further analysis esc-5672-left# ethanalyzer local interface mgmt write bootflash:mgmt.pcap Capturing on mgmt0 10 esc-5672-left# dir bootflash: | grep mgmt. Apr 28 21:54: mgmt.pcap esc-5672-left# ethanalyzer local read bootflash:mgmt.pcap :54: > SNMP get-next-request :54: > SNMP get-response :54: > DNS Standard query X25 ame :54: > SNMP get-next-request :54: > SNMP get-response :54: > UDP Source port: Destination port: 3200 © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Ethanalyzer example capture high priority traffic using display-filter to capture – displaying to terminal esc-5672-left# ethanalyzer local interface inbound-hi display-filter stp Capturing on inband :04: :2a:6a:d3:7e:db -> 01:00:0c:cc:cc:cd STP RST. Root = 32768/500/00:06:f6:75:b0:3c Cost = 9 Port = 0x8104 :04: :2a:6a:d3:7e:db -> 01:00:0c:cc:cc:cd STP RST. Root = 32768/501/00:2a:6a:d3:7e:fc Cost = 0 Port = 0x8104 :04: :2a:6a:d3:7e:db -> 01:00:0c:cc:cc:cd STP RST. Root = 32768/502/00:2a:6a:d3:7e:fc Cost = 0 Port = 0x8104 :04: :2a:6a:d3:7e:db -> 01:00:0c:cc:cc:cd STP RST. Root = 32768/503/00:2a:6a:d3:7e:fc Cost = 0 Port = 0x8104 <snip? 10 packets captured esc-5672-left# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

CPU statistics CLI view of in-band control plane data
esc-5672-left# show hardware internal cpu-mac inband counters eth Link encap:Ethernet HWaddr 00:2A:6A:F9:73:23 inet6 addr: fe80::22a:6aff:fef9:7323/64 Scope:Link UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST MTU:9216 Metric:1 RX packets:49 errors:0 dropped:0 overruns:0 frame:0 TX packets:10420 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4116 (4.0 KiB) TX bytes: (3.3 MiB) eth Link encap:Ethernet HWaddr 00:2A:6A:F9:73:24 inet6 addr: fe80::22a:6aff:fef9:7324/64 Scope:Link RX packets: errors:0 dropped:0 overruns:0 frame:0 TX packets: errors:0 dropped:0 overruns:0 carrier:0 RX bytes: (336.7 MiB) TX bytes: (104.9 MiB)

CRC Errors Basic understanding of hardware architecture is necessary for troubleshooting The following hardware overview is a subset of BRKARC-3452 – Cisco Nexus 5600/6000 Switch Architecture BRKARC-3454 – In-depth and personal with the Cisco Nexus 2000 Fabric Extender Architectures, Features, and Topologies © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Switch Fabric Mode and Latency
Recommend 10G fabric when most of ports are 10G and latency is important 40G Fabric Mode 10GE 40GE Cut-through with light traffic Store-N-Forwarding with heavy traffic Store-N-Forwarding Cut-through Egress Ingress 10G Fabric Mode 10GE 40GE Cut-through Store-N-Forwarding Egress Ingress

Cut-through Mode and CRC Errors
Cut-through switching changes how we troubleshoot problems in the switch. Ethernet CRC is at the end of the frame, so even a CRC error cannot cause a drop on a cut-through port. We are already forwarding the frame by the time the ingress mac can read the CRC value. Ethernet Header IPv4 IP Payload FCS Parsing CRC Bad corruption Forward

Cut-through Mode and CRC “Stomping”
In addition to receiving errored frames, the Nexus 5600/6000 can generate a bad CRC for several reasons: MTU violation IP length error Ethernet length error ethertype/len < 1500 / 0x5dc is interpreted as length Invalid Ethernet preamble Only received errors will count as RX CRC errors. You are more likely to see CRC errors in a network with a cut-through switch. The errors will pass through all cut-through switches and finally drop at the first store-and-forward buffer.

Example of received corrupted frame.
Traffic flow is from Eth1/6 to Eth1/32 Frames already coming in corrupted into switch esc-5672-right# show interface e1/6 ... RX 2 unicast packets 202 multicast packets 0 broadcast packets input packets bytes 0 jumbo packets 0 storm suppression bytes 0 runts 0 giants CRC 0 no buffer input error 0 short frame 0 overrun 0 underrun 0 ignored esc-5672-right# show hardware internal bigsur port ethernet 1/6 counters rx | inc CRC RX_PKT_CRC_NOT_STOMPED | | | 0 RX_PKT_CRC_STOMPED | | | 0 esc-5672-right# In older code, input error/crc were seen as output error in show interface But behavior changed since 6.0(2)N2(1) esc-5672-right# show hardware internal bigsur port ethernet 1/32 counters rx | inc FRAME TX_PKT_FRAME_ERROR | | | 0 esc-5672-right# Until CSCub87560, ingress CRCs used to be sent out as output errors. But after CSCub87560 made this change, output errors are reserved only for FEX fabric interfaces.

Example of switch stomping due to MTU violation.
Traffic flow is from Eth1/6 to Eth1/32 9000 byte frames already coming in on Eth1/6 but switch MTU is 1500 bytes esc-5672-right# sh queuing interface ethernet 1/6 | inc MTU q-size: , q-size-40g: , HW MTU: 1500 (1500 configured) esc-5672-right# show hardware internal bigsur port ethernet 1/6 counters rx | grep RX_PKT_SIZE_IS_819 RX_PKT_SIZE_IS_8192_TO_ | | | 0 esc-5672-right# show hardware internal bigsur asic 1 counters interrupt | grep -i mtu big_bmin_cl1_INT_p1_err_ig_mtu_vio | | | |0 esc-5672-right# Packets are truncated to 1500 bytes and stomped with CRC esc-5672-right# show hardware internal bigsur port ethernet 1/31 counters rx | grep TX_PKT_SIZE_IS_1519 TX_PKT_SIZE_IS_1519_TO_ | | | 0 esc-5672-right# show hardware internal bigsur asic 3 counters interrupt | grep crc big_bmen_glb_INT_pc1_p2_norm_crc_stomp | | | |0 big_fwe_psr1_P2_INT_pkt_err_eth_crc_stomp | | | |0 esc-5672-right# show hardware internal bigsur port ethernet 1/31 counters rx | inc FRAME TX_PKT_FRAME_ERROR | | | 0 esc-5672-right# Until CSCub87560, ingress CRCs used to be sent out as output errors. But after CSCub87560 made this change, output errors are reserved only for FEX fabric interfaces.

Finding the Source of CRC Errors
CRC errors are introduced in 3 ways: Bad physical connection copper, fiber, transceiver, phy “stomping” due to intentionally originated errors Received bad CRC “stomped” from neighboring cut-through switch. Start by finding any RX CRC counters. If none, then this switch is responsible for originating Use interrupt counters to find the reason and port, if intentional Log in to next switch upstream of CRC counters, check for RX CRC there. Use the above logic to determine if this switch is originating any errors. Finally, inspect optics/pluggables, fiber/cables and troubleshoot as a Layer 1 issue. Change cable and port to find where the problem follows. Store & Forwarding mode can be configured esc-5672-right(config)# hardware ethernet store-and-fwd-switching Enabling store-and-forward switching. Please copy the configuration and reload the switch esc-5672-right(config)#

Store and forward mode Incoming error’ed frames are discarded rather than being sent out with stomped CRC esc-5672-right(config)# hardware ethernet store-and-fwd-switching Enabling store-and-forward switching. Please copy the configuration and reload the switch esc-5672-right(config)# esc-5672-right# show hardware internal bigsur port ethernet 1/6 counters rx | grep CRC RX_PKT_CRC_NOT_STOMPED | | | 0 RX_PKT_CRC_STOMPED | | | 0 esc-5672-right# show interface ethernet 1/6 Ethernet1/6 is up … RX 0 unicast packets 16 multicast packets 0 broadcast packets input packets bytes 0 jumbo packets 0 storm suppression bytes 0 runts 0 giants CRC 0 no buffer input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble input discard 0 Rx pause

Store and forward mode Incoming jumbo frames are discarded rather than being truncated and stomped with CRC esc-5672-right(config)# hardware ethernet store-and-fwd-switching Enabling store-and-forward switching. Please copy the configuration and reload the switch esc-5672-right(config)# esc-5672-right# show hardware internal bigsur port ethernet 1/6 counters rx | grep RX_PKT_SIZE_IS_81 RX_PKT_SIZE_IS_8192_TO_ | | | 0 esc-5672-right# show interface ethernet 1/6 Ethernet1/6 is up … RX unicast packets 27 multicast packets 0 broadcast packets input packets bytes jumbo packets 0 storm suppression bytes 0 runts 0 giants 0 CRC 0 no buffer 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble input discard 0 Rx pause

(ARP/ND/Host route/(*,G))
Key Forwarding tables Host table: 256,000 -entry hashing table Host table: Shared between MAC, ARP, ND and /32 host route Host table carving: MAC, IP host LPM table (summary routes): 32,000 entries Mroute table: 64,000 entries Host Table LPM Table(32,000) Mroute Table(64,000) MAC Region (S,G) Summary Routes IP Host Region (ARP/ND/Host route/(*,G))

Nexus 5600/6000 Packet Processing Flow Ingress Pipeline
VLAN Table MAC Table ARP/ND Table IPv4/Ipv6 RoutingTable FabricPath/TRILL FC routing Table Ingress UPC Unified Crossbar Fabric Buffer allocation Buffer accounting PAUSE singal 1G/10G/40G Unicast VOQ (8 per egress port) Forwarding Lookup Buffer management Parser ACL Ingress Policing Packet Rewrite MAC 8K Multicast VOQ Security ACL QoS ACL SPAN Filter Policing/Stats Storm Control Buffer Ingress SPAN replication

Nexus 5600/6000 Packet Processing Flow Egress Pipeline
Unified Crossbar Fabric Egress UPC PAUSE signal Multicast Drop Unicast queue Buffer management Egress ACL Egress Policing Multicast replication Scheduling Editing MAC TX SPAN Replication Buffer To ingress Buffer management Multicast queue

Nexus 5600/6000 L2 Unicast Forwarding
Forwarding Manager(FWM) is responsible for programming hardware in Nexus 5600/6000 for forwarding Hardware MAC address table referred to as Station Management Table(STM) L2 forwarding highlights Check if VLAN is allowed on ingress interface(STP state) For a new MAC, add source address to STM table with Vlan info STM table is synchronized across all ASICs If destination MAC is a hit against “myipr” table, packets are sent to L3 pipeline. Otherwise L2 forwarding © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

esc-5672-left Eth1/28 Eth1/1 Eth1/14 Server A Server B Server C Problem: Server C does not see traffic/Loss from Server A Given: Server A is sending traffic toward Server C. Possibly other servers too(Server B) All servers have had resolved ARP entries resolved. All servers are configured to be in the same VLAN. © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Get front panel to internal port ASIC mapping esc-5672-left Eth1/28 Eth1/1 Eth1/14 Server A Server B Server C esc-5672-left# show hardware internal bigsur all-ports | egrep name|1/1|1/14|1/28 name |idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer 1gb1/1 |1 |0 |1 | 0 p |0 |b3 |en |up |1a000000|pass| 0.00 1gb1/14 |2 |0 |2 | 1 p |13 |b3 |en |up |1a00d000|pass| 0.00 1gb1/28 |3 |0 |3 | 3 p |27 |b3 |en |up |1a01b000|pass| 0.00 © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Check for STP and MAC address table esc-5672-left# show spanning-tree vlan 10 VLAN0010 Spanning tree enabled protocol rstp Root ID Priority Address b.54c2.44c2 Cost Port (port-channel672) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority (priority sys-id-ext 10) Address a.6af9.737c Interface Role Sts Cost Prio.Nbr Type Po Desg FWD (vPC peer-link) Network P2p Po Root FWD (vPC) P2p Eth1/ Desg FWD Edge P2p Eth1/ Desg FWD Edge P2p Eth1/ Desg FWD Edge P2p esc-5672-left# sh mac address-table dynamic vlan 10 | inc 0010 * dynamic F F Eth1/1 * dynamic F F Eth1/14 * dynamic F F Eth1/28 esc-5672-left# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Check for STP and MAC address table in hardware esc-5672-left# show platform fwm info vlanif 10 ethernet 1/1 vlanif vlan 1.10 if 1a stp state: forwarding esc-5672-left# show platform fwm info vlanif 10 ethernet 1/14 vlanif vlan 1.10 if 1a00d000 stp state: forwarding esc-5672-left# show platform fwm info vlanif 10 ethernet 1/28 vlanif vlan 1.10 if 1a01b000 stp state: forwarding esc-5672-left# show platform fwm info hw-stm asic 0 | grep Eth1/ :5469:0 1:0:1 2.a.bc (e:0) Eth1/ :8546:0 1:0:1 2.a.bc (e:0) Eth1/ :9954:0 1:0:1 2.a.bc (e:0) esc-5672-left# show platform fwm info hw-stm asic 1 | grep esc-5672-left# show platform fwm info hw-stm asic 2 | grep esc-5672-left# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

MAC address event history esc-5672-left# show platform fwm info mac mac vlan 1.10 mac : vlan 1.10 mac vlan 1.10 mac : learned-on Eth1/1 age 110 ref_map = 'vlan if' mac vlan 1.10 mac : nohit_count 0 hw_programmed 1 mac_clone 0 mac vlan 1.10 mac : old_if_index 'null' mac vlan 1.10 mac : pss_flags 0 mac vlan 1.10 mac cfg attrs - not-cli-cfg not-static movable no-drop no-regmac non-netstack-learnt not-secure not-src-drop mac vlan 1.10 mac : mcec_flags 0x1, mac_info_flags 0, rem_if 0, sync_count 1 rcv_count 0 mac vlan 1.10 mac : CDCE Address 3:0:0:bc:a:2 Mac history (Last 35 operations): Total operations: 4: Operation: Mac create (9) (flags: Loc (0x1) mac_info_flags (0x0) if: 0x1a hint: 0) at Sat May 2 04:23: Operation: Mac learned from hw (40) Operation: Mac sent to peer on local learn (15) at Sat May 2 04:27: © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Check FWM for any drops on interface Note some drops can be seen due to configuration but further investigation needed esc-5672-left# show platform fwm info pif ethernet 1/1 | inc asic Eth1/1 pd: slot 0 logical port num 0 slot_asic_num 1 global_asic_num 1 fw_inst 0 phy_fw_inst 0 fc 0 esc-5672-left# show platform fwm info pif ethernet 1/1 | inc drop Eth1/1 pd: tx stats: bytes frames discard 0 drop 0 Eth1/1 pd: rx stats: bytes frames discard 0 drop esc-5672-left# show platform fwm info pif ethernet 1/29 | inc drop Eth1/1 pd: tx stats: bytes frames discard 0 drop 0 Eth1/1 pd: rx stats: bytes frames discard 0 drop esc-5672-left# show platform fwm info asic-errors 1 <snip> Printing non zero Carmel error registers - 32 bits: BIG_DROP_IDS_CODE_0_1: res = [0] BIG_DROP_IDS_CODE_0_1: res = [0] © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Drops are due to FWM IDS check failure FWM dropped packet can be redirected to Sup for further inspection esc-5672-left# debug platform fwm pkt-drop-redirect drop-condition IDS_CODE_0_1 asic-id 1 esc-5672-left# ethanalyzer local interface inbound-low display-filter ip.addr== detail <snip> Header checksum: 0x9301 [incorrect, should be 0x385c] [Good: False] [Bad : True] [Expert Info (Error/Checksum): Bad checksum] [Message: Bad checksum] [Severity level: Error] [Group: Checksum] Source: ( ) Destination: ( ) User Datagram Protocol, Src Port: 1024 (1024), Dst Port: 1024 (1024) esc-5672-left# no debug platform fwm pkt-drop-redirect asic-id 1 Drops in this case were due to host sending frames with incorrect IPv4 checksum © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

MAC address event history esc-5672-left# debug platform fwm pkt-drop-redirect drop-condition IDS_CODE_0_1 asic-id 1 esc-5672-left# ethanalyzer local interface inbound-low display-filter ip.addr== detail <snip> Header checksum: 0x9301 [incorrect, should be 0x385c] [Good: False] [Bad : True] [Expert Info (Error/Checksum): Bad checksum] [Message: Bad checksum] [Severity level: Error] [Group: Checksum] Source: ( ) Destination: ( ) User Datagram Protocol, Src Port: 1024 (1024), Dst Port: 1024 (1024) esc-5672-left# no debug platform fwm pkt-drop-redirect asic-id 1 © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Host table exhaustion 2015 May 4 12:09:17 esc-5672-left %FWM-2-STM_LIMIT_REACHED: Unicast station table dynamic capacity reached (limit ) - creating mac bca1 on port Eth1/14 and vlan 10 disabling dynamic learn notifications for 180 seconds or till capacity reaches 1500 entries esc-5672-left# show mac address-table count MAC Entries for all vlans: Dynamic Address Count: Static Address (User-defined) Count: 0 Multicast MAC Address Count: 0 Total MAC Addresses in Use: Total PVLAN Clone MAC Address Count: 0 esc-5672-left# show platform fwm info stm-stats Global level learning: disabled Vlan level learning: enabled MAC Stats: (learning_disable ucast 1 mcast 0 learn_on_exceptions 0) STM Threshold - total ucast entries : STM Threshold - total mcast entries : 0 STM Threshold - dynamic ucast entries : STM Threshold - dynamic mcast entries(excl. cloned) : 0 STM Threshold - dynamic cloned mcast entries : 0 STM Threshold - dynamic mcast entries(combined) : 0 STM Threshold - ucast cloned entries : 0 STM Threshold - mcast cloned entries : 0 STM Threshold - ucast cloned adds : 0 STM Threshold - ucast cloned destroys : 0 STM Threshold - total limit : STM Threshold - dynamic ucast limit : © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Host Table N5600/6000 Host Table
N5600/6000 has a 256k host table divided in two parts: MAC region IP region: holds ARP, IPv6 ND, /32 Host Routes Default sizes are 128k for MAC region and 128k for IP region. Host Table MAC Region 256,000 Entries IP Host Region (ARP & ND & /32 Host route)

Host Table Host Table Carving
In virtualized environments, there can be a larger number of MAC addresses compared to IP addresses. One gateway (IP) with multiple VMs behind (MAC). MAC/ARP Hardware Resource Carving allows to change the size of MAC and IP regions to allocate more space for the MAC region. Host Table MAC Region 256,000 Entries IP Host Region (ARP & ND & /32 Host route)

Carving Profiles Use of pre-defined carving profiles.
HRT = Host Route Table = IP host region. STM = Station Table Management = MAC region. Template Profile Description hrt-128-stm-128 HRT size: 128k, STM size: 128k (default profile) hrt-96-stm-160 HRT size: 96k, STM size: 160k hrt-64-stm-192 HRT size: 64k, STM size: 192k hrt-32-stm-224 HRT size: 32k, STM size: 224k

MAC/ARP Resource Carving CLI
Specify the resource template to use Need to save the configuration and reload the switch to be applied. Show commands: show hardware profile route resource template show hardware profile route resource template default esc-5672-left(config)# hardware profile route resource service-template ? hrt-128-stm-128 Hrt: 128k, Stm: 128k (default size) hrt-224-stm-32 Hrt: 224k, Stm: 32k hrt-32-stm Hrt: 32k, Stm: 224k hrt-64-stm Hrt: 64k, Stm: 192k hrt-96-stm Hrt: 96k, Stm: 160k esc-5672-left(config)#

Nexus 5600/6000 L3 Forwarding L3 look up is built into the forwarding pipeline L3 License required to activate L3 features MyIPRTable contains the list of MAC addresses the switch can route for If the look up is a hit against MyIPRTable, packets are routed, if not they are bridged © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Host Table N5600/6000 L3 table look up order
Host Route Table(HRT) is looked up first If no hit in HRT table, Longest Prefix Match(LPM) table is Host Table MAC/STM table 128,000 Host Route Table 128,000 Summary Routes 32,000 MyIPRTABLE 512

esc-5672-left Eth1/3 Eth1/4 L3 Eth1/1 Server A Server B Problem: To check for L3 routing information in software and hardware Given: Server A( ) is in VLAN 10 esc-5672-left has SVI in VLAN 10 with HSRP and OSPF configured Server B( ) is learnt via ECMP OSPF route © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Check for local interface/HSRP state, MyIPRTable esc-5672-left# sh hsrp brief *:IPv6 group #:group belongs to a bundle P indicates configured to preempt. | Interface Grp Prio P State Active addr Standby addr Group addr Vlan Active local (conf) esc-5672-left# sh int vlan 10 Vlan10 is up, line protocol is up Hardware is EtherSVI, address is 002a.6af9.737c Internet Address is /24 MTU 1500 bytes, BW Kbit, DLY 10 usec esc-5672-left# show hsrp interface vlan 10 | inc Virtual Virtual IP address is (Cfged) Virtual mac address is c9f.f00a (Default MAC) esc-5672-left# show system internal forwarding myiprtable Index | BD No | Mac Addr | Ref Count a.6af9.737c a.6af9.737c <snip> c9f.f00a esc-5672-left# show platform fwm info l3lif vlan 10 | inc int-vlan|mac Vlan10: iftype SVI: int-vlan 94 l3-vdc-vlan 10 fhrp_enable 1 num_fhrp_grps 1 Vlan10: mac-address: 002a.6af9.737c Vlan10: fhrp-mac:0000.0c9f.f00a, l2-fm-state:L2FM_MAC_STATE_ACTIVE, remote l2-fm-state:L2FM_MAC_STATE_STANDBY Router/SVI MAC in the myiprtable, hit in this table means the packet need to be routed © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Check for routing information esc-5672-left# show ip route IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string> /24, ubest/mbest: 2/0 *via , Eth1/3, [110/44], 00:12:49, ospf-1, intra *via , Eth1/4, [110/44], 00:12:49, ospf-1, intra esc-5672-left# show forwarding route /24 IPv4 routes for table default/base Prefix | Next-hop | Interface | Labels / Ethernet1/3 Ethernet1/4 esc-5672-left# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Check for adjacency information esc-5672-left# show ip arp | inc 10.1. :13: b.54c2.d842 Ethernet1/3 :16: b.54c2.44c2 Ethernet1/4 esc-5672-left# show forwarding adjacency IPv4 adjacency information next-hop rewrite info interface b.54c2.d842 Ethernet1/3 esc-5672-left# show forwarding adjacency b.54c2.44c2 Ethernet1/4 esc-5672-left# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Check for route and adjacency in hardware esc-5672-left# show system internal forwarding ipv4 route Routes for table default/base Dev | Prefix | UC/MC Handle (Index) |AdjIdx(nhcount)| LIF / x4000d(0x41770)/0xdeadbeef xe(0x2) esc-5672-left# show system internal forwarding adjacency entry 0xe detail Index 0xe MAC 001B.54C2.D842 BD 117 esc-5672-left# esc-5672-left# show platform fwm info l3lif ethernet 1/3 | inc int-vlan Eth1/3: iftype L3-Port: int-vlan 117 l3-vdc-vlan 6 fhrp_enable 0 num_fhrp_grps 0 esc-5672-left# show platform fwm info l3lif ethernet 1/4 | inc int-vlan Eth1/4: iftype L3-Port: int-vlan 118 l3-vdc-vlan 7 fhrp_enable 0 num_fhrp_grps 0 © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

FIB TCAM exhaustion esc-5672-left# 2015 May 5 20:39:21 esc-5672-left %FWM-4-FIB_TCAM_RESOURCE_WARNING: FIB TCAM usage is at 90 percent 2015 May 5 20:39:29 esc-5672-left %FWM-2-FIB_TCAM_RESOURCE_EXHAUSTION: FIB TCAM exhausted, prefix insert failed esc-5672-left# show ip route summary IP Route Table for VRF "default" Total number of routes: 33031 Total number of paths: Best paths per protocol: Backup paths per protocol: am : None local : 4 direct : 4 broadcast : 9 ospf : 33024 Number of routes per mask-length: /8 : /24: /32: 20 esc-5672-left# show hardware profile status Max Mcast Routes = 8192. Used Mcast Routes = 5. Total LPM Entries = Used Unicast IPv4 LPM Entries = Used Unicast IPv6 LPM Entries = 2. <snip> esc-5672-left# show hardware profile status Max Mcast Routes = 8192. Used Mcast Routes = 5. Total LPM Entries = Used Unicast IPv4 LPM Entries = Used Unicast IPv6 LPM Entries = 2. Used Unicast Host4 Entries in LPM= 2. Used Unicast Host6 Entries in LPM= 0. Used Multicast IPv4 LPM Entries = 8. Reserved LPM Entries for future use = 3. Used Host Entries = 18. Used Unicast Host Entries in Host (Total) = 18. Used Unicast Host4 Entries in Host = 18. Used Unicast Host6 Entries in Host = 0. Used Multicast Host Entries in Host (Total) = 0. Used Multicast Host4 Entries in Host = 0. Used Multicast Host6 Entries in Host = 0. esc-5672-left# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Flexible Buffer Management Ingress Buffer
Shared buffer is good for burst absorption Dedicated buffer is good for predictable performance for each every port Flexible buffer management: dedicated plus shared “queue-limit” in QoS policy specify dedicated buffer. Rest are shared Long distance FCoE, contd.. UPC Ingress Buffer (16MB) SPAN Control Shared Packet Buffer Port 1 dedicated Port 2 dedicated Port 3 dedicated

Flexible Buffer Management Default Ingress Buffer Allocation
The table lists ingress buffer allocation with default QoS configuration. Buffer Pool 10 GE Port 40 GE Port Control traffic (per port) 64 KB 67.2 KB SPAN (per port) 38.4 KB 153.6 KB Class default (per port) 100 KB Shared buffer 13.2 MB 14.7 MB

Flexible Buffer Management Egress Buffer
9MB packet buffer at egress UPC. Shared among 3x40GE/12x10GE Unicast traffic can be buffered at egress and ingress Multicast is buffered at egress in case of congestion. contd. UPC Egress Buffer (9MB) Unicast Buffer Multicast Buffer hardware mutlicast-buffer-size <1-8 MB>

Flexible Buffer Management Default Egress Buffer Allocation
The table lists egress buffer allocation with default QoS configuration. Buffer Pool 10 GE Port 40 GE Port Unicast (per port) 363 KB 650KB with 10G fabric mode 635KB with 40G fabric mode Multicast (per ASIC) 4.3 MB 6.6 MB

Nexus 5600/6000 Queuing Nexus 5600/6000 utilize ingress/egress queuing for unicast Nexus 5600/6000 utilize egress queuing for multicast Ingress queuing is implemented by Virtual Output Queuing (VOQ) VOQ prevents head of line blocking One egress interface can be congested, but ingress buffer still accepts frames into other queues Ingress queuing is helpful for data flows where many ports talk to few, the load is spread across the sources Simple flowcontrol mechanism can be implemented end-to-end flowcontrol is necessary for FCoE © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 Queuing Queuing implication on troubleshooting:
For unicast traffic, drops occur at INGRESS! For multicast(flooded traffic), drops occur at EGRESS! © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 Forwarding and Queuing
esc-5672-left Eth1/28 Eth1/1 Eth1/14 Server A Server B Server C Problem: Server C does not see traffic/Loss from Server A Given: Server A is sending traffic toward Server C. Possibly other servers too(Server B) All servers have had resolved ARP entries resolved. All servers are configured to be in the same VLAN © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 Queuing Server A sending line rate bursts of packets to Server C esc-5672-left# sh queuing interface ethernet 1/1 ; sh queuing interface ethernet 1/28 Ethernet1/1 queuing information: <snip> RX Queuing qos-group 0 q-size: , q-size-40g: , HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: 0 Statistics: Pkts received over the port : Ucast pkts sent to the cross-bar : Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 30 Pkts discarded on ingress : 0 Per-priority-pause status : Rx (Inactive), Tx (Inactive) Ethernet1/28 queuing information: Pkts received over the port : 0 Ucast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : Pkts sent to the port : © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 Queuing Server A sending line rate bursts of packets to Server C But you are getting complaints about performance problems Oh no!!. You are seeing packet drops now esc-5672-left# sh queuing interface ethernet 1/1 Ethernet1/1 queuing information: TX Queuing qos-group sched-type oper-bandwidth WRR RX Queuing qos-group 0 q-size: , q-size-40g: , HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: 0 Statistics: Pkts received over the port : Ucast pkts sent to the cross-bar : 59007 Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 14 Pkts discarded on ingress : 40993 Per-priority-pause status : Rx (Inactive), Tx (Inactive) esc-5672-left# sh int ethernet 1/1 | inc discard|input 30 seconds input rate 0 bits/sec, 0 packets/sec input rate 0 bps, 0 pps; output rate 1.18 Kbps, 1 pps input packets bytes 0 input with dribble input discard © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 Queuing Ingress discards typically are due to an egress congestion Common causes Speed mismatch(Ex: 40G interface sending traffic to 10G host) Multiple interfaces sending line rate bursts to one interface Microbursts filling up ingress buffers Nexus 5600/6000 has analytical capability to troubleshoot congestion Identify congested interface Microburst monitoring SPAN on drop Latency monitoring Buffer usage monitoring © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 Nexus 5600/6000 switches support a very rich suite of data analytics Data Analytics is covered in detail in following breakout session BRKDCT Network visibility using advanced Analytics in Nexus switches(6/9 3:30pm) © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 ELAM Embedded Logic Analyzer(ELAM) is supported starting 7.x Tool to capture dataplane traffic Without ELAM, a SPAN or an inline sniffer would be required ELAM also captures switching decision for the traffic Implemented in hardware with a parallel snoop process of actual packet decision process Captures the first packet which matches trigger Cannot be used to troubleshoot packet loss/performance problems. There is no impact/penalty to switch or traffic due to ELAM Meant to be used for troubleshooting by TAC/development One will need to know port to internal ASIC mapping © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 ELAM Scenario Problem:
esc-5672-left Eth1/1 Eth1/14 Server A Server B Problem: Server A cannot ping/reach server B Given: Server A is sending ICMP traffic toward Server B. Servers have resolved ARP, no other apparent problems seen in switch © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 ELAM Scenario esc-5672-left Eth1/1 Eth1/14 Server A
Server B esc-5672-left# show hardware internal bigsur all-ports | egrep name|1/1|1/14 name |idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer 1gb1/1 |1 |0 |1 | 0 p |0 |b3 |en |up |1a000000|pass| 0.00 1gb1/14 |2 |0 |2 | 1 p |13 |b3 |en |up |1a00d000|pass| 0.00 © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 ELAM Multiple options available for ELAM
Be as specific as possible but ELAM can be triggered for all slots/ASIC instances, combination of IP/MAC addresses, VLAN, L3-4 protocol types etc Here ingress ELAM is being set to trigger on a specific source/destination IP address esc-5672-left# elam slot 1 asic bigsur instance 1 esc-5672-left(bigsur-elam)# trigger lu ingress ? arp ARP Frame Format ce CE Frame Format fc FC Frame Format ipv4 IPv4 Frame Format esc-5672-left(bigsur-elam)# trigger lu ingress ipv4 if source-ipv4-address_ipv destination-ipv4-address_ipv esc-5672-left(bigsur-elam)# start capture esc-5672-left(bigsur-elam)# show capture lu ELAM: Nothing captured esc-5672-left(bigsur-elam)# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 Port/MAC ACL(PACL/MACL)
Port Access-lists are supported to filter IP traffic MAC Access-lists are supported to filter non IP traffic In addition to filtering, PACL and MACL can be effectively used to troubleshoot data plane issues such as packet loss Can be applied to both L2/L3 interfaces No penalty to switch/traffic Supported only in ingress direction Avoids wastage of time due to setting up SPAN sessions Reduces time to resolution Remember to have explicit “permit ip any any” at the end After troubleshooting, remove the ACL off the interface first and then ACL © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 PACL Scenario Problem: Server A is having packet loss
esc-5672-left Eth1/1 Eth1/14 Server A Server B Problem: Server A is having packet loss Given: Server A is sending IP traffic toward Server B. Servers have resolved ARP, no other apparent problems seen in switch © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Nexus 5600/6000 PACL PACL example esc-5672-left# sh run aclmgr !Command: show running-config aclmgr !Time: Sat May 2 06:59: version 7.1(1)N1(1) ip access-list test statistics per-entry 10 permit ip / /32 20 permit ip any any interface Ethernet1/1 ip port access-group test in 100K IP packets sent from server AB and all accounted for on switch esc-5672-left# sh access-lists IPV4 ACL test statistics per-entry 10 permit ip / /32 [match=100000] 20 permit ip any any [match=0] esc-5672-left# © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

“If it ain’t broke, don’t fix it”
Or not… Software upgrade required for bug fixes, enhancements, new features N5600/6000 being single Supervisor system, ISSU restrictions apply No L3 license installed Switch needs to be in STP leaf state No LACP fast rate ISSU needs to be considered during design, not an after thought BU maintains a recommended software page Refer to bug tool kit, release notes, upgrade guide, lab testing etc. When in doubt, ask around(community forums) or contact TAC © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Software upgrades Use “install” all process to upgrade/downgrade
Keep an eye on compatibility matrix. Install process will warn.. This prevents loss of configuration. esc-5672-left(config)# boot kickstart bootflash:n6000-uk9-kickstart N1.1.bin Warning: Changing bootvariables and reloading is not recommended on this platform. Use install all command for NX-OS upgrades/downgrades. Performing image verification and compatibility check,please wait.... esc-5672-left(config)# boot system bootflash:n6000-uk N1.1.bin esc-5672-left(config)# end esc-5672-left# install all kickstart bootflash:n6000-uk9-kickstart N1.1.bin system bootflash:n6000-uk N1.1.bin Verifying image bootflash:/n6000-uk9-kickstart N1.1.bin for boot variable "kickstart". [####################] 100% -- SUCCESS <snip> © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Software upgrades Maintenance mode.. Available starting 7.1(0)N1(1)
Gracefully eject a switch for upgrades/maintenance esc-5672-left(config)# system mode maintenance BGP(v4/v6) is running... will be shutdown BGP ASN = 65338 config terminal router bgp 65338 shutdown end EIGRP(v4) is not running, nothing to be done <snip> vPC is running... will be shutdown vPC Domain id = 572 vpc domain 572 Interfaces will be shutdown Do you want to continue (y/n)? [n] © 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Don’t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit

Final Thoughts Learn more in the World of Solutions.
Visit after the event for updated PDFs, on- demand session videos, networking, and more! Follow Cisco Live! using social media: Facebook: Twitter: LinkedIn Group:

Cisco Live 2014 4/20/2017

4/20/2017 Cisco Live 2014

Objectives Understand troubleshooting on Nexus 5600 and 6000 platforms

Similar presentations

Presentation on theme: "Objectives Understand troubleshooting on Nexus 5600 and 6000 platforms"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Objectives Understand troubleshooting on Nexus 5600 and 6000 platforms

Similar presentations

Presentation on theme: "Objectives Understand troubleshooting on Nexus 5600 and 6000 platforms"— Presentation transcript:

Similar presentations

About project

Feedback