DC Compute SAN Carlos Lopez CCIE SAN, DC #21063

DC Compute SAN Carlos Lopez CCIE SAN, DC #21063
Cisco Live 2016 2/3/2018 DC Compute SAN Carlos Lopez CCIE SAN, DC #21063 David Kester CCIE SAN #19555 Ed Mazurek CCIE SNA/IP, SAN #6448

Introductions Carlos Lopez CCIE SAN, DC #21063 Technical Leader TAC
Cisco Live 2016 2/3/2018 Introductions Carlos Lopez CCIE SAN, DC #21063 Technical Leader TAC David Kester CCIE SAN #19555 Team Leader Storage Networking Ed Mazurek CCIE SNA/IP, SAN #6448 Technical Leader TAC TAC-Time

Agenda Components Topology Nexus FC NPV vs FCoE-NPV Bugs
Cisco Live 2016 2/3/2018 Agenda Components Topology Nexus FC NPV vs FCoE-NPV Bugs MDS Slow Drain Troubleshooting Enhancements TAC-Time

Cisco Live 2016 2/3/2018 Components TAC-Time

Cisco Multi-Protocol Product Portfolio: SAN, LAN, and Compute
LAN/SAN SAN 16G COMPUTE Cisco UCS 6300 Series FI Cisco Nexus 7000 Cisco Nexus 9000 Cisco MDS 9700 48x16G Line-Rate FC 48x10GE Line-Rate FCoE Cisco UCS 6200 Series FI Nexus 5672UP-16G Cisco MDS 9396S Cisco MDS 9148S Cisco Nexus 5600 Cisco Nexus 5500 Cisco MDS 9718 24x40GE Line-Rate FCoE Cisco Nexus 2000 Cisco Nexus 3000 Cisco MDS 9250i Cisco UCS B-Series Blade Servers Cisco UCS C-Series Rack Servers 16G FC: Nexus 2348UPQ 12+ Years of Proven NX-OS Operating System Cisco Prime Data Center Network Manager (DCNM) Highlight that MDS 9700 has 3x the performance – Industry’s Highest Performance and Capacity 1.5-Tbps/Slot MDS 9710: 384 Line-Rate 16G FC Ports MDS 9706: 192 Line-Rate 16G FC Ports Unmatched Flexibility – FC and FCoE Consistent, Simplified Features, Management, and Programmability

Cisco MDS 9000 Series 16G FC Director Switches
Cisco 48x10G Line-rate FCoE Module Cisco MDS 9718 Director Future Proof Reliable Multi-Protocol Flexibility Investment Protection Ease of Management Cisco 48x16G Line-rate FC Module Cisco MDS 9710 Director Cisco MDS 9706 Director Cisco 24x40G FCoE Line-rate FCoE Module Last year we introduced the start of our new 16G FC product family with the MDS 9710 and our 48-port 16G line rate FC module. This year we introduced a new compact 9706 director and a 48-port 10G line-rate FCoE module. These director class products are available through our OSM partners (EMC, IBM, Hitachi, NetApp, and HP). Our 9700 director series offers 3x the performance of our competitors’ director-class switches, with up to 1.5Tbps per slot and are future-proofed for 32G FC and 40G FCoE. They are also the industry’s most reliable directors with N+1 fabric redundancy. We offer unmatched flexibility with multi-protocol capability as both 9700 chassis can support a mix and match of FC and FCoE cards. Available on the Cisco GPL, we have the MDS 9250i multiservice switch and the MDS 9148S fabric switch. With the MDS 9250i, you get unmatched versatility of multi-protocol and services such as FCIP SAN extension. With MDS 9148S you get affordable pay as you grow flexibility and auto-provisioning for ease of use. So there you have it, a complete refreshed portfolio with industry-leading performance, reliability, and flexibility. Driving Innovations for the Next Decade with a complete 16G Portfolio Deploy Small, Medium, Large SANs with Cisco MDS 9000 Family

Cisco MDS 9000 Series 16G FC Fabric Switches
Cisco MDS 9250i 16G Multi-Service Fabric Switch Pay-as-You-Grow Enterprise Class Features Reliability Multi-Protocol Flexibility Ease of Management Cisco MDS 9396S 16G FC Fabric Switch Cisco MDS 9148S 16G FC Fabric Switch Driving Innovations for the Next Decade with a complete 16G Portfolio Deploy Small, Medium, Large SANs with Cisco MDS 9000 Family

Cisco MDS 9700 Directors Comparison
26 RU Winning Points 32G FC line-rate ready Interchangeable line cards Redundant hardware Common PSUs, Linecards Single OS, Management Better UCS Interoperability 14 RU 9 RU Hardware Feature MDS 9706 MDS 9710 MDS 9718 Line Card slots 4 8 16 Line rate 16Gbps FC or 10 Gbps FCoE 192 384 768 Line rate 40 Gbps FCoE 96 Fabric Module slots (available / default) 6 / 3 6 / 6 Sup Slots 2 Fabric Module location Rear Airflow Front to Back Power Supply slots Power Consumption (Typical/Max) 2425W / 2620W 4615W / 5020W 4742/8462W

Converged FEX Architecture Enabling Director-class resiliency at Converged Access
Multi-Protocol Storage and Host Connectivity: FC, FCoE and IP Converged Architecture includes Cisco Data center portfolio: SAN(MDS 9700), LAN(Nexus 2k-7k) and Compute (UCS Chassis) accessibility

MDS 9250i Multiservice Fabric Switch
Features Multi-Protocol Support 16G FC, 10GE FCoE, 1GE/10GE FCIP, iSCSI Intelligent Storage Services for FC and FCoE SANs Fiber Channel over IP (FCIP) IO Accelerator (IOA) Data Mobility Migration (DMM) Integrated Management via Data Center Network Manager (DCNM) FICON Certified Benefits Single Platform for deploying Storage Services across FC, FCoE and IP based Storage Area Networks (SANs) High-Bandwidth SAN Extension across MAN/WAN Vendor independent array migration tools Interoperate data between FC and FCoE arrays

MDS 9250i Overview Next Generation Multiservice Intelligent Services-oriented Fabric Switch Provides FCIP, IOA and DMM Integrated 40x16G FC, 8x10GE FCoE, and 2x1GE/10GE FCIP/iSCSI ports Enclosure: 2 RU; Redundant and hot-swappable power supplies and fan trays Ethernet 1/1-8 (10 GE FCoE) FC 1/1-40 (2/4/8/16G FC) IPStorage 1/2-2 (1GE/10GE FCIP/iSCSI) Console USB Mgmt0 The Cisco MDS 9250i switch provides the following types of ports: Console port —An RS-232 port that you can use to create a local management connection. MGMT 10/100/1000 Ethernet port—An Ethernet port that you can use to access and manage the switch by IP address, such as through Cisco Data Center Network Manager (DCNM). Fibre Channel ports —Fibre Channel (FC) ports that can be used to connect to the SAN or for in-band management. Fibre Channel over Ethernet— FCoE ports that you can use for FCoE connectivity. IP Storage ports—FCIP ports that you can use to IP Storage services which can be used ofr FCIP or iSCSI connectivity. USB drive—USB port on the front panel for code loads, configuration file backups, log dumps, and report capture.

Cisco MDS 9148S Fabric Switch
Back Dual Power Supplies and Fans for Enterprise-Class Availability Front 48 x 16G FC Line Rate Performance 12- to 48-ports in 12-port increments 1 RU VERSATILE EASY TO USE ENTERPRISE-CLASS Line-rate 16/8/4/2G FC Ports Industry-leading port range Start with 12-port base Scale up with 12-port license Or, full 48-port option available Automated Provisioning Quick Configuration Wizard Same OS and Management across Industry’s broadest SAN Portfolio Non-disruptive software upgrades Up to 32 Virtual SANs (VSANs) Inter-VSAN Routing (IVR), QOS, PortChannels, N-Port ID Virtualization (NPIV), N-Port Virtualization (NPV), Comprehensive Security Hardware-based slow-drain detection and recovery High-Performance, Easy to Deploy, Enterprise-class Fabric Switch New Cisco MDS 9148S The Cisco MDS 9148 Multilayer Fabric Switch is a highly affordable, versatile, easy-to-manage storage networking switch for entry-level and departmental SANs. The Cisco MDS 9148S, based on Cisco’s “switch on a chip” Storage Networking ASIC, delivers 16 Gbps dedicated Fibre Channel bandwidth per port. The Cisco MDS 9148S is available in a base configuration of 12-ports with on-demand “pay as you grow” 12-port activation license to scale up to 48-ports. A fully configured 48-port configuration is also available for ease of orderability. A Quick Configuration Wizard is available for simple out-of-the-box configuration Enterprise-class capabilities such as non-disruptive upgrades, Virtual SANs (VSANs), Inter-VSAN Routing (IVR), QOS, Port Channels, N-Port ID Virtualization (NPIV) available Hot-swappable redundant power supplies and fans provide high availability.

Introducing MDS 9396S 96-Port 16G Fabric Switch
2 RU Cisco MDS 9396S Versatile Start with 48-port base; Scale up with 12-port license Or full 96-port option available Easy to Use Automated provisioning Quick Configuration Wizard Same OS and management across industry’s broadest SAN portfolio Enterprise-Class Dual-power supplies and fans, non-disruptive software upgrades Up to 4095 B2B credits per port (MDS 9396S); up to 253 B2B credits per port (MDS 9148S) Up to 32 Virtual SAN (VSANs) Hardware-based slow-drain detection and recovery, Inter-VSAN routing, QoS, PortChannels, N-Port ID Virtualization (NPIV), N-Port Virtualization (NPV) Forward Error Correction, Link Encryption (FC TrustSec) Industry’s Most Affordable 16G Fabric Switch Family

MDS 9396S B2B credits All ports in a port group can have maximum of 500 B2B credits Enterprise license enables extended credits that means up to 4095 B2B credit per port in a port group Port group can have maximum of 4150 B2B credits. Best Practice: Avoid grouping all E ports in same port group/IOSlice Generic Formula: For every 1 KM distance with 1GB speed, we need .5 BB credit for standard FC frame (2112 bytes). CLI command: show port-resources module 1

Topologies – UCS-FI N5K MDS
TAC-Time

Separation makes sure that the design is highly available even when one of the fabrics goes down.

Redundant redundancy is not required

Working with TAC – Topology Commands
Cisco Live 2014 2/3/2018 Working with TAC – Topology Commands Use the ‘show topology’ command to display the interswitch links `show topology` FC Topology for VSAN 1 : Interface Peer Domain Peer Interface Peer IP Address(Switch Name) port-channel 6 0x62(98) port-channel (sw201A) F # show top vsan 237 FC Topology for VSAN 237 : Interface Peer Domain Peer Interface Peer IP Address(Switch Name) fc1/4 0x02(2) fc1/ (RTP-SAN s-1) fc1/5 0xdb(219) fc1/ (F i-1) fc1/8 0xde(222) fc1/ (F S-1) port-channel24 0x7f(127) port-channel (F ) port-channel101 0x9f(159) port-channel (F ) vfc10 0xe8(232) vfc (F MDS9718) vfc-po540 0xe8(232) vfc-po (F MDS9718) F # F # show topology isl vsan 237 __________Local_________ _________Remote_________ VSAN Cost I/F PC I/F Band PC Domain SwName Port Port SwName Domain PC Stat Stat Speed width - 0x33 F fc1/8 fc1/1 F xde up g - - 0x33 F fc1/5 fc1/1 F xdb up g - - 0x33 F fc1/4 fc1/1 RTP-SA 0x up g - x33 F fc1/ F x9f down up g x33 F fc1/6 fc6/7 F x9f up up 8g 8g 24 0x33 F fc1/9 fc1/8 F x7f up up 4g 12g 24 0x33 F241-1 fc1/17 fc1/9 F x7f up up 4g 12g 24 0x33 F241-1 fc1/25 fc1/10 F x7f up up 4g 12g

Working with TAC – Topology Commands
Cisco Live 2014 2/3/2018 Working with TAC – Topology Commands Use the ‘show fcs ie’ command to display all the switches in the VSAN Core# show fcs ie IE List for VSAN: 1 IE-WWN IE Mgmt-Id Mgmt-Addr (Switch-name) 20:01:00:0d:ec:39:19:c1 S(Rem) 0xfffc0e (sw204A) 20:01:00:0d:ec:39:1a:01 S(Rem) 0xfffc (sw202A) 20:01:00:0d:ec:fb:88:41 S(Loc) 0xfffc (sw200A) 20:01:00:2a:6a:8c:0b:01 S(Adj) 0xfffc (sw201A) Loc = Local = this switch Adj = Adjacent = connected switch Rem = Remote = more than one hop away Last two characters of the Mgmt-id is the domain ID.

Working with TAC –Topology Commands
Cisco Live 2014 2/3/2018 Working with TAC –Topology Commands Use the show fcns database npv command to find Cisco NPV switches. Core# show fcns database npv VSAN 1 NPV NODE-NAME :20:01:00:0d:ec:51:06:01 NPV IP_ADDR : NPV INTERFACE :port-channel 30 CORE SWITCH WWN :20:00:00:0d:ec:24:ef:c0 CORE INTERFACE :Po105 Show fcns database detail

Cisco Live 2014 2/3/2018 Working with TAC –Topology Commands Use the show npv flogi-table command to shows where the device is attached and it’s uplink. NPV# show npv flogi-table SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE fc1/ x0d :00:00:00:c9:71:04:4e 20:00:00:00:c9:71:04:4e Po30

Cisco Live 2014 2/3/2018 Working with TAC –Topology Commands Note: Ensure both links are connected to the same upstream IP address. This tells you which upstream switch the UCS FI is connected to in case you need to connect to upstream switch to verify the connectivity. NPV# show npv internal info external-interface all | grep addr: fabric mgmt addr:

Nexus FC NPV vs FCoE-NPV
TAC-Time

Cisco Live 2016 2/3/2018 Nexus FC NPV vs FCoE-NPV FC and NPV FCoE-NPV FCoE FC or FCoE Nexus or MDS NPIV N5K NPV FC Both FC and FCoE FCoE FC or FCoE Nexus or MDS NPIV N5K FCoE-NPV FCoE Only TAC-Time

Cisco Live 2016 2/3/2018 Nexus FC NPV vs FCoE-NPV Enabling FC and NPV FC or FCoE Nexus or MDS NPIV N5K NPV FC First Enable FCoE and then NPV N5K(config)# feature fcoe FC license checked out successfully fc_plugin extracted successfully FC plugin loaded successfully FCoE manager enabled successfully FC enabled on all modules successfully N5K(config)# feature npv Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y FCoE Also enable fcoe qos TAC-Time

Cisco Live 2016 2/3/2018 Nexus FC NPV vs FCoE-NPV Enabling FC and NPV FC or FCoE Nexus or MDS NPIV N5K NPV FC Actually, after enabling fcoe, then NPV, What really happens… N5K(config)# feature npv Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y When the switch is reloaded in the NPV mode, Some configuration is saved: switchname management ip configuration and vrf boot variable username / password details ntp configuration callhome configuration snmp-server details feature fcoe FCoE TAC-Time

Cisco Live 2016 2/3/2018 Nexus FC NPV vs FCoE-NPV Enabling FCoE-NPV FC or FCoE Nexus or MDS NPIV N5K FCoE-NPV FCoE Enable fcoe-npv n5k(config)# feature fcoe-npv FCoE NPV license checked out successfully fc_plugin extracted successfully FC plugin loaded successfully FCoE manager enabled successfully FCoE NPV enabled on all modules successfully No reload No reconfiguration fcoe qos still required TAC-Time

Nexus FC NPV vs FCoE-NPV Comparison
Protocols FC and/or FCoE FCoE License FC_FEATURES_PKG FCOE_NPV_PKG Command feature fcoe feature npv feature fcoe-npv Write Erase Reload no Nexus Models N5K, N6K N5K, N6K, N9K FCoE QoS Required Disable FKA on Core for ISSU No if uplink is FC yes TAC-Time

Cisco Live 2016 2/3/2018 Nexus FC NPV vs FCoE-NPV Enabling NPV FC or FCoE Nexus or MDS NPIV N5K NPV FC + NPV First enable FCoE feature and then NPV Requires full storage services license Write erase reload is done This method allows both FCoE and FC upstream and host NPV connections FCoE-NPV enable fcoe-npv Method is preferred if you have a pure Ethernet environment with FCoE No write erase is done No storage service license is required TAC-Time

Nested NPV – Can I connect two NPV Switches?
Cisco Live 2016 2/3/2018 Nested NPV – Can I connect two NPV Switches? After enabling NPV, NPIV can also be enabled Connecting two Cisco NPV switches is not supported Cisco NPIV NPV+NPIV FC or FCoE NPV Cisco NPIV NPV+NPIV FC or FCoE 3rd Party Vendor Unsupported Supported TAC-Time

Bugs TAC-Time

Cisco Live 2016 2/3/2018 Frequent Bugs CSCun Weak CBC mode and weak ciphers should be disabled in SSH server Symptom: SSH servers on Cisco Nexus devices may be flagged by security scanners due to the inclusion of SSH ciphers and HMAC algorithms that are considered to be weak. These may be identified as 'SSH Server CBC Mode Ciphers Enabled' and 'SSH Server weak MAC Algorithms Enabled' or similar. Conditions: This issue applies to Cisco Nexus 7000, Cisco Nexus 5000 and MDS series switches. SSH functionality is enabled by default in Cisco NX-OS. The current SSH server status is displayed using the show ssh server command. TAC-Time

Frequent Bugs CSCun Weak CBC mode and weak ciphers should be disabled in SSH server With the Fix: If an SSH client configured to use weak ciphers is used to log in to a Cisco device with this fix, the login may fail. The following messages are logged in the switch syslog: %DAEMON-2-SYSTEM_MSG: fatal: no matching cipher found: client 3des- cbc,blowfish-cbc server aes128-ctr,aes192-ctr,aes256-ctr - sshd Reconfigure any SSH clients not to use weak ciphers like 3des-cbc or blowfish- cbc. DCNM uses SSH to manage Cisco devices and must be upgraded to at least 7.2(1) to work with devices with this fix. Known-fixed-releases: 7.3(0)N1(1) 7.2(1)N1(1) 7.2(0)N1(1) 6.2(11c) 5.2(8g) TAC-Time

Frequent Bugs CSCue79881 - SNMP crashes on SNMP bulk get query
Symptom: The SNMP process may crash with the following messages displayed in the output of show logging log %KERN-2-SYSTEM_MSG: mts_is_q_space_available_new():1416:Total mtsbuf size for sap 28, exceeds limit 15 perc of kernel %KERN-2-SYSTEM_MSG: mts_acquire_q_space() failing - no space in sap 28, uuid 26 send_opc 3176, pid 3616, proc_name sctpt_rx_thr - kernel %KERN-2-SYSTEM_MSG: [sap 28][pid 4406][comm:snmpd] sap recovering failed and so Killed - kernel %SYSMGR-2-SERVICE_CRASHED: Service "snmpd" (PID 4406) hasn't caught signal 6 (core will be saved) TAC-Time

Conditions: This bug affects both Nexus and MDS switches. It has been observed when a monitoring device is using snmp-bulk-get requests on the entity- MIB for multiple FEX modules at one time, or if there is continuous polling from multiple polling stations on slow mibs. Some examples of mibs that may be affected by continuous snmp bulk walk are: qos mib entity mib entity-fru mib bridge mib TAC-Time

Cisco Live 2016 2/3/2018 Frequent Bugs CSCue SNMP crashes on SNMP bulk get query Workaround: A possible workaround is configuring the no snmp-server counter cache enable command. This command prevents SNMP bulk gets from getting cached via the use of MTS buffers. This will prevent the MTS buffers from getting consumed and resulting in a process crash. The result of the command is that the interface table might be slower to update the statistics (since caching is disabled). Note: This workaround is only available on Nexus 7000 switches. This defect is fixed in NX-OS releases , (Nexus 7000), 5.2.8g (MDS), and (Nexus 7000 and MDS), 6.0(2)N1(2a) (Nexus 5K and 6k) Further Problem Description: A possible way of verifying if you are affected by this bug is to issue the command show system internal mts buffers summary and check if notifications for sap 28 are increasing. TAC-Time

Frequent Bugs CSCus MDS 9700 show tech detail missing some commands Symptom: MDS 9710 and MDS 9706 show tech detail missing some commands, like 'show running-config' and 'show startup-config'. Conditions: MDS 9710 and MDS at NX-OS 6.2(11). Workaround: Collect show tech-support all along with show tech-support detail Fixed in NX-OS 6.2(11c) and above. TAC-Time

Frequent Nexus 5K/6K Bugs
Cisco Live 2016 2/3/2018 Frequent Nexus 5K/6K Bugs CSCuj Unified fc interfaces come up as Ethernet after disruptive upgrades Symptom: In Cisco Nexus 5000 series switches, a disruptive upgrade with reason incompatible image causes the Unified Ports configured as FC ports to come up as Ethernet ports after upgrade. However, the FC port configuration still exists in the running configuration. Conditions: Upgrade between any two incompatible images and the fc interfaces are unified interfaces requiring the slot and port commands, slot z port x - y mode fc Backup the configuration before upgrading TAC-Time

Cisco Live 2016 2/3/2018 Frequent Nexus 5K/6K Bugs CSCuj Unified fc interfaces come up as Ethernet after disruptive upgrades Proactive Workaround: Do ISSU only between compatible images. Please check the result of install command for image compatibility. Reactive Workaround: After the disruptive ISSU between incompatible images, do the following: a. copy startup-config bootflash: b. copy running-config startup-config c. reload After reload: d. copy bootflash: running-config e. copy running-config startup-config Now the device should have the same configurations as before upgrade. Consult Release Notes for non-disruptive upgrade path TAC-Time

Cisco Live 2016 2/3/2018 Frequent Nexus 5K/6K Bugs CSCup FC interfaces are not listed in IF-MIB snmp Symptom: FC interfaces are not listed in IF-MIB snmp walk. Device Manager is not working correctly with the Nexus 5548UP or 5596UP (GEM modules installed) when the expansion module ports are set to fibre channel mode. Hovering over the ports with the mouse in Device Manager will display for example, "Ethernet 1/17 Status: failed". Looking at the same ports via CLI will show that the ports are really in FC mode and not configured as Ethernet ports. TAC-Time

Cisco Live 2016 2/3/2018 Frequent Nexus 5K/6K Bugs CSCup FC interfaces are not listed in IF-MIB snmp Conditions: Nexus 5548UP or Nexus 5596UP running NX-OS 7.0(2)N1(1) with GEM Expansion module ports configured to operate in Fibre Channel mode Some ports are in Fibre Channel mode on the base chassis. More Info: NX-OS 7.0(1)N1(1) and all previous software versions are not affected by this defect. This is an NX-OS bug, not a Device Manager bug. Fixed in 7.0(6)N1 and above. TAC-Time

Cisco Live 2016 2/3/2018 Nexus 5K/6K Bug CSCur10558 Trunk Protocol Enable does not show in running config when disabled Resolution Summary: 1 Made the command “[no] trunk protocol enable“ hidden 2. Added appropriate warning message when the command is run on CLI Fixed in 7.3(0)N1(1) 7.2(1)N1(1) 7.1(3)N1(1) 7.0(7)N1(1) (9) 5548-1(config-if)# no trunk protocol Warning: This will globally disable the switch's ability to form any trunks and impacts existing trunk ports Do you wish to continue(y/n)? [n] MDS/N7K CSCul '[no] trunk protocol enable' not displayed in show running-config Implemented in NX-OS 6.2(9) and above. Made the command “[no] trunk protocol enable” hidden Added appropriate warning message when the command is run on the CLI TAC-Time

Cisco Live 2016 2/3/2018 Nexus 5K/6K Bug CSCur10558 Trunk Protocol Enable does not show in running config when disabled [no] trunk protocol is not in the running config .. …or the show tech detail. Tip: show tech will contain show port internal info all You should always have ... Epp state: Enabled Port Trunking Protocol (PTP) and Port Channel Protocol (PCP) use the EPP frame. No practical reason to disable fibre channel trunking. MDS/N7K CSCul '[no] trunk protocol enable' not displayed in show running-config Implemented in NX-OS 6.2(9) and above. TAC-Time

Cisco MDS NX-OS 7.3(0)D1(1) OUI Enhancement
Example: Adding OUIs Switch(config)# wwn oui 0x10001c OUI - A 24 bit globally unique number assigned by IEEE. Port-channel functionality includes Cisco OUI check of peer switch. TAC-Time

MDS Slow Drain Troubleshooting Enhancements
TAC-Time

MDS Slow Drain Troubleshooting Enhancements
NX-OS 6.2(9) and 6.2(13) added several enhancements system timeout no-credit-drop triggered at exact time by HW TxWait slowport-monitor New port-monitor counters txwait tx-slowport-oper-delay tx-slowport-count show tech-support slowdrain DCNM Slow Drain Analysis TAC tool MDS_show_tech_slowdrain_analysis For a more comprehensive information see: TAC-Time - SAN Congestion! Understanding, Troubleshooting, Mitigating in a Cisco Fabric (2016 Las Vegas) TAC-Time

system timeout no-credit-drop
Triggered by HW at exact time system timeout no-credit-drop 200 mode f no-credit-drop causes frames to be dropped immediately if the destination port is at 0 Tx credits for the time specified Previously no-credit-drop was triggered by SW process at 100ms intervals NX-OS 6.2(9) and later triggered by the HW at exact time the threshold is reached Should be used in conjunction with lowering congestion-drop threshold Recommended for F ports Can drastically improve ISL performance under slow drain conditions xxx_FORCE_TIMEOUT_ON/OFF counter By default no-credit-drop is not enabled TAC-Time

TxWait enhancements txwait
txwait is a counter that increments every 2.5us when port is at 0 Tx credits and there are frames queued for transmit txwait * 2.5 / = seconds of time the port was unable to transmit Only applies to the following: MDS 9500 with generation 4 linecards: MDS 9000 Family 32-Port 8-Gbps Advanced Fibre Channel Switching Module (DS-X K9) MDS 9000 Family 48-Port 8-Gbps Advanced Fibre Channel Switching Module (DS-X K9) MDS Port 16-Gbps Fibre Channel Switching Module (DS-X K9) MDS 9148S 16G Multilayer Fabric Switch MDS 9250i Multiservice Fabric Switch MDS 9396S 16G Multilayer Fabric Switch Others will return zero TAC-Time

TxWait enhancements - continued
txwait can be seen in the following: show interface counters Raw value in 2.5us units Percentage Tx credits are available for last 1s/1m/1h/72h show process creditmon txwait-history 60sec, 60min, 72hour graphs show logging onboard txwait SNMP fcIfTxWaitCount variable TAC-Time

txwait - show interface counters mds9710-1# show interface fc1/13 counters | i fc|wait fc1/13 us Txwaits due to lack of transmit credits * 2.5 / = seconds Cumulative since the interface counters were last cleared The above indicates the MDS was not able to transmit for over 15 seconds since the counters were last cleared TAC-Time

txwait - Percentage Tx credits are available for last 1s/1m/1h/72h Utilizes the underlying txwait counter MDS9710-1# show interface fc1/13 counters fc1/13 … 5 Transmit B2B credit transitions to zero 2 Receive B2B credit transitions to zero us TxWait due to lack of transmit credits Percentage Tx credits not available for last 1s/1m/1h/72h: 1%/5%/3%/2% 32 receive B2B credit remaining 128 transmit B2B credit remaining 128 low priority transmit B2B credit remaining TAC-Time

txwait-history mds9710-1# show process creditmon txwait-history module 1 port 13 TxWait history for port fc1/13: ============================== # ## 900 # ## 800 ## ## 700 ## ## 600 ### ### 500 ### ## ### 400 ### ## #### 300 ### ## #### 200 ### ## #### 100 ### ## #### Credit Not Available per second (last 60 seconds) # = TxWait (ms) Graphical display of time where Tx credits are not available Similar in format to cpu history 3 graphs per port Last 60 seconds Last 60 minutes Last 72 hours Utilizes the underlying txwait counter TAC-Time

slowport-monitor system timeout slowport-monitor <1-500> mode e|f – Must be configured! Events are captured every 100ms Last 10 events per port captured in slowport-monitor-events Logging onboard slowport-monitor-events captures more events Currently implemented for: Gen 3 LCs - DS-X K9 and DS-X92xx-96K9 modules Gen 4 LCs - DS-X K9 and DS-X K9 modules 9700 & 9396S (Gen 5) 9250i & 9148S Differences exist between Gen3, Gen4 and 9700/9250i/9148S/9396S TAC-Time

slowport-monitor – 9700/9250i/9148S/9396S (Gen 5 LCs) Gen5/9250i/9148S/9396S have enhanced HW capabilities Each 100ms interval the number of times Tx credits remained at 0 for the configured(admin) delay is counted. The average operational delay is determined – This is how long the port was at 0 Tx credits Recorded when at least one complete event occurred More events available via logging onboard slowport-monitor-events MDS9710-1# show process creditmon slowport-monitor-events Module: 01 Slowport Detected: YES ========================================================================= Interface = fc1/ | admin | slowport | oper | Timestamp | | delay | detection | delay | | | (ms) | count | (ms) | | | 5 | 1300 | 20 | 1. 04/01/15 23:03: | | 5 | 1296 | 19 | 2. 04/01/15 23:03: | | 5 | 1291 | 19 | 3. 04/01/15 23:03: | … | 5 | 1256 | 19 |10. 04/01/15 23:03: | Actual average delay Note: Oper delay limited by no- credit-drop threshold 4 events in last 100ms Configured delay(5ms) TAC-Time

Slow Drain Alerting and Mitigation
Port-monitor alerting Port-monitor allows monitoring of several counters relating to slow drain credit-loss-reco Credit loss recovery counter lr-rx The number of link resets received by the fc-port lr-tx Link resets transmitted by the fc-port timeout-discards Timeout discards counter tx-credit-not-available Credit not available counter(in 100ms increments) tx-discards Tx discards counter tx-slowport-count Number of slowport events tx-slowport-oper-delay Slowport operational delay txwait Amount of time at 0 Tx credits and packets queued rx-datarate Rx data rate as a percentage of link speed tx-datarate Tx data rate as a percentage of link speed New in 6.2(13) Note: There are other counters that are valuable and should also be considered for inclusion in monitoring but are not part of slow drain TAC-Time

Troubleshooting – Documentation for TAC
Cisco Live 2016 2/3/2018 Troubleshooting – Documentation for TAC show tech-support slowdrain Contains all the commands available that pertain to slow drain Contains “context” commands to understand the FC topology Contains name server commands to identify devices Contains active zonesets to understand device relationships Most useful when run from DCNM and gathered for the entire fabric SAN Client -> Tools -> Run CLI Commands… When opening up a case with the TAC please have this available! Used for MDS_show_tech_slowdrain_analysis TAC-Time

DCNM Slow Drain Analysis
DCNM 7.1(1) added Slow Drain Analysis DCNM 7.2(2) added improvements DCNM 10.0(1) added improvements Used for pulling fabric wide slow drain counters for a defined period of time Useful for ongoing slow drain problems Accessed from the Web Client Health -> Diagnostics -> Slow drain Analysis TAC-Time

MDS_show_tech_slowdrain_analysis
Slow draining end device! Red indicates level 3 (credit-loss) Yellow indicates level 1 (latency) Arrows indicate direction of congestion Orange indicates level 2 (timeout drops) Green indicates no congestion TAC-Time

Cisco Live 2016 2/3/2018 TAC-Time

DC Compute SAN Carlos Lopez CCIE SAN, DC #21063

Similar presentations

Presentation on theme: "DC Compute SAN Carlos Lopez CCIE SAN, DC #21063"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

DC Compute SAN Carlos Lopez CCIE SAN, DC #21063

Similar presentations

Presentation on theme: "DC Compute SAN Carlos Lopez CCIE SAN, DC #21063"— Presentation transcript:

Similar presentations

About project

Feedback