Cisco Catalyst 6500 Series Switches:

Cisco Catalyst 6500 Series Switches:
Carlos Nivon

Comunidad de Sopórte de Cisco – Webcast en vivo
Carlos Nivón

Gracias por su asistencia el día de hoy
La presentación incluirá algunas preguntas a la audiencia. Le invitamos cordialmente a participar activamente en las preguntas que le haremos durante la sesión

Copia de la presentación
Si desea bajar una copia de la presentación de hoy, vaya a la liga indicada en el chat o use ésta dirección

Chassis Overview

Cat 6500 slot Orientation Vertically Aligned Slots
6509-NEBS-A 6513 6509-NEBS (EOS) 6509 6506 6503 Horizontally Aligned Slots Vertically Aligned Slots

Supervisors, Line cards and
other Modules

Supervisor Engine 32 Access Layer Supervisor 32

Supervisor Engine 720 Supervisor 720 with Integrated Core Layer
Switch Fabric Supervisor 720 with Integrated Switch Fabric Core Layer

Ethernet and WAN Line Cards
Ethernet Line Cards 10/100 TX and 100 Fiber 10/100/1000 TX GE SFP GE GBIC 10GE Inline Power WAN Line Cards OSM FlexWAN SIP

Advanced Services Modules
Security Firewall Module IPSec VPN Shared Port Adapter Intrusion Detection SSL Application Networking Services CSM CSM-S ACE

Advanced Services Modules (Cont.)
Wireless Services WLSM MWAM CSG IP Telephony Network Monitoring CMM T1/E1 Services Modules NAM and NAM2 TAD

Catalyst 6500 Backplane Architecture

Classic 32-Gbps Shared-Bus Backplane
Line Card Multilayer Forwarding Table 32-Gbps Shared Switching Bus PFC Switching System Control Bus Results Bus Multilayer Switch Feature Card Bus ASIC Port or Bus ASIC Local Buffer Fabric Arbitration Port ASIC Network MGMT NMP/MCP Local Buffer Supervisor Engine 10/100 Ethernet Gigabit Ethernet

Crossbar Switch Fabric
Multilayer Forwarding Table C R O S B A CEF256 Fabric ASIC Port ASIC 1 x 8 Gbps PFC Switching System dCEF256 Fabric ASIC Port ASIC 1 x 8 Gbps Multilayer Switch Feature Card 1 x 8 Gbps Fabric ASIC Port ASIC Fabric Arbitration CEF720 Fabric ASIC Port ASIC 1 x 20 Gbps Network MGMT NMP/MCP 1 x 20 Gbps Fabric ASIC Port ASIC Supervisor Engine 720

Crossbar Switch Fabric Layout Nine-Slot Chassis
Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Slot5 Fabric ASIC Slot 5 Slot6 Fabric ASIC Slot 6 Type of card in slot: Fabric ASIC Fabric ASIC Fabric ASIC = Fabric (SFM/Sup) Slot7 Slot8 Slot9 = Line Card

Crossbar Switch Fabric 13-Slot Chassis
Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Slot7 Fabric ASIC Slot 7 Slot8 Fabric ASIC Slot 8 Type of card in slot: Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC = Fabric (SFM/Sup) Slot9 Slot10 Slot11 Slot12 Slot13 = Line Card

Introducing the Shared Bus and Switch Fabric Architectures

CEF Forwarding Architectures
Features of CEF forwarding architectures include the following: CEF Hardware-based centralized forwarding PFC on supervisor makes all forwarding decisions Handles centralized forwarding up to 30 Mpps dCEF Hardware-based distributed forwarding dCEF engine has a copy of the entire forwarding table at the line card All traffic is switched at a sustained 48 Mpps (for DFC3 on CEF720)

Supervisor Engine 720 Switch-Fabric Connectivity
MSFC3 30 to 400 Mpps Forwarding Performance Routing Table PFC3 Hardware Fwd Tables z CEF720 Series dCEF720 Series 20 Optional DFC3 Integrated DFC3 20 20 Integrated Switch Fabric 20 20 8 8 32-Gbps Switching Bus 8 CEF256 Series dCEF256 Series Classic Series Optional DFC3 Integrated DFC3

Supervisor Engine 32 Supervisor Engine 32 with Eight GE Uplinks
WS-SUP32-GE-3B Supervisor Engine 32 with Two 10-GE Uplinks WS-SUP32-10GE-3B

Supervisor Engine 32: Front Panel
8 x SFP based GE Uplink Ports 2 x USB Ports Compact Flash Slot 1 x 10/100/1000 GE Uplink Port RS-232 Console Port

Integrated PFC3 PFC3B Supervisor Engine 32

Integrated MSFC2a MSFC2a Supervisor Engine 32

Supervisor Engine 32 Line Card Compatibility
Architecture Supported? Classic YES CEF256 dCEF256 NO CEF720 dCEF720 SFM/SFM2 Services Modules Any DFC OSM* SIP FlexWAN Supervisor Engine 32 *OSM: Original Storage Manufacturer

Supervisor Engine 720 Overview
Console Port Uplink Ports Removable Storage Slots

Supervisor Engine 720 Options
Supervisor Engine 720-3B Supervisor Engine 720-3BXL Incorporates new PFC3B to provide the same features as the XL version but not as high a capacity for routes and flow information Incorporates new PFC3BXL, extending hardware features and system capacity for routes and flow information

Catalyst 6500 Supervisor Engine 720 PFC Options
Name PFC3A PFC3B PFC3B-XL Routes 256,000 1 million Number of ACLs 512 4000 NetFlow Entries 128,000 (64,000) 128,000 (115,000) 256,000 (230,000) ACE Counters No Yes MPLS Default Memory SP 512 MB + RP 512 MB SP 1 GB + RP 1 GB

Supervisor Engine 720 Switch Fabric
Integrated 720-Gbps switch fabric. CEF256 and dCEF256 connect in at 8 Gbps per fabric channel. CEF720 and dCEF720 connect in at 20 Gbps per fabric channel. Switch Fabric

Supervisor Engine 720 Hardware Features
IPv6 Software Features IPv6 addressing ICMP for IPv6 DNS for IPv6 V6 MTU path discovery SSH for IPv6 IPv6 Telnet IPv6 traceroute dCEF for IPv6 RIP for IPv6 IS-IS for IPv6 OSPF v3 for IPv6 BGP for IPv6 IPv6 Hardware Features 128,000 FIB entries IPv6 load sharing up to 16 paths EtherChannel hash across 48 bits IPv6 policing/NetFlow/classification STD and EXT V6 ACLs IPv6 QoS lookups IPv6 multicast IPv6-to-IPv4 Tunneling IPv6 edge over MPLS (6PE) IPv6 function located on PFC3

MPLS Hardware Features
MPLS applies to any Ethernet port on the following line cards: Classic Ethernet Line Cards CEF256 Ethernet Line Cards MPLS HARDWARE FEATURES Up to 1000 MPLS VPNs MPLS VPN (RFC 2457) on any Ethernet port MPLS multicast VPN MPLS label switch router (LSR) MPLS label edge router (LER) MPLS Traffic Engineering (TE) MPLS Ethernet over MPLS (EoMPLS) on PFC3B DSCP-to-EXP mapping dCEF256 Ethernet Line Cards CEF720 Ethernet Line Cards dCEF720 Ethernet Line Cards MPLS function located on PFC3

Catalyst 6500 Architecture Overview
Catalyst 6500 Line Cards

Optical Services Modules
Catalyst 6500 Line Cards C A T L Y S 6 5 10/100BASE-TX and 100BASE-FX 10/100/1000BASE-TX Gigabit Ethernet SFP L I N E C A R D S GE GBIC 10GE WAN Optical Services Modules In-line Power SIP

Classic and Crossbar Switch Fabric Line Cards
Shared Bus Connector Crossbar Connector Shared Bus Connector Classic CEF256

Switch Fabric Crossbar
Line Card Types 32-Gbps Shared Bus Classic Line Cards CEF256 Line Cards CEF720 Line Cards 8 20 20 dCEF256 Line Cards dCEF720 Line Cards Supervisor 8 8 20 20 Switch Fabric Crossbar

Classic Line Card Architecture
Classic line cards support a connection to the 32- Gbps shared bus only. 32-Gbps Shared Bus Gigabit Ethernet ASIC 10/100 ASIC 10/100 ASIC 10/100 ASIC 10/100 ASIC Buffer Buffer Buffer Buffer Ports 1–12 Ports 13–24 Ports 25–36 Ports 37–48 48-Port 10- and 100-MBps Line Card

CEF256 Line Card Architecture
Crossbar CEF256 line cards support a connection to the 32-Gbps shared bus and an 8-Gbps connection to the switch fabric. 32-Gbps Shared Bus 8 Optional DFC Daughter Card Fabric ASIC 32 Gbps Local Switching Bus Port ASIC Port ASIC Port ASIC Port ASIC 512-KB Buffer 512-KB Buffer 512-KB Buffer 512-KB Buffer Ports 1–4 Ports 5–8 Ports 9–12 Ports 13–16 16-Port Gigabit Ethernet Line Card

dCEF256 Line Card Architecture
Crossbar 8 8 dCEF256 line cards support two 8-Gbps connections to the switch fabric only. Fabric ASIC Fabric ASIC Integrated DFC and DFC3 32-Gbps Local Bus 32-Gbps Local Bus Port ASIC Port ASIC Port ASIC Port ASIC 512-KB Buffer 512-KB Buffer 512-KB Buffer 512-KB Buffer Ports 1–4 Ports 5–8 Ports 9–12 Ports 13–16 16-Port Gigabit Ethernet Line Card

CEF720 Line Card Architecture
Crossbar 20 20 32-Gbps Shared Bus Fabric ASIC Optional DFC3 Daughter Card Fabric ASIC Port ASIC Port ASIC Port ASIC Port ASIC Ports 1–12 Ports 13–24 Ports 25–36 Ports 37–48 48-Port Gigabit Ethernet Line Card

dCEF720 Line Card Architecture
Crossbar 20 20 dCEF720 line cards support two 20-Gbps connections to the switch fabric only. Integrated DFC Fabric ASIC Fabric ASIC Port ASIC Port ASIC Port ASIC Port ASIC Ports 1–12 Ports 13–24 Ports 25–36 Ports 37–48 48-Port Gigabit Ethernet Line Card

Line Card Packet Flow

Classic-to-Classic Centralized Forwarding
Layer 3 and Layer 4 Engine Supervisor Engine 720 Red 4 D 2 Port ASIC Port ASIC Layer 2 Engine Classic Module B 720-Gbps Switch Fabric X 1 3 PFC3 DBUS RBUS X Source Destination Blue VLAN Red VLAN Entire Packet Packet Header X S Classic Module A Port ASIC Port ASIC D Blue S

CEF256-to-CEF256 Centralized Forwarding
Port ASIC Port ASIC Layers 3 and 4 Engine Supervisor Engine 720 2 LCRBUS LCDBUS L2 Engine 720-Gbps Switch Fabric Fabric Interface 3 8Gbps CEF256 Module B 5 PFC3 DBUS Source Destination Blue VLAN Red VLAN Entire packet Packet header S RBUS D Fabric Interface 8Gbps CEF256 Module A 4 1 LCDBUS LCRBUS X X Port ASIC Port ASIC Note: Packet flow for a CEF256-to-CEF720 is similar. The main differences are the CEF720 module architecture and the speed of the fabric channel to the CEF720 module. Blue S

CEF720 and DFC3-to-CEF720 and DEFC3 Distributed Forwarding
Red D 5 Port ASIC Port ASIC CEF720 Module B and DFC3 Layers 3 and 4 Engine DFC3 Supervisor Engine 720 PFC3 720-Gbps Switch Fabric Fabric Interface and Replication Engine 20Gbps Layer 2 Engine 4 20Gbps Source Destination Blue VLAN Red VLAN Entire Packet Packet Header S CEF720 Module A and DFC3 D Fabric Interface and Replication Engine 2 Layer 2 Engine 3 1 Layers 3 and 4 Engine Port ASIC Port ASIC DFC3 Blue S

Catalyst 6500 Line Card Options
CEF720 √ dCEF256 CEF256 Classic Interface Type 10/100BASE-TX 100BASE-FX 10/100/1000BASE-TX 1000BASE GBIC 1000BASE SFP 10GE XENPAK 10BASE-FL Services Modules FlexWAN OSMs* SIP * OSM: Optical Services Module

Troubleshooting the Catalyst 6500

Basic Performance check

show Commands The switch supports two slots for the supervisor engines. A CLI command is provided to allow the administrator to inspect which of the SFMs is active: 6500# show fabric active Active fabric card in slot 5 No backup fabric card in the system The mode of operation in use by the SFM can also be inspected by issuing the following command: 6500# show fabric switching-mode Fabric module is not required for system to operate Modules are allowed to operate in bus mode Truncated mode is not allowed unless threshold is met Threshold for truncated mode operation is 2 SFM-capable cards Module Slot Switching Mode Crossbar Crossbar Crossbar DCEF

show Commands (Cont.) The status of the SFM can be inspected by using the following command: 6500# show fabric status slot channel speed module fabric status status G OK OK G OK OK G OK OK G OK OK The utilization of the SFM can be inspected by using the following command: 6500# show fabric utilization slot channel speed Ingress % Egress % G G G G

show Commands (Cont.) During troubleshooting, the SFM can be inspected for transmission errors: 6500# show fabric errors Module errors: slot channel crc hbeat sync DDR sync Fabric errors: slot channel sync buffer timeout 6500#

System Capacity Planning
C6500# show platform hardware capacity ? acl Show QoS/Security ACL capacity cpu Show CPU resources capacity eobc Show EOBC resources capacity fabric Show Switch Fabric resources capacity flash Show Flash/NVRAM resources capacity forwarding Show forwarding engine capacity interface Show Interface resources capacity monitor Show SPAN resources capacity multicast Show L3 Multicast resources capacity netflow Show Netflow capacity pfc Show PFC resources capacity power Show Power resources capacity qos Show QoS resources capacity rate-limit Show CPU Rate Limiters capacity system Show System resources capacity vlan Show VLAN resources capacity New CLI command that provides a dashboard view of system hardware capacity, as well as the current utilization of the system.

Oversubscription

Simplified Campus Example
6:1 WS-X6548-GE-TX (CEF256) 48 ports and 8-Gb 4:1 oversubscription 2x WS-X6548-GE-TX (CEF256) 48 ports, 8-Gbps backplane 8:1 oversubscription = 16 Gb 8:1 Access 1x Supervisor Engine 720 2x 1-Gb uplinks = 2 Gb 1.2:1 WS-X6724-SFP (CEF720) 24 ports and 20-Gb backplane 1.2:1 oversubscription Aggregation Total core-edge oversubscription ≈ 58:1 Traffic flows vertically, bidirectional Low overall bandwidth requirements

High CPU

High CPU Utilization Why should I be concerned about high CPU usage ?
It is very important to protect the control-plane for network stability, as resources (CPU, Memory and buffer) are shared by control-plane and data-plane traffic What are the usual symptoms of high CPU usage ? Control-plane instability e.g., OSPF flap Traffic loss Reduced switching/forwarding performance Slow response to Telnet / SSH SNMP poll miss At what percentage level at should I start troubleshooting ? It depends on the nature and level of the traffic. It is very essential to find a baseline CPU usage during normal working conditions, and start troubleshooting when it goes above specific threshold. E.g., Baseline RP CPU usage 25%. Start troubleshooting when the RP CPU usage is consistently at 40% or above.

Commands used to set baseline
High CPU Utilization Commands used to set baseline RP: show process cpu RP: show ibc RP: show msfc netint 1 Gbps Inband MSFC 3 Port ASIC Flash RP CPU C DRAM RP: show ip traffic RP: show interfaces C SP CPU Flash 1 Gbps Inband DRAM Sup720 SP: show process cpu SP: show msfc netint SP: show ibc Monitor the CPU usage in DFCs also using “remote command module <mod#> show process cpu” C = Controller

Total CPU usage (Process + Interrupt) CPU usage due to Interrupt
High CPU Utilization CPU utilization is due to: Process (e.g., due to recurring events, control-plane process) Interrupts (e.g., due to inappropriate switching path) Investigate CPU utilization via “show proc cpu” and find if the usage is due to process or interrupts DUT#show proc cpu CPU utilization for five seconds: 99%/90%; one minute: 9%; five minutes: 8% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process % 1.11% 0.23% 18 Virtual Exec Total CPU usage (Process + Interrupt) CPU usage due to Interrupt

High CPU utilization – Process
Process: ARP Input Caused by ARP flooding. Static route configured with interface instead of next-hop IP address. This will generate ARP request for every packet that is not reachable via more specific routes. ip route GigabitEthernet 2/5 DUT#show ip traffic | begin ARP ARP statistics: Rcvd: 6512 requests, 2092 replies, 0 reverse, 0 other Sent: 258 requests, 707 replies (0 proxy), 0 reverse Drop due to input queue full: 20 <snip> DUT#show interfaces | include line protocol|rate Vlan501 is up, line protocol is up 5 minute input rate bits/sec, 2535 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec Incrementing at very high rate Look for abnormal input rate

Process: IP Input Caused by traffic that needs to process-switched or destined to the CPU Common Reasons: Traffic with IP-options enabled Fragmentation (due to MTU mismatch) Broadcast storm Traffic that needs further CPU processing e.g., ACL Logging Traffic to which ICMP Redirect or Unreachable required e.g., TTL=1, ACL Deny etc. Configure Optimized ACL Logging (OAL) in PFC3 onwards

Due to aggressive polling. “show snmp” provides SNMP input and output stats Process: SNMP Engine DUT#show process cpu | include CPU|SNMP CPU utilization for five seconds: 71%/0%; one minute: 29%; five minutes: 8% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process % 31.11% 7.05% 0 SNMP ENGINE Process: BGP Scanner Walks the BGP table and confirms reachability of the next hops. It also checks conditional-advertisement to determine whether or not BGP should advertise condition prefixes, performs route dampening. It is normal to see this process spiking up for short duration, when the device carries huge internet routing table. DUT#show proc cpu | include BGP PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process % 0.00% 0.00% 0 BGP Router % 0.00% 0.00% 0 BGP I/O % 0.00% 0.00% 0 BGP Scanner When adding, removing or soft-reconfiguration of BGP peers BGP control traffic

Process: Exec and Virtual Exec DUT#show process cpu | include CPU|Virtual |Exec CPU utilization for five seconds: 30%/0%; one minute: 8%; five minutes: 5% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process % 2.12% 1.89% 0 Exec % 0.00% 0.00% 1 Virtual Exec Responsible for tty lines (console, auxiliary) High CPU when too many messages sent to console / vty Responsible for vty lines (telnet, SSH) Check if any debug is enabled via “show debug”. Issue “undebug all” if it is not needed DUT#show debugging Generic IP: IP packet debugging is on Disable logging via “no logging console” or “no logging terminal”

High CPU utilization – Traffic to RP CPU
DUT#show ip traffic IP statistics: Rcvd: total, local destination 0 format errors, 0 checksum errors, bad hop count 0 unknown protocol, not a gateway 0 security failures, 0 bad options, 120 with options Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble 0 fragmented, 0 couldn't fragment Bcast: 417 received, 0 sent Mcast: received, sent Sent: generated, 0 forwarded Drop: 0 encapsulation failed, 0 unresolved, 0 no adjacency 0 no route, 0 unicast RPF, 0 forced drop 0 options denied, 0 source IP address zero ICMP statistics: Rcvd: 0 format errors, 0 checksum errors, 17 redirects, 112 unreachable 812 echo, 812 echo reply, 0 mask requests, 0 mask replies, 0 quench 0 parameter, 0 timestamp, 0 info request, 0 other 0 irdp solicitations, 0 irdp advertisements 0 time exceeded, 0 timestamp replies, 0 info replies ARP statistics: Rcvd: requests, replies, 0 reverse, 0 other TTL<2 IP options Fragmentation Broadcasts ARP not resolved Ping Request Punts to generate ICMP redirect ARPs It also displays stats for : BGP, EIGRP, TCP, UDP, PIM, IGMP and OSPF Do this command few times to find the fastest growing counter

High CPU utilization – Traffic to RP CPU
Find the interface that's holding most of the buffers Commands to see packets getting punted DUT#show buffers assigned Header DataArea Pool Rcnt Size Link Enc Flags Input Output 46FDBC Small Vl None 46FE CBC4 Small Vl None . . . DUT#show buffers input-interface vlan 100 dump Buffer information for RxQ3 buffer at 0x378B3BC data_area 0x7C05EF0, refcount 1, next 0x0, flags 0x200 linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1 if_input 0x46C7C68 (Vlan100), if_output 0x0 (None) inputtime 2d03h (elapsed 00:00:01.024) outputtime 00:00: (elapsed never), oqnumber 65535 datagramstart 0x7C05F36, datagramsize 62, maximum size 2196 mac_start 0x7C05F36, addr_start 0x7C05F36, info_start 0x0 network_start 0x7C05F44, transport_start 0x7C05F58, caller_pc 0x6C1564 source: , destination: , id: 0x0000, ttl: 1, TOS: 192 prot: 17, source port 1985, destination port 1985 0: AFACEFAD /,o 12: 28: 44: CC43 C00C A0 60: FF74D B....tU 76: 5E A C00030 92: D5 8922DB03 E tU."[.`... 108: 07C107C1 001CECB A.A..l4.....d.. 124: F DB cisco...."[.A..P Find the traffic. Please remember that the traffic seen may be normal control-plane traffic, expected to be sent to RP CPU Packet details Remember, this command shows only the process-switched traffic

High CPU utilization – Interrupt
How to troubleshoot high CPU due to interrupts ? DUT#show proc cpu CPU utilization for five seconds: 99%/90%; one minute: 9%; five minutes: 8% Most of the times, packets punted to CPU has common factors. Packets received on the same vlan / interface or interfaces in the same module or same VRF etc. Packet have specific destination or destination prefixes learnt from a specific neighbor Packet have same L4 source or destination ports Anything else common ? Details on all supported Packet Capture Tools

Verify CEF is enabled globally and on all interfaces DUT#show cef state CEF Status: RP instance common CEF enabled IPv4 CEF Status: CEF enabled/running dCEF enabled/running CEF switching enabled/running DUT#show ip interfaces | include line pro|CEF switching Vlan2 is up, line protocol is up IP CEF switching is enabled Vlan3 is up, line protocol is up Verify if CEF is enabled globally and per interface

Switching path statistics – per interface basis DUT#show interface gig7/4 stats GigabitEthernet7/4 Switching path Pkts In Chars In Pkts Out Chars Out Processor Route cache Distributed cache Total DUT#show interface switching GigabitEthernet2/2 Protocol Path Pkts In Chars In Pkts Out Chars Out IP Process Cache misses 0 Fast Auton/SSE ARP Process Process switched SW CEF switched Hw-switched Process name Process switched Distributed switched packets

High CPU utilization – Hardware rate-limiters
UNICAST RATE LIMITERS LAYER 2 RATE LIMITERS CEF Receive Traffic destined to the Router L2PT encapsulation/decapsulation L2PT CEF Glean ARP packets Layer 2 PDUs: BPDU, CDP, VTP, PAGP, LACP, Pause, DTP, UDLD PDU CEF No Route Packets with no route in the FIB IP Errors Packets with IP checksum or length errors ICMP Redirect Packets that require ICMP redirects ICMP No Route ICMP unreachables for unroutable packets Similarly, Sup720/PFC3B or 3BXL supports multicast rate-limiters in hardware ICMP ACL Drop ICMP unreachables for admin deny packets RPF Failure Packets that fail uRPF check L3 Security CBAC, Auth-Proxy, and IPSEC traffic ACL Input NAT, TCP Int, Reflexive ACLs, Log on ACLs ACL Output NAT, TCP Int, Reflexive ACLs, Log on ACLs VACL Logging Notification of VACL denied packets GENERAL RATE LIMITERS IP Options Unicast traffic with IP Options set MTU Failure Packets requiring fragmentation Capture Used with Optimized ACL Logging TTL Failure Packets with TTL< 2

Common Reasons for High CPU Utilization
Same interface forwarding (to generate ICMP redirects) ACL log TTL<2 IP options Fragmentation ACL deny or no route packet (to generate ICMP unreachable) Forwarding exception (out of TCAM / Adjacency space) Feature exception (out of TCAM space / conflict) SW-supported feature (crypto, NBAR) Multicast RPF drops Platform-specific traffic handling Forwarding path issues – requires troubleshooting

NetDriver (Netdr) Debug
Be as specific as possible; on SP, remote login switch, then same set of commands) DUT#debug netdr capture ? acl (11) Capture packets matching an acl and-filter (3) Apply filters in an and function: all must match continuous (1) Capture packets continuously: cyclic overwrite destination-ip-address (10) Capture all packets matching ip dst address dstindex (7) Capture all packets matching destination index ethertype (8) Capture all packets matching ethertype interface (4) Capture packets related to this interface or-filter (3) Apply filters in an or function: only one must match rx (2) Capture incoming packets only source-ip-address (9) Capture all packets matching ip src address srcindex (6) Capture all packets matching source index tx (2) Capture outgoing packets only vlan (5) Capture packets matching this vlan number <cr> This debug should not be service-impacting 69

Does the CPU Inband Driver See the Packet?
DUT#show netdr captured-packets A total of 289 packets have been captured The capture buffer wrapped 0 times Total capture capacity: 4096 packets dump of incoming inband packet interface Vl1000, routine mistral_process_rx_packet_inlin dbus info: src_vlan 0x3E8(1000), src_indx 0x45(69), len 0x40(64) bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x43E8(17384) E E E80000 mistral hdr: req_token 0x0(0), src_index 0x45(69), rx_offset 0x76(118) requeue 0, obl_pkt 0, vlan 0x3E8(1000) destmac FF.FF.FF.FF.FF.FF, srcmac 00.A0.CC C4, protocol 0806 layer 3 data: A0CC21 94C FE E8 ... DUT#undebug netdr DUT#debug netdr clear-capture Example of inbound packet on interface VLAN 1000 ARP packet Make sure to turn it off afterwards Make sure to clear memory used up by captured packets 70

Enhanced crashinfo

Crashes Crashes will require TAC involvement
Open a TAC service request and collect the following info: Crashinfo file Core file (if configured so) Show tech-support What you were doing that made it crash!!

Example of Process Crash Output
Crashing process ID Crashing process name 00:05:29: %DUMPER-3-PROCINFO: pid = 16427: (sbin/tcp.proc), terminated due to signal SIGTRAP, trace trap (not reset when caught) (Signal from user) 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: zero at v v1 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: a a a a3 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R4 7BC 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: t t t t3 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: t t t t7 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: s s s s3 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R16 00FDDFA 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: s s s s7 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: t t k k1 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R B3F4C 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: gp sp s ra 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R FF90 00FDDF 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: sr lo hi bad 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R FC 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: cause pc epc 00:05:29: %DUMPER-3-REGISTERS_INFO: 16427: R B3F5C 00:05:29: %DUMPER-3-TRACE_BACK_INFO: 16427: (libc.so+0x2EF5C) (libc.so+0x12450) (s72033_rp-adventerprisek9_wan-58-dso-p.so+0x17C00) (libc.so+0x127AC) 00:05:30: %DUMPER-3-CRASHINFO_FILE_NAME: 16427: Crashinfo for process sbin/tcp.proc at bootflash:/crashinfo_tcp.proc 00:05:30: %DUMPER-3-CORE_FILE_NAME: 16427: Core for process sbin/tcp.proc at disk0:/tcp.proc dmp.Z 00:05:31: %DUMPER-5-DUMP_SUCCESS: 16427: Core dump success 00:05:31: %SYSMGR-3-ABNORMTERM: tcp.proc:1 (jid 91) abnormally terminated, restarted scheduled Crashinfo filename and location Core filename and location [SAMPLE BODY SLIDE NOTES] The following information is needed during the Host Sensor Solaris Agent installation: Agent name—The name should be determined during the design phase. Agent type—The Agent type should be determined during the design phase. The type is typically dependent on the purpose of the server to be protected. For example, you would select the WSE for an e-commerce server. Console communication parameters—The Console communication parameters should be determined during the design phase. These parameters include the IP address of the Console station, and the TCP port number used for Agent-to-Console communication. Console public key—The Console’s public key is required to activate the Agent. Have the key available during installation to avoid having to manually copy the key to the Agent. Warning: The Host Sensor Solaris Agent requires that the Console host be reachable by using the ping command. The installation process will fail if it is not. 73 4

Example of What Files to Collect After Crash
For previous slide tcp.proc process crash you need to collect the following files: Cat6K#dir bootflash: Directory of bootflash:/ 4 -rw Sep :28:42 -06:00 crashinfo_tcp.proc bytes total ( bytes free) Cat6K#dir disk0: Directory of disk0:/ 1 -rw Sep :26:54 -06:00 s72033-adventerprisek9_wan_dbg-vz.PP_R31_INTEG_050829 2 -rw Sep :50:54 -06:00 s72033-adventerprisek9_wan_dbg-vz.pikespeak_r31_0908_1 3 -rw Sep :50:04 -06:00 s72033-adventerprisek9_wan-vz SX1010 4 -rw Sep :28:42 -06:00 tcp.proc dmp.Z bytes total ( bytes free) Both filenames encode the process that crashed Crashinfo filename and location [SAMPLE BODY SLIDE NOTES] The following information is needed during the Host Sensor Solaris Agent installation: Agent name—The name should be determined during the design phase. Agent type—The Agent type should be determined during the design phase. The type is typically dependent on the purpose of the server to be protected. For example, you would select the WSE for an e-commerce server. Console communication parameters—The Console communication parameters should be determined during the design phase. These parameters include the IP address of the Console station, and the TCP port number used for Agent-to-Console communication. Console public key—The Console’s public key is required to activate the Agent. Have the key available during installation to avoid having to manually copy the key to the Agent. Warning: The Host Sensor Solaris Agent requires that the Console host be reachable by using the ping command. The installation process will fail if it is not. 74 4

Best Practices

Overview of Reliability in the Cisco Catalyst 6500 Series Switch

Cisco 6500 System Reliability
Resiliency (Layer 2 or Layer 3): SSO, NSF Fault Detection GOLD Soft HA Network Element Redundancy Operations OIR of Line Cards OIR of Sup OIR of PSU, Modules TDR NAIS Redundancy Supervisor Switch Fabric Service Modules Clock Fans Power Supplies Network Resilience Operational Processes Protection Schemes: HSRP/GLBP/VRRP, EtherChannel, 802.1s/w, PVST+

Using Route Processor Redundancy and RPR+

RPR and RPR+ The Catalyst 6500 supports failover between two supervisors installed in the switch. Two fault tolerant modes can be configured; Route Processor Redundancy (RPR) and Route Processor Redundancy Plus (RPR+). RPR Catalyst 6500 RPR provides failover generally within 2 to 4 minutes RPR+ requires both supervisors to be the same, and both must run the same IOS image. Sup720-A Sup720-B RPR+ RPR+ provides failover generally within seconds PSU PSU

Configuring RPR and RPR+
Configuration of RPR and RPR+ is achieved by entering redundancy configuration mode, then choosing the mode you wish to run. 6500# conf t Enter configuration commands, one per line. End with CNTL/Z. 6500(config)# redundancy 6500(config-red)# mode ? rpr Route Processor Redundancy rpr-plus Route Processor Redundancy Plus RPR RPR+ 6500(config-red)# mode rpr 6500(config-red)# mode rpr-plus

Confirming RPR, RPR+ Status
The redundant configuration status of the switch can be viewed using the following command: 6500# show redundancy states my state = 13 -ACTIVE peer state = 1 -DISABLED Mode = Simplex Unit = Primary Unit ID = 5 Redundancy Mode (Operational) = Route Processor Redundancy Plus Redundancy Mode (Configured) = Route Processor Redundancy Plus Split Mode = Disabled Manual Swact = Disabled Reason: Simplex mode Communications = Down Reason: Simplex mode client count = 11 client_notification_TMR = milliseconds keep_alive TMR = 9000 milliseconds keep_alive count = 0 keep_alive threshold = 18 RF debug mask = 0x0 Redundant State Configured

Catalyst 6500 Supervisor Redundancy
Using SSO and NSF

SSO Overview

SSO Overview DFC DFC DFC Active Supervisor Standby Supervisor
MSFC PFC Active and standby supervisors run in synchronized mode. Redundant MSFC is in hot-standby mode. Switch processors synchronize STP, port and VTP states. PFCs synchronize Layer 2 and Layer 3 FIB, Netflow and ACL tables. DFCs are not repopulated with Layer 2 and Layer 3 FIB, Netflow and ACL tables. Very fast failover (0 to 3 seconds) between supervisors but still need to rebuild routes on external routers. Active Supervisor Sup MSFC PFC Standby Supervisor Line Card DFC Line Card DFC Line Card DFC

SRM with SSO Overview RP RP RP RP SP SP SP SP PFCx PFCx PFCx PFCx DFCx
Active Standby Standby Active RP RP RP RP New RP builds table and reestablishes neighbor relationships. SP SP SP SP STP, Port, VTP States STP, Port, VTP States Layer 3 traffic forwards on last known FIB in hardware. PFCx PFCx PFCx PFCx Layer 2 and Layer 3 FIB, Netflow, ACL Tables Layer 2 and Layer 3 FIB, Netflow, ACL Tables DFCx DFCx DFCs not affected by supervisor failover Layer 2 and Layer 3 FIB, Netflow, ACL Tables Layer 2 and Layer 3 FIB, Netflow, ACL Tables Before Failover After Failover

NSF Overview Predictable traffic path No route flap PSU 1 PSU 2
Catalyst 6500 NSF-aware neighbor Linecard 1 Linecard 3 Failover time: 0 to 3 seconds Linecard 3 NSF-capable router Linecard 4 NSF-aware neighbor Primary Supervisor 720 Redundant Supervisor 720 Linecard 7 Linecard 8 Predictable traffic path No route flap Linecard 9 PSU 1 PSU 2 NSF-aware neighbors do not reconverge. NSF-aware neighbors help the NSF-capable router restart. NSF-aware neighbors continue forwarding traffic to the restarting router. NSF-capable router rebuilds Layer 3 routing protocol database from neighbor. Data is forwarded in hardware based on preswitchover CEF information while routing protocols reconverge.

NSF Configuration To configure SSO to use NSF:
6500(config)# redundancy 6500(config-red)# mode sso To verify the configuration: 6500# show redundancy states

BGP NSF Configuration To configure BGP NSF:
6500(config)# router bgp as-number 6500(config-router)# bgp graceful-restart To verify the configuration: 6500# show ip bgp neighbors x.x.x.x

OSPF NSF Configuration
To configure OSPF NSF: 6500(config)# router ospf processID 6500(config-router)# nsf To verify the configuration: 6500# show ip ospf

ISIS NSF Configuration
To configure ISIS NSF: 6500(config)# router isis tag 6500(config-router)# nsf [cisco | ietf] To verify the configuration: 6500# show running-config 6500# show isis nsf

EIGRP NSF Configuration
To configure EIGRP NSF: 6500(config)# router eigrp as-number 6500(config-router)# nsf To verify the configuration: 6500# show running-config 6500# show ip routing

Redundancy Modes RPR 2-4 minutes All releases RPR+ 30-60 seconds
SRM with SSO 0-3 seconds Layer 2 12.2(17b)SXA 12.2(17d)SXB NSF with SSO layers 2-4 12.2(18)SXD

Reasons to Use Storm Control

DoS Protection: Control Plane Protection
High rates of link level broadcast traffic impact switch CPU and the stability of the network: Storm control limits the rate of broadcast traffic received by the distribution switch. Broadcast traffic within the local switch remains unrestrained. Local subnet devices may still be affected, but the network remains alive. CONST_DIAG-SP-6-HM_MESSAGE: High traffic/CPU util seen on Module 5 [SP=40%,RP=99%,Traffic=0%]

DoS Protection: Storm Control
Storm control is also known as broadcast suppression: limits the volume of broadcast, multicast and/or unicast traffic protects the network from intentional and unintentional flood attacks and STP loops limits the combined rate of broadcast and multicast traffic to normal peak loads Dropped Packets Quantity Threshold Time 1 2 3 Seconds

Protecting the Distribution Layer
Configure storm control on distribution downlinks. Limit broadcast and multicast to 1.0% of a GigE link to ensure distribution CPU remains in the safe zone. ! Enable storm control storm-control broadcast level 1.0 storm-control multicast level 1.0 Broadcast Traffic CPU Impact Conservative Max Sup720 CPU Load

Configuring Storm Control
Storm control suppression is configured in interface configuration mode as follows: 6500(config-if)# storm-control ? broadcast Broadcast address storm control multicast Multicast address storm control unicast Unicast address storm control 6500(config-if)# storm-control broadcast ? level Set storm suppression level on this interface 6500(config-if)# storm-control broadcast level ? < > Enter Integer part of storm suppression level 6500(config-if)# storm-control multicast level ? 6500(config-if)# storm-control unicast level ?

Configuring Storm Control (Cont.)
Statistics for storm control suppression can be displayed as follows: 6500# show interface g1/9 counters broadcast Port TotalSuppDiscards Gi1/ 6500# show interface g1/9 counters multicast Gi1/ 6500# show interface g1/9 counters unicast Gi1/ 6500#

Fault Management

Fault Management on the Catalyst 6500
Improving resiliency in redundant and nonredundant deployments: Fault Management Enhanced System Stability Enhanced Network Stability Misconfigured system Memory corruption Software inconsistency Hardware faults Detection Isolation Correction Software enhancements for better fault detection Mechanisms to detect and correct soft failures in the system Proactive fault detection and isolation Routines to detect failures that the runtime software may not be able to detect

Fault Management Framework
Reports Faults and Takes Action Call Home, Syslogs, SNMP EEM Automates actions based on events that have occurred; TCL-based configurable fault policy GOLD Soft High Availability Troubleshooting Detects system problems proactively Detects and correct soft failures Provides intelligent troubleshooting and debugging mechanisms

Generic Online Diagnostics

Generic Online Diagnostics
GOLD implements a number of health checks both at system startup and while the system is running. GOLD complements existing HA features like NSF/SSO running in the background, and alerting HA features when disruption occurs. Diagnostic Results Bootup Diagnostics SYSLOG Message %DIAG-SP-3-MAJOR: Module 2: Online Diagnostics detected a Major Error. Please use diagnostic Module 2' to see test results. Check operational status of components Run Time Diagnostics On-demand diagnostics statically triggered by an administrator Scheduled diagnostics to run at a specific time Non-disruptive health diagnostics running in the background Diagnostic Action Invoke action to resolve issue i.e. reset component, invoke HA action, CallHome, etc

GOLD Fault Detection Framework for high availability :
Boot Up Diagnostics Quick go and no-go tests Disruptive and nondisruptive tests Proactive diagnostics serve as high availability triggers and take faulty hardware out of service. Health Monitoring Diagnostics Periodic background tests Nondisruptive tests Troubleshooting Tools: On-demand Diagnostics and Schedule Diagnostics Reactive diagnostics for troubleshooting Can run all the tests Include disruptive tests used in manufacturing

GOLD Test Suite Bootup Diagnostics: On-demand Diagnostics:
EARL learning tests (Sup & DFC) L2 tests (channel, BPDU, capture) L3 tests (IPv4, IPv6, MPLS) Span and multicast tests CAM lookup tests (FIB, NetFlow, QoS CAM) Port loopback test (all cards) Fabric snake tests Health Monitoring Diagnostics: SP-RP inband ping test (Sup’s SP/RP, EARL(L2&L3), RW engine) Fabric channel health test (fabric enabled line cards) MacNotification test (DFC line cards) Non-disruptive loopback test Scratch registers test (PLD & ASICs) On-demand Diagnostics: Exhaustive memory test Exhaustive TCAM search test Stress Testing All bootup and health monitoring tests can be run on-demand Scheduled Diagnostics: All bootup and health monitoring tests can be scheduled Scheduled switch-over

Trivia ¿Qué tienen en común la Copa Confederaciones FIFA con los Catalyst Switches de Cisco?

Sesión de Preguntas y Respuestas
El experto responderá verbalmente algunas de las preguntas que hayan realizado. Use el panel de preguntas y respuestas (Q&A) para preguntar a los expertos ahora

Nos interesa su opinión!!!
Habrá un sorteo con los que llenen el questionario de evaluación Tres asistentes recibirán un Regalo sorpresa Para llenar la evaluación haga click en el link que está en el chat. También saldrá automáticamente al cerrar el browser de la sesión.

Pregunte al Experto Si tiene preguntas adicionales pregunte aquí
Carlos responderá del martes 4 de diciembre al viernes 14 de diciembre del 2012.

Próximo Webcast en portugués
Tema: Resolución de problemas en el Session Initiation Protocol (SIP) Martes 6 de diciembre 7:00 a.m. Ciudad de México 8:30 a.m. Caracas 10:00 a.m Bs.As. 2:00 p.m. Madrid Michelle Jardim HOD=E&LANGUAGE_ID=P&SEMINAR_CODE=S17480& PRIORITY_CODE=

Respuesta a la Trivia ¿Qué tienen en común la Copa Confederaciones FIFA con los Catalyst Switches de Cisco? En 1999, Cisco lanzó la familia de switches inteligentes multi-gigabit Cisco Catalyst Ese mismo año México se convierte en la primera nación que gana la copa confederaciones FIFA en casa.

Muchas gracias por su asistencia
Por favor complete la encuesta de evaluación de este evento y gane premios

Cisco Catalyst 6500 Series Switches:

Similar presentations

Presentation on theme: "Cisco Catalyst 6500 Series Switches:"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Cisco Catalyst 6500 Series Switches:

Similar presentations

Presentation on theme: "Cisco Catalyst 6500 Series Switches:"— Presentation transcript:

Similar presentations

About project

Feedback