1. LAN Switching – Session 1
Aninda Chatterjee
Karthick Murugan

2. Agenda
IOS upgrade
High CPU
Layer 1 troubleshooting
Crashes

3. IOS Upgrade Downloading an IOS image – where and how Boot variables
Configuration register
ROMmon recovery

4. To download and search for Cisco software, go to www. cisco
To download and search for Cisco software, go to and hover on ‘Support’.

5. In the ‘Download’ section, type in the product model for which you need the software. This should list out all results for that product.

6. Choose the specific product that you are interested in
Choose the specific product that you are interested in. For example, we chose 6500 with MSFC3.
Choose what you need to download from the following. For example, to download different IOS software's for this product, choose ‘IOS Software’.

7. This will now list out all publicly available IOS trains that you can download.
Note, safe harbor versions are clearly mentioned here.

8. IOS Upgrade
A boot variable is a variable that the system uses to identify the path of the image with which it has to boot.
Incorrectly set boot variables can often lead to boot failure which results the system to boot into ROMmon.
You can check the currently set boot variable using the ‘show bootvar’ command (in some systems, this might be ‘show boot’).
Switch#show bootvar
BOOT variable = sup-bootdisk:s72033-advipservicesk9_wan- mz SXJ5.bin,1;
Another quick way to check the configured boot variable is by using ‘show run | in boot’
Switch#show run | in boot
boot system flash sup-bootdisk:s72033-advipservicesk9_wan- mz SXJ5.bin

9. Certain platforms allow more than one path as the boot variable
Certain platforms allow more than one path as the boot variable. In this case, the first path in the sequence is what is used to try and boot first. If that fails, the second is used and so on.
Switch#show bootvar
BOOT variable = sup-bootdisk:s72033-advipservicesk9_wan- mz SXJ5.bin,1;sup-bootdisk:s advipservicesk9_wan-mz SXJ3.bin,1;
Switch#show run | in boot
boot-start-marker
boot system flash sup-bootdisk:s72033-advipservicesk9_wan- mz SXJ5.bin
boot system flash sup-bootdisk:s72033-advipservicesk9_wan- mz SXJ3.bin

10. Notice, the correct syntax of configuring the boot variable is:
boot system flash <file system:image.bin>
For example:
boot system flash sup-bootdisk:s72033-advipservicesk9_wan- mz SXJ5.bin
In case you need to change the order of preference of your boot variables, you need to delete the existing configured variables and reconfigure in your preferred order.
Also note that some platforms limit you to only a single configurable boot variable. If you attempt to configure another, it will simply overwrite the existing boot variable. For example, the 3750 platform exhibits this behavior.

11. Configuration register
This is used for dictating how a system boots up, what options to use while booting (such as ignoring the configuration) among other things.
You can verify the value of the configuration register using the ‘show bootvar’ or the ‘show version’ commands.
Commonly used configuration registers:
0x2101 – This tells the switch to use the first available image in the file system (it ignores configured boot variable).
0x2102 – This tells the switch to use the configured boot variable.
0x2142 – This tells the switch to ignore the startup configuration while boot up, mainly used for password recovery.
Certain platforms do not allow you to change the configuration register value.

12. The ROMmon console looks like this: rommon 1> or Switch:
ROMmon recovery
In certain situations (like a corrupted software image or incorrectly set boot variables/configuration register), the system boots into ROMmon.
The ROMmon console looks like this:
rommon 1> or Switch:
First and foremost, check if you have an available software image in your file system.
rommon 1 > dir bootdisk:
Initializing ATA monitor library...
Directory of bootdisk:
rw- s72033-advipservicesk9_wan-mz SXJ3.bin

13. To check what file systems are available at ROMmon, use the ‘dev’ command:
rommon 3 > dev
Devices in device table:
id name
bootdisk: boot disk
disk0: PCMCIA Disk 0
disk1: PCMCIA Disk 1
eprom: eprom
usb: USB
If you find an available image, attempt to boot using that with the following:
rommon 2 > boot bootdisk:s72033-advipservicesk9_wan-mz SXJ3.bin

14. If, for some reason, you are unable to boot with an image or the image is not available, you can use the ROMmon TFTP recovery method which is available in some platforms (which have a management port).
xample09186a ecf.shtml
Some platforms also support USB drives. This can be used as a file system to temporarily boot your device.
As a last resort, if everything fails, you will have to use the Xmodem procedure to transfer your image over a console cable.
infrastructure/switching/blog/2012/01/24/loading-an-ios-on-a-switch-via-xmodem

15. High CPU
High CPU can be due to two reasons – interrupt driven traffic or due to a process.
It is critical to identify which of these is the trigger for high CPU to decide the next troubleshooting steps.
Process driven high CPU is because of a specific process (es) consuming a large cycle of the CPU. For some platforms, certain processes are expected to use a larger than normal cycle of the CPU (For example, the Hulc LED process on the 3750 devices).
Hung processes are also a common cause of process driven high CPU.
Interrupt driven high CPU is because of packets being software switched (punted to CPU) due to the device’s inability to switch it in hardware due to specific reasons.
This could also been seen due to excessive traffic being destined for the device(CPU) itself.

16. High CPU
Also note that any feature that is not supported by the platform, in hardware, will be punted to the CPU for processing, leading to possible high CPU.
Useful commands to troubleshoot high CPU:
Catalyst 3K:
show process cpu sorted – this gives you the CPU cycles being currently consumed in descending order which helps to identify the top process.
show process cpu history – this gives the CPU’s historical data.
show controller cpu-interface – this helps to identify which CPU queue and the number of packets received/dropped per queue.
debug platform cpu-queues <queue-name> - this helps to identify the packets punted to CPU from that specific queue.

17. High CPU
Catalyst 4500:
show platform health– this helps to identify the misbehaving process (which is exceeding its threshold).
show platform cpu packet statistics– this gives the number of packets received/dropped per queue.
debug platform packet all receive buffer– this captures the packets being punted to the CPU.
show platform cpu packet buffered- this displays the packets captured from the previous debug.

18. High CPU
Catalyst 6500:
debug netdr capture rx– this captures packets being punted to the CPU for software processing.
show netdr captured-packets – this displays the packets captured via the previous debug command.
debug netdr clear-capture – this clears the buffer which contained any previously captured packets.
Remember to undebug all at the end of the troubleshooting.

19. High CPU Start by looking at the CPU usage: Live lab
High CPU is observed on the switch and we need to troubleshoot and resolve the issue.
Start by looking at the CPU usage:

20. High CPU
From the output, we can see that it is interrupt driven. 80% of the CPU usage is due to packets being punted to the CPU. We can also see that the ARP input is abnormally high.
Look at CPU history to determine for how long the CPU has been high.

21. High CPU
We see that the CPU has been consistently high for the last 22 minutes or so. It is always a good practice to recollect any possible changes that were made around/before the start of the problem.
Since this is interrupt driven traffic, we need to identify what packets are being punted to the CPU and where they are coming from.
To capture these packets, issue the debug netdr capture rx command. This command is not CPU intensive since these packets are captured on a dedicated ASIC.
This debug captures the first 4096 packets hitting the CPU.
To view these, issue the show netdr captured-packets command.

22. High CPU

23. High CPU
There are several important fields to look at from this capture that are needed to troubleshoot.
src_index – this identifies the ingress interface of the packet
dest_indx – this identifies the look up result and determines where the packet needs to be sent.
srcmac – this identifies the source mac of the frame
destmac - this identifies the destination mac of the frame.
src_vlan – this identifies the source VLAN of the frame.
interface – this identifies the layer 3 ingress interface
flood – this identifies if the packet needs to be flooded or not.

24. Crashes
What is a crash? A crash is an unexpected reboot/reload of the device.
Most Cisco devices have the ability to generate a crash file which contains events/related information of the crash.
This file is stored in the file system of the device.
These files are needed by a TAC engineer, which will then be decoded using an internal tool to try and understand the reason for the crash.
You can retrieve these files as follows:
Catalyst 3K:
It will be inside the flash. If the file is very large, it is moved to crash_ext within the flash.
To verify if the crash file is generated, you can use dir flash:
You can extract this using more flash:<file-name> and send this output to the TAC engineer.

25. Crashes
Catalyst 4500:
The crash file generated is stored under the crashinfo file system.
To extract the file, you can use show platform crashdump.
Catalyst 6500:
This platform (based on the supervisor), maintains two CPUs – the route processor (RP) and the switch processor (SP). The crash of one leads to the crash of the other.
The crash generated is stored in sup-bootdisk, slavesup-bootdisk (for SP) and bootdisk, slavebootdisk (RP).
To extract the file, you can use more <file system>:<file-name>

26. Layer 1 troubleshooting
It is important to identify any layer 1 problems since these can affect the functioning of the higher layers.
The show interface output is the most handy command to look for any layer 1 trouble.
Several things that you should keep an eye on:
Collisions/late collisions: These are commonly seen due to duplex mismatches.
CRC – these are packets with corrupted checksums. The physical medium needs to be checked.
Input errors – these are corrupted packets. The physical medium needs to be checked.
Pause input/output – these indicate flow control.
Overruns: these are seen due to the input buffer being overwhelmed.
Output drops: These are seen due to the transmit buffer being overwhelmed.

27. Layer 1 troubleshooting