This section describes the online diagnostics available for troubleshooting your switch router. Online diagnostics provide the following types of tests:
Accessibility tests between the route processor and the interface modules and the route processor and the switch processor.
Online insertion and removal (OIR) diagnostic tests.
Snake tests through the switch router to ensure connectivity between the ports.
The switch router displays an error message on the console when it detects a hardware failure or problem.
Note Online diagnostic tests only run on the primary route processor.
Processor Loopback Test (Catalyst 8540 CSR)
The processor loopback test detects failures in the route processor to switch processor interface. The test sends a packet from the route processor to each switch processor which then loops back to the route processor. This test can run on the switch router without any port adapters or interface modules installed.
Note The size of the packet and frequency of the test are configurable to minimize the impact
on system performance.
Accessibility Test
The accessibility tests ensure connectivity, at a configurable interval, between all of the following:
Interface modules
Active switch processor
Standby switch processor, if it is present
OIR Test
Online insertion and removal (OIR) tests check the functioning of the switch fabric and interfaces on a per-port basis. The switch router performs these tests when the system boots up and when you insert an interface module into a slot. The OIR test sends a packet to the interface loopback and expects to receive it within a certain time period. If the packet does not reach the port within the expected time period, or the received packet is corrupted, an error is registered and the port is changed to an administrative down state. Packets that are 1000 bytes in size are used in the test.
The OIR tests support all the enhanced Gigabit Ethernet interface modules, with the exception of the OC-3c and OC12c ATM uplink interface module. In addition, OIR tests are not supported on the Fast Ethernet or Gigabit Ethernet interface modules.
The snake test detects and reports port-to-port connectivity failures. The snake test establishes connections across all the active ports in the switch router, originating and terminating at the primary route processor. The route processor establishes a connection by sending a packet to each port in turn, which then terminates at the route processor. If the packet does not reach the route processor within the expected time period, or the received packet is corrupted, further testing is performed to isolate and disable the port causing the problem. The frequency of the test is configurable to minimize the impact on system performance.
The snake test supports all the enhanced Gigabit Ethernet interface modules on the Catalyst 8540 MSR and Catalyst 8540 CSR.
Checking DDTs Database and Release Notes for Workarounds
This section describes different methods you can use to check for IOS software bugs (defect tracking tool numbers [DDTs]) in your version of IOS software. Often, your problems with the switch router have been fixed or a workaround has been determined in a more recent version of software.
There are two ways to check for known bugs in the IOS software:
Step 1 If you are not already logged in to Cisco.com, enter your user name and password at the login prompt.
Step 2 Read the "Bug Navigator II Help" instructions.
Step 3 From the "Cisco Hardware" list select your switch router under "Catalyst 8500 Series Switches." The Bug Navigator search tool replaces "Bug Navigator II Help" (in the right frame of the page).
Step 4 Select the following from the drop down menus:
Version
Revision
Severity
Note As an option, you can enter words or phrases (separated by commas) in the data entry field
to limit your search.
Step 5 Click the "Search" button.
The entire window will be replaced with a "Bug Search Results" window with a list of DDTs containing your search criteria. Look at the Bug reports listed in the "titles" column. An existing bug entry that describes the problem you are having may have been fixed in a more recent version of the IOS software. Look in the "Fixed-in" column for a later version of the IOS software. All you might have to do to solve your problem is upgrade your software.
If a software upgrade is not listed as a way to solve your problem, double-click on the bug title and read the DDT details; a workaround might be listed there.
Checking IOS Release Notes
Release notes describe the features and caveats for Cisco IOS software releases. The release notes are listed by both product and IOS release number.
Note All information pertains to both the Catalyst 8540 CSR and Catalyst 8510 CSR platforms,
unless differences between the platforms are noted in the text.
The "Caveats" section of the release note lists known caveats for the switch router by tracking the DDTS number and the release number, and indicates whether the caveat has been corrected.
The "Caveat Symptoms and Workarounds" section summarizes caveat symptoms and suggested workarounds for the switch routers. You can also search thorough this section online, using either a word string or the DDTS number.
Troubleshooting Hardware and Software Version Problems
A common error you may encounter is the incompatibility of hardware modules and the IOS software version needed to perform a particular function. Check theHardware and Software Compatibility Matrix document, available online, to confirm that you are using IOS software that supports the various hardware components installed in your switch router.
Verifying Hardware and Software Versions
Display the hardware and software versions to ensure that they are the most recent. Very old hardware and software versions (two or three versions back) can have caveats that have been fixed in more recent versions. Use the following EXEC commands to display the version information:
Command
Purpose
show version
Displays the software version information.
show hardware [detail]
Displays detailed hardware information including revision level and version.
show functional-image slot slot
Displays functional image information.
To verify the hardware and software versions, use the following steps:
Step 1 Display the system software version:
Switch# show version
Cisco Internetwork Operating System Software
IOS (tm) L3 Switch/Router Software (C8540CSR-IN-M), Version 12.0(10)W5(18c)
Step 5 Use the show functional-image command to display detailed information about the functional images for the route processors, switch processors, and Fast Ethernet and Gigabit Ethernet interface modules for the switch router (in this example, the Catalyst 8540 CSR). The following example shows how to display the functional image for the route processor in slot 4:
Step 6 Verify the FunctionalVersion and #HardwareRequired fields to determine the FPGA version and the hardware version required for the FPGA. Compare this with the hardware version using the show hardware command output displayed in Step 2 and Step 4. If the FPGA version does not support the hardware version, download a new FPGA image, upgrade the hardware, or both.
If you determine that you need to upgrade your IOS image, refer to the Cisco.com web page, and follow these steps to find and download the image you need:
Step 1 Login to the Cisco.com; the window will change and display additional features.
Step 2 Under Service and Support, select Software Center.
Step 3 The Technical Assistance Center page is displayed. Under Tools, select IOS Upgrade Planner.
The Cisco IOS Planner page provides greater flexibility to browse for your preferred software. You can view all major releases, all platforms, and all software features from a single interface. Choosing a platform, a maintenance release, or a software feature the planner automatically limits the other menu choices, based on your selections until you arrive at your preferred software.
Step 4 In the Select Platform column, select either of the following:
"C8540C" (for Catalyst 8510 CSR and Catalyst 8540 CSR platforms)
"C8540M" for Catalyst 8510 MSR and Catalyst 8540 MSR platforms)
Step 5 In the Select Release column, select the IOS software release you want to download.
Step 6 Read the requirements, and if your hardware configuration meets the requirements, click the agreement button.
Step 7 Select the file to download to your switch router.
To upgrade the IOS image on your platform, continue with the following section.
IOS Upgrade Procedures
If your IOS image is not the most recent, you can download the IOS image from Cisco.com to the switch router.
Formatting Flash memory on a new PCMCIA card or on any Flash memory device that has locked blocks or failed sectors.
Managing files on file systems, including setting the default file system, listing files on a file system and deleting and recovering files.
Modifying, Downloading, and Maintaining System Images
The following tasks are performed frequently to maintain system image files:
Copy images from Flash memory to a network server. You can store system images for backup, or other purposes, by copying them from a Flash memory device to a TFTP or rcp server.
Copy images from a network server to Flash memory. You perform this procedure when upgrading your system image or functional image.
Copy images between local Flash memory devices. You perform this procedure when moving a system image from one switch router to another or to synchronize the functional images on switch routers with redundant route processors.
Maintaining Functional Images
You can load functional images used by certain hardware controllers in the switch router. The following sections describe the function and maintenance of functional images:
Functional images provide the low-level operating functionality for various hardware controllers. On hardware controllers within system programmable devices, such as Field Programmable Gate Arrays (FPGAs) and Erasable Programmable Logic Devices (EPLDs), the hardware functional images can be reprogrammed independently of loading the system image, and without removing the devices from the controller.
All new hardware you purchase is shipped with the functional images loaded. Loading a different functional image is required only when upgrading or downgrading functional image versions.
FPGA Upgrade Procedures
If the functional image is not the most recent, you can download the functional image to the switch router from Cisco.com. Use the reprogram command to update the functional image to the processor or interface module. The following example shows how to reprogram the route processor in slot 4 with the functional image fi-c8540-rp.A.4-8.bin from the Flash PC card in slot 0:
Switch# reprogram slot0:fi-c8540-rp.A.4-8.bin 4
Note You can only enter the reprogram command from the console session prompt.
You can find the functional images and release notes for one of the following on Cisco.com, on the LAN Switching Products site:
Note After you have determined the hardware and software versions on the switch router, check
the release notes and DDTS database for symptoms resembling those you are observing.
Often, the problem has already been discovered and a workaround has been provided.
Troubleshooting Switch Route Processor Redundancy and Enhanced High System Availability (Catalyst 8540 CSR and Catalyst 8540 MSR)
The Catalyst 8540 MSR and Catalyst 8540 CSR support, redundant route processor operation with dual route processors. In addition, Enhanced High System Availability (EHSA) is provided in the switching fabric when three switch processors are installed in the chassis. The troubleshooting processes for these features are described in the following sections:
The Catalyst 8540 MSR and Catalyst 8540 CSR supports fault tolerance by allowing a secondary route processor to take over if the primary fails. This secondary, or redundant, route processor runs in standby mode. In standby mode, the secondary route processor is partially booted with the Cisco IOS software; however, no configuration is loaded.
At the time of a switchover, the secondary route processor takes over as primary and loads the configuration as follows:
If the running configurations on the primary and secondary route processors match, the new primary uses the running configuration file
If the running configurations on the primary and secondary route processors do not match, the new primary uses the last saved configuration file in its nonvolatile random-access memory (NVRAM) (not the NVRAM of the former primary)
The former primary then becomes the secondary route processor.
Note If the secondary route processor is unavailable, a major alarm is reported. Use the show
facility-alarm status command to display the redundancy alarm status.
For detailed redundant route processor configuration information, refer to the "Initially Configuring the ATM Switch Router" chapter in the ATM Switch Router Software Configuration Guide .
For redundant route processors to function correctly, your switch router's route processors must meet all of the following requirements:
Route processors must have identical hardware configurations
ROMMON must be version 12.0(4.6)W5(13) or later
Both route processors must have identical releases of IOS software
A common error you may encounter is the incompatibility of hardware modules and the IOS software version needed to perform a particular function. Check theHardware and Software Compatibility Matrix document, available on-line, to confirm that you are using IOS software that supports the various hardware components installed in your switch router.
Troubleshooting Hardware and Software Versions of Redundant Route Processors
To troubleshoot the route processor hardware and software versions for redundancy, use the following commands:
Command
Purpose
show version
Displays the configuration register value.
show hardware detail
Displays the hardware and software configurations of the primary and secondary route processors.
show redundancy
Displays the hardware and software configurations of the primary and secondary route processors.
To confirm that your switch router route processors meets the redundancy requirements, complete the following steps:
Step 1 Use the show version command to confirm the system hardware and software status of the primary route processor.
Switch# show version
Cisco Internetwork Operating System Software
IOS (tm) PNNI Software (cat8540m-WP-M), Version 12.1(2.3)W6(1.33), CISCO DEVELOP MENT TEST VERSION
System image file is "slot0:cat8540m-wp-mz.121-2.3.PE33"
cisco C8540MSR (R5000) processor with 262144K/256K bytes of memory.
R5000 CPU at 200Mhz, Implementation 35, Rev 2.1, 512KB L2 Cache
Last reset from power-on
3 Ethernet/IEEE 802.3 interface(s)
16 FastEthernet/IEEE 802.3 interface(s)
14 ATM network interface(s)
505K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
8192K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x0
Switch#
Step 2 Verify the ROM field. It indicates the release of IOS software loaded and running on the primary route processor.
Step 3 Use the show hardwaredetail command to compare the hardware versions of the primary and secondary route processors.
8540MSR# show hardware detail
C8540 named 8540MSR, Date: 18:42:12 UTC Fri Jan 5 2001
.
(Information Deleted)
.
slot: 4/* Controller-Type : Route Proc
Part Number: 73-2644-05 Revision: A0
Serial Number: MIC03140NXK Mfg Date: Apr 04 99
RMA Number: 0 H/W Version: 5.7
FPGA Version: 4.8
slot: 4/0 Controller-Type : Netclk Module
Part Number: 73-2868-03 Revision: A0
Serial Number: MIC03140NSU Mfg Date: Apr 04 99
RMA Number: 0 H/W Version: 3.1
FPGA Version: 4.8
.
(Information Deleted)
.
slot: 8/* Controller-Type : Route Proc
Part Number: 73-2644-05 Revision: A0
Serial Number: MIC03140NXH Mfg Date: Apr 04 99
RMA Number: 0 H/W Version: 5.7
FPGA Version: 4.8
slot: 8/0 Controller-Type : Netclk Module
Part Number: 73-2868-03 Revision: A0
Serial Number: MIC03140NVT Mfg Date: Apr 04 99
RMA Number: 0 H/W Version: 3.1
FPGA Version: 4.8
.
(Information Deleted)
.
Step 4 In the slots labeled Controller-Type : Route Proc, compare the Part Number, FPGA, and H/W Version fields. These numbers must all match, or redundancy will not function correctly on your switch router.
Step 5 Use the show redundancy command to check the configuration and status of the route processors.
Switch# show redundancy
This CPU is the PRIMARY
Primary
-------
Slot: 8
Uptime: 8 weeks, 4 days, 11 hours, 31 minutes
Image: PNNI Software (cat8540m-WP-M), Version 12.1(2.3)W 12.0(4a)W5(11a) RELEASE SOFTWARE
Image: PNNI Software (cat8540m-WP-M), Version 12.0(4a)W5(11a) RELEASE SOFTWARE
Step 6 Verify the Primary, Secondary, and Slot fields. They indicate in which slot the primary route processor is configured.
Step 7 Verify the Last Running Config. Sync and Last Startup Config. Sync fields. They indicate the last time the running configuration and startup configuration were synchronized between the route processors.
Troubleshoot Redundant Route Processor Functions
To troubleshoot the route processor functions for redundancy, use the following commands:
Command
Purpose
show atm vc interface atm 0
Displays the VC status between the primary and secondary route processors.
show atm status
Displays the status of the primary and secondary route processors.
Follow these steps to troubleshoot the route processor redundancy on the switch router:
Step 1 Use the show atm vc interface atm 0 command to confirm that the permanent virtual circuit (PVC) between switch route processors (SRPs) is up.
Switch# show atm vc interface atm 0
Interface VPI VCI Type X-Interface X-VPI X-VCI Encap Status
ATM0 0 35 PVC ATM0/0/0 0 16 ILMI DO
.
(Information Deleted)
.
ATM0 0 245 PVC ATM-SEC0 0 29 IPC UP
Switch#
Step 2 Verify the ATM-SEC0 field and confirm the PVC is up.
Step 3 Use the show atm status command to confirm the status of the PVC and other states.
Switch# show atm status
NUMBER OF INSTALLED CONNECTIONS: (P2P=Point to Point, P2MP=Point to MultiPoint,
MP2P=Multipoint to Point)
Type PVCs SoftPVCs SVCs TVCs PVPs SoftPVPs SVPs Total
P2P 982 0 0 0 0 0 0 982
P2MP 36 0 0 0 0 0 0 36
MP2P 0 0 0 0 0 0 0 0
TOTAL INSTALLED CONNECTIONS = 1018
PER-INTERFACE STATUS SUMMARY AT 16:42:51 UTC Fri Dec 8 2000:
Step 5 Verify that the ILMI Addr Reg State field is UpAndNormal.
If you determine that redundancy is configured incorrectly, refer to the "Initially Configuring the ATM Switch Router" chapter in the ATM Switch Router Software Configuration Guide.
Troubleshooting Switch Processor Redundancy with HSRP (Catalyst 8540 CSR and Catalyst 8540 MSR)
Enhanced High System Availability (EHSA) is provided in the switching fabric when three switch processors are installed in the chassis. These features and their configuration are described in the "Initially Configuring the ATM Switch Router" chapter in the ATM Switch Router Software Configuration Guide .
To troubleshoot the EHSA of the switch processor, use the following commands:
Command
Purpose
show preferred-switch-card-slots
Displays the configuration of the switch processors.
show switch fabric
Displays the switch fabric details of the switch router.
Follow these steps to troubleshoot the redundant switch processors on the switch router:
Step 1 Use the show preferred-switch-card-slotscommand to confirm the configuration of the switch processors.
Switch# show preferred-switch-card-slots
The currently preferred switch card slots are slot: 5 and slot: 7
The currently active switch card slots are slot: 5 and slot: 7
Switch#
Step 2 Use the show switch fabric command to confirm the status of the switch processors and their location.
Switch# show switch fabric
swc_presence_mask: 0x5
Switch mode: NR_20G
Number of Switch Cards present in the Chassis: 3
SWC SLOT SWC_TYPE SWC_STATUS
=================================================
5 EVEN ACTIVE
6 STANDBY STANDBY
7 ODD ACTIVE
MMC Switch Fabric (idb=0x62146E7C)
.
(Information Deleted)
.
Step 3 Check the field Number of Switch Cards present in the Chassis. If you only have two switch processors, your switch router does not have EHSA. With three switch processors installed, if either of the two active switch processors fail, the third processor takes over.
Step 4 Check the SWC_STATUS field. This fields identifies the active and standby switch processors.
