|
|
This document describes how to troubleshoot the Catalyst 8540 CSR and Catalyst 8510 CSR switch routers.
This document includes the following sections:
This section describes general troubleshooting guidelines for the Catalyst 8540 CSR and Catalyst 8510 CSR.
Display the hardware and software versions to ensure that they are the most recent. Very old hardware and software versions (two or three versions back) can have problems that have been fixed in more recent versions. Use the following EXEC commands to display the version information:
Follow these steps to verify the hardware and software versions:
Step 2 Use the show hardware command to display the hardware revision levels:
The hardware version is listed in the Hw Vrs column.
Step 3 Use the show hardware detail command to display detailed information about the hardware, including the functional image versions.
Step 4 Use the show functional-image command to display detailed information about the functional images for the route processors, switch processors, and Fast Ethernet and Gigabit Ethernet interface modules for the Catalyst 8540 CSR. The following example shows how to display the functional image for the route processor in slot 4:
Check the Functional Version and #HardwareRequired fields to determine the FPGA version and the hardware version required for the FPGA. Compare this with the hardware version with the show hardware command output displayed in Step 2 and Step 3. If the FPGA version does not support the hardware version, download a new FPGA image, upgrade the hardware, or both.
If the functional image is not the most recent, you can download the functional image to the switch router from CCO. Use the reprogram command to update the functional image to the processor or interface module. The following example shows how to reprogram the route processor in slot 4 with the functional image fi-c8540-rp.A.4-8.bin from the Flash PC card in slot 0:
 |
Note You can only issue the reprogram command from the console session prompt. |
You can find the functional images and release notes on CCO at the following URL:
http://www.cisco.com/cgi-bin/tablebuild.pl/cat8540c-fpga
|
Note After you have determined the hardware and software versions on the switch router, research the release notes and DDTS database for symptoms resembling those you are observing. Often, the problem has already been discovered and a workaround provided. |
Follow these steps to troubleshoot physical interface problems:
Step 2 Check the FastEthernet or GigabitEthernet field to see whether the interface is up.
If down, check for the following:
If administratively down, the interface has been administratively taken down. Use the no shutdown interface configuration command to reenable the interface.
Step 3 Check the line protocol field to see whether the status is up.
If down, check for the following:
For more information on interface configuration, refer to the Layer 3 Software Feature and Configuration Guide.
Follow these steps to troubleshoot interface and controller problems:
Figure 1 shows the difference between the show controllers command output for releases up to and including 12.0(5)W5(13) and the releases after 12.0(5)W5(13).
Step 2 Check the interface status line in the show interfaces command display. This line indicates that the interfaces are configured as automatic duplex, automatic speed, and 100BaseTX.
Step 3 Check the Chip Status Register field contents in the show controllers command output to verify that the link is up and the link is configured the same as the output from the show interfaces command.
In all cases when troubleshooting a problem, use the following commands:
You might see problems of cell transmission through the switch router detected by a build-up of cells on an internal virtual channel (VC). There are several reasons why these problems occur:
A switch processor can have a cell stuck problem in internal virtual channels (VCs) resulting from timing issues in the hardware and software on the Catalyst 8540 CSR. You might see more than one port affected on one or more interface modules. Online insertion and removal (OIR) of the interface module will temporarily fix the problem.
Follow these steps to troubleshoot cell stuck problems:
The show switch fabric command clears the counters after it displays. Entering the command again shows the current activity on the switch router.
Step 2 Issue the show switch fabric command again to show new activity.
Look at the values in the Rejected Cells and Invalid Cells fields. Note that the Rejected Cells and Invalid Cells fields counters are increasing. This means there might be a problem in the switch fabric.
Step 3 Verify that no ports are involved by issuing the show epc queuing and show epc status commands.
If the queues are empty and all of the ports show OK status, then the problem is not the ports, but with the switch processor.
You can resolve this problem by upgrading your system software image to release version 12.0(4a)WX5(11) or later, replacing the switch processors, or both.
If one or more Fast Ethernet or Gigabit Ethernet ports are not transmitting cells, then the failure might be a port stuck problem.
Follow these steps to troubleshoot a port stuck problem:
|
Note Be sure to use the show switch fabric command during the lightest possible traffic conditions because actual traffic might be using the memory buffers. |
The show switch fabric command clears the counters after it displays. Entering the command again shows the current activity on the switch router.
Step 2 Enter the show switch fabric command again.
Look at the values in the Rejected Cells and Invalid Cells fields. Note that the Rejected Cells and Invalid Cells fields counters are increasing. This means there might be a problem in the switch fabric.
Step 3 Use the show epc queuing and show epc status command to display interface queues and status.
The show epc queueing command output shows that no activity is going across interface GigabitEthernet 1/0/0. This is verified in the show epc status command output, which indicates that interface GigabitEthernet 1/0/0 is "not OK." You have confirmed that the problem is a stuck port.
|
Note You might see a few cells in the QCNT column in the show epc queueing command output. That is normal. Issue the command several times to verify that traffic is moving through the queues. If the QCNT column values are incrementing and incrementing for the VCIs belonging to a particular interface, the problem is probably a stuck port. |
You can remedy the port stuck condition by removing and reinserting the interface module. A shutdown/no shutdown command sequence on the problem interface will not resolve the problem.
To recover from a port stuck failure, perform the following tasks:
If the switch router is not configured to reset the port upon detecting a port stuck failure, the port will be isolated, thus preserving the integrity of the switch router.
If the switch router is configured to reset the port upon detection of a stuck port failure, the switch router will isolate the port from the rest of the functioning ports, and reset the port. This might affect other ports on the interface module.
|
Note If you configure the switch router as described in the nondefault behavior after a port stuck failure is detected, the switch router will not reset the Ethernet ports. The Ethernet interface must be configured to reset before the port stuck failure occurs. Also, the default behavior is to not reset the port if a port stuck failure is detected. If the Ethernet interface is not configured to reset when a port stuck failure is detected, schedule the switch router for downtime to remove and reinsert the module. |
To configure the switch route to automatically recover from port stuck failures, use the following interface configuration commands:
|
 |
Caution Due to the nature of the microcode architecture, do not configure low values for the wait time in the epc portstuck-wait command. The default value of 180 seconds has been carefully chosen, allowing for the hello intervals of protocols such as HSRP, EIGRP, and OSPF. Configuring a low value might lead to incorrectly detecting temporary port stuck failures as real port stuck failures and will likely cause temporary connectivity loss. It is highly recommended to keep this value to at least 60 seconds. Lower values are provided to allow for some specific network designs when you can absolutely rule out temporary port stuck failure scenarios, and also as a debugging aid. For most networks, 180 seconds will work very well. |
To troubleshoot a connectivity problem between a port and another port or end-station, use the following commands:
|
The Catalyst 8540 CSR and Catalyst 8510 CSR use Cisco Express Forwarding (CEF). Much of the internal troubleshooting determines whether the central CEF information in the route processor is consistent with the distributed information in the content addressable memory (CAM) on the interfaces.
Follow these steps to troubleshoot IP routing problems:
Step 2 Use the show epc ip-prefix, show epc ip-address, and show epc ifmapping commands to verify that the hardware is correct. Figure 3 shows output from these commands.
|
Caution Use the clear ip route command carefully. It causes a temporary increase in switch router activity which can lead to traffic disruptions. |
Step 3 Verify the CAM entry information using the show epc patricia ipucast and show epc cam commands.Figure 4 show the CAM entry information for the example network.
The DA MAC address in this entry is 0x00902141 and the interface number is 4.
Step 4 Use the show epc if-entry command on the ingress interface to confirm that the source address in the egress interface entry matches the source address shown by the show adjacency detail command output in Figure 2. Figure 6 shows the show epc if-entry command output.
Step 5 To remove inconsistencies between the adjacency table and the IP address table, use the clear arp or clear adjacencies command to rebuild these tables.
|
Caution Use the clear arp and clear adjacencies commands carefully. They cause a temporary increase in switch router activity which can lead to traffic disruptions. |
To troubleshoot an IP routing problem, use the following commands:
|
If a BVI is involved in the problem, use the following commands:
|
IPX troubleshooting is similar to IP troubleshooting. The key is to check the consistency between the route processor table information and CAM tables on the ports.
The show ipc route command displays the IPX routes through the switch router. In this example, the routes are 8511 and 8512, the SAP server number is 8541, the egress interface is GigabitEthernet 11/0/1, and the node address is 8510.0010.7bfa.5f1f.
|
Note The SAP entries reside in route processor memory, not in the port CAM tables. |
Step 2 Now use the show epc ipx-prefix, show epc ipx-node, and show epc ipx cam commands to display the CAM tables for the ingress interface in question. The route lookups and packet rewrites are performed by the ingress interface. The example shown in Figure 8 uses the route numbers and MAC addresses highlighted in Figure 7.
Step 3 Figure 9 shows the format of the CAM entry. The last two bytes of the entry contain the interface number.
Use the show epc ifmapping command output to verify the interface number for the egress interface.
Step 4 You can obtain detailed information about the IPX prefix and node for both directly connected and remote networks using the show epc patricia ipx detail command (see Figure 10).
Step 5 Use the show epc if-entry command on the ingress interface to display information about the egress interface. In the example shown in Figure 11, the ingress interface is GigabitEthernet 11/0/1 and the egress interface is GigabitEthernet 10/0/0.
To troubleshoot an IPX problem, use the following commands:
|
If a BVI is involved, use the following commands:
|
IP multicast troubleshooting is similar to IP troubleshooting. The key is to check the consistency between the route processor table information and CAM tables on the interfaces.
Follow these steps to troubleshoot IP multicast problems:
If you have any problems with these fields, check the interface configuration. For information about configuring interfaces, refer to the Layer 3 Software Feature and Configuration Guide.
Step 2 Display the IP multicast entries contained in the CPU using the show ip mroute command.
Step 3 Use the address and interface information from the show ip mroute command output in Step 2 to display the CAM information with the show epc ipmcast command.
Step 4 Display the status of the VC for the incoming interface displayed in the show ip mroute command output in Step 2.
If there are inconsistencies or non-zero invalid entries in the tables, you can use the clear ip mroute * command to rebuild the tables.
|
Caution Use the clear ip mroute command carefully. It causes a temporary increase in switch router activity, which can lead to traffic disruptions. |
To troubleshoot an IP multicast problem, use the following commands:
|
The switch router might be configured with Layer 2 ports using bridge groups.
To troubleshoot Layer 2 switching, perform the following steps:
Step 2 Use the show epc patricia command to display Layer 2 information in the CAM for each interface.
Verify that the information from the show epc patricia command output in Figure 13 is consistent with the command outputs in Figure 12.
If there are inconsistencies or non-zero invalid entries in the tables, you can use the clear bridge command to rebuild the tables.
|
Caution Use the clear bridge command carefully. It causes a temporary increase in switch router activity which can lead to traffic disruptions. |
Step 3 Display the interface table entry in the CAM using the show epc if-entry command. Figure 14 shows the CAM interface table entry for the example network interface.
Refer to the Layer 3 Software Feature and Configuration Guide if any changes are necessary to the configuration of the interface.
To troubleshoot a Layer 2 bridging problem, use the following commands:
|
If a BVI is involved, use the following commands:
|
The switch router software provides several show commands that can be used for troubleshooting. This section contains lists of show commands grouped by the features they troubleshoot.
To troubleshoot trunk port problems, use the following command:
|
To troubleshoot an AppleTalk problem, use the following commands:
|
If a BVI is involved, use the following commands:
|
In the 12.0(10)W5(18) system software release, some of the show commands commonly used for troubleshooting have been converted from show epc commands to show controllers commands. The show controllers commands are described in the next section. Table 1 provides the mapping of the command syntax conversion.
Table 1 show Command Conversion
|
This section contains descriptions of show controllers and show switch commands that are useful for troubleshooting.
To display the controller register values, use the show controllers EXEC command.
Syntax Description
Command Modes
Usage Guidelines
This command uses IPC to get the values of MAC registers and MII registers. If the interface processor stops responding to IPC, the counter values shown are no longer current.
Examples
The following is sample output from the show controllers command for a Fast Ethernet interface:
Table 2 describes some of the important fields in the previous display.
Table 2 show controllers Registers and Descriptions for Fast Ethernet Interfaces
|
The following is sample output from the show controllers command for a Gigabit Ethernet interface:
Table 3 describes some of the important fields in the previous display.
Table 3 show controllers Registers and Descriptions for Gigabit Ethernet Interfaces
|
To display the access control list (ACL) entries on an interface module, use the show controllers access-list command.
Syntax Description
Command Modes
Usage Guidelines
This command displays TCAM entries for an ACL configured on an interface. It interprets the contents of TCAM and displays them in the same format as the Cisco IOS ACL commands, such as the show access-lists command.
|
Note Because of optimization, there might not be a one-to-one mapping of the output of the show controllers access-list command and the original access list. |
Examples
The following is sample output from the show controllers access-list command:
This output corresponds to the following access-list configuration:
Table 4 describes the fields in the show controllers access-list display.
To display the IP address table on an interface module, use the show controllers adjacency privileged EXEC command.
Syntax Description
Command Modes
Usage Guidelines
This command displays the IP address table on each interface module. This table contains IP address entries present in the ARP table (displayed by the show arp command) and adjacency table (displayed by the show adjacency command). This table has a one-to-one correspondence with the adjacency table.
Examples
The following is sample output from the show controllers adjacency command:
To display the IP prefix table on interface modules, use the show controllers cef EXEC command.
Syntax Description
|
| 1 CAM = content addressible memory |
Command Modes
Usage Guidelines
This command displays the IP prefix table on each interface module. The table contains IP prefix entries present in the IP routing table (displayed by the show ip route command) and CEF table (displayed by the show ip cef command). This table has a one-to-one correspondence with the CEF table.
Examples
The following is sample output from the show controllers cef summary command.
The following is sample output from the show controllers cef missing command:
The following is sample output from the show controllers cef detail command:
Table 5 describes some of the fields in the display.
Table 5 show controllers cef Field Descriptions
|
To display the contents of the Layer 2 CAM on interface modules, use the show controllers database layer2 privileged EXEC command.
Syntax Description
Command Modes
Usage Guidelines
The CAM number should be specified for dual CAM interfaces, such as a single interface of a 2-port GigabitEthernet interface module. For GigabitEthernet interfaces, if the CAM number is not specified then the commands displays information from CAM 0 by default .
Examples
The following is sample output from the show controllers database layer2 command:
The following is sample output from the show controllers database layer2 detail command:
Table 5 describes some of the fields in the display.
Table 6 show controllers database layer2 Field Descriptions
|
To display the resident interface entry that corresponds to a second interface, use the show controllers if-entry EXEC command.
Syntax Description
|
Command Modes
Usage Guidelines
One interface table for each physical port is resident on each interface module. Each table maintains entries corresponding to all interfaces or subinterfaces in the system. Each entry is indexed by a unique-identifier assigned to each interface or subinterface. The interface entry maintains the following information:
Examples
The following is sample output from the show controllers if-entry command:
The display contains the following categories of information:
To display the IP multicast routing table information stored on an interface module, use the show controllers ipmcast EXEC command.
Syntax Description
Command Modes
Usage Guidelines
This command displays the IP multicast routing table entries for all sources within a group (*, G) or for a specified source (S, G) within a group.
Examples
The following is sample output from the show controllers ipmcast command:
Table 7 describes some of the fields in the display.
Table 7 show controllers ipmcast Field Descriptions
|
To display the IPX network entries for a specified interface, use the show controllers ipx-prefix EXEC command.
Syntax Description
|
Command Modes
Examples
The following is sample output from the show controllers ipx-prefix command:
To display the counters on the switch router interfaces, use the show controllers c8500 counters EXEC command.
Syntax Description
This command has no arguments or keywords
Command Modes
Usage Guidelines
This command does not use IPC to get the information and can be used to find the port state prior to using any IPC-based commands. The counts reflect the actual number that the interface has received; these counter values are not reset when the clear counters command is issued.
Examples
The following is sample output from the show controllers c8500 counters command:
To display the interface number assigned to each interface or subinterface, use the show controllers c8500 ifmapping EXEC command.
Syntax Description
This command has no arguments or keywords.
Command Modes
Usage Guidelines
The output from this command shows all the interfaces and subinterfaces in the system, along with the assigned interface number for each. This command displays information specific to the Catalyst 8510 and Catalyst 8540 switch routers.
Examples
The following is sample output from the show controllers c800 ifmapping command:
To display IP multicast routing table control layer information, VC sharing information, and statistics, use the show controllers c8500 ipmcast EXEC command.
Syntax Description
Command Modes
Usage Guidelines
When an IP multicast source address or the keyword all is specified, this command displays IP multicast routing table control layer information, such as the VC used for an IP multicast packet matching a particular entry.
When the keyword vc-sharing is specified, this command displays all the point-to-multipoint VCs used for an IP multicast group and the number of entries that are sharing this VC.
When the keyword stats is used, this command displays debugging information.
The show controllers c8500 ipmcast command displays information specific to the Catalyst 8510 and Catalyst 8540 switch routers.
Examples
The following sample output from the show controllers c8500 ipmcast command displays routing table control layer information for an IP multicast group:
Table 8 describes some of the fields in the previous display.
Table 8 show controllers c8500 ipmcast Field Descriptions
|
The following sample output from the show controllers c8500 ipmcast command displays VC sharing information for an IP multicast group:
The following sample output from the show controllers c8500 ipmcast command displays general debugging statistics:
To display an IPX node entry for all interfaces, use the show controllers c8500 ipx-node EXEC command.
Syntax Description
|
|
Command Modes
Usage Guidelines
This command displays information specific to the Catalyst 8510 and Catalyst 8540 switch routers.
Examples
The following is sample output from the show controllers c8500 ipx-node command:
To display IPX network entries for a specified network for all interfaces, use the show controllers c8500 ipx-prefix EXEC command.
Syntax Description
Command Modes
Usage Guidelines
This command displays information specific to the Catalyst 8510 and Catalyst 8540 switch routers.
Examples
The following is sample output from the show controllers c8500 ipx-prefix command for all entries:
The following is sample output from the show controllers c8500 ipx-prefix command for a specified IPX network:
To display Layer 2 information about a particular MAC address, use the show controllers c8500 mac privileged EXEC command.
Syntax Description
|
Command Modes
Usage Guidelines
This command displays information specific to the Catalyst 8510 and Catalyst 8540 switch routers.
Examples
The following is sample output from the show controllers c8500 mac command:
Table 9 describes the fields in the display.
Table 9 show controllers c8500 mac Field Descriptions
|
To display current queuing information for all well-known static VCs in the system, use the show controllers c8500 queuing EXEC command.
Syntax Description
This command has no arguments or keywords.
Command Modes
Usage Guidelines
This command displays information specific to the Catalyst 8510 and Catalyst 8540 switch routers.
Examples
The following is sample output from the show controllers c8500 queuing command:
Table 9 describes the fields in the display.
Table 10 show controllers c8500 queuing Field Descriptions
|
To display VLAN information, use the show controllers vlan-entry privileged EXEC command.
Syntax Description
Command Modes
Usage Guidelines
Use this command to display the VLAN table in the CAM on a specific interface.
Examples
The following is sample output from the show controllers vlan-entry command:
Table 11 describes the fields in the display.
To display bridge table information on the switch router, use the show switch bridge-table privileged EXEC command.
Syntax Description
Command Modes
Usage Guidelines
The IOS bridge table keeps track of the number of interprocess communications (IPCs) received to learn and age the MAC information.
Examples
The following is sample output from the show switch bridge-table command:
Table 12 describes the fields in the display.
Table 12 show switch bridge-table Field Descriptions
|
To indicate that a port detected as being stuck should be reset and reloaded, use the epc port-reload command. To indicate that a port detected as being stuck should be shut down, use the no form of the command.
Syntax Description
This command has no arguments or keywords.
Command Modes
Usage Guidelines
The epc port-reload command specifies how the system should respond when a port is detected as being stuck. If a port does not respond for a specified duration of time (set with the epc portstuck-wait command) to messages sent by the CPU, the port is declared stuck. If the epc port-reload command is in effect, the stuck port is reset and its microcode is reloaded so that it resumes normal operation. If the no epc port-reload command is in effect, the port is shut down.
Examples
The following example shows how to specify that a port detected as being stuck should be reset and reloaded:
Related Commands
To specify how long the port-stuck detection mechanism should wait after a port has stopped responding to CPU requests, use the epc portstuck-wait command. To set this value to its default, use the no epc portstuck-wait command.
Syntax Description
|
Defaults
Command Modes
Syntax Description
For a port to be declared stuck, no response to any requests must be received by the CPU for the length of time specified by the epc portstuck-wait command. Once this time has elapsed with no response, the port is declared stuck and the action specified in the epc port-reload command is taken.
Examples
The following example shows how to specify a wait time of four minutes before declaring a port stuck:
Related Commands
Access Registrar, AccessPath, Are You Ready, ATM Director, Browse with Me, CCDA, CCDE, CCDP, CCIE, CCNA, CCNP, CCSI, CD-PAC, CiscoLink, the Cisco NetWorks logo, Cisco Powered Network logo, Cisco Systems Networking Academy, Fast Step, FireRunner, Follow Me Browsing, FormShare, GigaStack, IGX, Intelligence in the Optical Core, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, iQ Logo, iQ Readiness Scorecard, Kernel Proxy, MGX, Natural Network Viewer, Network Registrar, the Networkers logo, Packet, PIX, Point and Click Internetworking, Policy Builder, RateMUX, ReyMaster, ReyView, ScriptShare, Secure Script, Shop with Me, SlideCast, SMARTnet, SVX, TrafficDirector, TransPath, VlanDirector, Voice LAN, Wavelength Router, WebViewer, Workgroup Director, and Workgroup Stack are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, Empowering the Internet Generation, are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, Cisco, the Cisco Certified Internetwork Expert Logo, Cisco IOS, the Cisco IOS logo, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Collision Free, Enterprise/Solver, EtherChannel, EtherSwitch, FastHub, FastLink, FastPAD, IOS, IP/TV, IPX, LightStream, LightSwitch, MICA, NetRanger, Post-Routing, Pre-Routing, Registrar, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, Inc. or its affiliates in the U.S. and certain other countries.
All other brands, names, or trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0010R)
Posted: Tue Dec 9 08:15:09 PST 2003
All contents are Copyright © 1992--2003 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.
