cc/td/doc/product/mels/15540
hometocprevnextglossaryfeedbacksearchhelp
PDF

Table Of Contents

Troubleshooting the Cisco ONS 15540 Processor Card

Verifying Processor Card Configuration

Recovering a Lost Password

Verifying NME Interface Configurations

Troubleshooting Processor Card Memory

Verifying Hardware and Software Versions

Verifying Hardware and Software Compatibility

Troubleshooting Redundant Processor Cards

Verifying Hardware and Software Versions of Redundant Processor Cards

Verifying Redundant Processor Card Functions

Checking the DDTs Database and Release Notes for Workarounds

Using Bug Navigator II

Checking Cisco IOS Release Notes


Troubleshooting the Cisco ONS 15540 Processor Card

This chapter provides troubleshooting information about the Cisco ONS 15540 processors.

The Cisco ONS 15540 uses a fixed optical backplane between the client side transponder modules and the trunk side mux/demux modules. This minimizes the required functionality of the processor card. The processor card is used primarily to boot the system, provide Cisco IOS software version and download control, and provide APS redundancy and environmental monitoring mechanisms. In a redundant system, the processor cards monitor each other using the backplane Ethernet signals.

Note There is no switch fabric on the Cisco ONS 15540 processors.

The chapter includes the following sections:

• Verifying Processor Card Configuration

• Recovering a Lost Password

• Verifying NME Interface Configurations

• Troubleshooting Processor Card Memory

• Verifying Hardware and Software Versions

• Verifying Hardware and Software Compatibility

• Troubleshooting Redundant Processor Cards

• Checking the DDTs Database and Release Notes for Workarounds

Note For information on slot assignments, processor card LEDs, alarm condition clear and reset button, interrupt clear and reset button, NME LEDs, and cabling, refer to the
Cisco ONS 15540 ESP Hardware Installation Guide. For default configuration of the various modules, refer to the Cisco ONS 15540 ESP Configuration Guide and Command Reference.

Verifying Processor Card Configuration

To display the processor configuration and status, use the show running-config command.

Command
Purpose

show running-config

Shows all components of the processor running a configuration.


The following example shows the show running-config command, which displays all the components of the processor configuration. For a detailed description of this command, refer to the Cisco IOS Configuration Fundamentals Command Reference.

Switch# show running-config
Building configuration...

Current configuration : 2773 bytes
!
version 12.1
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname Switch
!
boot bootldr bootflash:ons15540-i-mz.Apr18
no logging console
enable password lab
!
filter 1/0 1 8
ip subnet-zero
ip host-routing
no ip finger
no ip domain-lookup
!
oscp timer hello-interval 300
oscp timer inactivity-factor 25
oscp timer hello-holddown 50
patch Thru1/0 Wdm1/1
patch Thru1/1 Wdm1/2
!
!
interface Transparent3/0/0
mtu 1514
no ip address
shutdown
monitor enable
encap sonet rate oc3
!
[Information deleted]

!
interface Wave3/0
mtu 1514
no ip address
shutdown
!
[Information deleted]

!
interface Wdm1/0
mtu 1514
no ip address
[Information deleted]

!
interface Wavepatch3/0/0
mtu 1514
no ip address
shutdown
[Information deleted]

!
interface Filter1/0/0
mtu 1514
no ip address
[Information deleted]

!
interface Thru1/0
mtu 1514
no ip address
[Information deleted]

!
interface FastEthernet0
ip address 172.20.42.225 255.255.255.0
duplex auto
speed auto
!
ip default-gateway 172.20.42.206
ip classless
no ip http server
!
cdp timer 200
cdp holdtime 120
snmp-server engineID local 80000009030000AB0000007C
snmp-server community public RW
snmp-server enable traps rf
!
line con 0
transport input none
line aux 0 1
line vty 0 4
exec-timeout 0 0
password lab
login
length 0
!
end

Recovering a Lost Password

This section describes the procedure to recover a lost login or to enable a password. The procedure differs depending on the platform and the software used, but in all cases, password recovery requires that the system be taken out of operation and powered down.

If you need to perform the following procedure, make certain that there are secondary systems that can temporarily serve the functions of the system undergoing the procedure. If this is not possible, advise all potential users and, if possible, perform the procedure during low-use hours.

Note Make a note of your password, and store it in a secure place.

All of the procedures for recovering lost passwords depend on changing the configuration register of the system. This is done by reconfiguring the system software.

More recent Cisco platforms run from Flash memory or are netbooted from a network server and can ignore the contents of NVRAM (nonvolatile random-access memory) when booting. By ignoring the contents of NVRAM, you can bypass the configuration file (which contains the passwords) and gain complete access to the system. You can then recover the lost password or configure a new one.

Note If your password is encrypted, you cannot recover it. You must configure a new password.

Follow these steps to recover a password:

Step 1 Enter the show version command and the configuration register value in the privileged EXEC mode. The default value is 0x2102.

Step 2 Power up the Cisco ONS 15540.

Step 3 Press the Break key sequence or send a break signal, which is usually ^] within 60 seconds of turning the system on. If you do not see the > prompt with a system name, the terminal is not sending the correct break signal. In that case, check the terminal or terminal emulation setup.

Step 4 Enter the confreg command at the > prompt.

Step 5 Answer yes to the Do you wish to change configuration [y/n]? prompt.

Step 6 Answer no to all the questions that appear until you reach the Ignore system config info [y/n] prompt. Answer yes.

Step 7 Answer no to the remaining questions until you reach the Change boot characteristics [y/n]? prompt. Answer yes.

Step 8 Enter 2 at the enter to boot: prompt.

Step 9 Answer no to the Do you wish to change configuration [y/n]? prompt.

Step 10 Enter the reset command at the rommon> prompt.

Step 11 Enter the enable command at the Switch> prompt. You are in enable mode and see the Switch# prompt.

Step 12 Enter the show startup-config command to view your password.

Step 13 Proceed to Step 16 if your password is clear text. Or, continue with Step 14 if your password is encrypted.

Step 14 Enter the configure memory command to copy the NVRAM into memory if your password is encrypted.

Step 15 Enter the copy running-config startup-config command.

Step 16 Enter the configure terminal command.

Step 17 Enter the enable secret password command.

Step 18 Enter the config-register value command, where value is whatever value you entered in Step 1.

Step 19 Enter the exit command to exit configuration mode.

Step 20 Enter the copy running-config startup-config command.

Step 21 Enter the reload command at the prompt.

Verifying NME Interface Configurations

The administration interfaces provide simple command-line interfaces to all internal management and debugging facilities of the processor. To manage and debug the processor, you can use the NME (network management Ethernet) interface, the console port, and the auxiliary port.

For cable connection information for each of these interface ports, refer to the
Cisco ONS 15540 ESP Hardware Installation Guide. For initial configuration information, refer to the Cisco ONS 15540 ESP Configuration Guide and Command Reference.

The NME interface has a full duplex, auto sensing connection with troubleshooting LEDs on the processor faceplate.

You can configure and monitor the NME connection using the CLI. The NME connection appears in the configuration as FastEthernet 0 or FastEthernet 1 depending on the slot where the processor is installed.

To display the NME FastEthernet module configuration and status, use the following commands:

Command
Purpose

show interfaces FastEthernet 0

Displays the status of the physical interface.

show controllers

Displays the interface memory management and error counters on the FastEthernet interface.


Follow these steps to verify the NME interface:

Step 1 Use the show interfaces FastEthernet 0 slot/subcard/port command to check the NME interface configuration.

Switch# show interfaces FastEthernet 0
FastEthernet0 is up, line protocol is up
Hardware is AmdFE, address is 0000.ab00.0000 (bia 0000.ab00.0000)
Internet address is 172.20.42.118/24
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Half-duplex, 10Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
222 packets input, 15877 bytes
Received 194 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 abort
0 watchdog
0 input packets with dribble condition detected
28 packets output, 3384 bytes, 0 underruns(0/0/0)
2 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
2 lost carrier, 0 no carrier

Step 2 Check the FastEthernet field to see whether the interface is up. If it is down, check for the following:

•Disconnected or faulty cabling. Check cables.

•Hardware failure. Swap hardware.

If administratively down, the interface has been administratively taken down. Use the no shutdown interface configuration command to reenable the interface.

Step 3 Check the line protocol field to see whether the status is up.

If the interface is down, the line protocol software processes might have determined that the line is unusable or the local or remote interface might be misconfigured. See if the interface can be brought up by following the recommendations in Step 2.

Step 4 Check the duplex mode field. It should match the speed of the interface and be configured as auto-negotiation.

Step 5 Check the last input and last output fields. They show the number of hours, minutes, and seconds since the last packet was successfully received or transmitted by the interface.

Step 6 Check the output hang field. It shows the number of hours, minutes, and seconds since the last reset caused by a lengthy transmission.

Step 7 Check the CRC field. The presence of many CRC errors, but not many collisions, indicates excessive noise. If the number of errors is too high, check the cables for damage. If you are using UTP cable, make sure you are using Category 5 cables and not another type, such as Category 3.

Note Errors and the input and output difference should not exceed 0.5 to 2.0 percent of traffic on the interface.

Step 8 Check the collisions fields. These numbers indicate packet collisions and these numbers should be very low. The total number of collisions, with respect to the total number of output packets, should be 0.1 percent or less.

Step 9 Check the late collisions fields. Late collisions should never occur in a properly designed Ethernet network. They usually occur when Ethernet cables are too long or when there are too many repeaters in the network.

Step 10 Check carrier fields. These numbers indicate a lost carrier detect signal and can be caused by a malfunctioning interface that is not supplying the transmit clock signal or by a cable problem. If the system notices that the carrier detect line of an interface is up, but the line protocol is down, it periodically resets the interface in an effort to restart it. Interface resets can also occur when an interface is looped back or shut down.

Step 11 Check the buffer fields. These numbers indicate the number of received packets discarded because there was no buffer space. Broadcast storms on Ethernet networks, and bursts of noise on serial lines, are often responsible for no-input buffer events.

Step 12 Check the FastEthernet field to see whether the interface is up. If it is down, see if the interface can be brought up by following the recommendations in Step 2. If administratively down, the interface has been administratively taken down. Use the no shutdown interface configuration command to reenable the interface.

If you determine that the connection is configured incorrectly, refer to the
Cisco ONS 15540 ESP Configuration Guide and Command Reference.

In addition, you can use the show controllers command to troubleshoot the status of the NME interface configuration:

Switch# show controllers fastethernet 0
Interface FastEthernet0
Hardware is AMD Laguna
ADDR: 60B31408, FASTSEND: 0, MCI_INDEX: 0
DIST ROUTE ENABLED: 0
Route Cache Flag: 1
LADRF=0x0000 0x0100 0x0000 0x0000
CSR0 =0x00000072, CSR3 =0x00001044, CSR4 =0x0000491D, CSR15 =0x00000180
CSR80 =0x00009900, CSR114=0x00000000, CRDA =0x04D86460, CXDA =0x04D866C0
BCR9 =0x00000000 (half-duplex)
CSR5 =0x00000001, CSR7 =0x000008A8, CSR100=0x0000F000, CSR125=0x00005C3C
BCR2 =0x00000000, BCR9 =0x00000000, BCR18 =0x00001981, BCR22 =0x0000FF06
BCR25 =0x00000017, BCR26 =0x0000000C, BCR27 =0x00000000, BCR32 =0x00004800
HW filtering information:
Promiscuous Mode Disabled, PHY Addr Enabled, Broadcast Addr Enabled
PHY Addr=0000.AB00.0000, Multicast Filter=0x0000 0x0100 0x0000 0x0000
amdp2_instance=0x60B32F98, registers=0x46000000, ib=0x4D86180
rx ring entries=64, tx ring entries=128
rxring=0x4D861E0, rxr shadow=0x60B331A4, rx_head=40, rx_tail=0
txring=0x4D86620, txr shadow=0x60B332D0, tx_head=10, tx_tail=10, tx_count=0
Software MAC address filter(hash:length/addr/mask/hits):
spurious_idon=0, filtered_pak=0, throttled=0, enabled=0, disabled=0
rx_framing_err=0, rx_overflow_err=0, rx_buffer_err=0, rx_bpe_err=0
rx_soft_overflow_err=0, rx_no_enp=0, rx_discard=0, rx_miss_count=0
tx_one_col_err=0, tx_more_col_err=0, tx_no_enp=0, tx_deferred_err=0
tx_underrun_err=0, tx_late_collision_err=0, tx_loss_carrier_err=2
tx_exc_collision_err=0, tx_buff_err=0, fatal_tx_err=0
hsrp_conf=0, need_af_check=0
tx_limited=0(128)
PHY registers:
Register 0x00: 1000 786D 0000 6B60 01E1 0021 0004 2001
Register 0x08: 0000 0000 0000 0000 0000 0000 0000 0000
Register 0x10: 0013 0004 186A 001E 0440 2004 0001 0200
Register 0x18: 0008 0000 0000 0000

Troubleshooting Processor Card Memory

To troubleshoot the processor memory, use the following commands:

Command
Purpose

show flash

Displays the configuration register value.

show memory

Shows statistics about the Cisco ONS 15540 memory, including free pool statistics.

show buffers

Displays statistics for the buffer pools on the Cisco ONS 15540.


Troubleshooting Cisco ONS 15540 processor card memory is the same as troubleshooting any Cisco route processor. You can refer to the document Troubleshooting Hardware and Booting Problems at the following URL: http://www.cisco.com/univercd/cc/td/doc/cisintwk/itg_v1/tr1903.htm.

If the Cisco ONS 15540 fails, it is sometimes useful to get a full copy of the memory image, called a core dump, to identify the cause of the failure. Core dumps are generally only useful to your technical support representative. For troubleshooting information relating to system management and information about creating core dumps, refer to the Cisco IOS Configuration Fundamentals Command Reference.

Verifying Hardware and Software Versions

A common problem is an incompatibility between a hardware module and the Cisco IOS software version needed to perform a particular function. This section describes troubleshooting that problem.

Display the hardware and software versions to ensure that they are the most recent. Very old hardware and software versions (two or three versions back) can have caveats that have been fixed in more recent versions. Use the following EXEC commands to display version information:

Command
Purpose

show version

Displays the software version information.

show hardware [detail]

Displays detailed hardware information including revision level and version.


To verify hardware and software versions, use the following steps:

Step 1 Use the show version command to display the system software version on the active processor.

Switch# show version

Cisco Internetwork Operating System Software
IOS (tm) ONS-15540 Software (manopt-M0-M), 12.1(X:X)
Copyright (c) 1986-2001 by cisco Systems, Inc.
Compiled Fri 23-Feb-01 15:23 by ffrazer
Image text-base:0x60010950, data-base:0x604E8000
ROM:System Bootstrap, Version 12.1(X:X)
BOOTFLASH:ONS-15540 Software (manopt-M0-M), 12.1(X:X)
Switch uptime is 30 minutes
System returned to ROM by power-on
System image file is "tftp://test/eng/manopt-m0-mz.010223.6"
cisco (QUEENS-CPU) processor with 98304K/32768K bytes of memory.
R7000 CPU at 234Mhz, Implementation 39, Rev 2.1, 256KB L2, 2048KB L3 Cache
Last reset from power-on
2 Ethernet/IEEE 802.3 interface(s)
509K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
16384K bytes of Flash internal SIMM (Sector size 64K).
Configuration register is 0x102

Step 2 Verify the ROM field. It indicates the release of Cisco IOS software loaded and running on the active processor.

Step 3 Use the show hardware command to display the hardware revision levels for the processors.

Switch# show hardware
--------------------------------------------------------------------------------
named Switch, Date: 04:36:18 UTC Fri Apr 20 2001
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Slot Controller Type Part No. Rev Serial No. Mfg. Date RMA No. H/W Ver.
---- ------------------ ---------- --- ----------- ---------- ------- ----------
1 Mx-DMx-Mthrbd 01/01/2000 0x00 0.0
3 XpndrMotherboard 01/01/2000 0x00 0.0
6 Queens CPU 73-5621-02 03 CAB0505GZHD 02/16/2001 0x00 2.1
7 Queens CPU 73-5621-02 03 CAB0505GZHV 02/16/2001 0x00 2.1

--------------------------------------------------------------------------------
Back-Plane EEPROM
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Model Ver Serial No. MAC-Address MAC-Size RMA No. RMA Code MFG-Date
--------- --- ---------- ----------------- -------- ------- -------- -----------
0.0 00-ab-00-00-00- 1 0x00 0x00 01/01/2000

--------------------------------------------------------------------------------
Power-Supply Module
--------------------------------------------------------------------------------
Primary Power-Supply is : Not working
Backup Power-Supply is : Not working

Step 4 Verify that the hardware versions listed in the H/W Ver column for the processors in slots 6 and 7 are the same. If the hardware versions are not the same, continue with the "Verifying Hardware and Software Compatibility" section.

Step 5 Use the show hardware detail command to display detailed information about the processor hardware, including the functional image versions.

Switch# show hardware detail
--------------------------------------------------------------------------------
named Switch, Date: 04:36:29 UTC Fri Apr 20 2001
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Slot Number : 1
Controller Type : Mx-DMx-Mthrbd
On-Board Description :
Orderable Product Number:
Board Part Number :
Board Revision :
Serial Number :
Manufacturing Date : 01/01/2000
Hardware Version : 0.0
RMA Number : 0x00
RMA Failure Code : 0x00
Functional Image Version: 2.18

--------------------------------------------------------------------------------

[Information Deleted]

--------------------------------------------------------------------------------
Slot Number : 6
Controller Type : Queens CPU
On-Board Description : Queens_CPU_PHASE_0
Orderable Product Number: N/A
Board Part Number : 73-5621-02
Board Revision : 03
Serial Number : CAB0505GZHD
Manufacturing Date : 02/16/2001
Hardware Version : 2.1
RMA Number : 0x00
RMA Failure Code : 0x00
Functional Image Version: 1.8

--------------------------------------------------------------------------------
Slot Number : 7
Controller Type : Queens CPU
On-Board Description : Queens_CPU_PHASE_0
Orderable Product Number: N/A
Board Part Number : 73-5621-02
Board Revision : 03
Serial Number : CAB0505GZHV
Manufacturing Date : 02/16/2001
Hardware Version : 2.1
RMA Number : 0x00
RMA Failure Code : 0x00
Functional Image Version: 1.11

--------------------------------------------------------------------------------

[Information Deleted]

Step 6 Verify that the Hardware Version and Functional Image Version fields for the processors in slots 6 and 7 are the same. If they are not the same, continue with the following process to confirm that they are compatible.

Verifying Hardware and Software Compatibility

You can verify your hardware and software version compatibility by using the following EXEC command to display processor compatibility information:

Command
Purpose

show redundancy capability

Displays the software version compatibility information.


To verify hardware and software compatibility of the processors and modules, use the following steps:

Step 1 Use the show redundancy capability command to display the system software version compatibility with the various modules installed.

Switch# show redundancy capability

CPU capability support

Active CPU Sby CPU Sby Compat CPU capability description
---------- ---------- ----------- ----------------------------------------
96 MB 96 MB OK CPU DRAM size
32 MB 32 MB OK CPU PMEM size
  512 KB 512 KB OK CPU NVRAM size
16 MB 16 MB OK CPU Bootflash size
2.1 2.1 OK CPU hardware major.minor version
1.129 1.129 OK CPU functional major.minor version

Linecard driver major.minor versions, (counts:Active=18, Standby=18)

Active CPU Sby CPU Sby Compat Drv ID Driver description
---------- ---------- ----------- ------ -----------------------------------
1.1 1.1 OK 0x1000 CPU w/o Switch Fabric
1.1 1.1 OK 0x1001 Fixed Transponder, w/monitor
1.1 1.1 OK 0x1002 Fixed Transponder, no monitor
1.1 1.1 OK 0x1003 Pluggable Transponder, w/monitor
1.1 1.1 OK 0x1004 Pluggable Transponder, no monitor
1.1 1.1 OK 0x1005 Line Card Motherboard
1.1 1.1 OK 0x1006 Backplane
--More-- Active CPU Sby CPU Sby Compat Drv ID Driver description
---------- ---------- ----------- ------ -----------------------------------
1.1 1.1 OK 0x1007 32-ch Mux/Demux
1.1 1.1 OK 0x1008 Fixed 4-ch Mux/Demux, no OSC
1.1 1.1 OK 0x1009 Fixed 8-ch Mux/Demux, no OSC
1.1 1.1 OK 0x100A Modular 4-ch Mux/Demux, no OSC
1.1 1.1 OK 0x100B Modular 8-ch Mux/Demux, no OSC
1.1 1.1 OK 0x100C 32-ch Array Wave Guide
1.1 1.1 OK 0x100D Mux/Demux Motherboard
1.1 1.1 OK 0x100E Modular 4-ch Mux/Demux plus OSC
1.1 1.1 OK 0x100F Modular 8-ch Mux/Demux plus OSC
1.1 1.1 OK 0x1010 Mux-Demux Motherboard, no OSC
1.1 1.1 OK 0x1011 Line Card Motherboard, no splitter

Software sync client versions, listed as version range X-Y.
X indicates the oldest peer version it can communicate with.
Y indicates the current sync client version.
Sync client counts:Active=2, Standby=2

Active CPU Sby CPU Sby Compat Cl ID Redundancy Client description
---------- ---------- ----------- ----- ------------------------------------
ver 1-1 ver 1-1 OK 17 CPU Redundancy
ver 1-1 ver 1-1 OK 6 OIR Client

Backplane IDPROM comparison
Backplane IDPROM field Match Local CPU Peer CPU
--------------------------- ----- -------------------- --------------------
idversion YES 1 1
magic YES 153 153
card_type YES 4102 4102
order_part_num_str YES N/A N/A
description_str YES Manhattan_Backplane_PHASE_0
Manhattan_Backplane_PHASE_0
board_part_num_str YES 73-5655-03 73-5655-03
board_revision_str YES
serial_number_str YES XXXXXXXXX XXXXXXXXX
date_of_manufacture_str YES 02/16/2001 02/16/2001
deviation_numbers_str YES 0 0
manufacturing_use YES 0 0
rma_number_str YES 0x00 0x00
rma_failure_code_str YES 0x00 0x00
oem_str YES Cisco_Systems Cisco_Systems
clei_str YES
snmp_oid_substr YES 0 0
schematic_num_str YES 92-4113-03 92-4113-03
hardware_major_version YES 3 3
hardware_minor_version YES 0 0
engineering_use_str YES 1 1
crc16 OK 17248 42412
user_track_string YES lab lab
diagst YES ^A ^A
board_specific_revision YES 1 1
board_specific_magic_number YES 153 153
board_specific_length YES 56 56
mac_address_block_size YES 16 16
mac_address_base_str YES 00016442e6aa 00016442e6aa
cpu_number OK 0 1
optical_backplane_type YES 255 255
Switch#

Step 2 Check the CPU memory sizes and versions in the column, CPU capability description column. The numbers in the columns Active CPU and Sby CPU (Standby CPU) columns should match. If not, check the Sby Compat (Standby Compatibility) column. If this column indicates the values are OK, then these values will function as compatible redundant processors. If not, swap the processors with versions that are compatible.

Step 3 Check the CPU hardware major.minor versions and CPU functional major.minor versions in the column, CPU capability description column. The numbers in the Active CPU and Sby CPU (Standby CPU) columns should match. If not, check the Sby Compat (Standby Compatibility) columns. If this column indicates the values are OK, then these values will function as compatible redundant processors. If not, swap the processors with versions that are compatible.

Step 4 Check the information in the Linecard driver section of the display. This section shows the compatibility of the software versions installed on the active and standby processors with the various modules installed in the system.

Step 5 Check the Sby Compat (Standby Compatibility) and the Driver description columns. An OK in the Sby Compat column indicates the software version installed on the processors supports the drivers on the modules listed.

Step 6 Check the Software sync client version section of the display. The Active CPU, Sby CPU and Redundancy Client description columns indicate the software versions the two processors can use to synchronize their configurations. The version range in the display, shown as X-Y, indicates oldest-current peer client versions. For example, if the version lists 1-2, that indicates version 1 is the oldest version that the current version 2 could synchronize with its configuration.

Step 7 Check the Backplane IDPROM comparison section of the display. Check the Match column. This indicates which elements match, are acceptable, or fail. Some elements do not match but the range is acceptable. For example, the crc16 elements fields never match because the information in the IDPROMs of the two processors are different so the checksums never match. But they do appear as OK or compatible.

If any of the drivers are not supported or appear as OK, try updating the images installed on the processors. Use the information in the "Checking the DDTs Database and Release Notes for Workarounds" section to upgrade to a more recent version. That should solve a processor image compatibility problem.

Troubleshooting Redundant Processor Cards

The Cisco ONS 15540 supports fault tolerance by allowing a standby processor card to take over if the active processor card fails. This standby, or redundant, processor card runs in hot-standby mode. In hot-standby mode, the standby processor card is partially booted with the Cisco IOS software; however, no configuration is loaded.

At the time of a switchover, the standby processor card takes over as the active processor card and loads the configuration as follows:

•If the running configurations on the active and standby processor cards match, the new active processor card uses the running configuration file.

•If the running configurations on the active and standby processors do not match, the new active processor card uses the last saved configuration file in its NVRAM (not the NVRAM of the former active processor card).

The former active processor card then becomes the standby processor.

Note If the standby processor card is unavailable, a major alarm is reported. Use the show facility-alarm status command to display the redundancy alarm status.

For redundant processor cards to function correctly, your Cisco ONS 15540 processor cards must meet the following requirements:

•Both processor cards must have compatible hardware configurations.

•Both processor cards must have compatible releases of Cisco IOS software.

A common error you may encounter is the incompatibility of hardware modules and the Cisco IOS software version needed to perform a particular function.

Verifying Hardware and Software Versions of Redundant Processor Cards

To troubleshoot the processor card hardware and software versions for redundancy, use the following commands:

Command
Purpose

show version

Displays the processor card software version information.

show redundancy

Displays the hardware and software configurations of the active and standby processor cards.


To troubleshoot the hardware and software versions on redundant processor cards, use the following steps:

Step 1 Use the show version command to display the system software version on the active processor card as described in the "Verifying Hardware and Software Versions" section.

Step 2 Use the show redundancy command to check the configuration and status of the active and standby processor card.

Switch# show redundancy

Redundant system information
----------------------------
Available Uptime: 50 weeks, 5 days, 6 hours, 14 minutes
Time since last switchover: 30 weeks, 3 days, 2 hours, 20 minutes
Switchover Count: 2
Inter-CPU Communication State:UP
Last Running Config sync: 3 hours, 5 minutes
Running Config sync status: In Sync
Last Startup Config sync: 1 week, 3 days, 4 hours, 19 minutes
Startup Config sync status: In Sync

Last Restart Reason: Normal boot
This CPU is the Active CPU.
-------------------------------
Slot: 6
Time since CPU Initialized: 35 weeks, 5 days, 1 hours, 42 minutes
Image: Version 12.1(20010328:195418) ons15540-i-mz
Software Redundancy State: ACTIVE
Hardware State: ACTIVE
Hardware Severity: 0
Peer CPU is the Standby CPU.
-------------------------------
Slot: 7
Time since CPU Initialized: 7 hours, 14 minutes
Image: Version 12.1(20010328:195418) ons15540-i-mz
Software Redundancy State: STANDBY HOT
Hardware State: STANDBY
Hardware Severity: 0

Step 3 Verify the Last Running Config sync and Last Startup Config sync fields. They indicate the last time the running configuration and startup configuration were synchronized between the processor cards.

Step 4 Verify the active, standby, and Slot fields. They indicate in which slot the active processor card is configured.

Verifying Redundant Processor Card Functions

To troubleshoot the processor card function capabilities and redundancy, use the following commands:

Command
Purpose

show redundancy capability

Displays capabilities for the active and standby processor cards.

show redundancy clients

Displays internal redundancy software client information, which can be used to debug redundancy software.

show redundancy counters

Displays internal redundancy software counter information, which can be used to debug redundancy software.

show redundancy history

Displays the internal redundancy software history log, which can be useful for debugging redundancy software.

show redundancy running-config-file

Displays the running-config-file on the standby processor.

show redundancy states

Displays internal redundancy software state information.


Follow these steps to troubleshoot processor and redundancy capabilities on the system:

Step 1 Use the show redundancy capability command to display capabilities of the active or standby processor cards described in the "Verifying Hardware and Software Versions" section.

Step 2 Use the show redundancy clients command to display a list of internal redundancy clients.

Switch# show redundancy clients
clientID = 0 clientSeq = 0 RF_INTERNAL_MSG
clientID = 6 clientSeq = 16 OIR Client
clientID = 17 clientSeq = 40 CPU Redundancy
clientID = 19 clientSeq = 9999 RF_LAST_CLIENT

Step 3 Use the show redundancy counters command to display internal redundancy software counters.

Switch# show redundancy counters
Redundancy Facility OMs
comm link up = 0
comm link down down = 0

invalid client tx = 0
null tx by client = 0
tx failures = 0
tx msg length invalid = 0

client not rxing msgs = 0
rx peer msg routing errors = 0
null peer msg rx = 0
errored peer msg rx = 0

buffers tx = 0
tx buffers unavailable = 0
buffers rx = 0
buffer release errors = 0

duplicate client registers = 0
failed to register client = 0
Invalid client syncs = 0

Step 4 Use the show redundancy history command to display internal redundancy software history.

Switch# show redundancy history
Redundancy Facility Event Log:
00:00:00 client added: RF_INTERNAL_MSG(0) seq=0
00:00:00 client added: RF_LAST_CLIENT(19) seq=9999
00:00:16 client added: CPU Redundancy(17) seq=40
00:00:16 *my state = INITIALIZATION(2) *peer state = DISABLED(1)
00:00:16 RF_PROG_INITIALIZATION(0) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:16 RF_PROG_INITIALIZATION(0) CPU Redundancy(17) op=0 rc=11
00:00:16 RF_PROG_INITIALIZATION(0) RF_LAST_CLIENT(19) op=0 rc=11
00:00:16 *my state = NEGOTIATION(3) peer state = DISABLED(1)
00:00:16 RF_STATUS_PEER_PRESENCE(12) op=0
00:00:16 RF_EVENT_GO_ACTIVE(28) op=0
00:00:16 *my state = ACTIVE-FAST(9) peer state = DISABLED(1)
00:00:16 RF_STATUS_SPLIT_ENABLE(15) CPU Redundancy(17) op=0
00:00:16 RF_PROG_ACTIVE_FAST(6) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:16 RF_PROG_ACTIVE_FAST(6) CPU Redundancy(17) op=0 rc=11
00:00:16 RF_PROG_ACTIVE_FAST(6) RF_LAST_CLIENT(19) op=0 rc=11
00:00:16 *my state = ACTIVE-DRAIN(10) peer state = DISABLED(1)
00:00:16 RF_PROG_ACTIVE_DRAIN(7) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:16 RF_PROG_ACTIVE_DRAIN(7) CPU Redundancy(17) op=0 rc=11
00:00:16 RF_PROG_ACTIVE_DRAIN(7) RF_LAST_CLIENT(19) op=0 rc=11
00:00:16 *my state = ACTIVE_PRECONFIG(11) peer state = DISABLED(1)
00:00:16 RF_PROG_ACTIVE_PRECONFIG(8) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:16 RF_PROG_ACTIVE_PRECONFIG(8) CPU Redundancy(17) op=0 rc=11
00:00:16 RF_PROG_ACTIVE_PRECONFIG(8) RF_LAST_CLIENT(19) op=0 rc=11
00:00:16 *my state = ACTIVE_POSTCONFIG(12) peer state = DISABLED(1)
00:00:16 RF_PROG_ACTIVE_POSTCONFIG(9) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:16 RF_PROG_ACTIVE_POSTCONFIG(9) CPU Redundancy(17) op=0 rc=11
00:00:16 RF_PROG_ACTIVE_POSTCONFIG(9) RF_LAST_CLIENT(19) op=0 rc=11
00:00:16 *my state = ACTIVE(13) peer state = DISABLED(1)
00:00:16 RF_PROG_ACTIVE(10) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:16 RF_PROG_ACTIVE(10) CPU Redundancy(17) op=0 rc=11
00:00:16 RF_PROG_ACTIVE(10) RF_LAST_CLIENT(19) op=0 rc=11
00:00:16 client added: OIR Client(6) seq=16
00:00:19 RF_STATUS_PEER_PRESENCE(12) op=0
00:00:36 Configuration parsing complete
00:00:36 System initialization complete

Step 5 Use the show redundancy running-config-file command to display running configuration on the standby processor card.

sby-Switch# show redundancy running-config-file

Current configuration :601 bytes
!
version 12.1
no service pad
...
<Information deleted>
...
!
end

Step 6 Use the show redundancy states command to display internal redundancy software state information.

Switch# show redundancy states
my state = 13 -ACTIVE
peer state = 1 -DISABLED
Mode = Simplex
Unit = Primary
Unit ID = 6

Split Mode = Disabled
Manual Swact = Disabled Reason: Simplex mode
Communications = Down Reason: Simplex mode

client count = 4
client_notification_TMR = 30000 milliseconds
RF debug mask = 0x80

Refer to the Cisco ONS 15540 ESP Configuration Guide and Command Reference for the following:

•Configuring processor card redundancy

•Upgrading the software image on the redundant processor card

•Downloading the IOS image on the processor cards

Checking the DDTs Database and Release Notes for Workarounds

There are two methods you can use to check for Cisco IOS software bugs (defect tracking tool numbers [DDTs]) in your version of the Cisco IOS software. You can use the Bug Navigator II or check the release notes. Often, your problems with the Cisco ONS 15540 have been fixed or a workaround has been determined in a more recent version of software.

Using Bug Navigator II

Bug Navigator II is a DDTS search tool you can use to search the DDTS database and ask either of two types of questions:

•Symptom Diagnostics (for example, "What defect is causing my current symptoms?")

•Upgrade Planning (for example, "What software release is best for the features I am interested in?")

To search the DDTS database, you can access Bug Navigator II on the World Wide Web at http://www.cisco.com/support/bugtools/bugtool.shtml. Then perform the following steps:

Step 1 Enter your user name and password at the login prompt if you are not already logged in to Cisco.com.

Step 2 Read the Bug Navigator II Help instructions.

Step 3 Select your hardware from the Cisco Hardware list. The Bug Navigator search tool replaces Bug Navigator II Help (in the right frame of the page).

Step 4 Select the following from the drop-down menus:

•Version

•Revision

•Severity

Note As an option, you can enter words or phrases (separated by commas) in the data entry field to limit your search.

Step 5 Click the Search button.

The entire window is replaced with a Bug Search Results window with a list of DDTs containing your search criteria. Look at the Bug reports listed in the titles column. An existing bug entry that describes the problem you are having may have been fixed in a more recent version of the Cisco IOS software. Look in the Fixed-in column for a later version of the Cisco IOS software. All you might have to do to solve your problem is upgrade your software.

If a software upgrade is not listed as a way to solve your problem, double-click on the bug title and read the DDTs details; a workaround might be listed there.

Checking Cisco IOS Release Notes

Release notes describe the features and caveats for Cisco IOS software releases. The release notes are listed by both product and Cisco IOS release number.

The "Caveats" section of the release notes lists known caveats by tracking the DDTs number and the release number, and indicates whether the caveat has been corrected.

The "Caveat Symptoms and Workarounds" section summarizes caveat symptoms and suggested workarounds. You can also search through this section online, using either a word string or the DDTs number.

hometocprevnextglossaryfeedbacksearchhelp

Posted: Wed Mar 22 05:38:30 PST 2006
All contents are Copyright © 1992--2006 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.