Troubleshooting Processor Card Problems

Table Of Contents

Troubleshooting Processor Card Problems

2.1  Overview

2.2  Initial Troubleshooting Checklist

2.3  Verifying Processor Card Configuration

2.4  Recovering a Lost Password

2.5  Verifying NME Interface Configurations

2.6  Troubleshooting Processor Memory

2.7  Verifying Hardware and Software Versions

2.8  Verifying Hardware and Software Compatibility

2.9  Troubleshooting Redundant Processor Cards

2.9.1  Verifying Hardware and Software Versions of Redundant Processor Cards

2.9.2  Verifying Redundant Processor Card Functions

2.10  Troubleshooting Processor Cards

2.10.1  Active Processor Card Boot Failure

2.10.2  Standby Processor Card Boot Failure

2.10.3  Unable to Access Processor Card Console

2.10.4  Unable to Access Enable Mode on Active Processor Card

2.10.5  Unable to Access Enable Mode on Standby Processor Card

Troubleshooting Processor Card Problems

This chapter describes how to troubleshoot processor card problems. The chapter includes the following sections:

• Overview

• Initial Troubleshooting Checklist

• Verifying Processor Card Configuration

• Recovering a Lost Password

• Verifying NME Interface Configurations

• Troubleshooting Processor Memory

• Verifying Hardware and Software Versions

• Verifying Hardware and Software Compatibility

• Troubleshooting Redundant Processor Cards

• Troubleshooting Processor Cards

2.1  Overview

The Cisco ONS 15540 ESPx includes two processor cards for redundancy. Each processor consists of a number of subsystems, including a CPU, a system clock, Ethernet switch for communicating between processors and with the LRC (line card redundancy controller) on the mux/demux motherboards and line card motherboards, and a processor redundancy controller. The active processor controls the node, and all cards in the system make use of the system clock and synchronization signals from the active processor.

The processor card is equipped with a console port, a Fast Ethernet interface for Telnet access and network management, and an auxiliary port. There are two slots for Flash PC Cards or Flash disks.

On the processor card front panel are LEDs that display the status of critical, major, and minor signals, as well as the status of alarm cutoff and history conditions. The alarm signals from the processor go to an alarm daughterboard on the backplane, which has a connector for central office alarm facilities.

Note For information on slot assignments, processor card LEDs, alarm condition clear and reset button, interrupt clear and reset button, NME LEDs, and cabling, refer to the
Cisco ONS 15540 ESPx Hardware Installation Guide. For default configuration of the various modules, refer to the Cisco ONS 15540 ESPx Configuration Guide.

2.2  Initial Troubleshooting Checklist

Follow this initial checklist before proceeding with the troubleshooting procedures:

•Issue the show running-config command to check the running configuration.

•Ensure the LEDs on the processor cards show the proper state.

•Ensure the Ethernet and Console cables are connected properly.

•Issue the show facility-alarm status command to check for processor card, fan, or power supply alarms.

•Issue the show hardware detail command to verify the processor card functional image.

•Ensure online and power-on diagnostics do not report any alarms or failures for the processor card.

•Ensure the active and standby processor cards are compatible.

•Ensure the active and standby processor card have the same version of software installed.

2.3  Verifying Processor Card Configuration

To display the processor card configuration and status, use the show running-config command.

Command

Purpose

show running-config

Shows all components of the processor card running a configuration.

The following example shows the show running-config command, which displays all the components of the processor card configuration. For a detailed description of this command, refer to the Cisco IOS Configuration Fundamentals Command Reference.

Switch# show running-config
Building configuration...

Current configuration : 8344 bytes
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log datetime msec
no service password-encryption
!
hostname Switch
!
boot system bootflash:ons15540-i-mz.122-18.SV
logging snmp-authfail
enable password lab
!
no environment-monitor shutdown fan
diag online
ip subnet-zero
no ip domain-lookup
!
!

<Information deleted>
!
threshold-list sonet2
threshold name sonet-sdh section cv degrade index 0
value rate 7
--More--
threshold name sonet-sdh section cv failure index 1
!
threshold-list srik
threshold name sonet-sdh section cv degrade index 0
value rate 7
threshold name sonet-sdh section cv failure index 1
value rate 4
!
threshold-list temp
threshold name cvrd degrade index 0
value rate 5
aps trigger
!
redundancy
associate group spl
associate group `
aps working Wavepatch5/3/0
standby privilege-mode enable
!
!
interface Loopback0
ip address 2.2.2.2 255.255.255.0
!
interface FastEthernet0
ip address 172.25.22.55 255.255.255.254
duplex auto
speed auto
!
interface Filter0/0/0
no ip address
!
interface Oscfilter0/0
no ip address
!
<Information deleted>

log-adjacency-changes
network 20.1.1.0 0.0.0.255 area 0
network 30.1.1.0 0.0.0.255 area 0
!
router bgp 1
no synchronization
bgp log-neighbor-changes
no auto-summary
!
ip classless
ip route 0.0.0.0 0.0.0.0 FastEthernet0
no ip http server
!
!
access-list 100 deny tcp any any eq 3082
snmp-server community public RW
snmp-server enable traps snmp authentication warmstart
snmp-server enable traps tty
snmp-server enable traps threshold min-severity degrade
snmp-server enable traps bgp
snmp-server enable traps config
snmp-server enable traps syslog
snmp-server enable traps entity
snmp-server enable traps fru-ctrl
snmp-server enable traps topology throttle-interval 60
snmp-server enable traps optical monitor min-severity minor
snmp-server enable traps rf
snmp-server enable traps aps
snmp-server enable traps patch
snmp-server enable traps alarms
snmp-server host 172.25.18.22 version 2c WORD
snmp-server host 1.1.1.1 version 2c public alarms
snmp-server host 172.25.18.22 version 2c traps syslog
!
control-plane
!
patch Oscfilter1/0 Wave1
patch Filter1/0/3 Wavepatch4/3/1
!
line con 0
exec-timeout 0 0
line aux 0
line vty 0 4
exec-timeout 0 0
password lab
login
!
!
end

2.4  Recovering a Lost Password

This section describes the procedure to recover a lost login or to enable a password. The procedure differs depending on the platform and the software used, but in all cases, password recovery requires that the system be taken out of operation and powered down.

If you need to perform the following procedure, make certain that there are secondary systems that can temporarily serve the functions of the system undergoing the procedure. If this is not possible, advise all potential users and, if possible, perform the procedure during low-use hours.

Note Make a note of your password, and store it in a secure place.

All of the procedures for recovering lost passwords depend on changing the configuration register of the system. This is done by reconfiguring the system software.

More recent Cisco platforms run from Flash memory or are netbooted from a network server and can ignore the contents of NVRAM (nonvolatile random-access memory) when booting. By ignoring the contents of NVRAM, you can bypass the configuration file (which contains the passwords) and gain complete access to the system. You can then recover the lost password or configure a new one.

Note If your password is encrypted, you cannot recover it. You must configure a new password.

Follow these steps to recover a password:

Step 1 Enter the show version command and the configuration register value in the privileged EXEC mode. The default value is 0x2102.

Step 2 Power up the Cisco ONS 15540 ESPx.

Step 3 Press the Break key sequence or send a break signal, which is usually ^] within 60 seconds of turning the system on. If you do not see the > prompt with a system name, the terminal is not sending the correct break signal. In that case, check the terminal or terminal emulation setup.

Step 4 Enter the confreg command at the > prompt.

Step 5 Answer yes to the Do you wish to change configuration [y/n]? prompt.

Step 6 Answer no to all the questions that appear until you reach the Ignore system config info [y/n] prompt. Answer yes.

Step 7 Answer no to the remaining questions until you reach the Change boot characteristics [y/n]? prompt. Answer yes.

Step 8 Enter 2 at the enter to boot: prompt.

Step 9 Answer no to the Do you wish to change configuration [y/n]? prompt.

Step 10 Enter the reset command at the rommon> prompt.

Step 11 Enter the enable command at the Switch> prompt. You are in enable mode and see the Switch# prompt.

Step 12 Enter the show startup-config command to view your password.

Step 13 Proceed to Step 16 if your password is clear text. Or, continue with Step 14 if your password is encrypted.

Step 14 Enter the configure memory command to copy the NVRAM into memory if your password is encrypted.

Step 15 Enter the copy running-config startup-config command.

Step 16 Enter the configure terminal command.

Step 17 Enter the enable secret password command.

Step 18 Enter the config-register value command, where value is whatever value you entered in Step 1.

Step 19 Enter the exit command to exit configuration mode.

Step 20 Enter the copy running-config startup-config command.

Step 21 Enter the reload command at the prompt.

2.5  Verifying NME Interface Configurations

The administration interfaces provide simple command-line interfaces to all internal management and debugging facilities of the processor card. To manage and debug the processor card, you can use the NME (network management Ethernet) interface, the console port, and the auxiliary port.

For cable connection information for each of these interface ports, refer to the
Cisco ONS 15540 ESPx Hardware Installation Guide. For initial configuration information, refer to the Cisco ONS 15540 ESPx Configuration Guide.

The NME interface has a full duplex, auto-sensing connection with troubleshooting LEDs on the processor card faceplate.

You can configure and monitor the NME connection using the CLI. The NME connection appears in the configuration as fastethernet 0 for the active processor and as fastethernet-sby 0 for the standby processor.

To display the NME fastethernet module configuration and status, use the following commands:

Command

Purpose

show interfaces fastethernet 0

Displays the status of the physical interface.

show controllers fastethernet 0

Displays the interface memory management and error counters on the fastethernet interface.

Follow these steps to verify the NME interface:

Step 1 Use the show interfaces fastethernet 0 slot/subcard/port command to check the NME interface configuration.

Switch# show interfaces fastethernet 0
FastEthernet0 is up, line protocol is up
Hardware is AmdFE, address is 0001.6445.b110 (bia 0001.6445.b110)
Internet address is 172.25.22.55/31
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
671414 packets input, 45594677 bytes
Received 658232 broadcasts (0 IP multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
48004 packets output, 6459399 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Step 2 Check the FastEthernet field to see whether the interface is up. If it is down, check for the following:

•Disconnected or faulty cabling. Check cables.

•Hardware failure. Swap hardware.

If administratively down, the interface has been administratively taken down. Use the no shutdown interface configuration command to reenable the interface.

Step 3 Check the line protocol field to see whether the status is up.

If the interface is down, the line protocol software processes might have determined that the line is unusable or the local or remote interface might be misconfigured. See if the interface can be brought up by following the recommendations in Step 2.

Step 4 Check the duplex mode field. It should match the speed of the interface and be configured as autonegotiation.

Step 5 Check the last input and last output fields. They show the number of hours, minutes, and seconds since the last packet was successfully received or transmitted by the interface.

Step 6 Check the output hang field. It shows the number of hours, minutes, and seconds since the last reset caused by a lengthy transmission.

Step 7 Check the CRC field. The presence of many CRC errors, but not many collisions, indicates excessive noise. If the number of errors is too high, check the cables for damage. If you are using UTP cable, make sure you are using Category 5 cables and not another type, such as Category 3.

Note Errors and the input and output difference should not exceed 0.5 to 2.0 percent of traffic on the interface.

Step 8 Check the collisions fields. These numbers indicate packet collisions and these numbers should be very low. The total number of collisions, with respect to the total number of output packets, should be 0.1 percent or less.

Step 9 Check the late collisions fields. Late collisions should never occur in a properly designed Ethernet network. They usually occur when Ethernet cables are too long or when there are too many repeaters in the network.

Step 10 Check the carrier fields. These numbers indicate a lost carrier detect signal and can be caused by a malfunctioning interface that is not supplying the transmit clock signal or by a cable problem. If the system notices that the carrier detect line of an interface is up, but the line protocol is down, it periodically resets the interface in an effort to restart it. Interface resets can also occur when an interface is looped back or shut down.

Step 11 Check the buffer fields. These numbers indicate the number of received packets discarded because there was no buffer space. Broadcast storms on Ethernet networks, and bursts of noise on serial lines, are often responsible for no-input buffer events.

Step 12 Check the FastEthernet field to see whether the interface is up. If it is down, see if the interface can be brought up by following the recommendations in Step 2. If administratively down, the interface has been administratively taken down. Use the no shutdown interface configuration command to reenable the interface.

If you determine that the connection is configured incorrectly, refer to the
Cisco ONS 15540 ESPx Configuration Guide.

In addition, you can use the show controllers command to troubleshoot the status of the NME interface configuration:

Switch# show controllers fastethernet 0
Interface FastEthernet0
Interface FastEthernet0
Hardware is AMD Unknown
ADDR: 615687C0, FASTSEND: 0, MCI_INDEX: 0
DIST ROUTE ENABLED: 0
Route Cache Flag: 1
LADRF=0x0000 0x0100 0x0000 0x0000
CSR0 =0x00000072, CSR3 =0x00001044, CSR4 =0x0000491D, CSR15 =0x00000180
CSR80 =0x00009900, CSR114=0x00000000, CRDA =0x0658D3D0, CXDA =0x0658D630
BCR9 =0x00000001
CSR5 =0x00000001, CSR7 =0x00000820, CSR100=0x0000F000, CSR125=0x00005C3C
BCR2 =0x00001000, BCR9 =0x00000001, BCR18 =0x00001981, BCR22 =0x0000FF06
BCR25 =0x00000017, BCR26 =0x0000000C, BCR27 =0x00000000, BCR32 =0x00004400
HW filtering information:
Promiscuous Mode Disabled, PHY Addr Enabled, Broadcast Addr Enabled
PHY Addr=0001.6445.B110, Multicast Filter=0x0000 0x0100 0x0000 0x0000
amdp2_instance=0x61563920, registers=0x46000000, ib=0x658D0C0
rx ring entries=64, tx ring entries=128
rxring=0x658D120, rxr shadow=0x61563BC0, rx_head=43, rx_tail=0
txring=0x658D560, txr shadow=0x6156A580, tx_head=13, tx_tail=13, tx_count=0
Software MAC address filter(hash:length/addr/mask/hits):
spurious_idon=0, filtered_pak=0, throttled=0, enabled=0, disabled=0
rx_framing_err=0, rx_overflow_err=0, rx_buffer_err=0, rx_bpe_err=0
rx_soft_overflow_err=0, rx_no_enp=0, rx_discard=0, rx_miss_count=0
tx_one_col_err=0, tx_more_col_err=0, tx_no_enp=0, tx_deferred_err=0
tx_underrun_err=0, tx_late_collision_err=0, tx_loss_carrier_err=0
tx_exc_collision_err=0, tx_buff_err=0, fatal_tx_err=0
hsrp_conf=0, need_af_check=0
tx_limited=0(128)
PHY registers:
Register 0x00: 1000 786D 0000 6B60 01E1 41E1 0005 2801
Register 0x08: 0000 0000 0000 0000 0000 0000 0000 0000
Register 0x10: 001B 0004 186A 001E 2004 0000 0200
Register 0x18: 000D 0000 0000 0000 8300

2.6  Troubleshooting Processor Memory

To troubleshoot the processor memory, use the following commands:

Command

Purpose

show memory

Shows statistics about the Cisco ONS 15540 ESPx memory, including free pool statistics.

show buffers

Displays statistics for the buffer pools on the Cisco ONS 15540 ESPx.

Troubleshooting Cisco ONS 15540 ESPx processor card memory is the same as troubleshooting any Cisco route processor. You can refer to the document Troubleshooting Hardware and Booting Problems.

If the Cisco ONS 15540 ESPx fails, it is sometimes useful to get a full copy of the memory image, called a core dump, to identify the cause of the failure. Core dumps are generally only useful to your technical support representative. For troubleshooting information relating to system management and information about creating core dumps, refer to the Cisco IOS Configuration Fundamentals Command Reference.

2.7  Verifying Hardware and Software Versions

A common problem is an incompatibility between a hardware module and the Cisco IOS software version needed to perform a particular function. This section describes how to troubleshoot that problem.

Display the hardware and software versions to ensure that they are the most recent. Very old hardware and software versions (two or three versions back) can have caveats that have been fixed in more recent versions. Use the following EXEC commands to display version information:

Command

Purpose

show version

Displays the software version information.

show hardware [detail]

Displays detailed hardware information including revision level and version.

To verify hardware and software versions, use the following steps:

Step 1 Use the show version command to display the system software version on the active processor card.

Switch# show version

Cisco IOS Software, ONS-15540 Software (ONS15540-I-M), Version 12.2(18)SV
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2003 by Cisco Systems, Inc.
Compiled Fri 26-Sep-03 15:00 by hql

ROM: System Bootstrap, Version 12.1(10r)EV1, RELEASE SOFTWARE (fc1)

ESPx-BETA uptime is 3 days, 15 hours, 4 minutes
System returned to ROM by reload at 19:19:40 UTC Wed Oct 29 2003
System image file is "bootflash:ons15540-i-mz.122-18.SV"

Cisco ONS15540 (RM7000) processor with 98304K/32768K bytes of memory.
R7000 CPU at 234Mhz, Implementation 39, Rev 2.1, 256KB L2, 2048KB L3 Cache

Last reset from power-on
2 Ethernet interfaces
509K bytes of NVRAM.

20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
24576K bytes of Flash PCMCIA card at slot 1 (Sector size 128K).
16384K bytes of Flash internal SIMM (Sector size 256K).
Standby CPU is up
Standby CPU has 98304K/32768K bytes of memory.

Configuration register is 0x2102

Step 2 Verify the ROM field. It indicates the release of Cisco IOS software loaded and running on the active processor card.

Step 3 Use the show hardware command to display the hardware revision levels for the processor cards.

Switch# show hardware
--------------------------------------------------------------------------------
15540_Chassis_with_external_patch_support named Switch, Date: 11:48:03 UTC Sun
Nov 2 2003
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Back-Plane Information
-------------------------------------------------------------------------------
Orderable Product No. MAC-Address MAC-Size Serial No. Mfg. Date H/W Ver.
--------------------- ----------------- -------- ------------ ---------- ------
15540-CHSB= 00-0c-30-22-28-a0 16 TBC07392048 10/07/2003 3.2

-------------------------------------------------------------------------------
Slot Orderable Product No. Part No. Rev Serial No. Mfg. Date H/W Ver.
---- ------------------------- ---------- --- ------------ ---------- ---------
0/* 15540-LCMB-UNKNOWN 73-7793-02 11 CAB0604MD7C 01/29/2002 1.0
0/0 15540-MDXD-04A0= 74-2833-01 01 ANX0614000V 03/15/2002 1.0
0/1 15540-PSM-01 73-8207-01 PSM#1 01/01/2000 1.1
0/2 0 403011 11/08/3901 0.1
0/3 15540-MDXD-04G0= 74-2839-01 A0 ANX0629000C 07/16/2002 1.0
1/* 15540-LCMB-UNKNOWN 73-7793-01 11 CAB0543L1BY 10/26/2001 3.1
1/0 0 403698 01/23/2002 0.1
1/3 0 403015 11/08/2001 0.1
3/* 15540-LCMB-1400= 800-17218- A0 CNH0647006X 05/01/2003 4.1
3/0 15540-10GE-03B304 800-18905- 02 CAB0553M5WY 03/13/2002 5.0
3/1 15540-LCMB-TBD 800-xxxxx- 02 CAB061508J0 05/07/2003 5.60
6/* N/A 73-5621-03 03 CAB0517HL41 02/16/2001 3.5
7/* N/A 73-5621-02 03 SAK0447002V 02/16/2001 2.1
9/* 15540-TBD 73-7789-01 03 CAB0605MF12 02/09/2002 1.0
9/3 15540-TSP1-03B3= 68-1423-01 B0 CAB0549LR2M 12/19/2001 4.5

-------------------------------------------------------------------------------
Power-Supply Module
-------------------------------------------------------------------------------
Power-Supply A is : OK
Power-Supply B is : Not working

Step 4 Verify that the hardware versions listed in the H/W Ver column for the processor cards in slots 6 and 7 are the same. If the hardware versions are not the same, continue with the "Verifying Hardware and Software Compatibility" section.

Step 5 Use the show hardware detail linecard command to display detailed information about the processor card hardware, including the functional image versions.

Switch# show hardware detail linecard 6

--------------------------------------------------------------------------------
Slot Number : 6/*
Controller Type : 0x1000
On-Board Description : Queens_CPU_PHASE_0
Orderable Product Number: N/A
Board Part Number : 73-5621-02
Board Revision : 03
Serial Number : CAB0505GZHA
Manufacturing Date : 02/16/2001
Hardware Version : 2.5
RMA Number : 0x00
RMA Failure Code : 0x00
Functional Image Version: 1.27
Function-ID : 0

Step 6 Verify that the Hardware Version and Functional Image Version fields for the processor cards in slots 5 and 6 are the same. If they are not the same, refer to "Verifying Hardware and Software Compatibility" section to confirm that they are compatible.

2.8  Verifying Hardware and Software Compatibility

You can verify your hardware and software version compatibility by using the following EXEC command to display processor card compatibility information:

Command

Purpose

show redundancy capability

Displays the software version compatibility information.

To verify hardware and software compatibility of the processor cards and modules, use the following steps:

Step 1 Use the show redundancy capability command to display the system software version compatibility with the various modules installed.

Switch# show redundancy capability

CPU capability support

Active CPU Sby CPU Sby Compat CPU capability description
---------- ---------- ----------- ----------------------------------------
96 MB 96 MB OK CPU DRAM size
32 MB 32 MB OK CPU PMEM size
512 KB 512 KB OK CPU NVRAM size
16 MB 16 MB OK CPU Bootflash size
3.5 2.1 OK CPU hardware major.minor version
1.27 1.27 OK CPU functional major.minor version

Linecard driver major.minor versions, (counts: Active=43, Standby=43)

Active CPU Sby CPU Sby Compat Drv/Ch/F ID Driver description
---------- ---------- ----------- ----------- ------------------------------
1.1 1.1 OK 0x1000/0/0 CPU w/o Switch Fabric
1.1 1.1 OK 0x1001/1/0 Fixed Transponder, w/monitor
1.1 1.1 OK 0x1002/0/0 Fixed Transponder, no monitor
1.1 1.1 OK 0x1003/1/0 Pluggable Transponder, w/monit
1.1 1.1 OK 0x1004/0/0 Pluggable Transponder, no moni
2.1 2.1 OK 0x1005/0/0 Line Card Motherboard
1.1 1.1 OK 0x1006/0/0 Backplane
1.1 1.1 OK 0x1007/0/0 32-ch Mux/Demux
1.1 1.1 OK 0x1008/0/0 Fixed 4-ch Mux/Demux, no OSC
1.1 1.1 OK 0x1009/0/0 Fixed 8-ch Mux/Demux, no OSC
1.1 1.1 OK 0x100A/0/0 Modular 4-ch Mux/Demux, no OSC
1.1 1.1 OK 0x100B/0/0 Modular 8-ch Mux/Demux, no OSC
1.1 1.1 OK 0x100C/0/0 32-ch Array Wave Guide
2.1 2.1 OK 0x100D/0/0 Mux/Demux Motherboard
1.1 1.1 OK 0x100E/0/0 Modular 4-ch Mux/Demux plus OS
1.1 1.1 OK 0x100F/0/0 Modular 8-ch Mux/Demux plus OS
2.1 2.1 OK 0x1010/0/0 Mux-Demux Motherboard, no OSC
2.1 2.1 OK 0x1011/0/0 Line Card Motherboard, no prot
3.1 3.1 OK 0x1012/0/0 Down Link Motherboard
1.1 1.1 OK 0x1013/0/0 OC192 Down Link DaughterCard
2.1 2.1 OK 0x1014/1/0 10G Down Link DaughterCard
1.1 1.1 OK 0x1015/0/0 Modular 16-ch Mux/Demux, no OS
1.1 1.1 OK 0x1016/0/0 Modular 16-ch Mux/Demux plus O
2.1 2.1 OK 0x1017/0/0 Line Card Motherboard, no prot
1.1 1.1 OK 0x1018/1/0 Low bit rate Type-1 transponde
2.1 2.1 OK 0x1019/0/0 CN Tower Line Card Motherboard
2.1 2.1 OK 0x101A/0/0 Mux/Demux Motherboard
1.1 1.1 OK 0x101B/0/0 Modular 4-ch Mux/Demux no OSC
1.1 1.1 OK 0x101C/0/0 Modular 4-ch Mux/Demux plus OS
1.1 1.1 OK 0x101D/0/0 Modular 8-ch Mux/Demux no OSC
1.1 1.1 OK 0x101E/0/0 Modular 8-ch Mux/Demux plus OS
1.1 1.1 OK 0x101F/0/0 32-ch Array Wave Guide
2.1 2.1 OK 0x1020/0/0 Mux/Demux Motherboard, no OSC
1.1 1.1 OK 0x1021/0/0 POM Adapter
2.1 2.1 OK 0x1022/0/0 Down Link Motherboard, no prot
2.1 2.1 OK 0x1023/0/0 Down Link Motherboard, no prot
2.1 2.1 OK 0x1024/0/0 Line Card Motherboard, no prot
1.1 1.1 OK 0x1025/0/0 Modular 16-ch Mux/Demux, no OS
1.1 1.1 OK 0x1026/0/0 Modular 16-ch Mux/Demux plus O
1.1 1.1 OK 0x1027/0/0 PSM Trunk switch proctection m
1.1 1.1 OK 0x1028/1/0 non-plug type1 xpder with cont
1.1 1.1 OK 0x1029/1/0 Low bit rate type-1 xpdr w/con
1.1 1.1 OK 0x1000/0/1 ONS15540 Rommon

Software sync client versions, listed as version range X-Y.
X indicates the oldest peer version it can communicate with.
Y indicates the current sync client version.
Sync client counts: Active=6, Standby=6

Active CPU Sby CPU Sby Compat Cl ID Redundancy Client description
---------- ---------- ----------- ----- ------------------------------------
ver 1-2 ver 1-2 OK 17 CPU Redundancy
ver 1-1 ver 1-1 OK 19 Interface Sync
ver 1-1 ver 1-1 OK 36 MetOpt Password Sync
ver 1-2 ver 1-2 OK 18 Online Diagnostics
ver 1-2 ver 1-2 OK 6 OIR Client
ver 1-1 ver 1-1 OK 27 metopt cm db sync

Backplane IDPROM comparison
Backplane IDPROM field Match Local CPU Peer CPU
--------------------------- ----- -------------------- --------------------
idversion YES 1 1
magic YES 153 153
card_type YES 4102 4102
order_part_num_str YES 15540-CHSB= 15540-CHSB=
description_str YES 15540_Chassis_with_external_patch_support
15540_Chassis_with_exter
nal_patch_support
board_part_num_str YES 73-5655-04 73-5655-04
board_revision_str YES A0 A0
serial_number_str YES TBC07392048 TBC07392048
date_of_manufacture_str YES 10/07/2003 10/07/2003
deviation_numbers_str YES 0 0
manufacturing_use YES 0 0
rma_number_str YES 0x00 0x00
rma_failure_code_str YES 0x00 0x00
oem_str YES Cisco_Systems Cisco_Systems
clei_str YES 0 0
snmp_oid_substr YES 0 0
schematic_num_str YES 92-4113-03 92-4113-03
hardware_major_version YES 3 3
hardware_minor_version YES 2 2
engineering_use_str YES 0 0
crc16 OK 46433 21421
user_track_string YES 0 0
diagst YES ^A ^A
board_specific_revision YES 1 1
board_specific_magic_number YES 153 153
board_specific_length YES 57 57
mac_address_block_size YES 16 16
mac_address_base_str YES 000c302228a0 000c302228a0
cpu_number OK 0 1
optical_backplane_type YES 2 2

Step 2 Check the processor memory sizes and versions in the CPU capability description column. The numbers in the Active CPU and Sby CPU (Standby CPU) columns should match. If not, check the Sby Compat (Standby Compatibility) column. If this column indicates the values are OK, then these values will function as compatible redundant processor cards. If not, swap the processor cards with versions that are compatible.

Step 3 Check the CPU hardware major.minor versions and CPU functional major.minor versions in the CPU capability description column. The numbers in the Active CPU and Sby CPU (Standby CPU) columns should match. If not, check the Sby Compat (Standby Compatibility) columns. If this column indicates the values are OK, then these values will function as compatible redundant processor cards. If not, swap the processor cards with versions that are compatible.

Step 4 Check the information in the Linecard driver section of the display. This section shows the compatibility of the software versions installed on the active and standby processor cards with the various modules installed in the system.

Step 5 Check the Sby Compat (Standby Compatibility) and the Driver description columns. An OK in the Sby Compat column indicates the software version installed on the processor cards supports the drivers on the modules listed.

Step 6 Check the Software sync client version section of the display. The Active CPU, Sby CPU, and Redundancy Client description columns indicate the software versions the two processor cards can use to synchronize their configurations. The version range in the display, shown as X-Y, indicates oldest-current peer client versions. For example, if the version lists 1-2, that indicates version 1 is the oldest version that the current version 2 could synchronize with its configuration.

Step 7 Check the Backplane IDPROM comparison section of the display. Check the Match column. This indicates which elements match, are acceptable, or fail. Some elements do not match but the range is acceptable. For example, the crc16 elements fields never match because the information in the IDPROMs of the two processor cards are different so the checksums never match. But they do appear as OK or compatible.

If any of the drivers are not supported or appear as OK, try updating the images installed on the processor cards. Use the information in the "1.9  Checking Release Notes for Workarounds" section on page 1-12 to upgrade to a more recent version. That should solve a processor card image compatibility problem.

2.9  Troubleshooting Redundant Processor Cards

The Cisco ONS 15540 ESPx supports fault tolerance by allowing a standby processor card to take over if the active processor card fails. This standby, or redundant, processor card runs in hot-standby mode. In hot-standby mode, the standby processor card is partially booted with the Cisco IOS software; however, no configuration is loaded.

At the time of a switchover, the standby processor card takes over as the active processor card and loads the configuration as follows:

•If the running configurations on the active and standby processor card match, the new active processor card uses the running configuration file.

•If the running configurations on the active and standby processor cards do not match, the new active processor card uses the last saved configuration file in its NVRAM (not the NVRAM of the former active processor card).

The former active processor card then becomes the standby processor card.

Note If the standby processor card is unavailable, a major alarm is reported. Use the show facility-alarm status command to display the redundancy alarm status.

For redundant processor cards to function correctly, your Cisco ONS 15540 ESPx processor cards must meet the following requirements:

•Both processor cards must have compatible hardware configurations.

•ROMMON version 12.1(10r)EV.

•Both processor cards must have compatible releases of Cisco IOS software.

A common error you may encounter is the incompatibility of hardware modules and the Cisco IOS software version needed to perform a particular function.

2.9.1  Verifying Hardware and Software Versions of Redundant Processor Cards

To troubleshoot the processor card hardware and software versions for redundancy, use the following commands:

Command

Purpose

show version

Displays the processor card software version information.

show redundancy

Displays the hardware and software configurations of the active and standby processor cards.

To troubleshoot the hardware and software versions on the redundant processor card, use the following steps:

Step 1 Use the show version command to display the system software version on the active processor card as described in the "Verifying Hardware and Software Versions" section.

Step 2 Use the show redundancy summary command to check the configuration and status of the active and standby processor card.

Switch# show redundancy summary

Redundant system information
----------------------------
Available Uptime: 3 days, 15 hours, 10 minutes
sysUpTime (switchover clears): 3 days, 15 hours, 10 minutes
Switchover Count: 0

Inter-CPU Communication State: UP
Last Restart Reason: Normal boot

Last Running Config sync: 1 day, 16 hours, 56 minutes
Running Config sync status: In Sync
Last Startup Config sync: 3 days, 15 hours, 9 minutes
Startup Config sync status: In Sync

This CPU is the Active CPU.
-------------------------------
Slot: 6
Time since CPU Initialized: 3 days, 15 hours, 10 minutes
Image Version: ONS-15540 Software (ONS15540-I-M), Version 12.2(1
8)SV, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Image File: bootflash:ons15540-i-mz.122-18.SV
Software Redundancy State: ACTIVE
Hardware State: ACTIVE
Hardware Severity: 0

Peer CPU is the Standby CPU.
-------------------------------
Slot: 7
Time since CPU Initialized: 3 days, 14 hours, 46 minutes
Image Version: ONS-15540 Software (ONS15540-I-M), Version 12.2(1
8)SV, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Image File (on sby-CPU): bootflash:ons15540-i-mz.122-18.SV
Software Redundancy State: STANDBY HOT
Hardware State: STANDBY
Hardware Severity: 0
Privilege Mode: Enabled

Step 3 Verify the Last Running Config sync and Last Startup Config sync fields. They indicate the last time the running configuration and startup configuration were synchronized between the processor cards.

Step 4 Verify the active, standby, and Slot fields. They indicate in which slot the active processor card is configured.

2.9.2  Verifying Redundant Processor Card Functions

To troubleshoot the processor card function capabilities and redundancy, use the following commands:

Command

Purpose

show redundancy capability

Displays capabilities for the active and standby processor cards.

show redundancy clients

Displays internal redundancy software client information, which can be used to debug redundancy software.

show redundancy counters

Displays internal redundancy software counter information, which can be used to debug redundancy software.

show redundancy events

Displays internal redundancy software event information, which can be used to debug redundancy software.

show redundancy history

Displays the internal redundancy software history log, which can be useful for debugging redundancy software.

show redundancy states

Displays internal redundancy software state information.

show redundancy summary

Displays a summary of internal redundancy software counter information.

Follow these steps to troubleshoot processor card and redundancy capabilities on the system:

Step 1 Use the show redundancy capability command to display capabilities of the active or standby processor cards described in the "Verifying Hardware and Software Versions" section.

Step 2 Use the show redundancy clients command to display a list of internal redundancy clients.

Switch# show redundancy clients

clientID = 0 clientSeq = 0 RF_INTERNAL_MSG
clientID = 6 clientSeq = 180 OIR Client
clientID = 7 clientSeq = 190 APS
clientID = 17 clientSeq = 230 CPU Redundancy
clientID = 18 clientSeq = 280 Online Diagnostics
clientID = 19 clientSeq = 300 Interface Sync
clientID = 27 clientSeq = 330 metopt cm db sync
clientID = 35 clientSeq = 360 History RF Client
clientID = 36 clientSeq = 370 MetOpt Password Sync
clientID = 65000 clientSeq = 65000 RF_LAST_CLIENT

Step 3 Use the show redundancy counters command to display internal redundancy software counters.

Switch# show redundancy counters

Redundancy Facility OMs
comm link up = 0
comm link down down = 0

invalid client tx = 1
null tx by client = 0
tx failures = 0
tx msg length invalid = 0

client not rxing msgs = 0
rx peer msg routing errors = 0
null peer msg rx = 0
errored peer msg rx = 0

buffers tx = 1
tx buffers unavailable = 0
buffers rx = 1
buffer release errors = 0

duplicate client registers = 0
failed to register client = 0
Invalid client syncs = 0

Step 4 Use the show redundancy events command to display internal redundancy software events.

Switch# show redundancy events
Redundancy Facility Events :

RF_PROG_INITIALIZATION (100)
RF_PROG_STANDBY_COLD (101)
RF_PROG_STANDBY_CONFIG (102)
RF_PROG_STANDBY_FILESYS (103)
RF_PROG_STANDBY_BULK (104)
RF_PROG_STANDBY_HOT (105)
RF_PROG_ACTIVE_FAST (200)
RF_PROG_ACTIVE_DRAIN (201)
RF_PROG_ACTIVE_PRECONFIG (202)
RF_PROG_ACTIVE_POSTCONFIG (203)
RF_PROG_ACTIVE (204)
RF_PROG_PLATFORM_SYNC (300)
RF_PROG_EXTRALOAD (301)
RF_PROG_HANDBACK (302)
RF_STATUS_PEER_PRESENCE (400)
RF_STATUS_PEER_COMM (401)
RF_STATUS_SWACT_INHIBIT (402)
RF_STATUS_MAINTENANCE_ENABLE (403)
RF_STATUS_MANUAL_SWACT (404)
RF_STATUS_REDUNDANCY_MODE_CHANGE (405)
RF_STATUS_OPER_REDUNDANCY_MODE_CHANGE (406)
RF_REGISTRATION_STATUS (407)
RF_EVENT_NEGOTIATE (500)
RF_EVENT_START_PROGRESSION (501)
RF_EVENT_STANDBY_PROGRESSION (502)
RF_EVENT_CLIENT_PROGRESSION (503)
RF_EVENT_CONTINUE_PROGRESSION (504)
RF_EVENT_LOCAL_PROG_DONE (505)
RF_EVENT_PEER_PROG_DONE (506)
RF_EVENT_NOTIFICATION_TMO (507)
RF_EVENT_SWACT_INHIBIT_TMO (508)
RF_EVENT_KEEP_ALIVE (509)
RF_EVENT_KEEP_ALIVE_TMO (510)
RF_EVENT_GO_ACTIVE (511)
RF_EVENT_GO_STANDBY (512)
RF_EVENT_GO_ACTIVE_EXTRALOAD (513)
RF_EVENT_GO_ACTIVE_HANDBACK (514)

Step 5 Use the show redundancy history command to display internal redundancy software history.

Switch# show redundancy history

4w5d client added: RF_INTERNAL_MSG(0) seq=0
4w5d client added: RF_LAST_CLIENT(65000) seq=65000
00:00:00 client added: History RF Client(35) seq=360
00:00:16 client added: CPU Redundancy(17) seq=230
00:00:17 client added: Interface Sync(19) seq=300
00:00:17 client added: MetOpt Password Sync(36) seq=370
00:00:17 *my state = INITIALIZATION(2) *peer state = DISABLED(1)
00:00:17 RF_PROG_INITIALIZATION(100) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:17 RF_PROG_INITIALIZATION(100) CPU Redundancy(17) op=0 rc=11
00:00:17 RF_PROG_INITIALIZATION(100) Interface Sync(19) op=0 rc=11
00:00:17 RF_PROG_INITIALIZATION(100) History RF Client(35) op=0 rc=11
00:00:17 RF_PROG_INITIALIZATION(100) MetOpt Password Sync(36) op=0 rc=11
00:00:17 RF_PROG_INITIALIZATION(100) RF_LAST_CLIENT(65000) op=0 rc=11
00:00:17 *my state = NEGOTIATION(3) peer state = DISABLED(1)
00:00:17 RF_EVENT_GO_ACTIVE(511) op=0
00:00:17 *my state = ACTIVE-FAST(9) peer state = DISABLED(1)
00:00:17 RF_STATUS_MAINTENANCE_ENABLE(403) CPU Redundancy(17) op=0
00:00:17 RF_STATUS_MAINTENANCE_ENABLE(403) Interface Sync(19) op=0
00:00:17 RF_STATUS_MAINTENANCE_ENABLE(403) MetOpt Password Sync(36) op=0
00:00:17 RF_PROG_ACTIVE_FAST(200) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:18 client added: APS(7) seq=190
00:00:18 client added: Online Diagnostics(18) seq=280
00:00:18 client added: OIR Client(6) seq=180
00:01:01 client added: metopt cm db sync(27) seq=330
00:01:02 RF_PROG_ACTIVE_FAST(200) CPU Redundancy(17) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_FAST(200) Interface Sync(19) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_FAST(200) metopt cm db sync(27) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_FAST(200) History RF Client(35) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_FAST(200) MetOpt Password Sync(36) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_FAST(200) RF_LAST_CLIENT(65000) op=0 rc=11
00:01:02 *my state = ACTIVE-DRAIN(10) peer state = DISABLED(1)
00:01:02 RF_PROG_ACTIVE_DRAIN(201) RF_INTERNAL_MSG(0) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) OIR Client(6) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) APS(7) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) CPU Redundancy(17) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) Online Diagnostics(18) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) Interface Sync(19) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) metopt cm db sync(27) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) History RF Client(35) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) MetOpt Password Sync(36) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_DRAIN(201) RF_LAST_CLIENT(65000) op=0 rc=11
00:01:02 *my state = ACTIVE_PRECONFIG(11) peer state = DISABLED(1)
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) RF_INTERNAL_MSG(0) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) OIR Client(6) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) APS(7) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) CPU Redundancy(17) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) Online Diagnostics(18) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) Interface Sync(19) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) metopt cm db sync(27) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) History RF Client(35) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) MetOpt Password Sync(36) op=0 rc=11
00:01:02 RF_PROG_ACTIVE_PRECONFIG(202) RF_LAST_CLIENT(65000) op=0 rc=11
00:01:05 Configuration parsing complete
00:01:07 System initialization complete
00:01:07 *my state = ACTIVE_POSTCONFIG(12) peer state = DISABLED(1)
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) RF_INTERNAL_MSG(0) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) OIR Client(6) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) APS(7) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) CPU Redundancy(17) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) Online Diagnostics(18) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) Interface Sync(19) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) metopt cm db sync(27) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) History RF Client(35) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) MetOpt Password Sync(36) op=0 rc=11
00:01:07 RF_PROG_ACTIVE_POSTCONFIG(203) RF_LAST_CLIENT(65000) op=0 rc=11
00:01:07 *my state = ACTIVE(13) peer state = DISABLED(1)
00:01:07 RF_PROG_ACTIVE(204) RF_INTERNAL_MSG(0) op=0 rc=11
00:01:07 RF_PROG_ACTIVE(204) OIR Client(6) op=0 rc=11
00:01:07 RF_PROG_ACTIVE(204) APS(7) op=0 rc=11
00:01:07 RF_PROG_ACTIVE(204) CPU Redundancy(17) op=0 rc=11
00:01:07 RF_PROG_ACTIVE(204) Online Diagnostics(18) op=0 rc=11
00:01:07 RF_PROG_ACTIVE(204) Interface Sync(19) op=0 rc=11
00:01:07 RF_PROG_ACTIVE(204) metopt cm db sync(27) op=0 rc=11
00:01:08 RF_PROG_ACTIVE(204) History RF Client(35) op=0 rc=11
00:01:08 RF_PROG_ACTIVE(204) MetOpt Password Sync(36) op=0 rc=11
00:01:08 RF_PROG_ACTIVE(204) RF_LAST_CLIENT(65000) op=0 rc=11
00:01:08 RF_STATUS_PEER_PRESENCE(400) op=1
00:01:08 RF_STATUS_PEER_PRESENCE(400) OIR Client(6) op=1
00:01:08 RF_STATUS_PEER_PRESENCE(400) APS(7) op=1
00:01:08 RF_STATUS_PEER_PRESENCE(400) CPU Redundancy(17) op=1
00:01:08 RF_STATUS_PEER_PRESENCE(400) Online Diagnostics(18) op=1
00:01:08 RF_STATUS_PEER_PRESENCE(400) Interface Sync(19) op=1
00:01:08 RF_STATUS_PEER_PRESENCE(400) metopt cm db sync(27) op=1
00:01:08 RF_STATUS_PEER_PRESENCE(400) MetOpt Password Sync(36) op=1
23:13:35 RF_STATUS_PEER_PRESENCE(400) op=0
23:13:35 RF_STATUS_PEER_PRESENCE(400) OIR Client(6) op=0
23:13:35 RF_STATUS_PEER_PRESENCE(400) APS(7) op=0
23:13:35 RF_STATUS_PEER_PRESENCE(400) CPU Redundancy(17) op=0
23:13:35 RF_STATUS_PEER_PRESENCE(400) Online Diagnostics(18) op=0
23:13:35 RF_STATUS_PEER_PRESENCE(400) Interface Sync(19) op=0
23:13:35 RF_STATUS_PEER_PRESENCE(400) metopt cm db sync(27) op=0
23:13:35 RF_STATUS_PEER_PRESENCE(400) MetOpt Password Sync(36) op=0
23:13:35 Reloading peer (peer presence lost)
23:13:35 RF_STATUS_PEER_COMM(401) op=0
23:13:35 RF_STATUS_PEER_COMM(401) op=0

Step 6 Use the show redundancy states command to display internal redundancy software state information.

Switch# show redundancy states
my state = 13 -ACTIVE
peer state = 1 -DISABLED
Mode = Simplex
Unit ID = 6

Split Mode = Disabled
Manual Swact = Disabled Reason: Simplex mode
Communications = Down Reason: Simplex mode

client count = 10
client_notification_TMR = 30000 milliseconds
keep_alive TMR = 12000 milliseconds
keep_alive count = 0
keep_alive threshold = 17
RF debug mask = 0x0

Step 7 Use the show redundancy summary command to display a summary of internal redundancy software information.

Switch# show redundancy summary

Redundant system information
----------------------------
Available Uptime: 3 days, 14 hours, 48 minutes
sysUpTime (switchover clears): 3 days, 14 hours, 48 minutes
Switchover Count: 0

Inter-CPU Communication State: DOWN
Last Restart Reason: Normal boot

Last Running Config sync: never
Running Config sync status: Out of Sync
Last Startup Config sync: never
Startup Config sync status: Out of Sync

This CPU is the Active CPU.
-------------------------------
Slot: 6
Time since CPU Initialized: 3 days, 14 hours, 48 minutes
Image Version: ONS-15540 Software (ONS15540-I-M), Version 12.2(1
8)SV, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Image File: bootflash:ons15540-i-mz.122-18.SV
Software Redundancy State: ACTIVE
Hardware State: ACTIVE
Hardware Severity: 0

Peer CPU is the Standby CPU.
-------------------------------
Slot: 7
Time since CPU Initialized: Unknown, peer CPU not responding
Image Version: Unknown, peer CPU not responding
Image File (on sby-CPU): Unknown, peer CPU not responding
Software Redundancy State: DISABLED
Hardware State: NOT PLUGGED IN
Hardware Severity: 0
Privilege Mode: Enabled

Refer to the Cisco ONS 15540 ESPx Configuration Guide for the following:

•Configuring processor card redundancy

•Upgrading the software image on the redundant processor card

•Downloading the system image on the processor cards

2.10  Troubleshooting Processor Cards

This section contains troubleshooting procedures for processor card problems.

2.10.1  Active Processor Card Boot Failure

Symptom    The active processor card fails to boot.
Table 2-1 describes the potential causes of the symptom and the solutions.

Table 2-1 Active Processor Card Boot Failure

Possible Problem

Solution

Auto boot not configured

Manually boot the valid system image, and then use the config reg 0x2102 command to configure auto boot.

Invalid boot configuration

Manually boot the valid system image and check the boot system configuration. Correct the configuration if necessary.

2.10.2  Standby Processor Card Boot Failure

Symptom    The standby processor card fails to boot.
Table 2-2 describes the potential causes of the symptom and the solutions.

Table 2-2 Standby Processor Card Boot Failure

Possible Problem

Solution

Auto boot not configured

Manually boot the valid system image, and then use the config reg 0x2102 command to configure auto boot.

Invalid boot configuration

Manually boot the valid system image and check the boot system configuration. Correct the configuration if necessary.

Peer (active) processor card reset

Issue the show redundancy history, show redundancy state, show redundancy events, show redundancy clients, and show buffers commands and provide the outputs to Cisco technical support.

2.10.3  Unable to Access Processor Card Console

Symptom    The processor card console cannot be accessed.
Table 2-3 describes the potential causes of the symptom and the solutions.

Table 2-3 Unable to Access Switch Module Console

Possible Problem

Solution

Console cable

Verify that the console cable is connected properly, and replace if necessary.

Incorrect serial port setting

Check the serial port configuration, and correct the settings if necessary.

2.10.4  Unable to Access Enable Mode on Active Processor Card

Symptom    The system does not allow access to the enable mode.
Table 2-4 describes the potential causes of the symptom and the solution.

Table 2-4 Unable to Access Enable Mode

Possible Problem

Solution

Password incorrect

Perform the password recovery procedure. See the "Recovering a Lost Password" section.

2.10.5  Unable to Access Enable Mode on Standby Processor Card

Symptom    The system does not allow access to the enable mode on the standby processor card.
Table 2-4 describes the potential causes of the symptom and the solutions.

Table 2-5 Unable to Access Enable Mode on Standby Processor Card

Possible Problem

Solution

Password incorrect

Perform the password recovery procedure. See the "Recovering a Lost Password" section.

Password synchronization

Check the image on the active and standby processor cards, and update to the latest image if necessary. If the images are the same, issue the show tech and the show log commands and provide the outputs to Cisco technical support.

Standby privilege-mode not enabled

Enable the standby privilege-mode under redundancy configuration.

Command	Purpose
show interfaces fastethernet 0	Displays the status of the physical interface.
show controllers fastethernet 0	Displays the interface memory management and error counters on the fastethernet interface.

Command	Purpose
show memory	Shows statistics about the Cisco ONS 15540 ESPx memory, including free pool statistics.
show buffers	Displays statistics for the buffer pools on the Cisco ONS 15540 ESPx.

Command	Purpose
show version	Displays the software version information.
show hardware [detail]	Displays detailed hardware information including revision level and version.

Command	Purpose
show version	Displays the processor card software version information.
show redundancy	Displays the hardware and software configurations of the active and standby processor cards.

Command	Purpose
show redundancy capability	Displays capabilities for the active and standby processor cards.
show redundancy clients	Displays internal redundancy software client information, which can be used to debug redundancy software.
show redundancy counters	Displays internal redundancy software counter information, which can be used to debug redundancy software.
show redundancy events	Displays internal redundancy software event information, which can be used to debug redundancy software.
show redundancy history	Displays the internal redundancy software history log, which can be useful for debugging redundancy software.
show redundancy states	Displays internal redundancy software state information.
show redundancy summary	Displays a summary of internal redundancy software counter information.

Possible Problem	Solution
Auto boot not configured	Manually boot the valid system image, and then use the config reg 0x2102 command to configure auto boot.
Invalid boot configuration	Manually boot the valid system image and check the boot system configuration. Correct the configuration if necessary.

Possible Problem	Solution
Console cable	Verify that the console cable is connected properly, and replace if necessary.
Incorrect serial port setting	Check the serial port configuration, and correct the settings if necessary.

Possible Problem	Solution
Password incorrect	Perform the password recovery procedure. See the "Recovering a Lost Password" section.
Password synchronization	Check the image on the active and standby processor cards, and update to the latest image if necessary. If the images are the same, issue the show tech and the show log commands and provide the outputs to Cisco technical support.
Standby privilege-mode not enabled	Enable the standby privilege-mode under redundancy configuration.