|
Table Of Contents
Troubleshooting CPU Switch Module Problems
2.2 Initial Troubleshooting Checklist
2.3 Verifying CPU Switch Module Configuration
2.4 Recovering a Lost Password
2.5 Verifying NME Interface Configurations
2.6 Troubleshooting CPU Switch Module Memory
2.7 Verifying Hardware and Software Versions
2.8 Verifying Hardware and Software Compatibility
2.9 Troubleshooting Redundant CPU Switch Modules
2.9.1 Verifying Hardware and Software Versions of Redundant CPU Switch Modules
2.9.2 Verifying Redundant CPU Switch Module Functions
2.10 Troubleshooting CPU Switch Module Problems
2.10.1 Active CPU Switch Module Boot Failure
2.10.2 Standby CPU Switch Module Boot Failure
2.10.3 Unable to Access CPU Switch Module Console
2.10.4 Unable to Access Enable Mode on Active CPU Switch Module
2.10.5 Unable to Access Enable Mode on Standby CPU Switch Module
Troubleshooting CPU Switch Module Problems
This chapter describes how to troubleshoot CPU switch module problems. This chapter includes the following sections:
• Overview
• Initial Troubleshooting Checklist
• Verifying CPU Switch Module Configuration
• Verifying NME Interface Configurations
• Troubleshooting CPU Switch Module Memory
• Verifying Hardware and Software Versions
• Verifying Hardware and Software Compatibility
• Troubleshooting Redundant CPU Switch Modules
• Troubleshooting CPU Switch Module Problems
2.1 Overview
The Cisco ONS 15530 supports two CPU switch modules for redundancy, one in active mode and the other in hot-standby mode. CPU switch modules are installed in slot 5 and slot 6. Each CPU switch module has a processor, a switch fabric, a clock, an Ethernet switch for communication between CPU switch modules and with the LRC (line card redundancy controller) on the OADM modules and line cards, and an SRC (switch card redundancy controller). The active CPU switch module controls the system. All LRCs in the system use the system clock and synchronization signals from the active CPU switch module. Interfaces on the CPU switch modules permit access by 10/100 Ethernet, console terminal, or modem connections.
Note For information on slot assignments, CPU switch module LEDs, alarm condition clear and reset button, interrupt clear and reset button, NME LEDs, and cabling, refer to the
Cisco ONS 15530 Hardware Installation Guide. For default configuration of the various modules, refer to the Cisco ONS 15530 Configuration Guide and the Cisco ONS 15530 Command Reference.2.2 Initial Troubleshooting Checklist
Follow this initial checklist before proceeding with the troubleshooting procedures:
•Issue the show running-config command to check the running configuration.
•Ensure the LEDs on the CPU switch modules show the proper state.
•Ensure the Ethernet and Console cables are connected properly.
•Issue the show facility-alarm status command to check for CPU switch module, fan, or power supply alarms.
•Issue the show hardware detail command to verify the CPU switch module functional image.
•Ensure online and power-on diagnostics do not report any alarms or failures for the CPU switch module.
•Ensure the active and standby CPU switch modules are compatible.
•Ensure the active and standby CPU switch module have same version of software installed.
2.3 Verifying CPU Switch Module Configuration
To display the CPU switch module configuration and status, issue the show running-config command.
Command Purposeshow running-config
Shows all components of the CPU switch module running a configuration.
The following example shows the show running-config command, which displays all the components of the CPU switch module configuration. For a detailed description of this command, refer to the Cisco IOS Configuration Fundamentals Command Reference.
Switch# show running-config
Building configuration...
Current configuration : 2971 bytes
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
service internal
!
hostname top
!
boot system bootflash:ons15530-i-mz.sar-f-rep
boot bootldr bootflash:ons15530-i-mz.sar-f-rep
logging snmp-authfail
logging queue-limit 100
logging buffered 10000 debugging
enable password
!
diag online
no diag power-on
ip subnet-zero
ip ftp source-interface FastEthernet0
ip ftp username
ip ftp password
no ip domain-lookup
!
!
!
<information deleted>
control-plane
!
!
redundancy
associate group sp
aps working Wavepatch4/0/0
aps protection Wavepatch4/0/1
aps enable
standby privilege-mode enable
!
!
interface Loopback0
no ip address
!
interface FastEthernet0
ip address 172.25.22.125 255.255.255.0
duplex auto
speed auto
no cdp enable
!
interface Fastethernet-sby0
no ip address
shutdown
duplex auto
speed auto
!
<information deleted>
!
router ospf 100
log-adjacency-changes
redistribute connected subnets
redistribute static subnets
!
ip classless
ip route 0.0.0.0 0.0.0.0 FastEthernet0
ip route 172.25.18.0 255.255.255.0 FastEthernet0
no ip http server
!
!
!
snmp-server engineID local 80000009030000016447A1D1
snmp-server community public RW
snmp-server location san-jose-dev-test
snmp-server contact Edward.Ding : eding@cisco.com
snmp-server enable traps snmp authentication warmstart
snmp-server enable traps tty
snmp-server enable traps bgp
snmp-server enable traps oscp
snmp-server enable traps config
snmp-server enable traps syslog
snmp-server enable traps entity
snmp-server enable traps fru-ctrl
snmp-server enable traps topology throttle-interval 60
snmp-server enable traps rf
snmp-server enable traps aps
snmp-server enable traps patch
snmp-server enable traps alarms
banner motd ^C
*****^C
alias associate-group g400 ag400
!
line con 0
exec-timeout 0 0
line aux 0
line vty 0 4
exec-timeout 0 0
password lab
login
length 0
width 0
!
exception core-file /tftpboot/eding/CORE/h3
exception protocol ftp
exception dump 172.20.46.50
end
2.4 Recovering a Lost Password
This section describes the procedure to recover a lost login or to enable a password. The procedure differs depending on the platform and the software used, but in all cases, password recovery requires that the system be taken out of operation and powered down.
If you need to perform the following procedure, make certain that there are secondary systems that can temporarily serve the functions of the system undergoing the procedure. If this is not possible, advise all potential users and, if possible, perform the procedure during low-use hours.
Note Make a note of your password, and store it in a secure place.
All of the procedures for recovering lost passwords depend on changing the configuration register of the system. This is done by reconfiguring the system software.
More recent Cisco platforms run from Flash memory or are netbooted from a network server and can ignore the contents of NVRAM (nonvolatile random-access memory) when booting. By ignoring the contents of NVRAM, you can bypass the configuration file (which contains the passwords) and gain complete access to the system. You can then recover the lost password or configure a new one.
Note If your password is encrypted, you cannot recover it. You must configure a new password.
Follow these steps to recover a password:
Step 1 Enter the show version command and the configuration register value in the privileged EXEC mode. The default value is 0x2102.
Step 2 Power up the Cisco ONS 15530.
Step 3 Press the Break key sequence or send a break signal, which is usually ^] within 60 seconds of turning the system on. If you do not see the
>
prompt with a system name, the terminal is not sending the correct break signal. In that case, check the terminal or terminal emulation setup.Step 4 Enter the confreg command at the
>
prompt.Step 5 Answer yes to the
Do you wish to change configuration [y/n]?
prompt.Step 6 Answer no to all the questions that appear until you reach the
Ignore system config info [y/n]
prompt. Answer yes.Step 7 Answer no to the remaining questions until you reach the
Change boot characteristics [y/n]?
prompt. Answer yes.Step 8 Enter 2 at the
enter to boot:
prompt.Step 9 Answer no to the
Do you wish to change configuration [y/n]?
prompt.Step 10 Enter the reset command at the
rommon>
prompt.Step 11 Enter the enable command at the
Switch>
prompt. You are in enable mode and see theSwitch#
prompt.Step 12 Enter the show startup-config command to view your password.
Step 13 Proceed to Step 16 if your password is clear text. Or, continue with Step 14 if your password is encrypted.
Step 14 Enter the configure memory command to copy the NVRAM into memory if your password is encrypted.
Step 15 Enter the copy running-config startup-config command.
Step 16 Enter the configure terminal command.
Step 17 Enter the enable secret password command.
Step 18 Enter the config-register value command, where value is whatever value you entered in Step 1.
Step 19 Enter the exit command to exit configuration mode.
Step 20 Enter the copy running-config startup-config command.
Step 21 Enter the reload command at the prompt.
2.5 Verifying NME Interface Configurations
The administration interfaces provide simple command-line interfaces to all internal management and debugging facilities of the CPU switch module. To manage and debug the CPU switch module, you can use the NME (network management Ethernet) interface, the console port, and the auxiliary port.
For cable connection information for each of these interface ports, refer to the
Cisco ONS 15530 Hardware Installation Guide. For initial configuration information, refer to the Cisco ONS 15530 Configuration Guide and the Cisco ONS 15530 Command Reference.The NME interface has a full duplex, auto sensing connection with troubleshooting LEDs on the CPU switch module faceplate.
You can configure and monitor the NME connection using the CLI. The NME connection appears in the configuration as FastEthernet 0 or FastEthernet-sby 0 depending on the slot where the CPU switch module is installed.
To display the NME FastEthernet module configuration and status, use the following commands:
Follow these steps to verify the NME interface:
Step 1 Issue the show interfaces FastEthernet 0 slot/subcard/port command to check the NME interface configuration.
Switch# show interfaces FastEthernet 0
FastEthernet0 is up, line protocol is up
Hardware is Gt96k FE, address is 0009.7c1a.cb50 (bia 0009.7c1a.cb50)
Internet address is 172.25.22.125/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Half-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:06, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3000 bits/sec, 5 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
131803 packets input, 8271274 bytes
Received 131333 broadcasts (0 IP multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
3254 packets output, 200502 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Step 2 Check the FastEthernet field to see whether the interface is up. If it is down, check for the following:
•Disconnected or faulty cabling. Check cables.
•Hardware failure. Swap hardware.
If administratively down, the interface has been administratively taken down. Issue the no shutdown interface configuration command to reenable the interface.
Step 3 Check the line protocol field to see whether the status is up.
If the interface is down, the line protocol software processes might have determined that the line is unusable or the local or remote interface might be misconfigured. See if the interface can be brought up by following the recommendations in Step 2.
Step 4 Check the duplex mode field. It should match the speed of the interface and be configured as auto-negotiation.
Step 5 Check the last input and last output fields. They show the number of hours, minutes, and seconds since the last packet was successfully received or transmitted by the interface.
Step 6 Check the output hang field. It shows the number of hours, minutes, and seconds since the last reset caused by a lengthy transmission.
Step 7 Check the CRC field. The presence of many CRC errors, but not many collisions, indicates excessive noise. If the number of errors is too high, check the cables for damage. If you are using UTP cable, make sure you are using Category 5 cables and not another type, such as Category 3.
Note Errors and the input and output difference should not exceed 0.5 to 2.0 percent of traffic on the interface.
Step 8 Check the collisions fields. These numbers indicate packet collisions and these numbers should be very low. The total number of collisions, with respect to the total number of output packets, should be 0.1 percent or less.
Step 9 Check the late collisions fields. Late collisions should never occur in a properly designed Ethernet network. They usually occur when Ethernet cables are too long or when there are too many repeaters in the network.
Step 10 Check carrier fields. These numbers indicate a lost carrier detect signal and can be caused by a malfunctioning interface that is not supplying the transmit clock signal or by a cable problem. If the system notices that the carrier detect line of an interface is up, but the line protocol is down, it periodically resets the interface in an effort to restart it. Interface resets can also occur when an interface is looped back or shut down.
Step 11 Check the buffer fields. These numbers indicate the number of received packets discarded because there was no buffer space. Broadcast storms on Ethernet networks, and bursts of noise on serial lines, are often responsible for no-input buffer events.
Step 12 Check the FastEthernet field to see whether the interface is up. If it is down, see if the interface can be brought up by following the recommendations in Step 2. If administratively down, the interface has been administratively taken down. Issue the no shutdown interface configuration command to reenable the interface.
If you determine that the connection is configured incorrectly, refer to the
Cisco ONS 15530 Configuration Guide.In addition, you can use the show controllers command to troubleshoot the status of the NME interface configuration:
Switch# show controllers fastethernet 0
Interface FastEthernet0
Hardware is GT96K FE ADDR: 62118CA0, FASTSEND: 0, MCI_INDEX: 0
DIST ROUTE ENABLED: 0
Route Cache Flag: 1
GPIO 2 CONF= 7FFF7FFF
GPIO 2 IO= 3D003D
CIU arbit = 2A8
PHY add register = 0x3E0
PHY data register = 0xF1F003A
Port Configuration Register= 0x80
ENABLE HT8K HMOD0
Port Configuration Extend Register= 0xCD00
TX1:1 RXPRI=DE(00) ~FLCNTL ~FLNKP MFL64KB E
Port Command Register= 0x0
Port Status Register= 0x9
100MB HDPX FCTL EN LNK UP ~PAUSED TX oFF
Serial Parameter Register= 0x218823
Hash table pointer= 0x35A83C0
Source ADDR L= 0x0
Source ADDR H= 0x0
SDMA configuration register= 0x2200
RETX 0 RX BE TX BE FRINT BSIZE 4
SDMA command register= 0x1000080
SRT TXL EN RX
Interrupt MASK= 0x80003DCD
Interrupt Cause= 0x0
Serial 0 mask 3
Serial 0 cause 0
IP DIFFSERV P0L= 0x0 IP DIFFSERV P0H= 0x0
IP DIFFSERV P1L= 0x0 IP DIFFSERV P1H= 0x0
IP VLAN TAG PRI= 0xF0CC
IP VLAN TAG PRI= 0xF0CC
First rxd Q0= 0x35E85A0 Curr rxd Q0= 0x35E85A0
First rxd Q1= 0x35E88A0 Curr rxd Q1= 0x35E88A0
First rxd Q2= 0x35E8D00 Curr rxd Q2= 0x35E8D00
First rxd Q3= 0x35E9160 Curr rxd Q3= 0x35E9160
First txd Q0= 0x35E99D0 First txd Q1= 0x35E9E00
gt96kfe_instance=0x6211AA58, registers=0xB4088800
rx ring entries=64, tx ring entries=128
rxring0=0x35E8440, rxring1=0x35E88A0, rxring2=0x35E8D00, rxring3=0x35E9160
malloc rxring0=0x35E8440, rxring1=0x35E88A0, rxring2=0x35E8D00, rxring3=0x35E91
60
Head rxring0=0xD, rxring1=0x0, rxring2=0x0, rxring3=0x0
Tail rxring0=0x0, rxring1=0x0, rxring2=0x0, rxring3=0x0
Shadow rxring0=0x6211ACE0, rxring1=0x6211AE20, rxring2=0x6211AF60, rxring3=0x62
125CA0
tx_limited=0(128)
txring0=0x35E95C0, txring1=0x35E9E00
Head txring0=0x41, txring1=0x0
Tail txring0=0x41, txring1=0x0
Tail COUNT txring0=0x0, txring1=0x0
PHY registers:
Register 0x00: 1000 782D 0040 6212 01E1 40A1 0003 0000
Register 0x08:
Register 0x10: D000 0301 0000 0000 0000 017F 0100 0000
Register 0x18: 003A F33E 8F00 FF00 002A C000 20A0
MIB counters:
bytes_recvd =11564162
bytes_sent =214232
frames_recvd =156732
frames_sent =3265
total_bytes_recvd =11564162
total_frames_recvd =156735
bcast_frames_recvd =131833
mcast_frames_recvd =22545
crc_errors =0
ovr_sized_frames =0
fragments =3
jabber =0
collision =0
late_collision =0
64bytes_frames =146311
65_127bytes_frames =8619
128_255bytes_frames =1015
256_511bytes_frames =4056
512_1023bytes_frames =0
1023_maxbytes_frames =0
rx_error =0
dropped_frames =0
mcast_frames_tx =0
bcast_frames_tx =2803
sml_frame_recvd =0
Software MAC address filter(hash:length/addr/mask/hits):
0x00: 0 ffff.ffff.ffff 0000.0000.0000 131803
2.6 Troubleshooting CPU Switch Module Memory
To troubleshoot the CPU switch module memory, use the following commands:
Command Purposeshow memory
Shows statistics about the Cisco ONS 15530 memory, including free pool statistics.
show buffers
Displays statistics for the buffer pools on the Cisco ONS 15530.
Troubleshooting Cisco ONS 15530 CPU switch module memory is the same as troubleshooting any Cisco route processor. Refer to the "Troubleshooting Hardware and Booting Problems" chapter of the Cisco IOS Internetwork Troubleshooting Handbook for more information.
If the Cisco ONS 15530 fails, it is sometimes useful to get a full copy of the memory image, called a core dump, to identify the cause of the failure. Core dumps are generally only useful to your technical support representative. For troubleshooting information relating to system management and information about creating core dumps, refer to the Cisco IOS Configuration Fundamentals Command Reference.
2.7 Verifying Hardware and Software Versions
A common problem is an incompatibility between a hardware module and the Cisco IOS software version needed to perform a particular function. This section describes troubleshooting that problem.
Display the hardware and software versions to ensure that they are the most recent. Very old hardware and software versions (two or three versions back) can have caveats that have been fixed in more recent versions. Use the following EXEC commands to display version information:
To verify hardware and software versions, use the following steps:
Step 1 Issue the show version command to display the system software version on the active CPU switch module.
Switch# show version
Cisco Internetwork Operating System Software
IOS (tm) ONS-15530 Software (ONS15530-I-M), Version 12.2(20030711:0
04939) [sar-f-rep 108]
Copyright (c) 1986-2003 by cisco Systems, Inc.
Compiled Mon 14-Jul-03 14:44 by sar
Image text-base: 0x60010BDC, data-base: 0x60A30000
ROM: System Bootstrap, Version 12.1(10r)EV, RELEASE SOFTWARE (fc1)
top uptime is 8 hours, 2 minutes
System returned to ROM by RPR Switchover at 20:01:26 UTC Fri Jun 23 2000
System image file is "bootflash:ons15530-i-mz.sar-f-rep"
cisco ONS15530 (RM7000) processor with 49152K/16384K bytes of memory.
R7000 CPU at 234Mhz, Implementation 39, Rev 2.1, 256KB L2, 2048KB L3 Cache
Last reset from s/w nmi
2 FastEthernet/IEEE 802.3 interface(s)
509K bytes of non-volatile configuration memory.
16384K bytes of Flash internal SIMM (Sector size 256K).
Standby CPU is up
Standby CPU has 49152K/16384K bytes of memory.
error - a Software forced crash, PC 0x602C1830
ONS-15530 Software (ONS15530-I-M), Experimental Version 12.2(20030711:004939) [s
ar-f-rep 108]
Compiled Mon 14-Jul-03 14:44 by sar
Image text-base: 0x60010BDC, data-base: 0x60A30000
Stack trace from system failure:
FP: 0x625BF990, RA: 0x602C1830
FP: 0x625BF9C0, RA: 0x6008DB90
FP: 0x625BF9F8, RA: 0x625BFA88
FP: 0x625BF9F8, RA: 0x602BF5D0
FP: 0x625BFA18, RA: 0x60623578
FP: 0x625BFA60, RA: 0x6062376C
FP: 0x625BFCE8, RA: 0x60620998
FP: 0x625BFD58, RA: 0x6060B7D4
Configuration register is 0x2
Step 2 Verify the ROM field. It indicates the release of Cisco IOS software loaded and running on the active CPU switch module.
Step 3 Issue the show hardware command to display the hardware revision levels for the CPU switch modules.
Switch# show hardware
--------------------------------------------------------------------------------
ONS 15530 Chassis, ETSI Version named Switch, Date: 04:04:48 UTC Sat Jun 24 2000
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Back-Plane Information
--------------------------------------------------------------------------------
Orderable Product No. MAC-Address MAC-Size Serial No. Mfg. Date H/W Ver
--------------------- ----------------- -------- ------------ ---------- -------
15530-CHAS-E= 00-09-7c-1a-cb-50 16 TBC06101005 2002/06/24 3.1
--------------------------------------------------------------------------------
Slot Orderable Product No. Part No. Rev Serial No. Mfg. Date H/W Ver.
---- ------------------------- ---------- --- ------------ ---------- ----------
0/0 PROTO-HAMPTONS-MUX/DEMUX 73-7399-01 2 CAB0603MBAX 01/30/2002 1.0
0/1 PROTO-HAMPTONS-MUX/DEMUX 73-7399-01 2 CAB0603MB91 01/30/2002 1.0
5/* 15530-CPU= 73-6572-06 C0 CNH0651006X 01/21/2003 6.1
6/* 15530-CPU= 73-6572-06 C0 CNH0651006L 01/14/2003 6.1
Power Supply:
Slot Part No. Rev Serial No. RMA No. Hw Vrs Power Consumption
---- ---------------- ---- ----------- ----------- ------- -----------------
Power Supply 0 Not present
Unable to read idprom for 1
Power Supply 1 :
type : 600W AC
status : OK
Step 4 Verify that the hardware versions listed in the H/W Ver column for the CPU switch modules in slots 5 and 6 are the same. If the hardware versions are not the same, continue with the "Verifying Hardware and Software Compatibility" section.
Step 5 Issue the show hardware detail command to display detailed information about the CPU switch module hardware, including the functional image versions.
Switch# show hardware detail
--------------------------------------------------------------------------------
ONS 15530 Chassis, ETSI Version named Switch, Date: 04:05:37 UTC Sat Jun 24 2000
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Back-Plane Information
--------------------------------------------------------------------------------
Slot Number : N/A
Controller Type : 0x1106
On-Board Description : ONS 15530 Chassis, ETSI Version
Orderable Product Number: 15530-CHAS-E=
Board Part Number : 73-6573-03
Board Revision : 02
Serial Number : TBC06101005
Manufacturing Date : 2002/06/24
Hardware Version : 3.1
RMA Number : 0
RMA Failure Code : 0
MAC Address : 00-09-7c-1a-cb-50
MAC Address Block Size : 16
--------------------------------------------------------------------------------
Slot Number : 0/0
Controller Type : 0x1108
On-Board Description : Prototype-Hamptons-MUX/DEMUX
Orderable Product Number: PROTO-HAMPTONS-MUX/DEMUX
Board Part Number : 73-7399-01
Board Revision : 2
Serial Number : CAB0603MBAX
Manufacturing Date : 01/30/2002
Hardware Version : 1.0
RMA Number : 0x00
RMA Failure Code : 0x00
--------------------------------------------------------------------------------
Slot Number : 0/1
Controller Type : 0x1108
On-Board Description : Prototype-Hamptons-MUX/DEMUX
Orderable Product Number: PROTO-HAMPTONS-MUX/DEMUX
Board Part Number : 73-7399-01
Board Revision : 2
Serial Number : CAB0603MB91
Manufacturing Date : 01/30/2002
Hardware Version : 1.0
RMA Number : 0x00
RMA Failure Code : 0x00
--------------------------------------------------------------------------------
Slot Number : 5/*
Controller Type : 0x1100
On-Board Description : ONS 15530 CPU and Switch Board
Orderable Product Number: 15530-CPU=
Board Part Number : 73-6572-06
Board Revision : C0
Serial Number : CNH0651006X
Manufacturing Date : 01/21/2003
Hardware Version : 6.1
RMA Number :
RMA Failure Code :
Functional Image Version: 1.43
Function-ID : 0
--------------------------------------------------------------------------------
Slot Number : 6/*
Controller Type : 0x1100
On-Board Description : ONS 15530 CPU and Switch Board
Orderable Product Number: 15530-CPU=
Board Part Number : 73-6572-06
Board Revision : C0
Serial Number : CNH0651006L
Manufacturing Date : 01/14/2003
Hardware Version : 6.1
RMA Number :
RMA Failure Code :
Functional Image Version: 1.43
Function-ID : 0
Power Supply:
Slot Part No. Rev Serial No. RMA No. Hw Vrs Power Consumption
---- ---------------- ---- ----------- ----------- ------- -----------------
Power Supply 0 Not present
Unable to read idprom for 1
Power Supply 1 :
type : 600W AC
status : OK
Step 6 Verify that the Hardware Version and Functional Image Version fields for the CPU switch modules in slots 5 and 6 are the same. If they are not the same, continue with the following process to confirm that they are compatible.
Step 7 Use the show functional-image command to display detailed information about the functional images for the route processors, switch processors, and Fast Ethernet interface for the Cisco ONS 15530. The following example shows how to display the functional image for the route processor in slot 4:
Switch# show functional-image slot X
Step 8 Verify the FunctionalVersion and #HardwareRequired fields to determine the FPGA version and the hardware version required for the FPGA. Compare this with the hardware version using the show hardware command output. If the FPGA version does not support the hardware version, download a new FPGA image, upgrade the hardware, or both.
2.8 Verifying Hardware and Software Compatibility
You can verify your hardware and software version compatibility by using the following EXEC command to display CPU switch module compatibility information:
Command Purposeshow redundancy capability
Displays the software version compatibility information.
show functional-image slot slot
Displays functional image information.
To verify hardware and software compatibility of the CPU switch modules and modules, use the following steps:
Step 1 Issue the show redundancy capability command to display the system software version compatibility with the various modules installed.
Switch# show redundancy capability
CPU capability support
Active CPU Sby CPU Sby Compat CPU capability description
---------- ---------- ----------- ----------------------------------------
48 MB 48 MB OK CPU DRAM size
16 MB 16 MB OK CPU PMEM size
512 KB 512 KB OK CPU NVRAM size
16 MB 16 MB OK CPU Bootflash size
6.1 6.1 OK CPU hardware major.minor version
1.43 1.43 OK CPU functional major.minor version
Linecard driver major.minor versions, (counts: Active=13, Standby=13)
Active CPU Sby CPU Sby Compat Drv/Ch/F ID Driver description
---------- ---------- ----------- ----------- ------------------------------
1.3 1.3 OK 0x1100/0/0 CPU with Switch Fabric
2.3 2.3 OK 0x1101/0/0 10 Port ESCON line card
2.1 2.1 OK 0x110A/0/0 8 Port GE-FC line card
3.1 3.1 OK 0x1105/0/0 2.5G Transparent line card
1.9 1.9 OK 0x1105/1/0 2.5G Transparent line card
3.1 3.1 OK 0x1109/0/0 2.5G Transparent line card
1.9 1.9 OK 0x1109/1/0 2.5G Transparent line card
Active CPU Sby CPU Sby Compat Drv/Ch/F ID Driver description
---------- ---------- ----------- ----------- ------------------------------
1.3 1.3 OK 0x1103/0/0 OSC line card
0.1 0.1 OK 0x1107/1/0 OSC daughter card
2.1 2.1 OK 0x1102/0/0 10G trunk card
1.0 1.0 OK 0x110B/0/0 2.5G trunk card
2.1 2.1 OK 0x1110/0/0 PSM wdm splitter
1.1 1.1 OK 0x1100/0/1 ONS15530 Rommon
Software sync client versions, listed as version range X-Y.
X indicates the oldest peer version it can communicate with.
Y indicates the current sync client version.
Sync client counts: Active=6, Standby=6
Active CPU Sby CPU Sby Compat Cl ID Redundancy Client description
---------- ---------- ----------- ----- ------------------------------------
ver 1-2 ver 1-2 OK 17 CPU Redundancy
ver 1-1 ver 1-1 OK 19 Interface Sync
ver 1-1 ver 1-1 OK 36 MetOpt Password Sync
ver 1-2 ver 1-2 OK 18 Online Diagnostics
ver 1-2 ver 1-2 OK 6 OIR Client
ver 1-1 ver 1-1 OK 27 metopt cm db sync
Backplane IDPROM comparison
Backplane IDPROM field Match Local CPU Peer CPU
--------------------------- ----- -------------------- --------------------
idversion YES 1 1
magic YES 153 153
card_type YES 4358 4358
order_part_num_str YES 15530-CHAS-E= 15530-CHAS-E=
description_str YES ONS 15530 Chassis, ETSI Version
ONS 15530 Chassis, ETSI
Version
board_part_num_str YES 73-6573-03 73-6573-03
board_revision_str YES 02 02
serial_number_str YES TBC06101005 TBC06101005
date_of_manufacture_str YES 2002/06/24 2002/06/24
deviation_numbers_str YES 0 0
manufacturing_use YES 0 0
rma_number_str YES 0 0
rma_failure_code_str YES 0 0
oem_str YES Cisco_Systems Cisco_Systems
clei_str YES
snmp_oid_substr YES 3.326 3.326
schematic_num_str YES 92-4568-03 92-4568-03
hardware_major_version YES 3 3
Backplane IDPROM field Match Local CPU Peer CPU
--------------------------- ----- -------------------- --------------------
hardware_minor_version YES 1 1
engineering_use_str YES
crc16 OK 26352 9285
user_track_string YES
diagst YES ^A ^A
board_specific_revision YES 1 1
board_specific_magic_number YES 153 153
board_specific_length YES 56 56
mac_address_block_size YES 16 16
mac_address_base_str YES 00097c1acb50 00097c1acb50
cpu_number OK 0 1
optical_backplane_type YES 255 255
Step 2 Check the CPU memory sizes and versions in the CPU Capability Description column. The numbers in the Active CPU and Sby CPU (Standby CPU) columns should match. If not, check the Sby Compat (Standby Compatibility) column. If this column indicates the values are OK, then these values will function as compatible redundant CPU switch modules. If not, swap the CPU switch modules with versions that are compatible.
Step 3 Check the CPU hardware major.minor versions and CPU functional major.minor versions in the CPU Capability Description column. The numbers in the Active CPU and Sby CPU (Standby CPU) columns should match. If not, check the Sby Compat (Standby Compatibility) columns. If this column indicates the values are OK, then these values will function as compatible redundant CPU switch modules. If not, swap the CPU switch modules with versions that are compatible.
Step 4 Check the information in the Linecard driver section of the display. This section shows the compatibility of the software versions installed on the active and standby CPU switch modules with the various modules installed in the system.
Step 5 Check the Sby Compat (Standby Compatibility) and the Driver description columns. An OK in the Sby Compat column indicates the software version installed on the CPU switch modules supports the drivers on the modules listed.
Step 6 Check the Software sync client version section of the display. The Active CPU, Sby CPU and Redundancy Client description columns indicate the software versions the two CPU switch modules can use to synchronize their configurations. The version range in the display, shown as X-Y, indicates oldest-current peer client versions. For example, if the version lists 1-2, that indicates version 1 is the oldest version that the current version 2 could synchronize with its configuration.
Step 7 Check the Backplane IDPROM comparison section of the display. Check the Match column. This indicates which elements match, are acceptable, or fail. Some elements do not match but the range is acceptable. For example, the crc16 elements fields never match because the information in the IDPROMs of the two CPU switch modules are different so the checksums never match. But they do appear as OK or compatible.
If any of the drivers are not supported or appear as OK, try updating the images installed on the CPU switch modules. Use the information in the "1.10 Checking Release Notes for Workarounds" section on page 1-16 to upgrade to a more recent version. That should solve a CPU switch module image compatibility problem.
2.9 Troubleshooting Redundant CPU Switch Modules
The Cisco ONS 15530 supports fault tolerance by allowing a standby CPU switch module to take over if the active CPU switch module fails. This standby, or redundant, CPU switch module runs in hot-standby mode. In hot-standby mode, the standby CPU switch module is partially booted with the Cisco IOS software; however, no configuration is loaded.
At the time of a switchover, the standby CPU switch module takes over as the active CPU switch module and loads the configuration as follows:
•If the running configurations on the active and standby CPU switch module match, the new active CPU switch module uses the running configuration file.
•If the running configurations on the active and standby CPU switch modules do not match, the new active CPU switch module uses the last saved configuration file in its NVRAM (not the NVRAM of the former active CPU switch module).
The former active CPU switch module then becomes the standby CPU switch module.
Note If the standby CPU switch module is unavailable, a major alarm is reported. Issue the show facility-alarm status command to display the redundancy alarm status.
For redundant CPU switch modules to function correctly, your Cisco ONS 15530 CPU switch modules must meet the following requirements:
•Both CPU switch modules must have compatible hardware configurations.
•ROMMON version 12.1(10r)EV.
•Both CPU switch modules must have compatible releases of Cisco IOS software.
A common error you may encounter is the incompatibility of hardware modules and the Cisco IOS software version needed to perform a particular function.
2.9.1 Verifying Hardware and Software Versions of Redundant CPU Switch Modules
To troubleshoot the CPU switch module hardware and software versions for redundancy, use the following commands:
To confirm that your system CPU switch modules meet the redundancy requirements, complete the following steps:
Step 1 Use the show version command to confirm the system hardware and software status of the active CPU switch module.
Switch# show version
Cisco Internetwork Operating System Software
IOS (tm) ONS-15540 Software (manopt-M0-M), 12.1(X:X)
Copyright (c) 1986-2001 by cisco Systems, Inc.
Compiled Fri 23-Feb-01 15:23 by ffrazer
Image text-base:0x60010950, data-base:0x604E8000
ROM:System Bootstrap, Version 12.1(X:X)
BOOTFLASH:ONS-15540 Software (manopt-M0-M), 12.1(X:X)
Switch uptime is 30 minutes
System returned to ROM by power-on
System image file is "tftp://test/eng/manopt-m0-mz.010223.6"
cisco (QUEENS-CPU) processor with 98304K/32768K bytes of memory.
R7000 CPU at 234Mhz, Implementation 39, Rev 2.1, 256KB L2, 2048KB L3 Cache
Last reset from power-on
2 Ethernet/IEEE 802.3 interface(s)
509K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
16384K bytes of Flash internal SIMM (Sector size 64K).
Configuration register is 0x102
Step 2 Verify the ROM field. It indicates the release of Cisco IOS software loaded and running on the active CPU switch module.
Step 3 Use the show hardware detail command to compare the hardware versions of the active and standby CPU switch modules.
Switch# show hardware detail
--------------------------------------------------------------------------------
named Switch, Date: 04:36:29 UTC Fri Apr 20 2001
--------------------------------------------------------------------------------
.
{Information Deleted]
.
--------------------------------------------------------------------------------
Slot Number : 6
Controller Type : Queens CPU
On-Board Description : Queens_CPU_PHASE_0
Orderable Product Number: N/A
Board Part Number : 73-5621-02
Board Revision : 03
Serial Number : CAB0505GZHD
Manufacturing Date : 02/16/2001
Hardware Version : 2.1
RMA Number : 0x00
RMA Failure Code : 0x00
Functional Image Version: 1.8
--------------------------------------------------------------------------------
Slot Number : 7
Controller Type : Queens CPU
On-Board Description : Queens_CPU_PHASE_0
Orderable Product Number: N/A
Board Part Number : 73-5621-02
Board Revision : 03
Serial Number : CAB0505GZHV
Manufacturing Date : 02/16/2001
Hardware Version : 2.1
RMA Number : 0x00
RMA Failure Code : 0x00
Functional Image Version: 1.11
--------------------------------------------------------------------------------
Back-Plane EEPROM
--------------------------------------------------------------------------------
Slot Number : N/A
Controller Type : N/A
On-Board Description :
Orderable Product Number:
Board Part Number :
Board Revision :
Serial Number :
Manufacturing Date : 01/01/2000
Hardware Version : 0.0
RMA Number : 0x00
RMA Failure Code : 0x00
Optical Back-Plane Type : Unknown Optical Backplane
MAC Address : 00-ab-00-00-00-
MAC Address Block Size : 1
--------------------------------------------------------------------------------
Power-Supply Module
--------------------------------------------------------------------------------
Primary Power-Supply is : Not working
Backup Power-Supply is : Not working
Step 4 In the slots labeled 6 and 7, compare the Image version fields. These numbers must all match or be compatible, otherwise redundancy will not function correctly on your Cisco ONS 15530. For additional information, see the "Verifying Hardware and Software Versions" section.
To troubleshoot the hardware and software versions on redundant CPU switch module, use the following steps:
Step 1 Issue the show version command to display the system software version on the active CPU switch module as described in the "Verifying Hardware and Software Versions" section.
Step 2 Issue the show redundancy summary command to check the configuration and status of the active and standby CPU switch module.
Switch# show redundancy summary
Redundant system information
----------------------------
Available Uptime: 12 hours, 50 minutes
sysUpTime (switchover clears): 7 hours, 52 minutes
Switchover Count: 5
Inter-CPU Communication State: UP
Last Restart Reason: Switch over
Reported Switchover Reason: Active unit failed (error - a Software forced cra
sh, PC 0x602C1830)
Software state at switchover: STANDBY HOT
Last Running Config sync: 7 hours, 52 minutes
Running Config sync status: In Sync
Last Startup Config sync: 7 hours, 52 minutes
Startup Config sync status: In Sync
This CPU is the Active CPU.
-------------------------------
Slot: 5
Time since CPU Initialized: 8 hours, 7 minutes
Image Version: ONS-15530 Software (ONS15530-I-M), Experimental V
ersion 12.2(20030711:004939) [sar-f-rep 108]
Image File: bootflash:ons15530-i-mz.sar-f-rep
Software Redundancy State: ACTIVE
Hardware State: ACTIVE
Hardware Severity: 0
Peer CPU is the Standby CPU.
-------------------------------
Slot: 6
Time since CPU Initialized: 7 hours, 52 minutes
Image Version: ONS-15530 Software (ONS15530-I-M), Version 12.2(20030711:004939) [sar-f-rep 108]
Image File (on sby-CPU): bootflash:ons15530-i-mz.sar-f-rep
Software Redundancy State: STANDBY HOT
Hardware State: STANDBY
Hardware Severity: 0
Privilege Mode: Enabled
Step 3 Verify the Last Running Config sync and Last Startup Config sync fields. They indicate the last time the running configuration and startup configuration were synchronized between the CPU switch modules.
Step 4 Verify the active, standby, and Slot fields. They indicate in which slot the active CPU switch module is configured.
2.9.2 Verifying Redundant CPU Switch Module Functions
To troubleshoot the CPU switch module function capabilities and redundancy, use the following commands:
Follow these steps to troubleshoot CPU switch module and redundancy capabilities on the system:
Step 1 Issue the show redundancy capability command to display capabilities of the active or standby CPU switch modules described in the "Verifying Hardware and Software Versions" section.
Step 2 Check the CPU memory sizes and versions in the column, CPU capability description. The numbers in the columns Active CPU and Sby CPU (Standby CPU) should match. If not, check the column, Sby Compat (Standby Compatibility). If this column indicates the values are OK then these values will function as compatible redundant CPU switch modules. If not, swap the CPU switch modules with versions that are compatible.
Step 3 Check the CPU hardware and functional major.minor versions in the column, CPU capability description. The numbers in the columns Active CPU and Sby CPU (Standby CPU) should match. If not, check the column, Sby Compat (Standby Compatibility). If this column indicates the values are OK then these values will function as compatible redundant CPU switch modules. If not, swap the CPU switch modules with versions that are compatible.
Step 4 Check the information in the column Driver description. This column lists the hardware drivers on the system components that are supported by the CPU switch module version for both the Active and Sby (Standby) CPU switch modules. OK indicates both versions of CPU switch modules support these drivers.
Step 5 Check the Software sync client version section of the display. The Active and Sby CPU columns Redundancy Client description columns indicate the software versions the two CPU switch modules can use to synchronize their configurations. The version range in the display, shown as X-Y, indicates oldest-current peer client versions. For example, if the version lists 1-2, that indicates version 1 is the oldest version that the current version 2 could synchronize with its configuration.
Step 1 Check the IDPROM comparison section of the display. Check the Match column. This indicates which elements match, are acceptable, or fail. Some elements do not match but the range is acceptable. For example, the crc16 elements fields never match because the information in the IDPROMs of the two CPU switch modules are different so the checksums never match. But they do appear as OK or compatible.
Step 2 Issue the show redundancy clients command to display a list of internal redundancy clients.
Switch# show redundancy clients
clientID = 0 clientSeq = 0 RF_INTERNAL_MSG
clientID = 6 clientSeq = 180 OIR Client
clientID = 7 clientSeq = 190 APS
clientID = 17 clientSeq = 230 CPU Redundancy
clientID = 18 clientSeq = 280 Online Diagnostics
clientID = 19 clientSeq = 300 Interface Sync
clientID = 27 clientSeq = 330 metopt cm db sync
clientID = 35 clientSeq = 360 History RF Client
clientID = 36 clientSeq = 370 MetOpt Password Sync
clientID = 65000 clientSeq = 65000 RF_LAST_CLIENT
Step 3 Issue the show redundancy counters command to display internal redundancy software counters.
Switch# show redundancy counters
Redundancy Facility OMs
comm link up = 2
comm link down down = 1
invalid client tx = 1
null tx by client = 0
tx failures = 1
tx msg length invalid = 0
client not rxing msgs = 0
rx peer msg routing errors = 0
null peer msg rx = 0
errored peer msg rx = 0
buffers tx = 2668
tx buffers unavailable = 0
buffers rx = 10858
buffer release errors = 0
duplicate client registers = 0
failed to register client = 0
Invalid client syncs = 0
Step 4 Issue the show redundancy history command to display internal redundancy software history.
Switch# show redundancy history
4w5d client added: RF_INTERNAL_MSG(0) seq=0
4w5d client added: RF_LAST_CLIENT(65000) seq=65000
00:00:00 client added: History RF Client(35) seq=360
00:00:01 client added: CPU Redundancy(17) seq=230
00:00:02 client added: Interface Sync(19) seq=300
00:00:02 client added: MetOpt Password Sync(36) seq=370
00:00:02 *my state = INITIALIZATION(2) *peer state = DISABLED(1)
00:00:02 RF_PROG_INITIALIZATION(100) RF_INTERNAL_MSG(0) op=0 rc=11
00:00:02 RF_PROG_INITIALIZATION(100) CPU Redundancy(17) op=0 rc=11
00:00:02 RF_PROG_INITIALIZATION(100) Interface Sync(19) op=0 rc=11
00:00:02 RF_PROG_INITIALIZATION(100) History RF Client(35) op=0 rc=11
00:00:02 RF_PROG_INITIALIZATION(100) MetOpt Password Sync(36) op=0 rc=11
00:00:02 RF_PROG_INITIALIZATION(100) RF_LAST_CLIENT(65000) op=0 rc=11
00:00:02 *my state = NEGOTIATION(3) peer state = DISABLED(1)
00:00:02 RF_STATUS_PEER_PRESENCE(400) op=1
00:00:02 RF_STATUS_PEER_PRESENCE(400) CPU Redundancy(17) op=1
00:00:02 RF_STATUS_PEER_PRESENCE(400) Interface Sync(19) op=1
00:00:02 RF_STATUS_PEER_PRESENCE(400) MetOpt Password Sync(36) op=1
00:00:03 RF_STATUS_PEER_COMM(401) op=1
00:00:03 RF_STATUS_PEER_COMM(401) CPU Redundancy(17) op=1
00:00:03 RF_STATUS_PEER_COMM(401) Interface Sync(19) op=1
00:00:03 RF_STATUS_PEER_COMM(401) MetOpt Password Sync(36) op=1
00:15:12 RF_EVENT_PEER_PROG_DONE(506) RF_LAST_CLIENT(65000) op=105
00:15:16 *my state = ACTIVE(13) *peer state = STANDBY HOT(8)
Information deleted-------------------------------------------------------
Step 5 Issue the show redundancy running-config-file command to display running configuration on the standby CPU switch module.
sby-Switch# show redundancy running-config-file
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
service internal
!
hostname top
!
boot system bootflash:ons15530-i-mz.sar-f-rep
boot bootldr bootflash:ons15530-i-mz.sar-f-rep
logging snmp-authfail
logging queue-limit 100
logging buffered 10000 debugging
enable password lab
!
diag online
no diag power-on
ip subnet-zero
ip ftp source-interface FastEthernet0
ip ftp username rhino
ip ftp password godzilla
no ip domain-lookup
!
!
!
!
!
Information deleted-----------------------------------------------------
end
^@^@
Step 6 Issue the show redundancy states command to display internal redundancy software state information.
Switch# show redundancy states
my state = 13 -ACTIVE
peer state = 8 -STANDBY HOT
Mode = Duplex
Unit ID = 5
Split Mode = Disabled
Manual Swact = Enabled
Communications = Up
client count = 10
client_notification_TMR = 30000 milliseconds
keep_alive TMR = 12000 milliseconds
keep_alive count = 0
keep_alive threshold = 17
RF debug mask = 0x0
Refer to the Cisco ONS 15530 Configuration Guide and the Cisco ONS 15530 Command Reference for the following:
•Configuring CPU switch module redundancy
•Upgrading the software image on the redundant CPU switch module
•Downloading the system image on the CPU switch modules
2.10 Troubleshooting CPU Switch Module Problems
This section includes CPU switch module troubleshooting procedures.
2.10.1 Active CPU Switch Module Boot Failure
Symptom The active CPU switch module fails to boot.
Table 2-1 describes the potential causes of the symptom and the solutions.
2.10.2 Standby CPU Switch Module Boot Failure
Symptom The standby CPU switch module fails to boot.
Table 2-2 describes the potential causes of the symptom and the solutions.
2.10.3 Unable to Access CPU Switch Module Console
Symptom The CPU switch module console cannot be accessed.
Table 2-3 describes the potential causes of the symptom and the solutions.
2.10.4 Unable to Access Enable Mode on Active CPU Switch Module
Symptom The system does not allow access to the enable mode.
Table 2-4 describes the potential causes of the symptom and the solutions.
Table 2-4 Unable to Access Enable Mode
Possible Problem SolutionPassword incorrect.
Perform the password recovery procedure. See the "Recovering a Lost Password" section.
2.10.5 Unable to Access Enable Mode on Standby CPU Switch Module
Symptom The system does not allow access to the enable mode on the standby CPU switch module.
Table 2-4 describes the potential causes of the symptom and the solutions.
Table 2-5 Unable to Access Enable Mode
Possible Problem SolutionPassword incorrect.
Perform the password recovery procedure. See the "Recovering a Lost Password" section.
Password synchronization.
Check the image on the active and standby CPU switch modules. Update to the latest image if necessary. If the images are the same, issue the show tech and the show log cammands and provide the outputs to Cisco technical support.
Posted: Mon Apr 30 12:21:39 PDT 2007
All contents are Copyright © 1992--2007 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.