Troubleshooting APS Problems

This chapter describes how to troubleshoot APS (Automatic Protection Switching) problems. This chapter includes the following sections:

• Overview

• Initial Troubleshooting Checklist

• Troubleshooting Specific APS Problems

15.1 Overview

APS provides protection against signal transmission failure. The Cisco ONS 15530 supports the following APS features:

•1+1 path protection

•Splitter protection

•Line card protection

–Client based

–Y-cable based

–Switch fabric based

•Trunk fiber protection

•Redundant switch fabric protection

•Bidirectional and unidirectional path switching

For more information on APS support on the Cisco ONS 15530, refer to the Cisco ONS 15530 Configuration Guide.

15.2 Initial Troubleshooting Checklist

Follow this initial checklist before proceeding with the troubleshooting procedures:

•Issue show interfaces commands to ensure that the interfaces along the signal paths are administratively up and that there are no errors on the interfaces.

•Issue a show connect command to verify the status of the cross connections.

•Issue a show aps detail command on both nodes to verify the following:

–The working and protection interfaces are correct.

–The aps state field shows "enabled (associated)."

–The msg-channel field shows "Up" on the desired message channel.

–The direction field shows the same expected values (either "uni" or "bi") on both nodes.

•Check that the LEDs on the cards show the proper state.

•Issue a show facility-alarm status command to display the alarms on the interfaces.

•If ITU cards are present, check that the ITU cards are patched to the correct OADM ports. Issue a show patch detail command to verify that there are no frequency mismatches.

•Ensure that all optical connectors are clean. Refer to the Cisco ONS 15530 Cleaning Procedures for Fiber Optic Connections document.

15.3 Troubleshooting Specific APS Problems

This section contains troubleshooting information for specific APS problems.

15.3.1 APS Group State Enabled But Not Associated

Symptom The show aps group command or show aps detail command output show an APS group state is enabled but the group is not associated.

Table 15-1 describes the potential causes of the symptoms and the solutions.

Table 15-1 APS Group State Enabled But Not Associated
Possible Problem	Solution
Either the working or protection channel is not present.	Make sure that all the cards are properly seated and that the LEDs are showing the proper state.
For switch fabric based line card protection, the cross connections through the switch fabric are not configured correctly.	1. Issue a show connect command to verify that the working and protection cross connections are correctly configured. 2. Issue the connect command to correct any problems.

15.3.2 Bidirectional APS Configured But Remote Node Direction, Architecture, and Receive k1/k2 Are Unknown

Symptom The show aps group command or the show aps detail command output shows an APS group state is configured for bidirectional switching but the remote node direction, remote node architecture, and receive k1/k2 are unknown.

Table 15-2 describes the potential causes of the symptoms and the solutions.

Table 15-2 Bidirectional APS Configured But Remote Node Direction, Architecture, and Receive k1/k2 Are Unknown
Possible Problem	Solution
The configured message channel is not up (if the message channel is not IP).	1. Issue a show interfaces wave command or a show interfaces ethernetdcc command to check the status of the message channel interface. 2. If both the interface and the line protocol are down check the trunk fiber and the local and remote patched fibers to ensure that light is received on the message channel interface. 3. If the interface is up but the protocol is down, issue the debug oscp hello-packet command to determine whether OSCP Hello packets are received from and transmitted to the far end.
There is a problem with OSCP.	See Chapter 16, "Troubleshooting OSCP Problems."
The client signal has errors.	Issue the show interfaces command to check the error counters on the active interface. If they are increasing, the line could be bad.

15.3.3 Message Channel Interface Up But APS Msg-Channel Status Down

Symptom The configured message channel interface is up but the msg-channel status in the show aps group or show aps detail command output is down.

Table 15-3 describes the potential causes of the symptoms and the solutions.

Note Check both the local and remote systems for message channel problems.

Table 15-3 Message Channel Interface Up But APS msg-channel Status Down
Possible Problem	Solution
The line cards are not correctly patched to the OADM modules.	Check the patch connections on the shelf. Ensure that ITU trunk cards are connected to the correct filter ports on the OADM module.
The OSC modules are not correctly patched to the OADM modules.	Check that the OSC module is correctly patched to the OADM module.
The laser frequency on the 2.5-Gbps ITU trunk card is not configured correctly.	Issue a show interfaces command to verify that the laser frequency is correctly configured on the transponder line cards and the 2.5-Gbps ITU trunk cards. If not, issue a laser frequency command to configure the correct laser frequency.
The patches between the line cards or the OSC modules and the OADM modules are not configured in the CLI.	Issue a show patch command to verify the patch connections are correctly configured. If not, issue the patch command to correct the configuration.
The unused wavepatch on a splitter line card in a line card protected configuration is not disabled.	Issue the shutdown command to disable the unused wavepatch interfaces.
If far-end group names are used in the APS message channel configuration, the names are not configured correctly.	1. Issue the show aps group command or the show aps detail command to verify the far-end group name configuration. 2. Issue the aps message-channel command to correct the far-end group name configuration.
The message channel is IP and the NME¹ connection is down.	Issue the show interfaces fastethernet 0 command to verify the status of the NME. If the line or the protocol is down, see Chapter 2, "Troubleshooting CPU Switch Module Problems."
The optical connectors are dirty.	Refer to the Cisco ONS 15530 Cleaning Procedures for Fiber Optic Connections document.

¹NME = network management Ethernet

15.3.4 APS Does Not Switch to Protection Signal When the Working Signal Fails

Symptom When the working signal fails, APS does not switch over to the protection signal.

Table 15-4 describes the potential causes of the symptoms and the solutions.

Table 15-4 APS Does Not Switch to Protection Signal When the Working Signal Fails
Possible Problem	Solution
An APS switchover request is pending.	1. Issue the show aps group command or the show aps detail command to determine the pending APS request. 2. Issue the aps clear command to remove the APS request.
A trunk failure occurred on the protection signal.	Correct the failure on the protection signal. See Chapter 3, "Troubleshooting Transponder Line Card Problems," Chapter 8, "Troubleshooting 2.5-Gbps ITU Trunk Card Problems," Chapter 9, "Troubleshooting 10-Gbps ITU Trunk Card Problems." or Chapter 10, "Troubleshooting 10-Gbps ITU Tunable Trunk Card Problems."

15.3.5 Lockout from Protection Request Fails

Symptom A request to lock out an APS switchover to the protection path made with an aps lockout command failed.

Table 15-5 describes the potential cause of the symptom and the solution.

Table 15-5 Lockout from Protection Request Fails
Possible Problem	Solution
The active signal is already switched to the protection path.	1. Issue the aps switch group-name force protection-to-working command to ensure that the active signal is on the working path and then Issue the aps lockout command. 2. If the aps switch group-name force protection-to-working command fails, check the status of the working path using the show interfaces command and resolve the signal failure.

15.3.6 Several Unexpected APS Messages Received

Symptom Several unexpected APS messages display on the console.

Table 15-6 describes the potential cause of the symptom and the solution.

Table 15-6 Several Unexpected APS Messages Received
Possible Problem	Solution
A bidirectional APS group on one end is enabled and disabled while the other end is generating a high priority request.	Issue an aps enable/no aps enable command sequence to reinitialize the message channel.This is a temporary condition and should disappear when both ends reestablish APS communication.

15.3.7 Remote Switchover Does Not Occur After Local Switchover

Symptom The remote system does not switch over after the local system switches over.

Table 15-7 describes the potential causes of the symptoms and the solutions.

Table 15-7 Remote Switchover Does Not Occur After Local Switchover
Possible Problem	Solution
Both systems are not configured for bidirectional APS.	1. Issue show aps detail commands on both systems to verify the APS direction configuration. 2. Issue aps direction commands to correct the APS direction configuration, if necessary.
The protection path on the remote system has failed.	1. Issue a show interfaces command for the protection interface on the remote system. 2. Resolve any problems on the interface.

15.3.8 Manual or Forced Switchover Fails

Symptom A request for a manual or forced APS switchover fails.

Table 15-8 describes the potential cause of the symptom and the solution.

Table 15-8 Manual or Forced Switchover Fails
Possible Problem	Solution
A higher priority request is in effect. For bidirectional APS, the higher priority request might originate from the remote node.	1. Issue the show aps group command or the show aps detail command to determine if the request is user generated or system generated. 2. For user generated requests, issue the aps clear command to remove the higher priority request. 3. For system generated requests, correct the failure that is preventing the switchover.

15.3.9 Wave Interface or Waveethernetphy Interface Is Down and One Wavepatch Interface Is Up

Symptom Wave interface or waveethernetphy interface is down and one of the wavepatch interfaces is up when the APS group is bidirectional.

Table 15-9 describes the potential causes of the symptoms and the solutions.

Table 15-9 Wave Interface or Waveethernetphy Interface Is Down and One Wavepatch Interface Is Up
Possible Problem	Solution
A signal failure occurred on the receive side of the working path on the local system.	Correct the signal failure on the local system.
A signal failure occurred on the receive side of the protection path of the remote system.	Correct the signal failure on the remote system.

15.3.10 APS Group Transmitting k1k2 sf-lp to Peer APS Group

Symptom The transmit k1k2 field in the show aps group or show aps detail command output indicates sf-lp is sent to the peer APS group in a y-cable configuration.

Table 15-10 describes the potential causes of the symptoms and the solutions.

Table 15-10 APS Group Transmitting k1k2 sf-lp to Peer APS Group
Possible Problem	Solution
A trunk fiber break or misconfiguration is causing keepalive timeouts.	1. Check the show facility status command output for keepalive timeout alarms on the active interface. 2. Verify that there are no breaks on the trunk fiber. 3. Issue show interfaces commands on the active interfaces on the system to verify that the flow identifier is correctly configured. Issue the cdl flow identifier command to correct mismatches.
A fiber break is causing Tx-CRC threshold alarms.	1. Check the show facility status command output for Tx-CRC alarms on the active interface. 2. Verify that there are no breaks on the trunk or client fiber.
A failure occurred on the client receive signal.	1. Check the show facility status command output for loss of signal and loss of sync alarms on the active interface. 2. Verify that there are no breaks on the client fiber and that the connector are clean. Refer to the Cisco ONS 15530 Cleaning Procedures for Fiber Optic Connections document. 3. Ensure that the SFP optics are properly seated and that the LEDs are on. 4. Issue a show interfaces command to verify the protocol encapsulation. Issue the encapsulation command to correct any misconfiguration.

Table Of Contents