Troubleshooting

This chapter describes periodic maintenance procedures and general troubleshooting procedures.

After an alarm occurs, use the BPX switch software to isolate the problem. If a BPX switch part has failed, then it must be replaced (see "Replacing Parts").

Preventive Maintenance

You perform most monitoring and maintenance of the BPX switch via the BPX switch operating system software. Preventive maintenance of the BPX switch hardware is minimal and requires only that you periodically check:

The node supply voltage and internal cabinet temperature by using the dspasm command. The temperature should not exceed 50×C.

Software Error and Abort Tables

The BPX software logs noncritical errors into the Software Error Table and contains up to 12 entries.

The BPX 9.3.X software logs critical errors into a separate Software Abort Table, so that there is no chance of critical errors being overwritten by noncritical errors. It contains up to 12 entries. Use the command dspabortlog to display the contents of the Software Abort Table and the command clrabortlog to clear it.

Troubleshooting the BPX Switch

This section describes basic troubleshooting steps to be taken for some of the more obvious node failures (see Table 28-1). This is not an exhaustive set of procedures, and does not take into account any of the diagnostic or network tools available to troubleshoot the BPX switch. For information on commands and usage, refer to the Cisco WAN Switching Command Reference.

General Troubleshooting Procedures

The BPX switch runs self-tests continuously to ensure proper function. When the node finds an error condition that affects its operation, it downs the card or trunk affected. It then selects a standby card or alternate trunk if one is available.

The FAIL indicators on the cards indicate that the system has found these cards defective in some mode, and now considers them as failed cards. Use Table 28-1 to find the cause and obtain the information on replacing the failed component.

Warning Contact Cisco Customer Service before attempting to replace fuses on backplane.

Table 28-1 Troubleshooting the BPX Switch
Symptom	Probable Cause	Remedy
Front panel LED on individual card not lighted.	Card Fuse.	Check card fuse. Replace if defective. Try another card of the same type. If still no LED lighted, backplane card slot fuse may be defective.
No front panel LEDs are lighted.	AC Systems: Circuit Breakers on AC Power Supply Tray. DC Systems: Circuit breakers on Power Entry Module(s) switched off.	Switch on circuit breakers. If problem persists, pull all cards and power supplies out to see if a shorted card or supply exists.
	BPX switch power cord plug dislodged from AC receptacle.	Check that no one is working on the system, shut off source breaker, then reconnect power cord.
Power supply ac LED lit but dc LED not lit.	Power supply defective.	Check DC on LEDs on ASM. If out, remove and replace power supply. If on, PS LED probably defective.
Card front panel fail LED lit.	Card failed self-test.	Check status of card at NMS terminal using dspcds screen. If alarm confirmed, try card reset (resetcd command). Finally, remove and replace the card.
Card stby LED on.	Card is off-line.	Not a problem as long as primary card is active.
ASM major or minor LED on.	Service-affecting (major) or nonservice-affecting (minor) system fault.	Check NMS event log to identify problem reported.
	Failed card in local node.	See remedy for card fail LED indication.
	Network trunk failed.	Observe Port LEDs on each BNI or BXM (ports configured in trunk mode). Use NMS dsptrk to locate failure.
	Failure in remote node. May be another BPX switch.	Use NMS dspnw screen to locate node in alarm. Refer to Cisco WAN Switching Command Reference for additional information.
	Internal temperature is higher than normal resulting from blocked air flow or defective fan.	Check front and back of node cabinet for freedom of air flow. Replace any fan that may have failed or slowed. Use NMS dsppwr screen to check node temperature.
ASM hist LED lit.	If no other alarm indications, a fault occurred in the past but has been cleared.	Press ASM history clear button. Check NMS event log to determine cause.
BXM Port LED is red or orange (BXM configured for trunk mode).	Trunk is in local or remote alarm.	Use NMS dsptrk screen to confirm trouble.
BNI Port LED is red or orange.	Trunk is in local or remote alarm.	Use NMS dsptrk screen to confirm trouble. Use short BNC loopback cable at LM-BNI connectors for local test of trunk. Loop trunk at DSX-3 cross-connect to check cable.
No BXM card or port LED on.	No trunks or lines, as applicable on card, are upped. Card has not necessarily failed.	Up at least one of the trunks or lines, as applicable, associated with the card (Trunks if BXM configured for trunk mode, lines if BXM configured for port mode).
No BME card or port LED on.	No lines are upped. Card has not necessarily failed.	Up at least one of lines, as applicable, associated with the card.
No BNI card or port LED on.	No trunks on card are upped. Card not necessarily failed.	Up at least one of the trunks associated with the card.
BXM Port LED is red or orange (BXM configured for port mode)	Line is in local or remote alarm.	Use NMS dsplns screen to confirm trouble.
BME Port LED is red or orange	Line is in local or remote alarm.	Use NMS dsplns screen to confirm trouble.
BCC fail LED flashing	Downloading system software or configuration data.	Wait for download to complete.
BCC LAN LED flashing	Normal for node connected to NMS terminal over Ethernet. If it does not flash, there may be problems with node to NMS data path.	Check that the cabling to the NMS is firmly connected to the LAN port on the LM-BCC back card. An alternate connection is to the control port.
No BCC card LED on.	Preparing to download new software (momentary condition).	Wait for download to begin.
	Command issued to run a software revision that was not available in the network.	Check that proper s/w revision is available on another node or on NMS.

Displaying the Status of Cards in the Node

When a card indicates a failed condition on the alarm summary screen, use the Display Cards (dspcds) command to display the status of the circuit cards on a node. The information displayed for each card type includes the card slot number, software revision level, and the status of the card.

The possible status description for each card type are listed in Table 28-2. Refer to the Cisco WAN Switching Command Reference for more information on the Display Cards command.

Table 28-2 Card Status for the BPX Switch
Card Type	Status¹	Description
All card types	Active	Active card.
	Active - F	Active card with no terminal failure.
	Standby	Standby card.
	Standby - F	Standby card with no terminal failure.
	Standby - T	Standby card performing diagnostics.
	Standby - F -T	Standby card with no terminal failure performing diagnostics.
	Failed	Card with terminal failure.
All card types (continued)	Unavailable	Card is present but it may be in one of the following states: 1. The node does not recognize the card. 2. The card is running diagnostics.
	Down	Downed card.
	Empty	No card in that slot.
BCC	Same status as for all card types, plus:
	Updating	Standby BCC downloading the network configuration from an active BCC. Note: Red FAIL LED flashes during updating.
	Cleared	BCC is preparing to become active.
	Downloading Software	There are downloader commands that appear when the system is down loading software to the BCC.
	Minor	BCC Redundancy alarm indicates node is configured for redundancy but no standby BCC is equipped.

¹Cards with an F status (no terminal failure) are activated only when necessary. Cards with a failed status are never activated.

System Troubleshooting Tools

You can perform a number of manually initiated tests from the Cisco WAN Manager NMS console to assist in system troubleshooting. These tests may be included in a job so they can be scheduled to run remotely at a specified time if desired.

User-Initiated Tests

Several user-initiated tests can be used to diagnose system problems. These tests are self-contained in that they do not require the use of external test equipment. They also do not require you to place a loopback at the far end to test both directions of transmission. The user initiated tests are listed in Table 28-3.

Several display commands can be used to obtain information that may be helpful in troubleshooting system problems. The system troubleshooting commands are also listed in Table 28-3.

Table 28-3 System Troubleshooting Commands Available
Command	Description
Test Connection (tstcon)—Frame Relay	Performs a bidirectional test of the specified Frame Relay connection or range of connections by inserting a test pattern and comparing the returned pattern with the pattern transmitted. A pass or fail indication appears next to the tested connection in the Display Connections screen.
Test Connection (tstcon)—data	Same as above except for synchronous data connections.
Test Connection (tstcon)—voice	Same as above except for voice connections.
Test Delay (tstdelay)—Frame Relay	Measures the round-trip delay over the selected Frame Relay connection.
Test Port (tstport)—Frame Relay	Tests the operation of the selected Frame Relay port on the node.
Test Port (tstport)—data	Same as above except for synchronous data ports.
Display Connection States (dspconst)	Displays in real-time the status of all voice connections terminating at a specified node.
Display Breakout Box (dspbob)—Frame Relay	Displays in real-time the status of data and control leads on selected Frame Relay connection.
Display Breakout Box (dspbob)—data	Same as above for synchronous data connections.
Display Breakout Box (dspbob)—trunk	Same as above for network subrate trunks.
Display Buses (dspbuses)	Displays the status of system buses.
Display Slot Errors (dspsloterrs)	Displays any data errors associated with the slots in a BPX node.
Display Slot Alarms (dspslotalms)	Displays any alarms associated with the slots in a BPX node.
Display Trunk Errors (dsptrkerrs)	Displays any data errors associated with the network trunks connected to a node.

Loopback Tests

Various loopback paths can be set up to help diagnose transmission problems. These rely on the use of external test equipment to provide the source of a test signal.

You set up a local loopback path (LL) in the local node at the Frame Relay Module (FRM) card associated with the port or connection to be tested. You then apply a test signal to the input. This passes through the associated Interface Card (FRI), is sent to the FRM card over the system bus where it is looped back toward the input. This tests the cabling and the local node processing of the signal.

Table 28-4 System Loopback Tests
Command	Description
Add Local Loopback (addloclp)—Frame Relay port	Adds a loopback path at the Frame Relay port from the transmit side back to the receive side at the local node.
Add Local Loopback (addloclp)—Frame Relay connection	Does the same as above only for an individual Frame Relay connection.
Add Local Loopback (addloclp)—data	Adds a loopback path at the synchronous data port from the transmit side back to the receive side at the local node.
Add Local Loopback (addloclp)—voice	Adds a loopback path for an individual voice channel on a circuit line at the local node.
Add Remote Loopback (addrmtlp)—Frame Relay port	Adds a loopback path at the Frame Relay port from the transmit side back to the receive side at the remote node.
Add Remote Loopback (addrmtlp)—Frame Relay connection	Same as above except for an individual Frame Relay connection.
Add Remote Loopback (addrmtlp)—data	Adds a loopback path at the synchronous data port from the transmit side back to the receive side at the remote node.
Add Remote Loopback (addrmtlp)—voice	Adds a loopback path for an individual voice channel on a circuit line at the remote node.
Add External Loopback (addextlp)—data	Activates a near end or far end loopback on an external device, such as a DSU, connected to a synchronous data port.

A remote loopback path (RL) is set up in the remote node also at the Frame Relay card (FRM). But, in this case, the signal travels over the network and through the remote node processing equipment but does not include the remote node Interface Card (FRI) or associated cabling. These components would be tested using another local loopback at the remote node.

The external loopback command finds limited use in data applications where an external data interface unit (DSU or CSU) is attached to the local node data interface card, illustrated by the SDI card in Figure 28-1. The local node transmits the appropriate loopback codes out the circuit line towards the external device and then sets up the appropriate loopback path (see Figure 28-1).

Connection Testing

System software includes a Test Connection (tstcon) command for testing network connections. This test is initiated by the network operator from the NMS console and can be performed at any time, but it momentarily interrupts traffic on the connection during the test. Connection testing should be performed only when an alarm has been reported from the connection or during off-hours.

Test Connection tests both directions of transmission from end to end and displays a pass or fail indication for each connection tested. You can specify:

In addition to testing the connection, the Test Connection routine will attempt to isolate and repair any failure it detects. The controller card at the node where the Test Connection (tstcon) command is issued instructs the service card to build packets containing special test frames. These packets are sent across the network to the terminating node, which depacketizes them, repacketizes the frame, and sends them back to the originating node where the returned frame is analyzed.

If the returned test pattern is incorrect, the system goes into an automatic fault isolation mode. Controllers in the various nodes along the connection route communicate with each other over an overhead message channel separate from the normal circuits.

The test pattern continues to be transmitted and analyzed at each node along the path as it is transmitted and as it is received until the failed network element is identified. Redundant cards may be switched into operation and routing tables in associated network trunk cards may be reprogrammed in an attempt to correct the problem. If all else fails, the suspected path and/or network component is then reported to the network manager (NMS).

External Device Window

External devices connected to network nodes, such as bridges, routers, or subrate multiplexers may be accessed through the NMS Window command. This feature provides a direct command line interface to external devices from the NMS console. Depending on the capability of the external device, it is often possible to report status and alarms and to control or configure the device through an RS232 port connection.

The following example illustrates a Window display of a router connected to the local node. In this example, the window is used to initiate a ping of the router connection.

Troubleshooting SONET Automatic Protection System

APS Configuration Problems

Not Able to Correctly Set Up APS 1+1 Line Redundancy Configuration

Description: The addapsln user interface command fails to execute correctly for APS 1+1 line addition.

Initial Investigation: The addapsln command is used to set up the APS line redundancy configuration. For APS 1+1 configurations, BPX software supporting APS and BXM firmware supporting APS must be used.

•

BXM-Enhanced OC-3 or OC-12 front cards. BXM-155-4 or BXM-155-8 front card of revision C or higher. BXM-622-2 or BXM-622-1 of revision E or higher.

•

BXM OC-3 or OC-12 APS back cards (they have two connectors on the back instead of one and require the daughter backplane in order to fit into the BPX backframe.

•

Card redundancy (addcdred or addyred) must be set up on the card pair prior to addapsln, see section on Y-cable issues. APS does not use the special Y-cable, it uses straight cables on both ports to the remote port. The redundant card must be in adjacent slots.

•

Using a back card frame containing internal card cage stiffeners requires that only slots 2-5 and 10-13 be used for APS 1+1 configurations. This is due to the stiffeners preventing the daughter backplane from fitting into the back card frame.

•

A newer back card frame removes the slot restriction of having to put daughter backplane and APS back cards in slots 2 to 5 and 10 to 13.

Unable to Set Up APS 1:1 Line Redundancy Configuration

Description: The addapsln user interface command fails to execute correctly for APS 1:1 line addition.

Initial Investigation: For APS 1:1 configuration, two adjacent lines on the same card are used. No special hardware is required; however, the maximum connections supported must be reduced by half using the cnfcdaps command. FW and SW support of APS is required.

Workaround: APS 1:1 can be run on non-APS-enhanced BXM card by halving the number of channels the card can support (cnfcdaps). No special back cards are needed for APS 1:1.

For APS 1:1 configuration the APS line must be configured (addapsln) before a line (upln) or trunk (uptrk) can be upped. Conversely, the line or trunk must be downed before the APS line can be deleted (delapsln).

Operator Information about APS Architectures

Description: The cnfapsln user interface command does not let you configure any combination of APS architectures.

Initial Investigation: You can change the APS configuration by using the cnfapsln command; however, not all combinations are allowed. Here is a table of combinations allowed and disallowed.

Mode	APS 1:1	APS 1+1, 1+1 ignore K1	APS 1+1 Annex B
Revertive	non-Revertive	Revertive	non-Revertive	Revertive	non-Revertive
Bidirectional	Default	Not Valid	Valid option	Valid option	Not Valid	Default
Unidirectional	Not Valid	Not Valid	Valid option	Default	Not Valid	Not Valid

Once the APS configuration 1+1, 1:1, 1+1 Annex B, or 1+1 ignore K1 is chosen by the addapsln, it cannot be changed except by deleting the APS line (delapsln) and readding the APS line with the new configuration (addapsln).

Operational Problems

This section describe possible APS operational problems and troubleshooting techniques for each.

Initial Investigation of APS Switch Operations

There are ten reasons an APS switch might occur. You can view these logged reasons by using the dsplog command. When the BXM switches an APS line it returns an event message to the SWSW with the reason why it switched and which line is active.

This list shows the possible conditions that might cause/prevent a switch. The list is arranged starting from highest precedence and ending with lowest precedence.

Lock Out of Protection—Specifies an external user-requested switch that prevents switching from working line to protection line from taking place.

Forced Switch—Specifies an external user-requested switch that forces a switch from working line to protection line or vice versa even if there is an alarm on the destination line.

Signal Fail—Specifies an automatically initiated switch due to a signal failure condition on the incoming OC-N line including loss of signal, loss of frame, AIS-L defects, and a line BER exceeding 10-3.

Signal Degrade—Specifies an automatically initiated switch due to a "soft failure" condition resulting from the line BER exceeding a preselected threshold (cnfapsln command).

Manual Switch—Specifies an external user requested switch that requests a switch from working line to protection line or vice versa but only if there is no alarm on the destination line.

Wait To Restore—Specifies a state request switch due to a revertive switch back to the working line because the Wait-to-Restore timer has expired.

Reverse Request—Specifies a state request switch due to the other end of an APS bidirectional line performing an APS switch.

Do Not Revert—Specifies a state request due to the external user request being cleared, such as a forced switch while using nonrevertive switching.

10.

No Request—Specifies a state request due to the external user request being cleared, such as a forced switch while using revertive switching.

Unable to Perform APS External Switch After Forced or Manual APS Switch

Description: After performing a forced switch from the working line to the protection line (switchapsln Ln1 Ln2 3) and then another forced switch back to working line (switchapsln Ln1 Ln2 4), when the user again tries to perform a forced switch to the protection line, nothing happens.

Investigation: Once a forced switch is made from the working line to the protection line and back again, a clear switch (switchapsln Ln1 Ln2 1) must be issued in order to perform another forced switch. This applies to APS manual and lockout switching also.

With APS 1+1, when repetitive switchapsln commands are issued, up to two in a row can be executed sequentially, when alternating between options 3 and 4 (forced switch), or 5 and 6 (manual switch), but no more. Attempts to execute a third switchapsnln will not succeed, and the following error message is displayed:

If users desire to perform repetitive switchapls commands, they need to issue a clear switch between each W-P, P-W pair of commands, for example:

APS Manual Switch to a Line Does Not Occur Right Away

Description: After issuing a manual switch either to working or protection line, the switch did not occur because the destination line was in alarm. When the alarm is cleared on that line the switch does occur.

Explanation: The BXM firmware remembers the "last user switch request" (also called external request) and tries to switch to that line when it becomes available.

Switch Occurs After Lockout Issued

Description: With protection line active, the user issues an APS switch lockout and a switch occurs back to the working line.

Investigation: This is normal operation. When the protection line is active and an APS switch lockout is issued, a switch to the working line will happen. The lockout function locks the working line as active. Only an external (user-requested) APS clear switch (switchapsln Ln1 Ln2 1) will disable the lockout.

APS Switch Made to a Line in Alarm

Description: After performing a forced switch to a line with a line alarm, the switch is successful in making an alarmed line active with possible loss of traffic.

Investigation: It is normal operation for a forced switch to cause a switch to a line even though it may be faulty. This enables you to "force" a switch to standby line even if it is in alarm. A traffic outage may occur.

During a manual switch request, the BXM firmware decides whether the switch should occur and the switch may not occur if there is an alarm on the standby line. An APS clear switch will allow automatic switching to resume following a forced switch.

Reverse Switch

Description: User performs a forced or manual switch on local end of APS line in bidirectional mode, but other end indicates a reverse switch was performed.

Investigation: This is normal operation. A reverse switch in bidirectional mode occurs on the far end of the APS line when the local end of the APS line performs a switch for any reason.

APS Switch Occurs at the Same Time as a Y-Red Switch

A forced or manual switch is in effect. In dspapsln, the Last User Switch Request is forced or manual w->p or p->w. If a switchcdred/switchyred is performed (could be caused by card failure or physically removing card also) the front card switches and an APS switch occurs.

A clear switch is in effect. In dspapsln, the Last User Switch Request is clear. If a switchyred is performed (could be caused by card failure or physically removing card also) the front card switches and an APS switch occurs.

Explanation: Following a switchcdred/switchyred, or active card reset, the BXM card is instructed to perform an APS switch to align itself with the Last User Switch Request (switchapsln).When a Y-red (switchcdred) switch takes place on a BXM card pair being used for APS 1+1, the card being switched is sent configuration messages including the last user switch request. The BXM card will initially become active in an APS "clear" switch, mode following a switchcdred or reset.

This means that the APS switching is on automatic. However if the Last User Switch Request is a manual or forced switch, the software sends this request to the BXM, and the BXM will switch to this line if it is not already active. This switch is done to comply with the users last APS switch request.

In the second case, if the last user request is "clear," full automatic APS switching is in effect with the working line being active by default. When there is no last user switch request (switchapsln to protection, for example) to switch to any particular line, the working line becomes active.

APS Switch Occurs After Issuing an APS Clear Switch

Description: User issues an APS clear switch (switchapsln Ln1 Ln2 1) command while protection line is active and a switch occurs to the working line.

Explanation: This is normal operation. An APS clear switch request causes the APS switching mechanism in the BXM to initialize. This will cause a switch back to the working line if the working line is in better shape than the protection line. If the protection line is not faulty, no switch will occur.

APS Switch Occurs Even Though APS Forced Switch in Effect

Description: A forced switch to protection line is performed. LOS on protection line causes a switch back to working line even though a forced switch is in progress.

Explanation: Signal Fail on Protection line has higher priority than Forced switch. Whenever the protection line is in failure, there is a switch to working line, even if the working line is failed or there is a forced W->P in affect.

APS Line is Failing to Switch

Description: The user issues an APS forced or manual switch request but no switch occurs.

Investigation: This could be due to a forced, manual, or lockout switch being in progress and a clear switch is required (switchapsln Ln1 Ln2 1). Need to issue an APS clear switch (switchapsln) to exit forced, manual, or lockout switch state.

If running the ITUT APS standard protocol, which does not report an Architecture Mismatch APS alarm, the problem could be that one end of the line is bidirectional and the other is unidirectional.

Check that configuration is the same on both ends, specifically uni/bidirectional mode, 1:1/1+1 configuration.

Large Cell Loss When Performing a Front Card Switchover

Description: A line configured for APS 1+1 line redundancy has its active front card switched either due to card failure, switchyred (switchcdred), or resetting the card. A loss of cells is observed.

Investigation: Cell loss at card switchover is not due to faulty APS. It is a result of the card redundant switch (Y-red switch) and there are up to 250 ms worth of traffic disruption during BXM front card switchovers.

APS Service Switch Description

Description: What is an APS service switch? Does it work on APS 1:1 configurations?

Investigation: An APS service switch is applicable only to APS 1+1 configuration. It enables switching all APS lines on a card by using a single switchapsln command with an "s" option at the end of the command. All APS lines on this card pair are switched and made active on a single back card, allowing the other back card to be removed for service.

Important: Be sure that the associated front card is active for the back card that is to remain in the rack. You might have to perform a switchcdred so that the back card that the service switch switches to has its associated front card active. A service switch is not required in order to remove a BXM front card with APS 1+1 lines on it. The card redundancy will handle the switch to the other card without affecting the lines.

APS Line Does Not Seem to Switch and Active Line is in Alarm

Description: A major line alarm is indicated on the active line yet it remains active due to no APS switch to the redundant line.

Verify that the configuration is correct (dspapsln, cnfapsln). See preceding configuration problems.

Use dspapsln to check the APS line's status. The dspapsln display shows the active and standby line's alarm status. It also shows if there are any APS alarms.

If the active line alarm status shows OK but the standby line alarm status shows an alarm, then a switch will not occur due to the standby line alarm. Troubleshoot the standby line problem.

If the standby line alarm status shows OK but the active line alarm status shows an alarm then a switch should have occurred and there is a more obscure problem.

If there is an APS alarm shown under Current APS alarms, this could be the problem (see section on APS Alarms).

If APS 1+1 is configured, use dspcds to check the status of the protection line's card. If there is a problem with this card, a switch may not occur.

Verify the sequence of events by using dsplog and tracing the entries that contain information about this line or APS on this line.

If a switch was attempted and succeeded due to a Loss of Signal, the message "APS SignalFail switch from LN 1 to LN 2" should be logged.

If the switch failed, a message appears such as "Cannot do APS SigFail switch from LN 1 to LN 2".

Workaround: Perform a clear switch on each end of the APS line (switchapsln 2.1 1). This may get both ends in sync and clear up the problem.

A forced switch from working to protection may be performed (example: switchapsln 2.1 3).

Warning If the protection line is in LOS and you force a switch to it, traffic is lost.

If the line is an APS 1+1 line, then the front cards are redundant and the user may try a switchcdred (switchyred) to induce APS switching. This should normally have no effect on APS switching. APS switching and card redundancy switching are independent.

The BXM card may be reset in combination with an APS clear switch either before of after the reset at both ends of the APS line. Perform an APS clear switch on both on both ends of the line. Reset the BXM cards (resetcd h).

BXM Back Card LED Green and Yellow Indications

Description: Prior to an APS switch, the active card LED is green and the standby card LED is yellow. After the APS switch, both LEDs are green

Explanation: The BXM back card LED is meant to show whether the card is currently being used at this time. Green means that this card is in use. Yellow means that the card is not in use and could be removed for service. If the standby line's card's LED is green it means that part of this card is being used at this time. This could happen due to the APS 1+1 cross-over circuit where the working line's front card is active but the protection line itself is active. The working line's back card is being used to shunt traffic to the protection line's back card.

BXM Port LED States

Scenario: For an APS 1+1 or APS 1:1 line pair, the port LEDS are the same color on the working and protection line.

Explanation: To switch software, the APS line pair is a single logical line. Although required to send BXM messages to both lines, these messages are the same message. Thus switch software cannot send different LED states to the BXM for the same APS line. The BXM firmware makes the protection line LED state the same as the working line LED state.

BME Connection Diagnostics

•

tstconseg and tstdelay commands may be used to troubleshoot a leaf connection both from the BME end point as well as on the other end point.

•

tstconseg is available on the root connection only on the non-BME end point.

Troubleshooting VSI Problems

This section describes how different types of channels are allocated (VSI, Automatic Routing Management), and how to troubleshooting some problems related to VSI. Note that some or all of the commands discussed in this section require service-level or above user privileges. To access these commands, you must have debug (Service or StrataCom level) privileges and passwords. Check with TAC for assistance.

How Channels Are Allocated and Deallocated

To understand channel allocation and deallocations problems, it is important to understand how the channels are distributed. The BXM card can support x number of channels. The value x varies between different models of BXMs.

How Networking Channels Are Allocated

Networking channels are assigned for trunk interfaces only. This includes physical, feeder, and virtual. Every physical and feeder trunk that is active is assigned 271 networking channels. For virtual trunks, the first virtual trunk upped on a port is assigned 271 networking channels. Every subsequent one requires an additional one. So if the second virtual trunk on the same port is upped, one more networking channel is reserved for that virtual trunk.

How Automatic Routing Management Channels Are Allocated/Configured

When a port or trunk interface is upped, a default value of 256 PVC channels are assigned. You can use the cnfrsrc command to change this value to fit your needs. Note that this is only the number of PVC channels configured. Every time a connection is added on the port or trunk interface, a counter is incremented to keep count of the number of PVCs used. This counter can never exceed the number configured. For the trunk interface, connections are rerouted if the new value configured is less than the old value. For the port interface, cnfrsrc will not allow you to decrease the configured value to be less than the used value. You will need to delete connections before decreasing the PVC value.

How SVC Channels are Allocated and Configured

You can configure the number of SVC channels by using the cnftrk or the cnfport command. SVC and VSI channels cannot coexist. The command will block you from configuring channels if there are VSI channels allocated.

How VSI Channels Are Assigned for VSI Master to Slave VCs

When a VSI shelf is added with the addshelf command on the feeder interface, 12 LCNs are reserved for master-to-slave VCs. The reason for 12 LCNs is that one LCN is needed to communicate to an active BXM (with VSI functionality). The BPX has 15 slots possible, two of which are used for the BCC and one used for the ASM card. The worst case is if the BPX has all BXM cards in the node, therefore the master end point (that is, the card with the VSI shelf added) needs 12 LCNs to communicate with all the cards on the node. The command dspvsich will display all the LCNs reserved for master to slave VCs and interslave VCs.

How VSI Channels Are Configured/Allocated

VSI channels are configured through the cnfrsrc command. The user specifies a VSI min and a VSI max for the partition. The number of channels that is allocated is max (sum_of_min, max_of_max) as shown in the following example:

How Background Redundancy Channels Are Allocated

The formula for getting the LCN is num_chans + 1. The channels are used for Y-redundancy cards to communicate with each other.

How IP Channels Are Allocated

IP channels are used for ALL5 messaging. The LCNs are reserved within switch software. The formula for getting the LCN is num_chans + 14 + port (0 based). Twelve LCNs are reserved for IP channels, one for each port.

How ILMI/LMI Channels Are Allocated

How ILMI Channels Are Allocated for VSI Partitions on Trunk Interfaces

When ILMI functionality is enabled for a VSI partition on a trunk interface, a new ILMI session is started on the BXM card for the trunk interface. The LCN for this session is allocated from the LCNs available for the Automatic Routing Management partition. This LCN is allocated from the port-based pool; not from the card-based pool.

Note that no new LCN is allocated when ILMI functionality is enabled for VSI partitions on port interfaces. This is because the ILMI functionality for VSI partitions on port interfaces use the same ILMI functionality that is started for Automatic Routing Management. These use the preallocated LCN as discussed in the preceding section.

How VSI Channels Are Assigned for Interslave VCs

Interslave VCs are assigned with LCNs that are reserved within switch software. These LCNs are not taken from the pool. The formula for getting the LCN is num_chans + 26 + dest_slot where num_chans is the number of channels the card supports

mc_vsi_end_lcn

This value is shown in the dsplogcd command. If the value is 0, then there are no VSI channels configured on the card. If it is not zero, then there are VSI channels. It marks the first VSI channel.

num chans

This value is shown in the dsplogcd command as "Physical Chans." It is reported to switch software from the card. Each BXM will vary in the number of channels that it supports.

How Port Group Enters the Channel Assignment Picture

The dsplogcd command is for service level users and above. You must have "service" level privileges to use it.

There are some models of BXM cards that will support more than one port group. The commands dsplogcd and dspcd will indicate the number of port groups supported. Even though each card supports x channels, there is a hardware limitation of how many channels can be supported between certain ports. A set of ports are grouped into port groups; that is, a BXM 8-port OC-3 card has two port groups, consisting of ports 1-4, and 5-8 respectively. Each port group will have an upper limit of the number of channels it can support, typically.

cnfrsrc Fails with "Available Channels is 0"

When the user thinks that there are channels available, but the cnfrsrc command says that the number of available channels is 0. The user will not be able to allocate any more VSI channels.

This might not be a problem because the user might not have accounted for hidden channel assignments like networking and VSI VCs. Execute the dspchuse command to see where all the channels are allocated. Note any channel assignment that looks suspicious. Verify this page with the channels configured from the cnftrk and cnfrsrc commands.

Workarounds: The workaround depends on where the problem is. If the problem is with PVCs, try the cnfrsrc command and change the number of pvcs. Since switchcc will rebuild the channel database, try executing switchcc.

•

Capture the dspchuse screen and compare against the cnfrsrc and cnftrk commands.

•

Verify the number of trunks that are upped. This will indicate the number of networking channels assigned.

•

Note the number of VSI shelves added. For each VSI shelf added, 12 LCNs are reserved on the BXM attached to the controller and 1 LCN is reserved for all the other active BXM cards. Capture the dspvsich command. For example:

–

On slot 13 - 25 lcns are reserved => 12 for each vsi shelf, and 1 for the shelf added to slot 11.

–

On slot 11 - 14 lcns are reserved => 12 for the vsi shelf, and 2 for the 2 shelves added on slot 13.

–

On slot 9 - 3 lcns are reserved => 2 for the 2 shelves added on slot 13, and 1 for the 1 shelf added on slot 11.

Disabling the partition will not recalculate the end_lcn value. The end_lcn is recalculated by a card reset or a switchcc or node rebuild.

cnfrsrc Fails with "Automatic Routing Management is Currently Using the Channel Space"

This error is indicates that there are Automatic Routing Management channels currently configured on the space that the user wants for VSI.

For example: The BXM card supports 100 channels, with 50 of the channels configured for PVCs and 50 for VSI ranging from 51-100. Assume that the card has five connections on channel 45-49. Now change the configuration of PVCs to 10. The command will work since only five are currently used. The available channels on the card is now 40. If cnfrsrc is executed now to increase the number of VSI channels, the command will fail, because channels 45-49 are currently in use.

To check if a specific connection is using a channel out of range, use the following procedure:

Step 1

Use the dcct command to verify the channel number (LCN) used by the connection.

Step 3

In normal conditions, the value of mc_vsi_end_lcn should be greater than LCN.

To check if any connection in the port or trunk card is using a channel out of range, use the following procedure:

Step 2

Use the dspchmap command to display the map of LCNs used by connection in the card. In normal conditions, no LCN higher than mv_vsi_end_lcn should be associated with an Automatic Routing Management connection or trunk xlat.

Workarounds: The only workaround is to delete the connections currently using the high end of the channel range. On the trunk interface, causing the connections to reroute will likely cause the lower LCN range to be used first. On the port interface, delete and readd the connection.

Troubleshooting Commands

A list of the troubleshooting commands are described in Table 28-5. For more information, refer to the Cisco WAN Switching Command Reference, Release 9.3.30.

Table 28-5 Troubleshooting Command List
Name	Description
addalmslot	Adds an alarm slot.
addextlp	Adds external loopback
addloclp	Adds a local loopback.
addlocrmtlp	Adds a local-remote loopback.
addrmtlp	Adds a remote loopback.
clrabortlog	Clears the critical errors from the Software Abort Table.
clrchstats	Clears the channel statistics.
clrclkalm	Clears the clock alarm.
clrclnalm	Clears the circuit line alarm.
clrclnerrs	Clears the circuit line errors.
clreventq	Clears the events queues.
clrlnalm	Clears the line alarm.
clrlnerrs	Clears the line errors.
clrlog	Clears the log.
clrmsgalm	Clears the message alarm.
clrphyslnalm	Clears the physical line alarms.
clrphyslnerrs	Clears the physical line errors.
clrportstats	Clears the port statistics.
clrslotalms	Clears the slot alarms.
clrsloterrs	Clears the slot errors.
clrtrkalm	Clears the trunk alarm.
clrtrkerrs	Clears the trunk errors.
clrtrkstats	Clears the trunk statistics.
cnflnalm	Configures the line alarm.
cnfoamlpbk	Configures the OAM loopback test.
cnfslotalm	Configures the slot alarm.
cnftrkalm	Configures the trunk alarm.
dellp	Deletes the loopback.
dncd	Downs the card.
dspabortlog	Displays the critical errors in the Software Abort Table.
dspalms	Displays the alarms.
dspbuses	Displays the buses.
dspclnerrs	Displays the circuit line errors.
dspcntrstats	Displays in real-time all counter statistics of a specified entity.
dspeventq	Displays the event queue names and the data in each.
dspfrcbob	Displays the FRC-2/FRM-2 breakout box.
dsplog	Displays the event log.
dsplnalmcnf	Displays the line alarm configuration.
dsplnerrs	Displays the line errors.
dspoamlpbk	Displays the OAM loopback test.
dsppwr	Displays the power.
dspslotalms	Display slot alarms
dspsloterrs	Displays the slot errors.
dspslotstatcnf	Displays the slot statistics configuration.
dspsv3	Displays the Cisco WAN Manager L3 (layer 3) Link Control Blocks.
dsptrafficgen	Displays whether the Traffic Generation feature for a card slot is enabled.
dsptrkerrs	Displays individual or all trunk errors.
prtclnerrs	Prints the circuit line errors.
prtlnerrs	Prints the line errors.
prtlog	Prints the log.
prttrkerrs	Prints the trunk errors.
resetcd	Resets the card.
resetpc	Resets the Port Concentrator.
switchcc	Switches the controller card.
tstcon	Tests the connection.
tstconseg	Tests the connection segment.
tstdelay	Tests the delay.
tstpcs	Tests the Port Concentrator Shelf.
tstport	Tests the port.
tstubus	Tests the cell bus.

Table Of Contents