cc/td/doc/product/wanbu/bpx8600/9_3_0
hometocprevnextglossaryfeedbacksearchhelp
PDF

Table of Contents

Troubleshooting

Troubleshooting

This chapter describes periodic maintenance procedures and general troubleshooting procedures:

After an alarm occurs, use the BPX switch software to isolate the problem. If an BPX switch part has failed, then it must be replaced. See Chapter 30, Replacing Parts.

Preventive Maintenance

You perform most monitoring and maintenance of the BPX switch via the BPX switch operating system software. Preventive maintenance of the BPX switch hardware is minimal and requires only that you periodically check:

    1. The node supply voltage and internal cabinet temperature by using the dspasm command. The temperature should not exceed 50\xb0 C.

    2. The event log by using the dsplog command.

    3. The network alarm status by using the dspalms command.

Troubleshooting the BPX Switch

This section describes basic troubleshooting steps to be taken for some of the more obvious node failures (refer to Table 29-1). This is not an exhaustive set of procedures, and does not take into account any of the diagnostic or network tools available to troubleshoot the BPX switch. Refer to the Cisco WAN Switching Command Reference for information on commands and command usage.


Caution   Do not perform any disruptive tests or repairs to the BPX switch on your own. Before proceeding with troubleshooting, call Customer Service so they can provide you with assistance in locating the fault and provide repair information.

General Troubleshooting Procedures

The BPX switch runs self tests continuously to ensure proper function. When the node finds an error condition that affects its operation, it downs the card or trunk affected. It then selects a standby card or alternate trunk if one is available.

The FAIL indicators on the cards indicate that the system has found these cards defective in some mode, and now considers them as failed cards. Use Table 29-1 to find the cause and obtain the information on replacing the failed component.

Never remove the active BCC until the standby BCC has entered the Standby mode. Using the dspcd command is the only reliable way to determine that the standby BCC has finished updating and has entered the Standby mode.--


Caution    When using Table 29-1 for troubleshooting, call Cisco Customer Service before performing any disruptive testing or attempting to repair the BPX switch. This ensures that you have isolated the correct problem area. It also enables Cisco Customer Service to provide assistance in performing the necessary procedures.


Warning Contact Cisco Customer Service before attempting to replace fuses on backplane.


Table 29-1:
Troubleshooting the BPX Switch
Symptom Probable Cause Remedy

Front panel LED on individual card not lighted.

Card Fuse.

Check card fuse. Replace if defective.

Try another card of the same type. If still no LED lighted, backplane card slot fuse may be defective.

No front panel LEDs are lighted.

AC Systems:

Circuit Breakers on AC Power Supply Tray.

DC Systems:

Circuit breakers on Power Entry Module(s) switched off.

Switch on circuit breakers. If problem persists, pull all cards and power supplies out to see if a shorted card or supply exists.

BPX switch power cord plug dislodged from AC receptacle.

Check that no one is working on the system, shut off source breaker, then reconnect power cord.

Power supply ac LED lit but dc LED not lit.

Power supply defective.

Check DC on LEDs on ASM. If out, remove and replace power supply. If on, PS LED probably defective.

Card front panel fail LED lit.

Card failed self-test.

Check status of card at NMS terminal using dspcds screen. If alarm confirmed, try card reset (resetcd command). Finally, remove and replace the card.

Card stby LED on.

Card is off-line.

Not a problem as long as primary card is active.

ASM major or minor LED on.

Service-affecting (major) or non-service affecting (minor) system fault.

Check NMS event log to identify problem reported.

Failed card in local node.

See remedy for card fail LED indication.

Network trunk failed.

Observe Port LEDs on each BNI or BXM (ports configured in trunk mode).
Use NMS dsptrk to locate failure.

Failure in remote node. May be another BPX switch.

Use NMS dspnw screen to locate node in alarm. Refer to Cisco WAN Switching Command Reference for additional information.

Internal temperature is higher than normal resulting from blocked air flow or defective fan.

Check front and back of node cabinet for freedom of air flow. Replace any fan that may have failed or slowed. Use NMS dsppwr screen to check node temperature.

ASM hist LED lit.

If no other alarm indications, a fault occurred in the past but has been cleared.

Press ASM history clear button. Check NMS event log to determine cause.

BXM Port LED is red or orange (BXM configured for trunk mode).

Trunk is in local or remote alarm.

Use NMS dsptrk screen to confirm trouble.

BNI Port LED is red or orange.

Trunk is in local or remote alarm.

Use NMS dsptrk screen to confirm trouble. Use short BNC loopback cable at LM-BNI connectors for local test of trunk. Loop trunk at DSX-3 crossconnect to check cable.

No BXM card or port LED on.

No trunks or lines, as applicable on card are upped. Card has not necessarily failed.

Up at least one of the trunks or lines, as applicable, associated with the card (Trunks if BXM configured for trunk mode, lines if BXM configured for port mode).

No BME card or port LED on.

No lines are upped. Card has not necessarily failed.

Up at least one of lines, as applicable, associated with the card.

No BNI card or port LED on.

No trunks on card are upped. Card not necessarily failed.

Up at least one of the trunks associated with the card.

BXM Port LED is red or orange (BXM configured for port mode)

Line is in local or remote alarm.

Use NMS dsplns screen to confirm trouble.

BME Port LED is red or orange

Line is in local or remote alarm.

Use NMS dsplns screen to confirm trouble.

BCC fail LED flashing

Downloading system software or configuration data.

Wait for download to complete.

BCC LAN LED flashing

Normal for node connected to NMS terminal over Ethernet. If it does not flash, there may be problems with node to NMS data path.

Check that the cabling to the NMS is firmly connected to the LAN port on the LM-BCC back card. An alternate connection is to the control port.

No BCC card LED on.

Preparing to download new software (momentary condition).

Wait for download to begin.

Command issued to run a software rev. that was not available in the network.

Check that proper s/w rev. is available on another node or on NMS.

Displaying the Status of Cards in the Node

When a card indicates a failed condition on the alarm summary screen, use the Display Cards (dspcds) command to display the status of the circuit cards on a node. The information displayed for each card type includes the card slot number, software revision level, and the status of the card.

The possible status description for each card type are listed in Table 29-2. Refer to the Cisco WAN Switching Command Reference for more information on the Display Cards command.


Table 29-2: Card Status for the BPX Switch
Card Type Status1 Description

All card types

Active

Active card.

Active - F

Active card with no terminal failure.

Standby

Standby card.

Standby - F

Standby card with no terminal failure.

Standby - T

Standby card performing diagnostics.

Standby - F -T

Standby card with no terminal failure performing diagnostics.

Failed

Card with terminal failure.

Unavailable

Card is present but it may be in one of the following states:

a. The node does not recognize the card.

b. The card is running diagnostics.

Down

Downed card.

Empty

No card in that slot.

BCC

Same status as for all card types, plus:

Updating

Standby BCC downloading the network configuration from an active BCC.

Note: Red FAIL LED flashes during updating.

Cleared

BCC is preparing to become active.

Downloading Software

There are downloader commands that appear when the system is down- loading software to the BCC.

Minor

BCC Redundancy alarm indicates node is configured for redundancy but no standby BCC is equipped.

1Cards with an F status (no terminal failure) are activated only when necessary. Cards with a failed status are never activated.

System Troubleshooting Tools

You can perform a number of manually-initiated tests from the Cisco WAN Manager NMS console to assist in system troubleshooting. These tests may be included in a job so they can be scheduled to run remotely at a specified time if desired.

User-initiated Tests

Several user-initiated tests can be used to diagnose system problems. These tests are self-contained in that they do not require the use of external test equipment. They also do not require you to place a loopback at the far end to test both directions of transmission. These tests are listed in Table 29-3.

Several display commands can be used to obtain information that may be helpful in troubleshooting system problems. These are also listed in Table 29-3.


Table 29-3: System Troubleshooting Commands Available
Command Description

Test Connection (tstcon)—frame relay

Performs a bi-directional test of the specified frame relay connection or range of connections by inserting a test pattern and comparing the returned pattern with the pattern transmitted. A pass or fail indication appears next to the tested connection in the Display Connections screen.

Test Connection (tstcon)—data

Same as above except for synchronous data connections.

Test Connection (tstcon)—voice

Same as above except for voice connections.

Test Delay (tstdelay)—frame relay

Measures the round-trip delay over the selected frame relay connection.

Test Port (tstport)—frame relay

Tests the operation of the selected frame relay port on the node.

Test Port (tstport)—data

Same as above except for synchronous data ports.

Display Connection States (dspconst)

Displays in real-time the status of all voice connections terminating at a specified node.

Display Breakout Box (dspbob)—frame relay

Displays in real-time the status of data and control leads on selected frame relay connection.

Display Breakout Box (dspbob)—data

Same as above for synchronous data connections.

Display Breakout Box (dspbob)—trunk

Same as above for network subrate trunks.

Display Buses (dspbuses)

Displays the status of system buses.

Display Slot Errors (dspsloterrs)

Displays any data errors associated with the slots in a BPX node.

Display Slot Alarms (dspslotalms)

Displays any alarms associated with the slots in a BPX node.

Display Trunk Errors (dsptrkerrs)

Displays any data errors associated with the network trunks connected to a node.

Loopback Tests

Various loopback paths can be set up to help diagnose transmission problems. These rely on using external test equipment to provide the source of a test signal.

The available loopback commands are listed in Table 29-4.

You set up a local loopback path (LL) in the local node at the PAD card (FRP) associated with the port or connection to be tested. You then apply a test signal to the input. This passes through the associated Interface Card (FRI), is sent to the Frame Relay PAD card (FRP) over the system bus where it is looped back towards the input. This tests the cabling and the local node processing of the signal.


Table 29-4: System Loopback Tests
Command Description

Add Local Loopback (addloclp)—frame relay port

Adds a loopback path at the frame relay port from the transmit side back to the receive side at the local node.

Add Local Loopback (addloclp)—frame relay connection

Does the same as above only for an individual frame relay connection.

Add Local Loopback (addloclp)—data

Adds a loopback path at the synchronous data port from the transmit side back to the receive side at the local node.

Add Local Loopback (addloclp)—voice

Adds a loopback path for an individual voice channel on a circuit line at the local node.

Add Remote Loopback (addrmtlp)—frame relay port

Adds a loopback path at the frame relay port from the transmit side back to the receive side at the remote node.

Add Remote Loopback (addrmtlp)—frame relay connection

Does the same as above only for an individual frame relay connection.

Add Remote Loopback (addrmtlp)—data

Adds a loopback path at the synchronous data port from the transmit side back to the receive side at the remote node.

Add Remote Loopback (addrmtlp)—voice

Adds a loopback path for an individual voice channel on a circuit line at the remote node.

Add External Loopback (addextlp)—data

Activates a near end or far end loopback on an external device, such as a DSU, connected to a synchronous data port.

A remote loopback path (RL) is set up in the remote node also at the PAD card (FRP). But, in this case, the signal travels over the network and through the remote node processing equipment but does not include the remote node Interface Card (FRI) or associated cabling. These components would be tested using another local loopback at the remote node.

The external loopback command finds limited use in data applications where an external data interface unit (DSU or CSU) is attached to the local node data interface card, illustrated by the SDI card in . The local node transmits the appropriate loopback codes out the circuit line towards the external device and then sets up the appropriate loopback path.


Figure 29-1: Network Loopback Paths


Connection Testing

System software includes a Test Connection (tstcon) command for testing network connections. This test is initiated by the network operator from the NMS console and can be performed at any time but it momentarily interrupts traffic on the connection during the test. Testing a connection should be performed only when an alarm has been reported from the connection or during off-hours.

Test Connection tests both directions of transmission from end-to-end and displays a pass or fail indication for each connection tested. You may specify:

In addition to testing the connection, the Test Connection routine will attempt to isolate and repair any failure it detects. The controller card at the node where the Test Connection (tstcon) command is issued instructs the service card to build packets containing special test frames. These packets are sent across the network to the terminating node, which depacketizes them, repacketizes the frame, and sends them back to the originating node where the returned frame is analyzed.

If the returned test pattern is incorrect, the system goes into an automatic fault isolation mode. Controllers in the various nodes along the connection route communicate with each other over an overhead message channel separate from the normal circuits.

The test pattern continues to be transmitted and analyzed at each node along the path as it is transmitted and as it is received until the failed network element is identified. Redundant cards may be switched into operation and routing tables in associated network trunk cards may be reprogrammed in an attempt to correct the problem. If all else fails, the suspected path and/or network component is then reported to the network manager (NMS).

External Device Window

External devices connected to network nodes, such as bridges, routers, or sub-rate multiplexers may be accessed through the NMS Window command. This feature provides a direct command line interface to external devices from the NMS console. Depending on the capability of the external device, it is often possible to report status and alarms and to control or configure the device through an RS232 port connection.

The following example illustrates a Window display of a router connected to the local node. In this example, the window is used to initiate a ping of the router connection.

Example: NMS Window to a Local Router Protocol [ip]: Target IP address: 192.9.202.1 Repeat count [5]: Datagram size [100]: Timeout in seconds [2]: Extended commands [n]: Type escape sequence to abort. ^^ Sending 5, 100-byte ICMP Echos to 192.9.202.1, timeout is 2 seconds: . . . . . Success rate is 100 percent

Troubleshooting SONET Automatic Protection System

Introduction

For APS line redundancy, these problems can occur:

The following sections describe possible APS configuration problems.

Not Able to Correctly Set Up APS 1+1 Line Redundancy Configuration

Description: The addapsln user interface command fails to execute correctly for APS 1+1 line addition.

Initial Investigation: The addapsln command is used to setup the APS line redundancy configuration. For APS 1+1 configurations, BPX software supporting APS and BXM firmware supporting APS must be used.

These hardware requirements must be met:

Unable to set up APS 1:1 line redundancy configuration

Description: The addapsln user interface command fails to execute correctly for APS 1:1 line addition.

Initial Investigation: For APS 1:1 configuration, two adjacent lines on the same card are used. No special hardware is required however the maximum connections supported must be reduced by half using the cnfcdaps command. FW and SW support of APS is required.

Workaround: APS 1:1 can be run on non APS enhanced BXM card by halving the number of channels the card can support (cnfcdaps). No special backcards are needed for APS 1:1.

For APS 1:1 configuration the APS line must be configured (addapsln) before a line (upln) or trunk (uptrk) can be upped. Conversely, the line or trunk must be downed before the APS line can be deleted (delapsln).

Use dspapsln to verify that the APS line has been added.

Operator information about APS architectures

Description: The cnfapsln user interface command does not let you configure any combination of APS architectures.

Initial Investigation: You can change the APS configuration by using the cnfapsln command, however not all combinations are allowed. Here is a table of combinations allowed and disallowed.

Mode APS 1:1 APS 1+1, 1+1 ignore K1 APS 1+1 Annex B
Revertive Non-revertive Revertive Non-revertive Revertive Non-revertive

Bi-

directional

Default

Not Valid

Valid option

Valid option

Not Valid

Default

Uni-

directional

Not Valid

Not Valid

Valid option

Default

Not Valid

Not Valid

Once the APS configuration 1+1, 1:1, 1+1 Annex B, or 1+1 ignore K1 is chosen by the addapsln, it cannot be changed except by deleting the APS line (delapsln) and re-adding the APS line with the new configuration (addapsln).

Operational Problems

This section describe possible APS operational problems and troubleshooting techniques for each.

Initial Investigation of APS Switch Operations

There are ten reasons an APS switch may occur. You can view these logged reasons by using the dsplog command. When the BXM switches an APS line it returns an event message to the SWSW with the reason why it switched and which line is active.

This list shows the possible conditions which may cause/prevent a switch. The list is arranged starting from highest precedence and ending with lowest precedence.

    1. Lock out of Protection
    An external user requested switch which prevents switching from working line to protection line from taking place.

    2. Forced Switch
    An external user requested switch which forces a switch from working line to protection line or vice-versa even if there is an alarm on the destination line.

    3. Signal Fail
    An automatically initiated switch due to a signal failure condition on the incoming OC-N line including loss of signal, loss of frame, AIS-L defects, and a line BER exceeding 10-3.

    4. Signal Degrade
    An automatically initiated switch due to a "soft failure" condition resulting from the line BER exceeding a pre-selected threshold (cnfapsln).

    5. Manual Switch
    An external user requested switch which requests a switch from working line to protection line or vice-versa but only if there is no alarm on the destination line.

    6. Wait To Restore
    A state request switch due to the a revertive switch back to the working line because the wait-to-restore timer has expired.

    7. Exercise
    Not supported

    8. Reverse Request
    A state request switch due to the other end of an APS bi-directional line performing an APS switch.

    9. Do not Revert
    A state request due to the external user request being cleared (such as a forced switch) while using non-revertive switching.

    10. No Request
    A state request due to the external user request being cleared (such as a forced switch) while using revertive switching.

Unable to perform APS external switch after forced or manual APS switch

Description: You perform a forced switch from the working line to the protection line (switchapsln Ln1 Ln2 3) and then another forced switch back to working line (switchapsln Ln1 Ln2 4). After this the user again tries to perform a forced switch to the protection line but sees nothing happen.

Investigation: Once a forced switch is made from the working line to the protection line and back again, a clear switch (switchapsln Ln1 Ln2 1) must be issued in order to perform another forced switch. This applies to APS manual and lockout switching also.

With APS 1+1, when repetitive switchapsln commands are issued, up to two in a row can be executed sequentially, when alternating between options 3 and 4 (forced switch), or 5 and 6 (manual switch), but no more. Attempts to execute a third switchapsnln will not succeed, and the following error message is displayed:

"Cannot request manual W->P when manual P->W switch in progress"

If users desire to perform repetitive switchapls commands, they need to issue a clear switch between each W-P, P-W pair of commands, for example:

switchapsln 2.1 1

APS manual switch to a line does not occur right away

Description: You have issued a manual switch either to working or protection line. The switch did not occur because the destination line was in alarm. When the alarm is cleared on that line the switch does occur.

Explanation: The BXM firmware remembers the "last user switch request" (also called external request) and tries to switch to that line when it becomes available.

Switch occurs after lockout issued

Description: With protection line active, the user issues an APS switch lockout and a switch occurs back to the working line.

Investigation: This is normal operation. When the protection line is active and an APS switch lockout is issued, a switch to the working line will happen. The lockout function locks the working line as active. Only an external (user request) APS clear switch (switchapsln Ln1 Ln2 1) will disable the lockout.

APS switch made to a line in alarm

Description: You perform a forced switch to a line with a line alarm. The switch is successful making an alarmed line active with possible loss of traffic.

Investigation: It is normal operation for a forced switch to cause a switch to a line even though it may be faulty. This enables you to "force" a switch to standby line even if it is in alarm. A traffic outage may occur.

During a manual switch request, the BXM firmware decides whether the switch should occur and the switch may not occur if there is an alarm on the standby line. An APS clear switch will allow automatic switching to resume following a forced switch.

Reverse switch

Description: User performs a forced or manual switch on local end of APS line in bidirectional mode but other end indicates a reverse switch was performed.

Investigation: This is normal operation. A reverse switch in bidirectional mode occurs on the far end of the APS line when the local end of the APS line performs a switch for any reason.

APS switch occurs at the same time as a yred switch

Description: Two related scenarios could cause this to occur.

    1. A forced or manual switch is in effect. In dspapsln, the Last User Switch Request is forced or manual w->p or p->w. If a switchcdred/switchyred is performed (could be caused by card failure or physically removing card also) the front card switches and an APS switch occurs.

    2. A clear switch is in effect. In dspapsln, the Last User Switch Request is clear. If a switchyred is performed (could be caused by card failure or physically removing card also) the front card switches and an APS switch occurs.

Explanation: Following a switchcdred/switchyred, or active card reset the BXM card will be instructed to perform an APS switch to align itself with the Last User Switch Request (switchapsln).When a yred (switchcdred) switch takes place on a BXM card pair being used for APS 1+1, the card being switched is sent configuration messages including the last user switch request. The BXM card will initially become active in an APS "clear" switch mode following a switchcdred or reset.

This means that the APS switching is on automatic. However if the Last User Switch Request is a manual or forced switch, the software sends this request to the BXM, and the BXM will switch to this line if it is not already active. This switch is done to comply with the users last APS switch request.

In the second case, if the last user request is "clear", full automatic APS switching is in effect with the working line being active by default. When there is no last user switch request (switchapsln to protection, for example) to switch to any particular line, the working line will become active.

APS switch occurs after issuing an APS clear switch

Description: User issues an APS clear switch (switchapsln Ln1 Ln2 1) command while protection line is active and a switch occurs to the working line.

Explanation: This is normal operation. An APS clear switch request causes the APS switching mechanism in the BXM to initialize. This will cause a switch back to the working line if the working line is in better shape than the protection line. If the protection line is not faulty, no switch will occur.

APS Switch Occurs even though APS Forced switch in effect

Description: A forced switch to protection line is performed. LOS on protection line causes a switch back to working line even though a forced switch is in progress

Explanation: Signal Fail on Protection line has higher priority than Forced switch. Whenever the protection line is in failure, there will be a switch to working line, even if the working line is failed or there is a forced W->P in effect.

APS line is failing to switch

Description: The user issues an APS forced or manual switch request but no switch occurs

Investigation: This could be due to a forced, manual, or lockout switch being in progress and a clear switch is required (switchapsln Ln1 Ln2 1). Need to issue an APS clear switch (switchapsln) to exit forced, manual, or lockout switch state.

If running the ITUT APS standard protocol which does not report an Architecture Mismatch APS alarm the problem could be that one end of the line is bi-directional and the other is uni-directional.

Check that configuration is the same on both ends, specifically uni/bidirectional mode, 1:1/1+1 configuration.

A manual switch will not occur if the standby line is in alarm.

Large cell loss when performing a front card switchover

Description: A line configured for APS 1+1 line redundancy has its active front card switched either due to card failure, switchyred (switchcdred), or resetting the card. A loss of cells is observed.

Investigation: Cell loss at card switchover is not due to faulty APS. It is a result of the card redundant switch (YRED switch) and there will be up to 250ms worth of traffic disruption during BXM front card switchovers.

APS service switch description

Description: What is an APS service switch? Does it work on APS 1:1 configurations?

Investigation: An APS service switch is applicable only to APS 1+1 configuration. It lets you switch all the APS lines on a card by using a single switchapsln command with an "s" option at the end of the command. All APS lines on this card pair will be switched and made active on a single backcard allowing the other backcard to be removed for service.

IMPORTANT: Be sure that the associated front card is active for the backcard that is to remain in the rack. You might have to perform a switchcdred so that the backcard that the service switch switches to has its associated front card active. A service switch is not required in order to remove a BXM front card with APS 1+1 lines on it. The card redundancy will handle the switch to the other card without affecting the lines.

APS line does not seem to switch and active line is in alarm

Description: A major line alarm is indicated on the active line yet it remains active due to no APS switch to the redundant line.

Initial Investigation

    1. Verify that the configuration is correct (dspapsln, cnfapsln). See preceding configuration problems.

    2. Use dspapsln to check the APS line's status. The dspapsln display shows the active and standby line's alarm status. It also shows if there are any APS alarms.
    If the active line alarm status shows OK but the standby line alarm status shows an alarm, then a switch will not occur due to the standby line alarm. Troubleshoot the standby line problem.
    If the standby line alarm status shows OK but the active line alarm status shows an alarm then a switch should have occurred and there is a more obscure problem.
    If there is an APS alarm shown under Current APS alarms then this could be the problem, see section on APS Alarms.
    If APS 1+1 is configured, use dspcds to check the status of the protection line's card. If there is a problem with this card a switch may not occur.

    3. Verify the sequence of events by using dsplog and tracing the entries which contain information about this line or APS on this line.
    If a switch was attempted and succeeded due to a Loss of Signal, the message "APS SignalFail switch from LN 1 to LN 2" should be logged.
    If the switch failed there will be a message such as "Cannot do APS SigFail switch from LN 1 to LN 2".

Work Around: Perform a clear switch on each end of the APS line (switchapsln 2.1 1). This may get both ends in sync and clear up the problem.

A forced switch from working to protection may be performed (example: switchapsln 2.1 3). WARNING: If the protection line is in LOS and we force a switch to it, traffic will be lost.

If the line is an APS 1+1 line, then the front cards are redundant and the user may try a switchcdred (switchyred) to induce APS switching. This should normally have no affect on APS switching. APS switching and card redundancy switching are independent.

The BXM card may be reset in combination with an APS clear switch either before of after the reset at both ends of the APS line. Perform an APS clear switch on both on both ends of the line. Reset the BXM cards (resetcd h).

BXM backcard LED green and yellow indications

Description: Prior to an APS switch the active card LED is green and the standby card LED is yellow. After the APS switch, both LEDs are green

Explanation: The BXM backcard LED is meant to show whether the card is currently being used by at this time. Green means that this card is in use. Yellow means that the card is not in use and could be removed for service. If the standby line's card's LED is green it means that part of this card is being used at this time. This could happen due to the APS 1+1 cross over circuit where the working line's front card is active but the protection line itself is active. The working line's backcard is being used to shunt traffic to the protection line's backcard.

BXM Port LED states

Scenario: For an APS 1+1 or APS 1:1 line pair, the port LEDS are the same color on working and protection line.

Explanation: To switch software, the APS line pair is a single logical line. Although required to send BXM messages to both lines, these messages will be the same message. Thus switch software cannot send different LED states to the BXM for the same APS line. The BXM firmware makes the protection line LED state the same as the working line LED state.

BME Connection Diagnostics

Troubleshooting VSI Problems

Ths section describes how different types of channels are allocated (VSI, Automatic Routing Management), and how to troubleshooting some problems related to VSI. Note that some or all of the commands discussed in this section require service-level or above user privileges. To access these commands, you must have debug (Service or StrataCom level) privileges and passwords. Check with the TAC for assistance.

How Channels are Allocated and Deallocated

To understand channel allocation and deallocations problems, it's important to understand how the channels are distributed. The BXM card can support x number of channels. The value x varies between different models of BXMs.

How Networking Channels are Allocated

Networking channels are assigned for trunk interfaces only. This includes physical, feeder, and virtual. Every physical and feeder trunk that is active is assigned 271 networking channels. For virtual trunks, the first virtual trunk upped on a port is assigned 271 networking channels. Every subsequent one requires an additional one. So if the second virtual trunk on the same port is upped, one more networking channel is reserved for that virtual trunk.

How Automatic Routing Management Channels are Allocated/Configured

When a port or trunk interface is upped, a default value of 256 PVC channels are assigned. You can use the cnfrsrc command to change this value to fit your needs. Note that this is only the number of PVC channels configured. Every time a connection is added on the port or trunk interface, a counter is incremented to keep count of the number of PVCs used. This counter can never exceed the number configured. For the trunk interface, connections will be rerouted if the new value configured is less than the old value. For the port interface, cnfrsrc will not allow you to decrease the configured value to be less than the used value. You will need to delete connections before decreasing the PVC value.

How SVC Channels are Allocated and Configured

You can configure the number of SVC channels by using the cnftrk or the cnfport command. SVC and VSI channels cannot co-exist. The command will block you from configuring channels if there are VSI channels allocated.

How VSI Channels are Assigned for VSI Master to Slave VCs

When a VSI shelf is added with the addshelf command on the feeder interface, 12 LCNs are reserved for master to slave VCs. The reason for 12 LCNs is that one LCN is needed to communicate to an active BXM (with VSI functionality). The BPX has 15 slots possible, two of which are used for the BCC and one used for the ASM card. The worse case is if the BPX has all BXM cards in the node, therefore the master endpoint (that is, the card with the VSI shelf added) needs 12 LCNs to communicate with all the cards on the node. The command dspvsich will display all the LCNs reserved for master to slave VCs and interslave VCs.

How VSI Channels Are Configured/Allocated

VSI channels are configured through the cnfrsrc command. The user specifies a vsi min and a vsi max for the partition. The number of channels that is allocated is max (sum_of_min, max_of_max).

For example:

port group 1: port 1: min max partition 1: 1000 1000 port 2: partition 1: 2000 1000 port group 2: port 3: partition 1: 2000 5000 port 4: partition 1: 2000 4000

For portgroup 1:

sum_of_min = 3000; max_of_max = 1000

For portgroup 2:

sum_of_min = 4000; max_of_max = 5000

Therefore, the number of channels allocated for VSI is 8000.

How Background Redundancy Channels are Allocated

The formula for getting the LCN is num_chans + 1. These channels are used for y-redundancy cards to communicate with each other.

How IP Channels are Allocated

IP channels are used for ALL5 messaging. The LCNs are reserved within switch software. The formula for getting the LCN is num_chans + 14 + port (0 based). Twelve (12) LCNs are reserved for IP channels, one for each port.

How ILMI/LMI Channels are Allocated

The formula for getting the LCN is num_chans + 2 + port.

How ILMI Channels are Allocated for VSI Partitions on Trunk Interfaces

When ILMI functionality is enabled for a VSI partition on a trunk interface, a new ILMI session is started on the BXM card for the trunk interface. The LCN for this session is allocated from the LCNs available for the AutoRoute partition. This LCN is allocated from the port-based pool; not from the card-based pool.

Note that no new LCN is allocated when ILMI functionality is enabled for VSI partitions on port interfaces. This is because the ILMI functionality for VSI partitions on port interfaces use the same ILMI functionality that is started for AutoRoute. These use the pre-allocated LCN as discussed in the preceding section.

How VSI Channels are Assigned for Interslave VCs

Interslave vcs are assigned with LCNs that are reserved within switch software. These lcns are not taken from the pool. The formula for getting the lcn is num_chans + 26 + dest_slot where num_chans is the number of channels the card supports

mc_vsi_end_lcn

This value is shown in the dsplogcd command. If the value is 0, then there are no vsi channels configured on the card. If it is not zero, then there are VSI channels. It marks the first VSI channel.

num chans

This value is shown in the dsplogcd command as "Physical Chans". It is reported to switch software from the card. Each BXM will vary in the number of channels that it supports.

How Port Group Enters the Channel Assignment Picture

The dsplogcd command is for service level users and above. You must have "service" level privileges to use it.

There are some models of BXM cards which will support more than 1 port group. The command dsplogcd and dspcd will indicate the number of port groups supported. Even though each card supports x channels, there is a hardward limitation of how many channels can be supported between certain ports. A set of ports are grouped into port groups; that is, a BXM 8-port OC-3 card has two port groups, consisting of ports 1-4, and 5-8 respectively. Each port group will have an upper limit of the number of channels it can support, majority of the time it's

(num_chans / num_of_port_groups).

cnfrsrc fails with "available channels is 0"

When the user thinks that there are channels available, but cnfrsrc says that the number of available channels is 0. The user will not be able to allocate any more vsi channels.

This might not be a problem because the user might not have accounted for hidden channel assignments like networking and VSI vcs. Execute the dspchuse command to see where all the channels are allocated. Note any channel assignment that looks suspicious. Verify this page with the channels configured from the cnftrk and cnfrsrc command.

The dspchuse command is available to users in this release .

Workarounds

The work around depends on where the problem is. If it's with PVCs, try cnfrsrc and change the number of pvcs. Since switchcc, will rebuild the channel database, try executing switchcc.

Here is a list of things that should be done:

Verify if anyone has disable a partition.

Disabling the partition will not recalculate the end_lcn value. The end_lcn will be recalculated by a card reset or a switchcc or node rebuild.

cnfrsrc fails with "Automatic Routing Management is currently using the channel space"

This error is indicating that there are Automatic Routing Management channels currently configured on the space that the user wants for VSI.

For example: Let's say the BXM card supports 100 channels. Currently 50 of the channels are configured for PVCs and 50 for VSI ranging from 51-100. Let's suppose that the card has 5 connections on channel 45-49. Now change the configuration of PVCs to 10. The command will work since only five (5) are currently used. The available channels on the card is now 40. If cnfrsrc is executed now to increase the number of VSI channels, the command will fail, because channels 45-49 are currently in use.

To check if a specific connection is using a channel out of range:

To check if any connection in the port or trunk card is using a channel out of range.

Workarounds

The only work around is to somehow delete the connections currently using the high end of the channel range. On the trunk interface, causing the connections to reroute will likely cause the lower lcn range to be used first. On the port interface, deleting and re-adding the connection.

Troubleshooting Commands


Table 29-5:
Troubleshooting Command List
Command Full Name

addalmslot

Add alarm slot

addextlp

Add external loopback

addloclp

Add local loopback

addlocrmtlp

Add local-remote loopback

addrmtlp

Add remote loopback

clrchstats

Clear channel statistics

clrclkalm

Clear clock alarm

clrclnalm

Clear circuit line alarm

clrclnerrs

Clear circuit line errors

clreventq

Clear the events queues

clrlnalm

Clear line alarm

clrlnerrs

Clear line errors

clrlog

Clear log

clrmsgalm

Clear message alarm

clrphyslnalm

Clear physical line alarms

clrphyslnerrs

Clear physical line errors

clrportstats

Clear port statistics

clrslotalms

Clear slot alarms

clrsloterrs

Clear slot errors

clrtrkalm

Clear trunk alarm

clrtrkerrs

Clear trunk errors

clrtrkstats

Clear trunk statistics

cnflnalm

Configure line alarm

cnfoamlpbk

Configure OAM loopback test

cnfslotalm

Configure slot alarm

cnftrkalm

Configure trunk alarm

dellp

Delete loopback

dncd

Down card

dspalms

Display alarms

dspbuses

Display Buses

dspclnerrs

Display circuit line errors

dspeventq

Display the event queue names and the data in each.

dspfrcbob

Display FRC-2/FRM-2 breakout box

dsplog

Display event log

dsplnalmcnf

Display line alarm configuration

dsplnerrs

Display line errors

dspoamlpbk

Display OAM loopback test

dsppwr

Display power

dspslotalms

Display slot alarms

dspsloterrs

Display slot errors

dspslotstatcnf

Display slot statistics configuration

dspsv3

Display Cisco WAN Manager L3 (layer 3) Link Control Blocks

dsptrafficgen

Display whether Traffic Generation feature for card slot is enabled

dsptrkerrs

Display individual or all trunk errors

prtclnerrs

Print circuit line errors

prtlnerrs

Print line errors

prtlog

Print log

prttrkerrs

Print trunk errors

resetcd

Reset card

resetpc

Reset Port Concentrator

switchcc

Switch controller card

tstcon

Test connection

tstconseg

Test connection segment

tstdelay

Test delay

tstpcs

Test Port Concentrator Shelf

tstport

Test port

tstubus

Test cell bus


hometocprevnextglossaryfeedbacksearchhelp
Posted: Fri Jul 27 16:34:56 PDT 2001
All contents are Copyright © 1992--2001 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.