cc/td/doc/product/wanbu/igx8400/9_3_30
hometocprevnextglossaryfeedbacksearchhelp
PDF

Table of Contents

Troubleshooting the IGX
Checking AC Power Supplies
Troubleshooting an IGX Node
Where to Go Next

Troubleshooting the IGX


This chapter describes how to diagnose problems. When a troubleshooting table in this chapter recommends replacement, refer to the procedures in Chapter 5, "Replacing Parts on the IGX."

The IGX operating system software does most of the IGX monitoring and maintenance. The only action that qualifies as preventive maintenance is checking the power supplies.

Checking AC Power Supplies

You cannot directly measure voltages on the AC power supplies in an IGX node. If a problem exists with one of the supplies, one or both the DC and AC LEDs goes off. Refer to "Replacing Parts on the IGX," for instructions on reseating or replacing an AC power supply.

After you install new or additional cards in the node, check the LEDs on the power supplies to make sure that the cards have not put an excessive load on the power supplies.

Troubleshooting an IGX Node

This section describes elementary troubleshooting procedures and briefly describes the commands used when troubleshooting an IGX node. (These commands are described in detail in the Cisco WAN Switching Command Reference.) This set of procedures is not exhaustive and does not take into account any of the diagnostic or network tools available to troubleshoot the IGX node.


Caution    Do not perform any disruptive tests or repairs to the IGX node without first calling the Technical Assistance Center (TAC). For further information, refer to the "Obtaining Technical Assistance" section.

This section contains the following topics:

General Troubleshooting Procedures

The IGX node regularly runs self-tests to ensure proper function. When the node finds an error condition that affects operation, it deactivates the affected card or line then selects a standby card or redundant line if one is available.


Caution    A FAIL LED that comes on on a card indicates that an error occurred. Try resetting the light with the resetcd f command. If the FAIL LED comes on again, use the table below to find the cause or call the Cisco TAC to obtain information on isolating the problem and possibly replacing a component. For information on contacting the Cisco TAC, see the "Obtaining Technical Assistance" section.

Symptom Probable Cause Solution

No indicators on IGX boards on—console screen blank. LEDS on power supplies not on.

IGX circuit breakers off.

1. Switch on power switch.

2. Switch off, then switch on the power switch to reset the breaker. Check to see if any overload condition exists (shorted connections, crow-barred power supplies).

IGX power cord dislodged from plug.

Reconnect power cord to AC receptacle.

Power supplies not functioning.

Replace power supplies.

Front card FAIL LED on.

Front card experienced an error:

  • NPM card
  • UXM-E card
  • NTM card
  • UFM card
  • FRM card
  • UVM card
  • CVM card
  • HDM card
  • LDM card
  • ARM card

Indicates that an error occurred. First, reset the card with the resetcd f command. If the LED comes on again, call the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.

 

Note If an NPM fails in a nonredundant IGX system, reboot the system.

Front card indicator on—replacement card does not fix problem.

Defective backplane is possible.

If a new card does not fix the problem, the backplane might be suspect (very uncommon failure). Contact the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.

SDI card FAIL indicator on.

SDI card failed:

  • SDI (EIA/TIA232) card
  • SDI (EIA/TIA449) card
  • SDI (V.35) card

Indicates an error occurred. Check alarm status of card. First, reset the card with the resetcd f command. If the LED comes on again, reseat card. If the LED comes on again, call the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.

The FAIL indicator on any of the following card sets came on, but the replacement card does not fix the problem: UXM-E, UFM, ARM, CVM, LDM.

Defective backplane is possible (very uncommon).

Note All cards connect to the backplane, and if the backplane is defective, it can cause a fault on any card.

If new card does not fix the problem, contact the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.

Blown backplane fuse (very uncommon).

See the backplane fuse section in Chapter 5, "Replacing Parts on the IGX."

Power Supply AC or DC Okay LED off.

Possible power supply defect.

Re-seat supply per instructions in Chapter 5, "Replacing Parts on the IGX." Remove and replace power supply if defective.

PE-BC wiring or card defective.

1. If power supplies output checks out, then PE-BC wiring connections or card is suspect.

2. Make sure that the plug connection to the PE-BC card is secure; tighten if it is not.

3. If the plug connection is secure and the Power Supply Monitor FAIL indicator is still on, then remove and replace the SCM card.

Command-line display incorrectly shows wrong IGX system type.

Jumper switch W6 in wrong position (see the "Making Frame Relay Connections" section).

To indicate an IGX 8420 node, the jumper must be in place. To indicate an IGX 8430 node, the jumper must be absent.

SCM circuitry that reads W6 setting might be defective.

Verify SCM circuitry with a known good SCM.

Neither Okay LED on a power supply is on.

Defective fan or fans in cooling assembly is allowing the temperature in the enclosure to rise above 40\xb0 C.

Verify that fan tray fans are working. If not, replace tray according to instructions in Chapter 5, "Replacing Parts on the IGX."

Defective power supply fan in power supply allowing the power supply temperatures to rise above 40\xb0 C.

1. If the system cooling fan assembly is functioning correctly, then a power supply fan is suspect. Remove cover over power supplies to determine if fan is rotating. Replacing a power supply fan is not a field repair. Replace the supply.

2. Issue a dsppwr command at the control terminal to check to see if the power supply fans are rotating, if not, remove and replace the power supply with the defective fan.

3. If power supply fans are functioning, then the power supply temperature sensor is defective. Remove and replace the temperature sensor

Defective SCM card.

1. If both the enclosure fan assembly and the power supply fans are working correctly, then the SCM card is suspect.

2. Remove and replace the SCM card.

Console screen blank, IGX indicator lights on.

Control terminal switched off.

Switch on the control terminal.

Control terminal power cord disconnected.

Reconnect the control terminal power cord to 208/240 VAC power outlet.

EIA/TIA-232 cable loose or disconnected from the control terminal port on the SCM, or from the control terminal.

Reconnect the EIA/TIA-232 cable to Control Terminal port on the SCM back card or to the control terminal itself.

Control terminal malfunctioning.

Refer to the control terminal manufacturer's manual.

Printer not functioning.

Printer switched off.

Switch on the printer.

Printer out of paper.

Renew the paper supply.

Printer power cord disconnected.

Reconnect the printer cord to 208/240 VAC power outlet.

EIA/TIA-232 cable loose or disconnected from the control terminal port on the SCM, or from the printer.

Reconnect EIA/TIA-232 cable to the Control Terminal port on the SCM back card or to the printer itself.

Printer malfunctioning.

Refer to the printer manufacturer's manual.

Modem not functioning.

Modem switched off.

Switch on the modem.

Modem power cord disconnected.

Reconnect modem power cord.

EIA/TIA-232 cable loose or disconnected from the Control Terminal port on the SCM, or from the modem.

Reconnect the EIA/TIA-232 cable to the Control Terminal port on the SCM back card or the modem itself.

Telephone hookup cable disconnected.

Reconnect the telephone hookup cable.

Modem malfunctioning.

Refer to the modem manufacturer's manual.

DIP switches not set correctly.

Refer to the modem manufacturer's manual.

Data Frame Multiplexing (DFM) is not functional.

DFM has been not been enabled by the Cisco TAC.

Contact the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.

DFM has been enabled but does not function.

DFM only runs on speeds up to 64 kbps.

Readjust the speed.

Background noise or music sounds choppy.

VAD problem—VDP needs sensitivity adjustment.

Contact the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.

High-speed modem drops to low speed.

ADPCM is taking over.

Contact the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.

Bundled (Frame Relay) connections have failed.

One or more bundled connections have failed.

Contact the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.

Troubleshooting the IGX Console Alarms

The initial mode of troubleshooting the IGX node uses the console alarms displayed on the console screen.This section provides you with a procedure for isolating the alarms and thereby isolating the fault. Any repair to the IGX node must be performed by personnel qualified by Cisco.

Isolating Alarm Faults


Caution   Contact the Cisco TAC before performing any disruptive testing or attempting to repair the IGX node, to ensure that you have isolated the correct problem area and to enable Cisco personnel to provide you with assistance in performing the necessary procedures. For more information, refer to the "Obtaining Technical Assistance" section.

When a MAJOR/MINOR alarm flashes on the console screen, complete the following steps to determine the probable cause of the alarm:


Step 1   Use dspnw command to identify the nodes.

Step 2   Use vt command to place yourself at the affected node, and use the dspalms command to identify the alarm type.

    a. If the alarm display indicates a failed connection, go to the "Troubleshooting Failed Connections" section.

    b. If the alarm display indicates a failed circuit line, go to the "Troubleshooting Failed Circuit Lines" section.

    c. If the alarm display indicates a failed trunk, go to the "Troubleshooting Failed Trunks" section.

    d. If the alarm display indicates a failed card, go to the "Troubleshooting Failed Cards" section.

    e. If the alarm display indicates an unreachable node, go to the "Troubleshooting Unreachable Nodes" section.



Troubleshooting Failed Connections


Step 1   Use the dspcons command to identify which connections have failed and to determine the remote-end connection assignments.

Step 2   Use the dsplog command to determine the cause of failure of the connections. These failures could consist of failed circuit lines, trunks cards, or clock overspeeds.

    a. If the connections have failed due to a circuit line failure, go to the "Troubleshooting Failed Circuit Lines" section.

    b. If the connections have failed due to a packet line failure, go to the "Troubleshooting Failed Trunks" section.

    c. If the connections have failed due to a card failure, go to the "Troubleshooting Failed Cards" section.

    d. If connections have failed due to a clock over speed condition, go to the "Troubleshooting Clock Over Speed" section.



Troubleshooting Failed Circuit Lines


Step 1   Use the dspclns command to identify the circuit line number and failure type.

    a. If the failure is a circuit line local CGA (no pulses received at the local end of circuit line) go to the "Troubleshooting Circuit Line Local or Remote CGAs" section.

    b. If the failure is a circuit line remote CGA (no pulses received at the remote end of circuit line), go to the "Troubleshooting Circuit Line Local or Remote CGAs" section.

    c. If the failure is circuit line frame slips (indicating excessive frame slips on the T1 between the IGX node and the PBX), go to the "Troubleshooting Circuit Line Frame Slips" section on this page.

    d. If the failure is circuit line bipolar errors (indicating excessive bipolar errors on this circuit line), go to the "Troubleshooting Circuit Bipolar Errors" section on this page.



Troubleshooting Circuit Line Local or Remote CGAs

Step 1   Use the dsplog command to determine date, time of day, and the duration of the CGA alarm.

Step 2   Determine if the PBX T1 subrate or the PBX E1 interface went down at the time the CGA alarm was logged by the IGX node.

Step 3   Check cabling between IGX node and the PBX and make necessary repairs, if defective.

Step 4   Make a note of the steps taken and call the Cisco TAC (see the "Obtaining Technical Assistance" section).



Troubleshooting Circuit Line Frame Slips

Step 1   Use the dsplog command to determine date, time of day, and duration of the frame slip alarm. Also determine if the clock source for this line has changed due to line failure in the network.

Step 2   Use the dspclnerrs command to quantify frame slips and rate information.

Step 3   Use the dspclnhist command to obtain historical information on frame slips.

Step 4   Use the dspcurclk command to identify the current clock source and path to the current clock source.

Step 5   Use the clrclnalm command to clear the circuit line alarms

Step 6   Make a note of the steps taken, and call the Cisco TAC (see the "Obtaining Technical Assistance" section).



Troubleshooting Circuit Bipolar Errors

Step 1   Use the dsplog command to determine when the bipolar error threshold was exceeded, and the duration of the alarm.

Step 2   Use the dspclnerrs command to quantify the bipolar errors.

Step 3   Use the dspclnhist command to obtain historical information on bipolar errors.

Step 4   Check cabling between IGX node and the PBX for loose connections, and tighten it if it is loose.

Step 5   Use the clrclnalm command to clear line alarms.

Step 6   Make a note of the steps taken, and contact the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.



Troubleshooting Failed Trunks


Step 1   Use the dsptrks command to identify the remote end node name, trunk numbers at each end, and the type of failure.

    a. If the display shows a communication failure, go to the "Troubleshooting Communication Failure and CGAs" section.

    b. If the display shows a local CGA, go to the "Troubleshooting Communication Failure and CGAs" section.

    c. If the display shows a remote CGA, go to the "Troubleshooting Communication Failure and CGAs" section.

    d. If the display shows a bipolar error, go to the "Troubleshooting Bipolar Errors, Frame Slip Errors, and Out-of-Frame Errors" section.

    e. If the display shows a frame slip error, go to the "Troubleshooting Bipolar Errors, Frame Slip Errors, and Out-of-Frame Errors" section.

    f. If the display shows an out-of-frame error, go to the "Troubleshooting Bipolar Errors, Frame Slip Errors, and Out-of-Frame Errors" section.

    g. If the display shows a time-stamped packet drop error, go to the "Troubleshooting Packet Drops" section.

    h. If the display shows a non-time-stamped packet drop error, go to the "Troubleshooting Packet Drops" section.

    i. If the display shows a loopback, go to the "Troubleshooting a Loopback" section.



Troubleshooting Communication Failure and CGAs

A Local CGA indicates no pulses at the local end of the trunk; a remote CGA indicates no pulses at the remote end of the trunk.


Step 1   Use the dsplog command to determine when the communication failure or CGA occurred, and identify connections that might have failed because of lack of bandwidth on an alternate route.

Step 2   Use the dsptrkerrs command at each end of the packet line to quantify errors, and determine if they are unidirectional or bidirectional.

Step 3   Call telephone carrier and request span testing. Ask the carrier to perform BER tests using multiple test patterns, including standard quasi, all 1, and 3 and 24 patterns.

Step 4   Make a note of the steps taken, and contact the Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.



Troubleshooting Bipolar Errors, Frame Slip Errors, and Out-of-Frame Errors

A bipolar error indicates excessive bipolar errors on the trunk. a frame slip error indicates excessive frame slips on the trunk. An out-of-frame error indicates excessive out-of-frame errors on the trunk.


Step 1   Use the dsplog command to determine the date, time of day, and duration of the alarm.

Step 2   Use the dsptrkerrs command at each end of the trunk to quantify errors, and determine whether they are unidirectional or bidirectional.

Step 3   Use the dsptrkhist command at each end of the trunk to collect historical information on line errors.

Step 4   Use the clrtrkalm command to clear trunk alarms.

Step 5   Contact the Cisco TAC for assistance. Cisco personnel can monitor line errors and might advise disruptive testing to be scheduled with the telephone carrier. For more information, refer to the "Obtaining Technical Assistance" section.

Step 6   Call the telephone carrier and request span testing. Ask the carrier to perform bit error rate (BER) tests using multiple test patterns, including standard quasi, all 1, and 3 and 24 patterns.

Step 7   If the telephone carrier is unable to isolate the problem on the span, contact Cisco TAC for assistance. For more information, refer to the "Obtaining Technical Assistance" section.



Troubleshooting Packet Drops

Time-stamped and non-time-stamped packet drop errors indicate that packet drops have exceeded the threshold for generating an alarm.


Step 1   Use the dsplog command to determine when the dropped packet alarm threshold was exceeded, and to determine the duration of the alarm.

Step 2   Use the dspload command alarm to determine the current loading of this trunk.

Step 3   Make a note of steps taken and call Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.



Troubleshooting a Loopback

Step 1   Determine if company personnel are performing span tests with CSU loopbacks, demarcation, or DSX panel.

Step 2   If company personnel are performing loopback tests, ask them to indicate when they have completed testing, and monitor the system to ensure that the loopback indication disappears when testing is complete.

Step 3   If company personnel are not performing loopback tests, the telephone carrier most likely has the E1 span in loopback mode.

Step 4   Call the telephone carrier to verify that they are testing the E1 span, and ask them to indicate when they have completed their tests. Monitor the system to ensure that the loopback indication disappears when testing is completed.

Step 5   Make a note of the alarm steps taken, and call Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.



Troubleshooting Failed Cards


Step 1   Use the dspcds command to determine which card has failed, along with its status (active or standby).

Step 2   Use the dsplog command to determine the time of day the card failed and whether or not any connections using this card are also in a failed condition.

Step 3   If the failed card is an HDM or LDM card, use the dspbob command at each end of the connection, using this card to verify that data is passing. For a CDP, CVM, or UVM, use the dspchstats command.

Step 4   If a card has failed, make a note of the steps taken, and call Cisco TAC (see the "Obtaining Technical Assistance" section).



Troubleshooting Unreachable Nodes


Step 1   At any node, use the dsplog command to determine the date and time of day that the node became unreachable. A node is usually unreachable because of a trunk failure or a power outage.

Step 2   Contact personnel at that node to determine if there was a power failure at the time logged by the IGX node.

Step 3   If there was a power failure, check that NPM comes up, and run diagnostics.

Step 4   If there was not a power failure, call Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.



Troubleshooting Clock Over Speed


Step 1   Use the dspbob command to determine the incoming baud rate for this connection.

Step 2   Use the dspcon command to verify that the console incoming baud rate is the same as the configured baud rate.

Step 3   Reconfigure the incoming baud rate to match the configured baud rate.

Step 4   Make a note of the steps taken, and call Cisco TAC. For more information, refer to the "Obtaining Technical Assistance" section.



Displaying a Summary of Alarms

The first step in troubleshooting an IGX node is to check the condition of the system by displaying alarm conditions throughout the system. To see a summary of all the alarms on an IGX node, use the dspalms (display current node alarms) command. The alarms summary includes the following:

To display alarms, enter the dspalms command.

If the screen indicates a failure, refer to the commands in Table 4-1 to further isolate the fault.

Table 4-1   Fault Isolation and Diagnostic Commands

Failure Diagnostic Commands

Connection

dspcons (display connections)

Line alarm

dspclns (display circuit lines)

Trunk

dsptrks (display trunks)

Cards

dspcds (display cards)

Power monitor and fans

dsppwr (display power supply status)

Remote node

dspnw (display network)

Unreachable nodes

dspnw (display network)

Remote node alarms

dspnw (display network)

Displaying the Status of Cards

When a card indicates a failed condition on the alarm summary screen, use the dspcds command to display the status of the cards on a node. The information displayed for each card type includes the slot number, software revision level, and card status. (Note that you cannot use dspcds in a job.)


Note   If the dspcds command or any other command incorrectly states the IGX model (for example, stating that an IGX 8420 node is an IGX 8430 node), check the jumper switch W6 on the SCM. A jumpered W6 indicates an IGX 8420 node. An open W6 indicates an IGX 8430 node. Chapter 4, "Installing the IGX," documents this aspect of the SCM.

Table 4-2 lists all possible dspcds command status descriptions for all card types, including CVM.

Table 4-2   dspcds Command Status Descriptions

dspcds Command Status Description

Active

Active card.

Active—F

Active card with nonterminal failure.

Standby

Standby card.

Standby—F

Standby card with nonterminal failure.

Standby—T

Standby card performing diagnostics.

Standby—F-T

Standby card with nonterminal failure performing diagnostics.

Failed

Card with terminal failure.

Unavailable

Card is present but can be in any of the following states:

  • The node does not recognize the card (might need to be reseated).
  • The card is running diagnostics.

Down

Downed card.

Empty

No card in that slot.

Active—T

Active card performing diagnostics.

Updating
(NPM only)

Standby NPM downloading the network configuration from an active NPM.

Note Red fail LED flashes during updating.

Cleared
(NPM only)

NPM is preparing to become active.

Loading software
(NPM only)

There are downloader commands that appear when the system is downloading software to the NPM.


Note   Cards with an "F" status (nonterminal failure) are activated only when necessary (for example, when there is no card of that type available). Cards with a failed status are never activated.

To display cards, issue the dspcds command. The dspcds command cannot be included in a job. Refer to the Cisco WAN Switching Command Reference for more information.

User-Initiated Tests

Several user commands help you test the node status. The CLI commands are:

For details on these commands, see the Cisco WAN Switching Command Reference.

Loopback Tests

Loopback tests are available to help diagnose the state of the IGX system. The CLI commands for activating these tests are:

For detailed information on these commands, see the Cisco WAN Switching Command Reference.

Card Testing with External Test Equipment

The HDM/SDI or LDM/LDI card set can be tested as a pair at the local node using external test equipment such as a bit error rate tester (BERT). This can be useful in isolating "dribbling" error rates to the cards, the Frame Relay data input, or the transmission facility. This test checks the data path from the electrical interface at the port through the card set to the cellbus in both directions of transmission.


Note   This is a disruptive test. Notify your network administrator before performing this test.

To perform this test, proceed as follows:


Step 1   Disconnect the cable connection to the SDI or LDI, and connect the BERT in its place.

Step 2   Set up an internal loopback on the Frame Relay port to be tested, using the addloclp command.

Step 3   Turn on the BERT, make sure that it indicates circuit continuity, and observe the indicated error rate.

Step 4   If there are any errors indicated, first replace the back card and retest. If the errors remain, then replace the front card and retest.

Step 5   When the test is complete, disconnect the BERT and reconnect the data cable. Release the local loopback by using the dellp command.

Step 6   Repeat at the node at the other end of the connection, if necessary.



Where to Go Next

For information on part replacement in the Cisco IGX 8400 series, see "Moving a Node."

For software configuration and service provisioning information, see the Cisco IGX 8400 Series Provisioning Guide, "Introduction to the Cisco IGX 8400 Series."

For more information on switch software commands, refer to the Cisco WAN Switching Command Reference, Chapter 1, "Command Line Fundamentals ."


hometocprevnextglossaryfeedbacksearchhelp
Posted: Thu May 15 08:27:26 PDT 2003
All contents are Copyright © 1992--2003 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.