Troubleshooting

This chapter provides information about isolating faults in the Cisco 6100 with NI-2 system. Most problems in a Cisco 6100 with NI-2 system can be traced to one of the system's field-replaceable units (FRUs).

Hot-Swappable FRUs

However, hot-swapping some FRUs causes an interruption in service. See Table 4-1.

FRU	Does Hot Swap Interrupt Service?	Notes
NI-2 card	Yes	-
Line card	Yes	Service is interrupted only for subscribers served by that line card.
System I/O card	Yes	You must turn off system power to replace the system I/O card. Note that the MAC address is stored on the system I/O card.
Fan tray	No	-
Air filter	No	-

Basic Checks

Before using the troubleshooting tables in this chapter, make the following basic checks:

Contacting the Cisco TAC for Help

In certain situations, the troubleshooting tables in this chapter direct you to contact the Cisco Technical Assistance Center (TAC) for help. If you have a maintenance contract or if your hardware is under warranty, call the TAC at +1 800 553 2447 (North America only), +1 408 526 7209, or visit this URL for a worldwide list of TAC regional telephone numbers:

System-wide Problems

This section offers suggestions for problems that affect the entire Cisco 6100 system.

Symptoms	Steps to Take
System fails to come up.	1. Check the POWER LED on the NI-2 card and the STATUS LEDs on the line cards. If all LEDs are off, troubleshoot the power source. 2. If any green LEDs are on, the system has power. Check the STATUS LED on the NI-2 card. If the STATUS LED is off, refer to the "NI-2 Card Problems" section. 3. If the Cisco 6100 is set to boot from a remote device over the network, make sure the remote device is up, that its network connection is solid, and that it contains the boot file. (The Cisco 6100 tries to boot over the network for a configured period, usually 5 to 15 minutes. If it is unable to boot over the network, it will eventually boot from bootflash.) 4. Try to establish a console connection to the Cisco 6100. If you cannot connect, see the steps for the symptom "You cannot establish a console or Telnet connection to the system." 5. If you achieve a console connection: Examine the command prompt. If the prompt says `rommon 1>`, the problem could be in flash memory, in bootflash, in an incorrectly set boot configuration register, or in an incorrect file name in a boot system command in the startup-config file. Refer to the Configuration Guide for Cisco DSLAMs with NI-2 for information on setting and interpreting configuration registers, configuring flash memory, and editing the startup-config file. To select an image to boot the system from flash, enter dir flash:. In the resulting display, find the name of the software image. Then enter boot flash:imagename, replacing imagename with the name of the software image. The system boots from flash. If you see a normal Cisco IOS prompt, which usually contains the name of the system (default is `DSLAM>`), enter show oir status. If the results indicate that the card is loading software, wait a few minutes for the port to come back up.
You cannot establish a console or Telnet connection to the system.	1. For a console problem, check the terminal settings against the list of settings in the "Connect a Console Terminal" section. 2. For a Telnet problem: If you are connecting to the Cisco 6100 through the Ethernet interface, check the configuration of your LAN for both the Cisco 6100 and the Telnet source. If you are connecting to the Cisco 6100 through an ATM interface, make sure that PVCs are set up between the two devices and that the map-list is correctly configured. Enter the command show running-config to display this information. If the Telnet source and the Cisco 6100 are on different networks, make sure static routes are configured at both ends so that the two devices can communicate. To check, use the ping command to ping each device from the other (that is, ping the Cisco 6100 from the Telnet source, and ping the Telnet source from the Cisco 6100). Alternatively, ping your default gateways from each end of the connection. 3. For both console and Telnet problems: Check the cabling and connectors between the terminal or Telnet source and the Cisco 6100. See "Connector and Pinout Specifications" to check pinouts. Press the Reset button on the NI-2 card faceplate to reset the card. If the problem persists, replace the NI-2 card.
System experiences a critical, major, or minor alarm.	1. Enter the command show facility-alarm status. Note the affected slot and port, if any, and the description of the problem. 2. If no slot number is indicated, enter show environment all and examine the results for an indication of which FRU is at fault. Refer to the "FRU-Specific Problems" section for instructions on troubleshooting that FRU. 3. If ATM0/0, ATM0/1, ATM0/2, or ATM0/3 is indicated: Enter a show int command for the interface (for example, show int atm0/1). Results may indicate a SONET problem (Loss of Signal, for example). Refer to the "NI-2 Card Problems" section for instructions on troubleshooting the NI-2 card. Enter show controllers commands for all trunk and subtending ports. (The ports are atm0/1, atm0/2, and, in a DS3 system, atm0/3.) For example, show controllers atm0/1. In the resulting display, check that the framing mode is set to the same value on this interface as at the other end of the connection. Also check that cell payload scrambling is on (on DS3 interfaces only). 4. If a line card slot is indicated, consult the "Line Card Problems" section.
A trunk or subtending port fails to come up. (OC-3c or DS3.)	1. Check the cable connections at both ends. Refer to "Cable and Port Mapping Specifications," to check pinouts. 2. To check the interface status and configuration, enter show interface atm slot#/port#. Check the following in the resulting display: If the port Admin Status is down, enter the commands below to correct the problem, replacing slot/port ID atm 0/1 with your slot/port ID: DSLAM> configure terminal Enter configuration commands, one per line. End with CNTL/Z. DSLAM(config-if)# int atm 0/1 DSLAM(config-if)# no shutdown DSLAM(config-if)# If the port IF Status is down, check for disconnected or faulty cables. (Optical cables connect to the NI-2 card; DS3 coaxial cables connect to the system I/O card on the back of the chassis.) If the Line Protocol is down, the line protocol software processes might have determined that the line is unusable; try swapping the cable. Another possibility is that clocking might be misconfigured, or the clocking source might have failed. Check the CRC field. The presence of many CRC errors but not many collisions is an indication of excessive noise. If the number is too high (greater than 0.5 to 2 percent of total traffic on the interface), check the cables to determine if any are damaged. If you need more information on interface configuration, refer to the Configuration Guide for Cisco DSLAMs with NI-2 and the ATM Switch Router Software Configuration Guide. 3. For a DS3 interface: enter show controllers atmslot#/port#. Check the following in the resulting display: Framing mode must be the same at both ends of the connection. Cell payload scrambling must be on at both ends of the connection. 4. Check the status and configuration of the interface at the far end. 5. If you need to run a loopback test, do the following: In interface configuration mode, enter loopback diagnostic or loopback line. Set the external test equipment to loop data through the Cisco 6100 port. Obtain loopback results from your external test equipment. Enter no loopback diagnostic to take the port out of loopback mode. 6. In interface configuration mode, reset the trunk port by executing the shutdown command followed by the no shutdown command. 7. Replace the NI-2 card. 8. If the problem with a DS3 interface persists, troubleshoot the system I/O card. See the "System I/O Card Problems" section.
System overheats.	Troubleshoot the fan tray. See the "Fan Tray Problems" section.
System experiences a clocking problem.	Troubleshoot the NI-2 card. See the "NI-2 Card Problems" section.
System experiences a power problem.	Troubleshoot the fuse and alarm panel.

FRU-Specific Problems

The following sections describe symptoms that might occur and the steps that you need to take if you experience problems with any Cisco 6100 field replaceable unit (FRU).

NI-2 Card Problems

If you need to remove or replace an NI-2 card, follow the steps below and also refer to the NI-2 Card FRU Installation and Replacement Notes for the Cisco 6100, Cisco 6130, Cisco 6160, and Cisco 6260 for instructions.

Symptom	Steps to Take
Power LED is off.	1. Check the Status LEDs on the line cards. If all LEDs are off, troubleshoot the power source. 2. If the line card STATUS LEDs are lit, remove the NI-2 card from its slot and check for bent or broken pins on both the card and the backplane. If you find damaged pins on the card, replace it. If you find damaged pins on the backplane, contact the Cisco TAC.
Status LED is off, indicating that the NI-2 card failed to boot or failed its power-on self test.	Press the Reset button on the NI-2 card. If the problem persists, replace the card.
Critical LED, Major LED, or Minor LED is on.	See the "System-wide Problems" section.
A trunk or subtending port fails to come up.	See the "System-wide Problems" section.
NI-2 card cannot be fully inserted into its slot.	Inspect connectors on both the card and the backplane, looking for obstructions, bent pins, or other damage. If you find damage to a line card connector, replace the line card. If you find damage to a backplane connector, contact the Cisco TAC.

Line Card Problems

Symptom	Steps to Take
All LEDs are off.	1. Check the POWER LED on the NI-2 card. If all LEDs are off, troubleshoot the power source. 2. If the NI-2 card Power LED is off, remove the card from its slot and check for bent or broken pins on both the card and the backplane. If you find damaged pins on the card, replace it. If you find damaged pins on the backplane, contact the Cisco TAC.
All ports on a card fail to come up (modems do not train). Status LED might be red, indicating that the line card failed to boot or failed its power-on self test.	1. Enter the show ipc nodes command to find out whether there is a communication problem between the line card and the NI-2 card. There should be an entry in the resulting display for each line card ("SMB IP Slot n") and for the NI-2 card ("IPC Master"). If there is an entry for each card, go to step 4. 2. If one or more cards are not listed, enter show oir status. If the resulting display indicates that the card in question is loading new code, wait a few minutes and reenter the command. (Usually, 2 to 3 minutes is long enough to wait.) The card's status should change to `running`. 3. Enter the command show dsl int atm slot#/port#. If the status says `Microcode downloading`, wait 10 minutes. When the download is complete, the card will reboot and come up normally. 4. Check the ALARM LEDs on the NI-2 card or enter show facility-alarm status to determine the alarm status of the system. If any alarms are indicated, see the "System-wide Problems" section for instructions on how to troubleshoot alarms. 5. Enter the command show dsl status and examine the results to ensure that the card is configured to be in its current slot. (In the Names column, the slot number appears as part of the port ID. For example, in ATM19/2, the slot number is 19. Nothing is displayed for slots that are not configured.) If necessary, use the slot command to update the configuration, or move the card to the correct slot. If the line card is a 4xflexi¹, it will not function unless you use the slot command to configure either CAP or DMT operation. 6. Reset the line card by disconnecting it from the backplane and reseating it in its slot. 7. Install the card in another slot. 8. If the problem persists, replace the line card.
Port fails to come up (modems do not train), or port LED flashes continuously.	1. Enter the command show dsl int atm slot#/port# to display the port configuration. Check the display to ensure that the port is properly provisioned. Make sure the port is configured to be running ("no shutdown" or IOS admin state = up). Also check the line status; if this reads "No CPE detected," troubleshoot the CPE device and the loop as described below under the symptom "You suspect a problem with the CPE or the subscriber loop." 2. In interface configuration mode, reset the port by executing the shutdown command followed by the no shutdown command. 3. Connect the subscriber to another port on the Cisco 6100. If the modems do not train, troubleshoot the CPE device and the loop as described below under the symptom "You suspect a problem with the CPE or the subscriber loop." If the modems train, go to the next step to troubleshoot the port that failed. 4. To test the modem on the line card, enter dsl test atmslot#/port# self. Results are displayed automatically after a few seconds. If the modem fails the test, replace the line card.
You suspect a problem with the CPE or the subscriber loop.	1. Disconnect the local loop and replace it with a test setup that consists of a modem or CPE that is known to work and a few thousand feet of wire. If the modems train over the shorter distance, the problem lies in the local loop or in the CPE—see the steps that follow. If the modems do not train, the port is probably at fault. Replace the line card. 2. Make sure that the CPE at the subscriber site is powered up. 3. Make sure that the CPE is compatible with the Cisco 6100, and that the software version running on the CPE is compatible with the software version running on the Cisco 6100. 4. Power cycle the CPE. When you turn the power on, the WAN Link LED on the CPE should blink. If the LED does not blink, check the CPE's configuration—if the interface is shut down, bring it up. 5. Perform a continuity check to find out if the cabling to the CPE is connected and intact. For example, if there is a phone on the line, check for a dial tone. 6. If there is no POTS, check DC resistance by shorting tip and ring at the remote end. 7. Make sure there are no load coils on the local loop. 8. Is the local loop too long? The maximum length range is 15,000 to 25,000 feet (4572 to 7620 meters). Within that range, wire gauge, crosstalk, and multiple bridge taps reduce the distance over which the modems can train. 9. Is the local loop too short? DMT modems train best over loops of several thousand feet. In a test situation with a loop just a few feet in length, the modems may fail to train. Add wire to the loop. 10. Replace the CPE.
Modems train at a low bit rate, or modems retrain continuously, or the line experiences too many errors.	1. Enter the command show dsl int atmslot#/port# to display the port configuration. Check the display to ensure that the port is properly provisioned. Look in particular for these statistics: Attenuation: typically this is 20 to 50 dB. If the attenuation value is higher than 50 dB, it might be depressing the bit rate. Repair or replace the cables and connectors in the loop. SNR margin: 3 to 6 dB is optimum. Use the dmt margin command to adjust SNR margin. Correction ratio: under DSL Statistics, look at the Received Superframes and Corrected Superframes values. A ratio of more than 1 corrected superframe for every 10 superframes received is too high. One or more of these adjustments might correct the problem: increase the SNR margin using the dmt margin command; increase error correction using the dmt check-bytes command; or increase interleaving using the dmt interleaving-delay command. Errored seconds: a rate of 10 to 20 errored seconds per minute or more is likely to cause retraining. (1 or 2 errored seconds every 15 minutes is a good rate.) See the steps that follow on crosstalk and impulse noise for suggestions on how to compensate. CRC errors: normal rates vary system by system. If the CRC error rate is higher than usual, it might cause excessive retraining. See the steps that follow on crosstalk and impulse noise for suggestions on how to compensate. 2. Crosstalk is caused by interference between services in adjacent cables. It affects random bits rather than chunks of data; upstream and downstream traffic can be affected differently. If crosstalk is increasing the bit error rate (BER), you can compensate in several ways: Reduce the bit rate using the dmt bitrate command. (See the Command Reference for Cisco DSLAMs with NI-2 for information on the dmt bitrate command.) Increase the SNR margin using the dmt margin command. (See the Command Reference for Cisco DSLAMs with NI-2 for information on the dmt margin command.) Turn on or increase error correction. Use the dmt check-bytes command. If the codeword size is not set to auto, you might need to use the dmt codeword-size command to adjust the codeword setting. (See the Command Reference for Cisco DSLAMs with NI-2 for information on these commands.) 3. If you experience impulse noise or clipping, both of which affect chunks of data rather than random bits, you can compensate by turning on or increasing interleaving and error correction. (However, note that this approach adds delay.) Use the dmt interleaving-delay command for interleaving. Use the dmt encoding-trellis command or the dmt check-bytes command for error correction. If you use dmt check-bytes and if the codeword size is not set to auto, you might need to use the dmt codeword-size command to adjust the codeword setting.
	4. If errors or retraining occur while the line is ringing, use the dmt interleaving-delay command to turn on interleaving. 5. In interface configuration mode, reset the port by executing the shutdown command followed by the no shutdown command. 6. Connect the customer to a different port on the Cisco 6100. If the modems train, replace the line card with the faulty port. 7. Troubleshoot the CPE device and the loop as described above under the symptom "You suspect a problem with the CPE or the subscriber loop." 8. Replace the line card.
Card cannot be fully inserted into its slot.	1. Remove the card and reinsert it, pushing firmly on both the top and the bottom of the faceplate. (The card might jam in the slot if you apply pressure to the top only.) If the chassis is full, you might need to push sideways to insert the card. 2. Inspect connectors on both the card and the backplane, looking for obstructions, bent pins, or other damage. If you find damage to a line card connector, replace the line card. If you find damage to a backplane connector, contact the Cisco TAC.
Card experiences problems in one slot but operates normally in another.	There may be a fault in your backplane. Contact the Cisco TAC.

14xflexi = quad-port flexi ATU-C line card

System I/O Card Problems

If you need to remove or replace the system I/O card, follow the steps below and also refer to the NI-2 System I/O Card FRU Installation and Replacement Notes for instructions.

Symptom	Steps to Take
A trunk port fails to come up.	See the "System-wide Problems" section.
A subtending port fails to come up.	See the "System-wide Problems" section.
An alarm relay or BITS circuit fails.	1. Check the connection at the wire-wrap connector on the system I/O card. See Table 1-8 for the pin assignments of the wire-wrap connector. 2. Check the connection at the other end of the circuit. 3. For a BITS problem, troubleshoot the T1 line at the other end of the circuit. 4. Replace the system I/O card (except in the case of a circuit breaker alarm problem; the system I/O card does not provide the circuit breaker trip alarm).
This message appears on the console or in the system log file: 00:00:28:%C6100-4-COOKIE: Corrupt or missing MAC address cookie using random base 007e.eb7d.e700	1. Make sure the software release running on your system is compatible with the hardware. The Cisco 6100 requires Cisco IOS Release 12.1(2)DA or greater. 2. Enter the command show hardware chassis. In the resulting display, look for data from the system I/O card EEPROM. If the system can read the contents of the EEPROM, the system I/O card is likely to be healthy. 3. If the system I/O card is missing, install it. 4. If the system I/O card is present, reseat it. First, turn off system power. Use a screwdriver to unfasten the screws that secure the system I/O card cover, and pull the card off the connectors. Push it firmly back into place and tighten the screws. 5. If the problem persists, replace the system I/O card.

Fan Tray Problems

If you need to remove or replace the fan tray or fan module, follow the steps below and also refer to the Cisco 6100/6130 Fan Tray FRU Installation and Replacement Notes for instructions.

Symptom	Steps to Take
Fan modules do not run.	1. Make sure the fan module is fully inserted into the fan tray and screwed in place. 2. Replace the fan module.
Fan modules run but the system overheats.	1. Make sure that the air intake vents at the bottom front of the chassis and the exhaust vents on the top of the chassis are free of blockages. 2. Make sure that the ambient temperature and other environmental factors affecting the system are within the ranges specified in the "Environmental Requirements" section. 3. Make sure that all cards, blank faceplates, and covers are in place. The cooling system cannot operate effectively unless the chassis is fully enclosed. 4. Check the LED on the fan module. If the LED is not green, the fan module has failed. Replace the fan module. 5. Check the air filter at the bottom of the fan tray, and if necessary clean or replace it. See the "Replacing the Air Filter in the Fan Tray" section for instructions on cleaning or replacing the air filter. 6. Reduce the ambient temperature.
Fan module status LED is not green.	The fan module has failed or is not receiving power. If other components in the system are receiving power (look for green LEDs), replace the fan module.
Air filter is damaged, dirty, or clogged.	See the "Replacing the Air Filter in the Fan Tray" section for instructions on cleaning or replacing the air filter.

Alarms

Tables 4-2 through 4-7 describe alarms for the Cisco 6100. In each table, the text in the Alarm column is the text that appears in the description field of the alarm message. Alarm messages appear on the console screen as the alarms occur; to see a list of current alarms, enter show facility-alarm status.

Refer to the problem tables in this chapter for more detailed troubleshooting instructions.

Alarm	Keyword	Severity	Description
Chassis temperature too high	OVER_TEMP	Major	An overtemperature condition has been detected. (Temperature is measured on the NI-2 card.)

Alarm	Keyword	Severity	Description
Provisioned slot is empty	MODULE_MISMATCH	Major	The card in this slot does not match the type configured for this slot.
Line card not equal to provisioning	MODULE_MISSING	Minor	This slot is configured for a card, but no card is present.
Invalid line card for this slot	MODULE_INVALID	Minor	The card detected in this slot cannot operate in this slot or is incompatible with the system configuration.
Flexi line card not provisioned	FLEX_MODULE_NOT_PROVISIONED	Info	You must use the slot command to configure a 4xflexi as either CAP or DMT.

Alarm	Keyword	Severity	Description
Loss of active clock sync	LOSS_OF_SYNC	Major	Loss of timing reference. The configured clock source is not available, so the system is using its internal clock.
BITS clock failure	BITS_FAILED	Major	BITS clock failure (LOS or AIS). The configured clock source is not available, so the system is using its internal clock.

The source of the OC-3c alarms in Table 4-5 is one of the following interfaces: ATM0/1 (the trunk) or ATM0/2 (the subtending interface).

Table 4-5 OC-3c/STM-1 Network Interface Alarms

Alarm	Keyword	Severity	Description
Loss of Cell Delineation	SONET_LOCD	Critical	Loss of cell delineation on a SONET line.
Path RDI Received	SONET_PRDI	Critical	Path Remote Defect Indication was received on a SONET line. This is equivalent to Path Far End Receive Failure (FERF).
Path AIS Received	SONET_PAIS	Critical	Path Alarm Indication Signal was received on a SONET line.
Loss of Pointer	SONET_LOP	Critical	Loss of pointer condition on a SONET line.
Line RDI	SONET_LRDI	Critical	Line Remote Defect Indication received on a SONET line. This is equivalent to line Far End Receive Failure (FERF).
Line AIS Received	SONET_LAIS	Critical	Line Alarm Indication Signal received on a SONET line.
Loss of Frame	SONET_LOF	Critical	LOF condition on a SONET line.
Loss of Signal	SONET_LOS	Critical	LOS detected on the SONET line.
Signal Label Mismatch	SONET_SIGNAL_ LABEL	Minor	Incorrect payload type signal label mismatch on a SONET line.

The source of the DS3 alarms in Table 4-6 is one of the following interfaces: ATM0/1 (the trunk), ATM0/2 (subtending interface), or ATM0/2 (subtending interface).

Table 4-6 DS3/E3 Network Interface Alarms

Alarm	Keyword	Severity	Description
Loss of Cell Delineation	DS3/E3_LOCD	Critical	The DS3 line is experiencing a loss of cell delineation.
RAI Received	DS3/E3_RAI	Critical	The DS3 line is receiving a remote alarm indication.
Yellow Alarm Received	DS3/E3_YELLOW	Critical	The DS3 line is receiving a yellow alarm, indicating that another device has detected a failure that might be in this device.
AIS Received	DS3/E3_AIS	Critical	The DS3 line is receiving an Alarm Indication Signal.
OOF Received	DS3/E3_OOF	Critical	The DS3 line has detected an Out of Frame condition.
LOS Detected	DS3/E3_LOS	Critical	The DS3 line has detected Loss of Signal at the framer.
PLCP LOF Detected	DS3_PLCP_LOF	Critical	The DS3 line has detected a Physical Layer Convergence Procedure Loss of Frame error.

Alarm	Keyword	Severity	Description
Not detected or missing	FAN_TRAY_REMOVED	Major	The fan tray has been removed from the chassis or is missing.
Multiple fan failures	FAN_FAIL_MULTI	Major	Two or more fan modules (out of three) in the fan tray have failed. When fan modules fail, the remaining fan modules run at full speed until the fault is corrected.
Single fan failure	FAN_FAIL_SING	Minor	A single fan module (one of three) in the fan tray has failed. When fan modules fail, the remaining fan modules run at full speed until the fault is corrected.

Table of Contents