Troubleshooting

This chapter explains how to isolate faults in the Cisco 6160 system. Most problems in a Cisco 6160 system can be traced to one of the system field-replaceable units (FRUs), which include the following:

•xDSL Transmission Unit—central office (xTU-C) line cards

–Quad-port flexi ATU-C line card (4xflexi)

–Quad-port STU-C line card (4xSDSL)

–Octal-port DMT ATU-C line card (8xDMT)

–Octal-port single-pair high-speed digital subscriber line, also known as symmetric high bit rate digital subscriber loop (G.SHDSL) line card (8xG.SHDSL)

–Octal-port ITU-C line card (8xISDL)

•Second generation network interface (NI-2) cards

–DS3+T1/E1 IMA

–DS3/2DS3

–OC-3c/2DS3

–OC-3c/OC-3c

•I/O cards

–DS3/2DS3+8xT1 IMA

–DS3/2DS3

•Power entry module (PEM)

•Blower tray

•Air filter

Warning Only trained and qualified personnel should be allowed to install, replace, or service this equipment.

This chapter consists of three major sections:

• System-Wide Problems: Troubleshooting problems that affect the entire system.

• FRU-Specific Problems: Troubleshooting faults that affect subsystems.

• Alarms: Descriptions of alarms that can signal problems or help with troubleshooting.

5.1 Hot-Swappable FRUs

All Cisco 6160 FRUs are hot swappable, except:

•The I/O card(s)

•The PEM, if only one PEM is operating

However, hot swapping some FRUs causes an interruption in service. See Table 5-1.

Table 5-1 Service Interruptions Caused by Replacing FRUs
FRU	Does Hot Swapping Interrupt Service?	Notes
xDSL line cards	Yes	Service is interrupted only for subscribers served by that line card.
NI-2 card	Yes	Hot swapping active NI-2 cards will interrupt service for the entire system until the NI-2 card is replaced or the redundant NI-2 takes over system operations. You can hot swap standby NI-2 cards without interrupting service.
I/O card	See next column.	You must turn off system power to replace the I/O card. Note that the MAC address is stored on the I/O card.
PEM	See next column.	You can replace a PEM without interrupting service if another PEM in the chassis continues to operate normally.
Blower tray	No	—
Air filter	No	—

5.2 Basic Checks

Before using the troubleshooting tables in this chapter, make the following basic checks:

•Are the ports properly configured? Refer to these sources for configuration instructions:

–Configuration Guide for Cisco DSLAMs with NI-2

–Command Reference for Cisco DSLAMs with NI-2

–ATM Switch Router Software Configuration Guide

–ATM Switch Router Command Reference Guide

–Cisco IOS Release 12.x information on the World Wide Web http://www.cisco.com/univercd/cc/td/doc/product/dsl_prod/ios_dsl/index.htm

•Are power leads and data cables firmly connected at both ends?

•Are all cards firmly seated and locked in the chassis?

•Are the I/O card, PEMs, and blower tray properly installed and secured to the chassis?

5.3 Contacting the Cisco TAC for Help

In certain situations, the troubleshooting tables in this chapter direct you to contact the Cisco Technical Assistance Center (TAC) for help. If you have a maintenance contract or if your hardware is under warranty, call the TAC at +1 800 553 2447 (North America only), +1 408 526 7209, or visit this URL for a worldwide list of TAC regional telephone numbers:

http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml

5.4 System-Wide Problems

This section offers suggestions for problems that affect the entire Cisco 6160 system.


Symptom	Steps to Take
System fails to come up.	1. Check the POWER LED on the NI-2 card, the STATUS LEDs on the line cards, and the INPUT OK LED on each PEM. If all LEDs are off, or if the red OUT FAIL LED on either PEM is on, troubleshoot the PEMs. See the "PEM Problems" section. 2. If any green LEDs are on, the system has power. Check the STATUS LED on the NI-2 card. If the STATUS LED is off, see the "I/O Card Problems" section. 3. If the Cisco 6160 is set to boot from a remote device over the network, make sure the remote device is up, that its network connection is solid, and that the remote device contains the boot file. (The Cisco 6160 tries to boot over the network for a configured period, usually 5 to 15 minutes. If it is unable to boot over the network, it will eventually boot from bootflash.) 4. Try to establish a console connection to the Cisco 6160. If you cannot connect, see the steps in the "You cannot establish a console or Telnet connection to the system." section. 5. If you achieve a console connection: –Examine the command prompt. If the prompt says `rommon 1>`, the problem could be in Flash memory, in bootflash, in an incorrectly set boot configuration register, or in an incorrect file name in a boot system command in the startup-config file. Refer to the Configuration Guide for Cisco DSLAMs with NI-2 for information on setting and interpreting configuration registers, configuring Flash memory, and editing the startup-config file. To get out of the ROM monitor, enter dir flash:. In the resulting display, find the name of the software image. Then enter boot flash:imagename, replacing imagename with the name of the software image. The system boots from Flash. –If you see a normal Cisco IOS prompt, which usually contains the name of the system (default is `DSLAM>`), enter show oir status. If the results indicate that the card is loading software, wait a few minutes for the port to come back up.
You cannot establish a console or Telnet connection to the system.	1. For a console problem, check the terminal settings against the list of settings in the "Connect the Console Terminal" section on page 3-24. 2. For a Telnet problem: –If you are connecting to the Cisco 6160 through the Ethernet interface, check the configuration of your LAN for both the Cisco 6160 and the Telnet source. –If you are connecting to the Cisco 6160 through an ATM interface, make sure that PVCs are set up between the two devices and that the map-list is correctly configured. Enter the command show running-config to display this information. –If the Telnet source and the Cisco 6160 are on different networks, make sure static routes are configured at both ends so that the two devices can communicate. To check, use the ping command to ping each device from the other (that is, ping the Cisco 6160 from the Telnet source, and ping the Telnet source from the Cisco 6160). Alternatively, ping your default gateways from each end of the connection. 3. For both console and Telnet problems: –Check the cabling and connectors between the terminal or Telnet source and the Cisco 6160. See Appendix C, "Connector and Pinout Specifications" to check pinouts. –Press the Reset button on the NI-2 card faceplate to reset the card. –If the problem persists, replace the NI-2 card.
System experiences a critical, major, or minor alarm.	1. Enter the command show facility-alarm status. Note the affected slot and port, if any, and the description of the problem. 2. If no slot number is indicated, enter show environment all and examine the results for an indication of which FRU is at fault. See the "FRU-Specific Problems" section for instructions on troubleshooting that FRU. 3. If ATM0/0, ATM0/1, ATM0/2, or ATM0/3 is indicated: –Enter a show int command for the interface (for example, show int atm0/1). Results could indicate a SONET problem (Loss of Signal, for example). See the "I/O Card Problems" section for instructions on troubleshooting the NI-2 card. –Enter show controllers commands for all trunk and subtending ports. (The ports are atm0/1, atm0/2, and, in a DS3 system, atm0/3.) For example, show controllers atm0/1. In the resulting display, check that the framing mode is set to the same value on this interface as at the other end of the connection. Also check that cell payload scrambling is on (DS3 interfaces only). 4. If a line card slot is indicated, consult the "NI-2 Card Problems" section.
A trunk or subtending port fails to come up (DS3 or OC-3c).	1. Verify that the trunk port selection is correct. 2. Check the cable connections at both ends. See the "Port Mapping Table" section on page B-1 to check pinouts. 3. To check the interface status and configuration, enter show interface atm slot#/port#. Check the following in the resulting display: –If the port Admin Status is down, enter the commands below to correct the problem, replacing slot/port ID atm 0/1 with your slot/port ID: `DSLAM> configure terminal` `Enter configuration commands, one per line.` `End with CNTL/Z.` `DSLAM(config-if)# int atm 0/1` `DSLAM(config-if)# no shutdown` `DSLAM(config-if)#` –If the port IF Status is down, check for disconnected or faulty cables. (Optical cables connect to the NI-2 card; DS3 coaxial cables connect to the I/O card on the back of the chassis.) –If the Line Protocol is down, the line protocol software processes might have determined that the line is unusable. Try swapping the cable. Another possibility is that clocking might be misconfigured, or the clocking source might have failed. –Check the CRC field. The presence of many CRC errors but not many collisions is an indication of excessive noise. If the number is too high (greater than 0.5 to 2 percent of total traffic on the interface), check the cables to determine if any are damaged. If you need more information on interface configuration, refer to the Configuration Guide for Cisco DSLAMs with NI-2 and the ATM Switch Router Software Configuration Guide. 4. For a DS3 interface: enter show controllers atmslot#/port#. Check the following in the resulting display: –Framing mode must be the same at both ends of the connection. –Cell payload scrambling must be on at both ends of the connection. 5. Check the status and configuration of the interface at the far end. 6. If you need to run a loopback test, do the following: –In interface configuration mode, enter loopback diagnostic (or loopback line). –Set the external test equipment to loop data through the Cisco 6160 port. –Obtain loopback results from your external test equipment. –Enter no loopback diagnostic (or no loopback line) to take the port out of loopback mode. 7. In interface configuration mode, reset the trunk port by executing the shutdown command followed by the no shutdown command. 8. Replace the NI-2 card. 9. If the problem with an DS3 interface persists, troubleshoot the I/O card. See the "I/O Card Problems" section.
A trunk or subtending port fails to come up (T1/E1).	1. Verify that the trunk port selection is correct. 2. Check the cable connections at both ends. See the "Port Mapping Table" section on page B-1 to check pinouts. 3. To check the interface status and configuration, enter show interface atm slot#/port#. Check the following information in the resulting display: –If the port Admin Status is down, enter the commands below to correct the problem, replacing slot/port ID atm 0/2 with your slot/port ID: `DSLAM> configure terminal` `Enter configuration commands, one per line.` `End with CNTL/Z.` `DSLAM(config-if)# int atm 0/2` `DSLAM(config-if)# no shutdown` `DSLAM(config-if)#` –If the port IF Status is down, check for disconnected or faulty wire. T1/E1 twisted pair wire connects to the I/O card. –If the Line Protocol is down, the line protocol software processes might have determined that the line is unusable; try swapping the cable. Another possibility is that clocking might be misconfigured, or the clocking source might have failed. –Check the CRC field. The presence of many CRC errors but not many collisions is an indication of excessive noise. If the number is too high (greater than 0.5 to 2 percent of total traffic on the interface), check the cables to determine if any are damaged. If you need more information on interface configuration, refer to the Configuration Guide for Cisco DSLAMs with NI-2 and the ATM Switch Router Software Configuration Guide. 4. Enter show controllers atmslot#/port#. Check the following information in the resulting display: –Framing mode must be the same at both ends of the connection. –Line coding must be the same at both ends of the connection (T1/E1). –Cell payload scrambling must be on at both ends of the connection. 5. Check the status and configuration of the interface at the far end. 6. If you need to run a loopback test to troubleshoot the T1/E1 links, do the following: –In interface configuration mode, enter loopback diagnostic (or loopback line). –Set the external test equipment to loop data through the Cisco 6015 port. –Obtain loopback results from your external test equipment. –Enter no loopback diagnostic (or no loopback line) to take the port out of loopback mode. 7. In interface configuration mode, reset the trunk port by executing the shutdown command followed by the no shutdown command. 8. Replace the NI-2 card. 9. If the problem with a T1/E1 interface persists, troubleshoot the I/O card. See the "I/O Card Problems" section.
A trunk or subtending port fails to come up (T1/E1 IMA).	1. Before checking IMA interface problems, troubleshoot the T1/E1 links that are configured in the IMA group, as described in the previous section. Verify that the links are up. 2. Check an IMA interface status with show ima interface atm0/imaX command, where X is the IMA interface ID. Look at the following items in the resulting display: –MinNumLinks (minimum number of links) should be equal to or below the number of links you have configured in your IMA group. You can change the minimum number of links parameter from 1 to 8 in the IMA group. –NeTxClkMode (near end transmit clock mode) should match the FeTxClkMode (far end transmit clock mode), which will be either ctc or itc. Both sides of the IMA link must use the same IMA clock mode. –DiffDelayMaxObs (maximum observed differential delay) should be less than DiffDelayMax (maximum differential delay configured for the group). If the differential delay observed is more than the maximum allowed, one or more links will not be allowed in the IMA group. Configure more allowable differential delay (up to 69 ms in T1, 55 ms in E1) using T1/E1 lines with less differential delay, or reconfigure similarly-delayed lines into IMA groups if possible. –The IMA Link Information table shows the interfaces configured in your IMA group, the near end and far end receive state, and any IMA alarms received at the near end. The NeRxState and FeRxState must both be active for each line to be active in the IMA group. The whole IMA group will be up when the number of active links on both sides are equal to or greater than the minimum number of configured links (MinNumLinks) for that IMA group. 3. It can take several seconds for an IMA group to synchronize between two IMA systems. Wait several minutes after configuring or reconfiguring a group to allow the synchronization to occur. 4. If synchronization does not occur in any one link or in the group after several minutes, or links cannot be successfully added to an active group, and the above IMA parameters have been checked and are found to be correct, clear the interface with a clear interface atm0/imaX command. This will reset the IMA group and start the IMA synchronization process again with all links in the group. 5. To check the interface status and configuration, enter show interface atm slot#/imagroup#. Check the following information in the resulting display: –If the port Admin Status is down, enter the commands below to correct the problem, replacing slot/port ID atm0/ima2 with your slot/port ID: `DSLAM> configure terminal` `Enter configuration commands, one per line.` `End with CNTL/Z.` `DSLAM(config-if)# int atm0/ima2` `DSLAM(config-if)# no shutdown` `DSLAM(config-if)#` If you need more information on interface configuration, refer to the Configuration Guide for Cisco DSLAMs with NI-2 and the ATM Switch Router Software Configuration Guide.
	6. In interface configuration mode, reset the port by executing the shutdown command followed by the no shutdown command. 7. Replace the NI-2 card. 8. If the problem with a T1/E1 interface persists, troubleshoot the I/O card. See the "I/O Card Problems" section.
System overheats	Troubleshoot the blower tray. See the "Blower Tray and Air Filter Problems" section.
System experiences a clocking problem	Troubleshoot the NI-2 card. See the "NI-2 Card Problems" section.
System experiences a power problem	Troubleshoot the PEMs. See the "Alarms" section.

5.5 FRU-Specific Problems

The following sections describe symptoms that might occur and the steps that you need to take if you experience problems with any Cisco 6160 FRU.

5.5.1 xTU-C Line Card Problems

Use the following table to diagnose and troubleshoot any problems with the xTU-C line cards.

Note If you need to remove or replace an xTU-C line card, see Chapter 6, "Upgrading and Maintaining the Cisco 6160 System" for instructions.


Symptom	Steps to Take
All LEDs are off.	1. Check the POWER LED on the NI-2 card and the INPUT OK LEDs on the PEMs. If none of the LEDS are lit, or if the red OUT FAIL LED on either PEM is lit, troubleshoot the PEM—see the "Alarms" section. 2. If the NI-2 card POWER LED is off, remove the card from its slot and check for bent or broken pins on both the card and the backplane. If you find damaged pins on the card, replace it. If you find damaged pins on the backplane, contact the Cisco TAC.
All ports on a card fail to come up (modems do not train). STATUS LED might be red, indicating that the line card failed to boot or failed its power-on self test.	1. Enter the show ipc nodes command to find out whether there is a communication problem between the line card and the NI-2 card. There should be an entry in the resulting display for each line card ("SMB IP Slot n") and for the NI-2 card ("IPC Master"). If there is an entry for each card, go to Step 4. 2. If one or more cards are not listed, enter show oir status. If the resulting display indicates that the card in question is loading new code, wait a few minutes and reenter the command. (Usually, 2 to 3 minutes is long enough to wait.) The card status should change to `running`. 3. Enter the command show dsl int atm slot#/port#. If the status says `Microcode downloading`, wait 10 minutes. When the download is complete, the card reboots and comes up normally. 4. Check the ALARM LEDs on the NI-2 card or enter show facility-alarm status to determine the alarm status of the system. If any alarms are indicated, see the "System-Wide Problems" section for instructions on how to troubleshoot alarms. 5. Enter the command show dsl status and examine the results to ensure that the card is configured to be in its current slot. (In the Names column, the slot number appears as part of the port ID. For example, in ATM19/2, the slot number is 19. Nothing is displayed for slots that are not configured.) If necessary, use the slot command to update the configuration, or move the card to the correct slot. If the line card is a 4xflexi, it will not function unless you use the slot command to configure either CAP or DMT operation. 6. Reset the line card by disconnecting it from the backplane and reseating it in its slot. 7. Install the card in another slot. 8. If the problem persists, replace the line card.
Port fails to come up (modems do not train), or port LED flashes continuously.	1. Enter the command show dsl int atm slot#/port# to display the port configuration. Check the display to ensure that the port is properly provisioned. Make sure the port is configured to be running ("no shutdown" or IOS admin state = up). Also check the line status; if this reads "No CPE detected," troubleshoot the CPE device and the loop as described in the "You suspect a problem with the CPE or the subscriber loop." section. 2. In interface configuration mode, reset the port by executing the shutdown command followed by the no shutdown command. 3. Connect the subscriber to another port on the Cisco 6160. If the modems do not train, troubleshoot the CPE device and the loop as described in the "You suspect a problem with the CPE or the subscriber loop." section. If the modems train, go to the next step to troubleshoot the port that failed. 4. To test the modem on the line card, enter dsl test atmslot#/port# self. Results are displayed automatically after a few seconds. If the modem fails the test, replace the line card.
You suspect a problem with the CPE or the subscriber loop.	1. Disconnect the local loop and replace it with a test setup that consists of a modem or CPE that is known to work, and a few thousand feet of wire. If the modems train over the shorter distance, the problem lies in the local loop or in the CPE. See the steps that follow. If the modems do not train, the port is probably at fault. Replace the line card. 2. Make sure that the CPE at the subscriber site is powered up. 3. Make sure that the CPE is compatible with the Cisco 6160, and that the software version running on the CPE is compatible with the software version running on the Cisco 6160. 4. Power cycle the CPE. When you turn the power on, the WAN Link LED on the CPE should blink. If the LED does not blink, check the CPE configuration—if the interface is shut down, bring it up. 5. Perform a continuity check to find out if the cabling to the CPE is connected and intact. For example, if there is a phone on the line, check for a dial tone. 6. If there is no POTS, check DC resistance by shorting tip and ring at the remote end. 7. Make sure there are no load coils on the local loop. 8. Check the length of the local loop; is it too long? The maximum length range is 15,000 to 25,000 feet (4572 to 7620 meters). Within that range, wire gauge, crosstalk, and multiple bridge taps reduce the distance over which the modems can train. 9. Check the length of the local loop; is it too short? DMT modems train best over loops of several thousand feet. In a test situation with a loop just a few feet in length, the modems might fail to train. Add wire to the loop. 10. Replace the CPE.
Modems train at a low bit rate, modems retrain continuously, or the line experiences too many errors. Note These steps apply to xTU-C line cards in DMT mode.	1. Enter the command show dsl int atmslot#/port# to display the port configuration. Check the display to ensure that the port is properly provisioned. Look in particular for these statistics: –Attenuation: typically this is 20 to 50 dB. If the attenuation value is higher than 50 dB, it might be depressing the bit rate. Repair or replace the cables and connectors in the loop. –SNR margin: 3 to 6 dB is optimum. Use the dmt margin command to adjust SNR margin. –Correction ratio: under DSL Statistics, look at the Received Superframes and Corrected Superframes values. A ratio of more than 1 corrected superframe for every 10 superframes received is too high. One or more of these adjustments might correct the problem: increase the SNR margin using the dmt margin command; increase error correction using the dmt check-bytes command; or increase interleaving using the dmt interleaving-delay command. –Errored seconds: a rate of 10 to 20 errored seconds per minute or more is likely to cause retraining. (1 or 2 errored seconds every 15 minutes is a good rate.) See the steps that follow on crosstalk and impulse noise for suggestions on how to compensate. –CRC errors: normal rates vary system by system. If the CRC error rate is higher than usual, it might cause excessive retraining. See the steps that follow on crosstalk and impulse noise for suggestions on how to compensate. 2. Crosstalk is caused by interference between services in adjacent cables. It affects random bits rather than chunks of data; upstream and downstream traffic can be affected differently. If crosstalk is increasing the bit error rate (BER), you can compensate in several ways: –Reduce the bit rate using the dmt bitrate command. (See the Command Reference for Cisco DSLAMs with NI-2 for information on the dmt bitrate command.) –Increase the SNR margin using the dmt margin command. (See the Command Reference for Cisco DSLAMs with NI-2 for information on the dmt margin command.) –Turn on or increase error correction. Use the dmt check-bytes command. If the codeword size is not set to auto, you might need to use the dmt codeword-size command to adjust the codeword setting. (See the Command Reference for Cisco DSLAMs with NI-2 for information on these commands.) 3. If you experience impulse noise or clipping, both of which affect chunks of data rather than random bits, you can compensate by turning on or increasing interleaving and error correction. (However, note that this approach adds delay.) Use the dmt interleaving-delay command for interleaving. Use the dmt encoding-trellis command or the dmt check-bytes command for error correction. If you use dmt check-bytes and if the codeword size is not set to auto, you might need to use the dmt codeword-size command to adjust the codeword setting. 4. If errors or retraining occur while the line is ringing, use the dmt interleaving-delay command to turn on interleaving. 5. In interface configuration mode, reset the port by executing the shutdown command followed by the no shutdown command.
	6. Connect the customer to a different port on the Cisco 6160. If the modems train, replace the line card with the faulty port. 7. Troubleshoot the CPE device and the loop as described above under the symptom "You suspect a problem with the CPE or the subscriber loop." section. 8. Replace the line card.
Card cannot be fully inserted into its slot.	1. Remove the card and reinsert it, pushing firmly on both the top and the bottom of the faceplate. (The card might jam in the slot if you apply pressure to the top only.) If the chassis is full, you might need to push sideways to insert the card. 2. Inspect connectors on both the card and the backplane, looking for obstructions, bent pins, or other damage. If you find damage to a line card connector, replace the line card. If you find damage to a backplane connector, contact the Cisco TAC.
Card experiences problems in one slot but operates normally in another.	There might be a fault in your backplane. Contact the Cisco TAC.

5.5.2 NI-2 Card Problems

Use the following table to diagnose and troubleshoot any problems with the NI-2 cards.

Note If you need to remove or replace an NI-2 card, see Chapter 6, "Upgrading and Maintaining the Cisco 6160 System" for instructions.


Symptom	Steps to Take
POWER LED is off.	1. Check the STATUS LEDs on the line cards and the INPUT OK LED on each PEM. If all LEDs are off, or if the red OUT FAIL LED on either PEM is on, troubleshoot the PEMs—see the "PEM Problems" section. 2. If the line card STATUS LEDs and the PEM INPUT OK LEDs are lit, remove the NI-2 card from its slot and check for bent or broken pins on both the card and the backplane. If you find damaged pins on the card, replace it. If you find damaged pins on the backplane, contact the Cisco TAC.
STATUS LED is off, indicating that the NI-2 failed to boot or failed its power-on self test.	Press the Reset button on the NI-2. If the problem persists, replace the card.
CRITICAL LED, MAJOR LED, or MINOR LED is on.	See the "System-Wide Problems" section.
A trunk or subtending port fails to come up.	See the "System-Wide Problems" section.
NI-2 cannot be fully inserted into its slot.	Inspect connectors on both the card and the backplane, looking for obstructions, bent pins, or other damage. If you find damage to a line card connector, replace the line card. If you find damage to a backplane connector, contact the Cisco TAC.

5.5.3 NI-2 Redundancy Problems

Table 5-2 provides information about symptoms and corrective actions related to NI-2 card cold redundancy problems.

Note If you need to remove or replace an NI-2 card, see Chapter 6, "Upgrading and Maintaining the Cisco 6160 System" for instructions.

Table 5-2 Troubleshooting NI-2 Card Cold Redundancy Problems
Symptom	Steps to Take
Both NI-2 cards go active.	Ensure that both NI-2 cards are Revision AO or later.
DS3 traffic interrupted.	1. Ensure that both NI-2 cards are Revision AO or later. 2. Ensure that the correct boot image is installed on the NI-2 cards. Refer to the Upgrading DSLAMs for NI-2 Card and APS Link Redundancy document for instructions.
Line card communication is interrupted.	1. Ensure that both NI-2 cards are Revision AO or later. 2. Ensure that the correct boot image is installed on the NI-2 cards. Refer to the Upgrading DSLAMs for NI-2 Card and APS Link Redundancy document for instructions.
Transient environmental alarms occur.	Ensure that the correct boot image is installed on the NI-2 cards. Refer to the Upgrading DSLAMs for NI-2 Card and APS Link Redundancy document for instructions.
TFTP boot fails.	The NI-2 card is not configured to boot from Flash. Refer to the Upgrading DSLAMs for NI-2 Card and APS Link Redundancy document for instructions.
Line cards reboot or reset upon switchover.	The software versions on the primary and secondary NI-2 cards might be mismatched. Use the Cisco IOS command show version to determine the IOS release currently installed on each of the NI-2 cards. Refer to the Upgrading DSLAMs for NI-2 Card and APS Link Redundancy document for more information on upgrading software images.
Redundant NI2 will not become active	Be sure that you have enabled the command no auto-sync bootflash in the redundancy configuration mode if you are using OC3 NI2's with different product numbers (for example, an NI2-155SM-155SM or NI2-155MM-155MM with an NI2-155SM-155SM2 or NI2-155MM-155MM2). Different types of NI2's require different bootflash image types to operate properly.

5.5.4 I/O Card Problems

Use the following table to diagnose and troubleshoot any problems with the I/O cards.

Note If you need to remove or replace the I/O card, see Chapter 6, "Upgrading and Maintaining the Cisco 6160 System" for instructions.


Symptom	Steps to Take
A trunk port fails to come up.	See the "System-Wide Problems" section.
A subtending port fails to come up.	See the "System-Wide Problems" section.
An alarm relay or BITS circuit fails.	1. Check the connection at the wire wrap connector on the I/O card. Refer to the pin assignments of the wire wrap connector. 2. If the problem involves the circuit breaker trip alarm, check the connection at the small green terminal block behind each PEM. 3. Check the connection at the other end of the circuit. 4. For a BITS problem, troubleshoot the T1 line at the other end of the circuit. 5. Replace the I/O card (except in the case of a circuit breaker alarm problem; the I/O card does not provide the circuit breaker trip alarm).
This message appears on the console or in the system log file: `00:00:28:%C6100-4-COOKIE: Corrupt or missing MAC address cookie using random base 007e.eb7d.e700`	1. Make sure the software release running on your system is compatible with the hardware. The Cisco 6160 requires Cisco IOS Release 12.1(1)DA or greater. 2. Enter the command show hardware chassis. In the resulting display, look for data from the I/O card EEPROM. If the system can read the contents of the EEPROM, the I/O card is likely to be healthy. 3. If the I/O card is missing, install it. 4. If the I/O card is present, reseat it. First, turn off system power. Use a screwdriver to unfasten the screws that secure the I/O card cover, and pull the card off the connectors. Push it firmly back into place and tighten the screws. 5. If the problem persists, replace the I/O card.

5.5.5 PEM Problems

Use the following table to diagnose and troubleshoot any problems with the PEM.

Note If you need to remove or replace a PEM, see Chapter 6, "Upgrading and Maintaining the Cisco 6160 System" for instructions.


Symptom	Steps to Take
INPUT OK LED on PEM fails to go on.	1. Make sure the circuit breaker on the PEM is turned to the ON (1) position. 2. Make sure the PEM is properly seated in the chassis and screwed in place. 3. Make sure the power leads are properly connected to power connectors on the PEM. (If the connections are loose or their polarity is reversed on the DC PEM, the chassis does not receive power.) 4. Check the power cable for breaks, cracked insulation, loose lugs, and signs of wear. Replace the power cable if it is not in good condition. 5. Check the power source. 6. Turn the circuit breaker on the PEM to the OFF (0) position, and move it to the other PEM slot. If the PEM still fails, replace it.
OUT FAIL LED on PEM goes on.	1. Make sure the PEM is fully inserted into its slot and firmly screwed in place. 2. Replace the PEM. 3. If the problem persists, there might be a fault in your backplane. Contact the Cisco TAC.
Circuit breaker on PEM trips.	A tripped circuit breaker indicates a serious overcurrent situation. The probable cause is a faulty backplane, a faulty power connector on the backplane, or a faulty PEM. Contact the Cisco TAC for assistance.
PEM has problems in one slot but operates normally in the other.	There might be a fault in your backplane. Contact the Cisco TAC.

5.5.6 Blower Tray and Air Filter Problems

Use the following table to diagnose and troubleshoot any problems with the blower tray and air filter.

Note If you need to remove or replace the blower tray or air filter as part of the troubleshooting steps below, see Chapter 6, "Upgrading and Maintaining the Cisco 6160 System" for instructions.


Symptom	Steps to Take
Blowers do not run.	1. Make sure the blower trays are fully inserted into the chassis and screwed in place. 2. Replace the blower tray.
Blowers run, but the system overheats.	1. Make sure that the air intake vents at the bottom front of the chassis and the exhaust vents on the top of the chassis are free of blockages, as explained in the "Ventilation" section. 2. Make sure that the ambient temperature and other environmental factors affecting the system are within the ranges specified in the "Environmental Requirements" section. 3. Make sure that all cards, blank faceplates, and covers are in place. The cooling system cannot operate effectively unless the chassis is fully enclosed. 4. Check the LED on the blower tray. If the LEDs show a problem, replace the indicated blower tray. 5. Check the air filter at the bottom of the blower tray, and if necessary clean or replace it. See the "Cleaning or Replacing the Air Filter" section on page 6-37 for instructions on cleaning or replacing the air filter. 6. Reduce the ambient temperature.
Air filter is damaged, dirty, or clogged.	See Chapter 6, "Upgrading and Maintaining the Cisco 6160 System," for instructions on cleaning or replacing air filters.

5.6 Alarms

Table 5-3 through 5-12 describe alarms for the Cisco 6160. In each table, the text in the Alarm column is the text that appears in the description field of the alarm message. Alarm messages appear on the console screen as the alarms occur; to see a list of current alarms, enter show facility-alarm status.

Refer to the problem tables in this chapter for more detailed troubleshooting instructions.

Table 5-3 Chassis Alarm
Alarm	Severity	Description
Chassis temperature too high	Major	A temperature too high for safe operation of the chassis has been detected. (Temperature is measured on the NI-2 card.)

Table 5-4 Card Slot Alarms
Alarm	Severity	Description
Provisioned slot is empty	Major	The line card in this slot does not match the type configured for this slot.
Line card not equal to provisioning	Minor	This slot is configured for a line card, but no line card is present.
Invalid line card for this slot	Minor	The line card detected in this slot cannot operate in this slot or is incompatible with the system configuration.
Line card was detected	Info	A line card is present in the slot and its software was successfully loaded. This alarm appears for all line cards when they are operating normally.
Flexi ATU-C line card not provisioned	Info	You must use the slot command to configure a 4xflexi as either CAP or DMT.

The source of the DS3 alarms in Table 5-5 is one of the following interfaces: ATM0/1 (the trunk), ATM0/2 (subtending interface), or ATM0/2 (subtending interface).

Table 5-5 DS3 Network Interface Alarms
Alarm	Severity	Description
Loss of Cell Delineation	Critical	The DS3 line is experiencing a loss of cell delineation.
RAI Received	Critical	The DS3 line is receiving a remote alarm indication.
Yellow Alarm Received	Critical	The DS3 line is receiving a yellow alarm, indicating that another device has detected a failure that might be in this device.
AIS Received	Critical	The DS3 line is receiving an alarm indication signal.
OOF Received	Critical	The DS3 line has detected an out of frame condition.
LOS Detected	Critical	The DS3 line has detected loss of signal at the framer.
PLCP LOF Detected	Critical	The DS3 line has detected a physical layer convergence procedure loss of frame error.

The source of the OC-3c alarms in Table 5-6 is one of the following interfaces: ATM0/1 (the trunk) or ATM0/2 (the subtending interface).

Table 5-6 OC-3c Network Interface Alarms
Alarm	Severity	Description
Loss of Cell Delineation	Critical	Loss of cell delineation on a SONET line.
Path RDI Received	Critical	Path remote defect indication was received on a SONET line. This is equivalent to path FERF¹.
Path AIS Received	Critical	Path alarm indication signal was received on a SONET line.
Loss of Pointer	Critical	Loss of pointer condition on a SONET line.
Line RDI	Critical	Line remote defect indication received on a SONET line. This is equivalent to line FERF.
Line AIS Received	Critical	Line alarm indication signal received on a SONET line.
Loss of Frame	Critical	LOF condition on a SONET line.
Loss of Signal	Critical	LOS detected on the SONET line.
Signal Label Mismatch	Minor	Incorrect payload type signal label mismatch on a SONET line.

¹FERF = far end receive failure

The source of the T1/E1 network interface alarms in Table 5-7 is one of the following interfaces: ATM0/2 through ATM0/9.

Table 5-7 T1/E1 Network Interface Alarms
Alarm	Severity	Description
LOS Detected	Critical	The T1/E1 line has detected loss of signal at the framer.
AIS Received	Critical	The T1/E1 line is receiving an alarm indication signal.
RAI Received	Critical	The T1/E1 line is receiving a remote alarm indication.
LOF Detected (red alarm)	Critical	The T1/E1 line has detected a loss of frame error.
LCD	Critical	The T1/E1 line is experiencing a loss of cell delineation.

The source of the IMA link alarms in Table 5-8 is one of the following interfaces: ATM0/ima0 through ATM0/ima3.

Table 5-8 IMA Link Network Interface Alarms
Alarm	Severity	Description
LOS Detected	Critical	The IMA link has detected loss of signal at the framer.
LOF Detected	Critical	The IMA link has detected a physical layer convergence procedure loss of frame error.
AIS Received	Critical	The IMA link is receiving an alarm indication signal.
RAI Received	Critical	The IMA link is receiving a remote alarm indication.
LCD	Critical	The IMA link is experiencing a loss of cell delineation.
LIF¹	Critical	The IMA link is experiencing an LIF defect at the near end.
LODS²	Critical	The link differential delay between the link and the other links in the group is over the tolerable value for the link differential delay.
Rx Failed	Critical	A persistent detection of a defect at the receiver. The criteria for entering the condition are implementation specific.
Tx Link Misconnected	Critical	The Tx link is not connected to the same far end IMA unit as the other Tx links in the group. The detection is implementation specific.
Rx Link Misconnected	Critical	The Rx link is not connected to the same far end IMA unit as the other Rx links in the group. The detection is implementation specific.
Remote Defect Indication-IMA	Critical	One of the available remote defect indicators (including IMA link specific defect) is in the link-related "Link Information" field.
Rx Fault	Critical	Implementation specific Rx fault declared at the near end.
Tx Fault	Critical	Implementation specific Tx fault declared at the near end.
Tx Link Unusable—FE³	Critical	The far end is reporting that Tx is unusable.
Rx Link Unusable—FE	Critical	The far end is reporting that Rx is unusable.

¹LIF = loss of IMA frame

²LODS = link out of delay sync

³FE = far end

Table 5-9 Redundancy Alarms
Alarm	Severity	Description
Active/Standby NI2 type mismatch	Major	The NI-2 cards installed in slot 10 and slot 11 are different types; for example, one is an OC-3c/OC-3c NI-2 card and the other is an OC-3c/2DS3 NI-2 card.
Standby NI2 missing	Major	The secondary NI-2 card slot has been provisioned for a standby NI-2 card, but the card is not installed in the chassis.
Redundancy process failed	Major	An internal redundancy software process has failed and redundancy may not be available.
Peer APS mode mismatch	Minor	The OC-3 trunk is connected to a device not configured for nonrevertive, unidirectional APS.
Both units ACTIVE	Info	Both the primary and the secondary NI-2 cards have come up in an active state.

Table 5-10 IOS Controller Alarms
Alarm	Severity	Description
Loss of active clock sync	Major	Loss of timing reference. The configured clock source is not available, so the system is using its internal clock.
BITS clock failure	Major	BITS clock failure (LOS or AIS). The configured clock source is not available, so the system is using its internal clock.

Table 5-11 Power Alarms
Alarm	Severity	Description
Fan voltage out of range	Major	The voltage going to the blowers from one or both PEMs is outside the acceptable range.
Power supply fault	Major	One or both PEMs are reporting a fault caused by high temperature, output failure, or input failure.
Temperature exceeds limit	Major	The PEM temperature is beyond the acceptable range.

Table 5-12 Blower Tray Alarms
Alarm	Severity	Description
Blower tray not detected or missing	Major	The blower tray has been removed from the chassis or is missing.
Multiple fan failures	Major	More than one fan in the blower tray has failed. When fans fail, the remaining fans run at full speed until the fault is corrected.
Single fan failure	Minor	One fan in the blower tray has failed. When a fan fails, the remaining fans run at full speed until the fault is corrected.

Table Of Contents