|
|
This chapter describes some of the tools provided for detecting and identifying network and/or equipment problems that are available to the network operator.
There are considerably more advanced tools built into the system software that are available only to Cisco Customer Service personnel. These advanced tools require in-depth knowledge or the hardware and software and are used generally to locate the less common types of system problems.
This chapter contains the following:
In a network with Cisco BPX 8600 series broadband switches, Cisco IGX 8400 series multiband switches, and Cisco IPX narrowband switches, it is recommended that at least one node be configured to transmit alarms automatically to Cisco Customer Service. illustrates the hardware configuration required for implementation. This can be a Cisco IGX 8400 series multiband switch, or a Cisco IPX narrowband switch.
When an alarm occurs on the network, the autodial modem automatically dials the specified telephone number. An auto-answer modem at Cisco Customer Service answers the call and directs it to a dedicated personal computer. The alarm is logged under the network ID (an ASCII character string) specified by the network administrator and approved by Cisco Customer Service personnel.
If the auto-answer modem at Customer Service is busy when an alarm arrives, then the autodial modem will keep dialing until the call is completed. A suggested modem is the Codex V.34 RSA 28.8 Kbaud modem. Connections to the node are detailed in the appropriate Cisco BPX 8600 Series Reference Manual or Cisco IGX 8400 Series Reference Manual.
The Cisco BPX 8600 series broadband, Cisco IGX 8400 series multiband, and Cisco IPX narrowband system software provides a variety of tools for monitoring the network and detecting and locating network and system problems. Almost all network troubleshooting can be performed from the console of the Cisco WAN Manager Network Management Station (NMS). These tools are described in general in the following paragraphs.
 |
Note For more specific information on using the various commands referenced in this section, refer to the Cisco WAN Switching Command Reference publication. For details on the Cisco WAN Manager NMS, refer to the Cisco WAN Manager Operations publication. |
The Cisco WAN Manager NMS displays a map of the network topology indicating where the nodes are located and the packet trunks that provide the connections between the nodes (refer to ). This topology map displays, in color, any network faults detected by the system.
All nodes in the network are accessible from the Cisco WAN Manager NMS and any network or system fault detected will be reported to the NMS. In addition the network operator can access any routing node in the network by using a Virtual Terminal (vt) command to create a virtual connection from the NMS console directly to the desired node. Non-routing nodes are accessed via "telnet" command on a LAN interface, and are managed via in-band SNMP. This allows the operator to enter commands and view system status as if he were locally connected to the node.
Whenever an alarm condition is detected in the network, the NMS console will sound an alarm tone, a Major Alarm or Minor Alarm notice will flash, and one or more of the icons on the topology map will change color to alert the network operator of the fault. Additionally, each node can be equipped to output a set of relay contact closures to operate customer alarm systems to notify network operators of an alarm condition in the network. See the Cisco IPX Reference, the Cisco IGX 8400 Series Reference, or the Cisco BPX 8600 Series Reference publications for further information.
A Major or Minor alarm indication after one or more of the node names will be displayed if there is an alarm anywhere in the network. This screen will identify the node(s) that detected the alarm condition. Another method of identifying the location of a network alarm is to issue a Display Networks (dspnw) command as shown in the following example:
Example: Display Network Screen
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:34 PST
NodeName Alarm Trunk Trunk Trunk
bpxa 1.1-1.1/bpxe 6.2-3.2/bpxc 5.1-6.1/bpxb
6.1-11.2/bpxd 1.2-5.1/bpxb 1.3-1.3/bpxc
5.2-12.2/bpxg
bpxb 6.1-5.1/bpxa 5.1-1.2/bpxa 6.2-3.1/bpxc
4.1-4.1/bpxc 4.2-4.2/bpxc 4.3-4.3/bpxc
5.3-1.2/bpxe
bpxc 3.2-6.2/bpxa 3.1-6.2/bpxb 2.1-14.1/bpxd
2.2-16/ipxb 2.3-14.3/bpxd 1.3-1.3/bpxa
4.1-4.1/bpxb 4.2-4.2/bpxb 4.3-4.3/bpxb
bpxd 14.1-2.1/bpxc 14.2-3/ipxa 14.3-2.3/bpxc
11.2-6.1/bpxa
bpxe 1.1-1.1/bpxa 1.2-5.3/bpxb 1.3-2.1/bpxf
bpxf Minor 2.1-1.3/bpxe
This Command: dspnw
Continue?
Next Command:
Once the location (node) where the alarm has been detected is identified, the network operator can use the Virtual Terminal feature to observe the status at that location. The Display Alarms (dspalms) command and associated screen will be used here. The following example illustrates a Minor Alarm at a node named D1.beta:
Example: Display Alarms Screen
bpxf VT YourID:1 BPX 15 9.1 Mar. 9 1997 14:36 PST
Alarm summary (Configured alarm slots: None)
Connections Failed: None
Groups Failed: None
TRK Alarms: None
Line Alarms: None
Cards Failed: None
Slots Alarmed: None
Missing Cards: None
Remote Node Alarms: None
Remote Domain Alarms: None
Interface Shelf Alarms: 1 Minor
ASM Alarms: None
Last Command: dspalms
If the Display Nodes (dspnds) or Display Network (dspnw) command screen indicates a service-affecting problem with either a network trunk or CPE service access line, there are two similar screens that will display their status. The following example illustrates the Display Trunk (dsptrks) command screen with a Major Alarm on trunk 5.2. It also indicates whether the alarm is a Red (Local) or Yellow (Remote) alarm.
Example: Display Trunk Status Screen
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:44 PST
TRK Type Current Line Alarm Status Other End
4.1 E3 Clear - OK bpxc/4.1
4.2 E3 Clear - OK bpxc/4.2
4.3 E3 Clear - OK bpxc/4.3
5.1 T3 Clear - OK bpxa/1.2
5.2 T3 Major - Loss of Sig (RED) axis31(AXIS)
5.3 T3 Clear - OK bpxe/1.2
6.1 OC3 Clear - OK bpxa/5.1
6.2 OC3 Clear - OK bpxc/3.1
Last Command: dsptrks
The following example illustrates the Display Lines (dsplns) command screen with a Major Alarm on line 12.2. It also indicates whether the line alarm is a Red (Local) or Yellow (Remote) alarm.
Example: Display Line Status Screen
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:44 PST
Line Type Current Line Alarm Status
10.1 E3 Clear - OK
10.2 E3 Clear - OK
12.1 T3 Clear - OK
12.2 T3 Major -
13.1 T3 Clear - OK
13.2 T3 Clear - OK
14.1 OC3 Clear - OK
14.2 OC3 Clear - OK
Last Command: dsplns
Network trunk and line inputs to each node are constantly monitored for error conditions such as loss of signal, bipolar errors, frame errors, etc. These trunk and line errors may not be severe enough to cause a Red or Yellow Alarm but can be observed using the Display Trunk Errors (dsptrkerrs) and Display Line Errors (dsplnerrs) command screens. The following example illustrates a typical summary trunk errors screen. Errored tens of seconds statistics are also kept and are displayed by a second trunk or circuit line screen.
Example: Display Trunk Errors Screen
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:44 PST
Total Errors
Code Rx Cell Out of Loss of Frame HCS Tx Cell Cell Cell
TRK Errors Dropped Frames Signal BitErrs Errors Dropped Errors Oofs
1.1 0 0 0 0 - 0 0 - -
1.3 0 0 0 0 - 0 0 - -
2.1 0 0 0 0 - 0 0 - -
2.2 0 0 0 0 - 0 0 - -
2.3 0 0 0 0 - 0 0 - -
3.1 - 0 0 0 - 0 0 - -
3.2 - 0 0 0 - 7 0 - -
4.1 0 0 0 0 - 0 0 - -
4.2 0 0 0 0 - 0 0 - -
4.3 0 0 0 0 - 0 0 - -
Last Command: dsptrkerrs
Next Command:
The system software automatically determines the card configuration of the node and displays the card type in each shelf slot. Each card is shipped with a model and revision number burned into Read-Only Memory to assist in tracking each individual card for inventory purposes.
Each card in a node is periodically tested in a background mode so as not to disrupt normal traffic. A short, automatic self-test is performed to identify card failures and, if the node is equipped with a redundant card set, transfer operation to the alternate card before any downtime is detected. The Display Cards (dspcds) screen, in the following example, shows what cards are equipped and their status: active, failed, or standby (for redundant card sets).
Example: Display Cards Screen
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:44 PST
FrontCard BackCard FrontCard BackCard
Type Rev Type Rev Status Type Rev Type Rev Status
1 Empty 9 Empty
2 Empty 10 ASI-E3 JA11 E3-2 BE Active
3 Empty 11 Empty
4 BNI-E3 CD03 E3-3 BE Active 12 ASI-T3 JA11 T3-2 BE Active
5 BNI-T3 CB04 T3-3 AA Active 13 ASI-T3 JX11 T3-2 BE Active
6 BNI-OC3BHA MMF-2 AH Active 14 ASI-OC3BWA MMF-2 DJ Active
7 BCC AVC LMBCC P01 Standby 15 ASM ACA LMASM P01 Active
8 BCC A0405 LMBCC AC Active
Last Command: dspcds
Next Command:
If any card is removed for any reason, including replacement, the node configuration database will remember what type of card was initially installed in each slot to minimize the possibility of installing the wrong card type in a slot.
Additionally, all node power supply voltage outputs are monitored and an alarm is generated if any output fails or if any voltage falls out of tolerance. The status of each power supply in the node is displayed on the Display Power Supply Status (dsppwr) command screen.
Each node is cooled by various fan assemblies. In the Cisco IGX 8400 series multiband switch and in the Cisco IPX narrowband switch cabinets and in the Cisco BPX 8600 Series wideband switch enclosure, there is a temperature sensor that monitors the ambient temperature inside the enclosure. If there should be a failure of one or more of the fans, causing the temperature to rise, an alarm will be generated so the fan or assembly can be replaced before the temperature exceeds the maximum allowed. The cabinet temperature can be observed in the Cisco IPX narrowband switch, the Cisco IGX 8400 series multiband switch, and the Cisco BPX 8600 series broadband switch using the dsppwr command screen as shown in the following example. In addition, the Cisco BPX 8600 series broadband switch provides a Display Assembly Modules (dspasm) screen.
Example: Display Power Supply Status Screen
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:44 PST
Power Status Cabinet Temperature
ASM Status: Active 31 87
Power voltage A/B: 0 / 48 V C 60 | | 140 F
e | | a
PSU Ins Type Rev SerNum Failure n 50 |--| 122 h
A N N/A N/A N/A N/A t | | r
B Y 240V 0C 26229 None i 40 | | 104 e
g | | n
Fan Status r 30 | | 86 h
a | | e
FAN 1 2 3 d 20 | | 68 i
3300 3240 3240 RPM e \Q--' t
Last Command: dsppwr
Next Command:
The endpoints and status of each connection that terminates on a node can be displayed using the Display Connections (dspcons) command and associated screen. This lists the connections by number and remote node. It also displays the status, OK or failed, and the Class of Service assigned. If desired, a filter can be applied so that this screen displays only certain types of connections, for example (Options: -v,-d,-a,-atfr,-g,+d,-abit,-fabit,-fail, nodename, -down,start_channel). The following is an example of a typical Display Connections (dspcons) command screen.
Example: Display Connections Screen
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:44 PST
Local Remote Remote Only Route
Channel NodeName Channel State atfr Avoid COS O
5.2.6.100 bpxb 13.2.10.1000 Failed cbr
5.2.6.101 bpxb 13.2.10.1001 Failed abr
10.1.1.1 bpxa 13.1.1.1 Ok abr 0 L
10.1.1.5 bpxb 10.1.1.5 Ok vbr
10.1.7.* bpxd 4.1.7.* Ok abr 0 L
10.1.9.* bpxa 13.1.9.* Ok vbr 0 L
10.1.10.10 bpxa 13.1.10.10 Ok cbr 0 L
10.1.11.* bpxa 13.1.11.* Ok abr 0 L
10.1.17.200 bpxa 13.1.17.201 Ok abr 0 L
10.1.23.102 bpxd 4.1.23.102 Ok abr-Grp 0 R
10.1.23.103 bpxd 4.1.23.103 Ok abr-Grp 0 R
10.1.23.104 bpxd 4.1.23.104 Ok abr-Grp 0 R
10.1.23.105 bpxd 4.1.23.105 Ok abr-Grp 0 R
This Command: dspcons -atfr
Continue?
A record of all events is kept in each node as well as in the Cisco WAN Manager NMS. This Event Log is useful in maintaining the network as it lists in chronological order, all network events, and displays the alarm/event category, major, minor, information, etc.
You can use the HP OpenView Event Browsers to display a network-wide list of events and alarms or filter the list as desired. Alternatively, the dsplog command can be used at any selected node to observe an event log for that particular node as indicated in the following example. The node's log database can generally hold events for approximately 30 days after which the most recent log entry replaces the oldest entry. An example of the dsplog command is shown in the following example:
Example: Event Log Screen for a Node
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:44 PST
Most recent log entries (most recent at top)
Class Description Date Time
Info User YourID logged in (Virtual Terminal) 12/09/95 15:33:29
Info User richard logged out (Virtual Terminal) 12/09/95 15:05:55
Info User richard logged in (Virtual Terminal) 12/09/95 15:05:32
Info User rachel logged out (Virtual Terminal) 12/09/95 14:57:42
Info User rachel logged in (Virtual Terminal) 12/09/95 14:57:17
Clear Communication Break with bpxb Cleared 12/08/95 16:31:22
Minor Communication Break with bpxb 12/08/95 16:31:12
Info User StrataCom charlie in (Local) 12/08/95 16:25:14
Info User StrataCom nancy out (Local) 12/08/95 16:09:06
Info User StrataCom evans out (Virtual Terminal) 12/08/95 15:46:15
Info User StrataCom nancy in (Virtual Terminal) 12/08/95 15:46:04
Info Clock switch to Line 5.1 of bpxb via TRK 5.1 12/08/95 15:34:59
Clear TRK 5.1 OK 12/08/95 15:34:28
This Command: dsplog
Continue?
Cisco WAN switching system software automatically assigns connections to the network trunks as they are added. If there is a choice between several trunks to the same destination, the software will attempt to balance the loading on the trunks. If an attempt to add a connection fails, it often is the result of inadequate bandwidth available for the connection.
Trunk loading on each trunk can be displayed by using the Display Trunk Loading (dspload) command, which displays the loading in both directions of transmission for each of the packet types. This screen is useful for determining the capacity of each trunk to accommodate additional frame relay traffic. The following example illustrates a typical trunk loading screen:
Example: Display Trunk Loading
bpxb VT YourID:1 BPX 15 9.1 Mar. 9 1999 14:44 PST
Trunk loads for node 'bpx1'
Units Used Available Reserved
TRK Xmt Rcv Xmt Rcv Xmt Rcv Xmt Rcv
1.1 Cell Cell 88320 88304 6688 6704 992 992
1.2 Cell Cell 63808 63808 31200 31200 992 992
1.3 Cell Cell 0 0 95008 95008 992 992
5.1 Cell Cell 273008 273008 79200 79200 992 992
5.2 Cell Cell 0 0 352208 352208 992 992
6.1 Cell Cell 0 0 352208 352208 992 992
6.2 Cell Cell 326592 326592 25616 25616 992 992
Last Command: dspload
Next Command:
There are a number of manually-initiated tests that can be performed from the Cisco WAN Manager NMS console to assist in system troubleshooting. These tests may be included in a job so they can be scheduled to run remotely at a specified time if desired.
There are several user-initiated tests that can be used to diagnose system problems. These tests are self-contained in that they do not require the use of external test equipment. They also do not require the operator to place a loopback at the far end to test both directions of transmission. These tests are listed in Table 7-1.
There are also several display commands that can be used to obtain information that may be helpful in troubleshooting system problems. These are also listed in Table 7-1.
| Command | Description |
|---|---|
Test Connection (tstcon)—frame relay | Performs a bi-directional test of the specified frame relay connection or range of connections by inserting an IPX-generated test pattern and comparing the returned pattern with the pattern transmitted. A pass or fail indication appears next to the tested connection in the Display Connections screen. |
Test Connection (tstcon)—data | Same as above except for synchronous data connections. |
Test Connection (tstcon)—voice | Same as above except for voice connections. |
Test Delay (tstdelay)—frame relay | Measures the round-trip delay over the selected frame relay connection. |
Test Port (tstport)—frame relay | Tests the operation of the selected frame relay port on the node. |
Test Port (tstport)- data | Same as above except for synchronous data ports. |
Display Connection States (dspconst) | Displays in real-time the status of all voice connections terminating at a specified node. |
Display Breakout Box (dspbob)—frame relay | Displays in real-time the status of data and control leads on selected frame relay connection. |
Display Breakout Box (dspbob)—data | Same as above for synchronous data connections. |
Display Breakout Box (dspbob)—trunk | Same as above for network subrate trunks. |
Display Buses (dspbuses) | Displays the status of the IPX system buses. |
Display Slot Errors (dspsloterrs) | Displays any data errors associated with the slots in a BPX node. |
Display Slot Alarms (dspslotalms) | Displays any alarms associated with the slots in a BPX node. |
Display Trunk Errors (dsptrkerrs) | Displays any data errors associated with the network trunks connected to a node. |
There are also various loopback paths that can be set up to help diagnose transmission problems. They rely on using external test equipment to provide the source of a test signal. The available loopback commands are listed in Table 7-2. illustrates the various loopback paths using, in this example, a frame relay card set (FRP/FRI) in an IPX node.
A local loopback path (LL) is set up in the local node at the PAD card (FRP) associated with the port or connection to be tested. A test signal applied to the input passes through the associated Interface Card (FRI), is sent to the Frame Relay PAD card (FRP) over the system bus where it is looped back towards the input. This tests the cabling and the local node processing of the signal.
| Command | Description |
|---|---|
Add Local Loopback (addloclp) —frame relay port | Adds a loopback path at the frame relay port from the transmit side back to the receive side at the local node. |
Add Local Loopback (addloclp)—frame relay connection | Does the same as above only for an individual frame relay connection. |
Add Local Loopback (addloclp)—data | Adds a loopback path at the synchronous data port from the transmit side back to the receive side at the local node. |
Add Local Loopback (addloclp) —voice | Adds a loopback path for an individual voice channel on a circuit line at the local node. |
Add Remote Loopback (addrmtlp)—frame relay port | Adds a loopback path at the frame relay port from the transmit side back to the receive side at the remote node. |
Add Remote Loopback (addrmtlp)—frame relay connection | Does the same as above only for an individual frame relay connection. |
Add Remote Loopback (addrmtlp)—data | Adds a loopback path at the synchronous data port from the transmit side back to the receive side at the remote node. |
Add Remote Loopback (addrmtlp) —voice | Adds a loopback path for an individual voice channel on a circuit line at the remote node. |
Add External Loopback (addextlp)—data | Activates a near end or far end loopback on an external device, such as a DSU, connected to a synchronous data port. |
A remote loopback path (RL) is set up in the remote node also at the PAD card (FRP). But, in this case, the signal travels over the network and through the remote node processing equipment but does not include the remote node Interface Card (FRI) or associated cabling. These components would be tested using another local loopback at the remote node.
The external loopback command finds limited use in data applications where an external data interface unit (DSU or CSU) is attached to the local node data interface card, illustrated by the SDI card in . The local node transmits the appropriate loopback codes out the circuit line towards the external device and then sets up the appropriate loopback path.
System software includes a Test Connection (tstcon) command for testing network connections. This test is initiated by the network operator from the NMS console and can be performed at any time but it momentarily interrupts traffic on the connection during the test. Testing a connection should be performed only when an alarm has been reported from the connection or during off-hours.
Test Connection tests both directions of transmission from end-to-end and displays a pass or fail indication for each connection tested. The test may be specified for a single connection or for a group of connections. The operator may specify a single connection, all connections, all connections of a particular type (voice, data, or frame relay), or a starting and ending connection number.
In addition to testing the connection, the Test Connection routine will attempt to isolate and repair any failure it detects. The controller card at the node where the Test Connection (tstcon) command is issued instructs the service card to build packets containing special test frames. These packets are sent across the network to the terminating node, which depacketizes them, repacketizes the frame, and sends them back to the originating node where the returned frame is analyzed.
If the returned test pattern is incorrect, the system goes into an automatic fault isolation mode. Controllers in the various nodes along the connection route communicate with each other over an overhead message channel separate from the normal circuits.
The test pattern continues to be transmitted and analyzed at each node along the path as it is transmitted and as it is received until the failed network element is identified. Redundant cards may be switched into operation and routing tables in associated network trunk cards may be reprogrammed in an attempt to correct the problem. If all else fails, the suspected path and/or network component is then reported to the network manager (NMS).
External devices connected to network nodes, such as bridges, routers, or sub-rate multiplexers may be accessed through the NMS Window command. This feature provides a direct command line interface to external devices from the NMS console. Depending on the capability of the external device, it is often possible to report status and alarms and to control or configure the device through an RS232 port connection.
The following example illustrates a Window display of a router connected to the local node. In this example, the window is used to initiate a ping of the router connection.
Example: NMS Window to a Local Router
Protocol [ip]:
Target IP address: 192.9.202.1
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Type escape sequence to abort. ^^
Sending 5, 100-byte ICMP Echos to 192.9.202.1, timeout is 2 seconds:
. . . . .
Success rate is 100 percent
Cisco WAN Manager collects network statistical data on the operation of the network and stores them in its database. They are available for display on the Cisco WAN Manager console in either tabular form or as bar charts. Statistics can be a useful source of information for troubleshooting problems that do not necessarily cause a major or minor alarm indication or for locating intermittent failures that may occur at random.
There are four classes of statistics:
Table 7-3 lists the statistics categories and the general nature of the statistics collected in each category. Note this is not a complete list of statistics but merely indicates some of the various conditions monitored. Refer to the Cisco WAN Manager Operations document for a complete listing.
Most statistics are collected on-demand and must be enabled by the system operator. The operator can set the collection interval, the sampling times, and the number of collection buckets to tailor the statistics for either long-term network performance evaluation or short term for network troubleshooting.
| Statistics Category | Types of Statistics |
|---|---|
Trunk statistics | Various trunk errors, bipolar violations, frame bit errors, loss of signal, etc. |
| Packet errors and out of frame |
| FastPackets and ATM cells of various types transmitted/dropped |
| Transmitted ATM cell counts |
| Received ATM cell counts |
| Cells with CLP and EFCN set |
| ATM header error counts |
| DS3 PLCP error counts |
| Bdata queue dropped cells. |
Line statistics | Various circuit line errors, bipolar violations, frame bit errors, loss of signal, etc. |
Connection statistics | Packets transmitted and received |
| Transmitted and received data bytes |
| Frame relay frames transmitted/discarded |
| Frames transmitted with FECN or BECN or DE set |
| Packets with CLP bit set dropped |
| Seconds in service |
Frame Relay Port | Frames transmitted and received |
| Bytes transmitted and received |
| Frames received with CRC or other errors |
| Frames discarded at the connection ingress |
| Frames discarded at the connection egress |
| Frames discarded at the port egress |
| LMI messages sent or dropped for various errors |
| DE frames dropped |
Posted: Tue Jan 16 11:15:30 PST 2001
All contents are Copyright © 1992--2001 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.