|
|
This chapter provides a very high-level view of the VCO/4K system and its environment. It will assist you in troubleshooting the VCO/4K system when you have not been presented with an error message documented in the Cisco VCO/4K System Messages or when the nature of the problem does not fit one of the scenarios presented in "Problem Scenarios."
The primary purpose of this chapter is to isolate the problem to a non-VCO/4K system category (i.e., the customer or telco world external to the VCO/4K) or to a major component of the VCO/4K system (VCO subsystem, SS7 subsystem, host application software, a peripheral, etc.).
The emphasis is on identifying problems that result in call loss.
This chapter covers the hierarchy of possible causes for system malfunctions and the diagnostic tools available. "Diagnostic Tools" contains descriptions of the diagnostic tools available.
The VCO/4K incorporates error detection features which output messages on the bottom display lines of the system console. These messages are defined in the Cisco VCO/4K System Messages. Messages displayed are time stamped, logged to a specified storage device (based on selections in the File System Configuration screen), and sent to the system printer. The system software maintains the error log for 30 days. You can recall the log for display or selectively print it using the Maintenance menu.
The system log on the VCO/4K stores the following information:
Systems with redundant control maintain "shadow" error logs for both controllers. The administrator or technician can specify which controller's log file to display or print through the Maintenance menu. System logs include a designation as to whether the message was generated by the left or the right controller.
The system log file provides information on general alarm conditions. It contains combination messages with both ALM and FRM prefix codes to indicate alarm conditions. These messages are written to the log file only at the initial occurrence of the alarm condition; similarly, messages are generated only for the clearing of the last occurrence of the alarm. An optional periodic alarm report can be written to the log file five minutes after system initialization and at 30-minute intervals thereafter. This option is activated or deactivated from the System Feature Configuration screen (refer to the Cisco VCO/4K System Administrator's Guide).
The Cisco VCO/4K System Administrator's Guide discusses administration screens that display alarm conditions and system log file alarm messages.
If problems are detected and repaired in the early stages, system downtime is greatly minimized. System logs are also accessible through a remote maintenance terminal and modem connection.
However, a true indication of overall system performance requires a history of system performance. Record your configuration data and monitor your system's performance on an ongoing basis. Cisco Systems recommends that users keep the printed output of daily error and status logs for a month. Both the VCO subsystem and the SS7 subsystem create an error log. They provide an excellent history of performance problems and maintenance activities requiring system reinitialization.
 |
Note To assure that a continuous hard-copy record of the system error logs is always available, Cisco Systems recommends not turning off (deselecting or powering OFF) the system printer except for maintenance purposes. You can also write system log files to either floppy or hard disk for later use depending on the File System Configuration screen selections you make (refer to the Cisco VCO/4K System Administrator's Guide for more information). |
|
Note During periods of high traffic volume, performing database maintenance might not be desirable. Database maintenance can overload the Combined Controller, causing calls to be dropped or lost. |
The Alarm Condition ($F0) report notifies the host of alarms. This report provides the same level of information to the host as the System Alarms Display provides to the system administrator. Alarm codes within the report map to the same ALM alarm messages that appear on the System Alarms Display and in log file messages. Refer to the Cisco VCO/4K Standard Programming Reference or Cisco VCO/4K Extended Programming Reference for a description of the $F0 report.
Figure 3-1 presents a high-level view of VCO/4K System troubleshooting.
|
|
When performance monitoring indicates a system problem, the administrator or technician should compare the symptoms of the problem against the following possible hierarchy of causes:
The human factor is the most likely cause of a system malfunction. Failure to follow recommended procedures for installing, programming, and maintaining the system can result in problems which are sometimes difficult to trace.
The VCO/4K is a system of integrated components. Its operation depends on data entered into the system database by the administrator. The systems coupled to external CO facilities through a main distribution frame that should be carefully mapped and updated as changes are made to the system configuration.
The technical documentation set contains information about, and organizational tools for, installing and maintaining a VCO/4K system. Technicians responsible for maintaining the system should be thoroughly familiar with the documents listed in the "Related and Referenced Documentation" section
Analog and digital interface circuits connect the VCO/4K to external telecommunication environments. Problem isolation requires establishing whether the fault stems from external causes or with internal circuit cards and adapters.
The following subsections present a hierarchy of external causes of malfunctions leading to call loss.
Poor signal quality and/or loss of line/trunk service can adversely affect system operation. Line/trunk connections can also be broken or miswired (tip and ring reversal) at the MDF/Digital Cross Connect serving the system or through the MDF/Digital Cross Connect cables running to the Storage/Control I/O Module cards.
|
Note Check the jumpers on the UTC-2 and ECT/LCT circuit card for Ground Start/Ground Connect or Loop Start /Loop Connect operation. The jumper settings must correspond to database entries for the individual ports set via the Trunk Card Configuration screen. Refer to the card-specific technical description in the Cisco VCO/4K Card Technical Descriptions for jumper settings. |
Lines, trunks, and channel banks connect the VCO/4K to the local Central Office (CO) or to specialized telecommunications equipment. Problems associated with CO facilities include:
When individual CO interface circuits fail, calls are blocked from obtaining service or completing a connection to the terminating number. CO traffic reports log the loss of service.
When a block of interface circuits fails, the problem is usually the failure of a VCO/4K interface card. The exception to this general rule is the failure of a digital span, which causes the loss of up to 24 channels. A digital span can be lost at the channel bank, the digital switch, or at its interface point with the VCO/4K.
A line/trunk with poor transmission characteristics can cause intermittent problems that may be difficult to detect without using special transmission test sets. Answer supervision and in-band call setup signaling can be lost even though voice transmission is intelligible.
If the VCO/4K is implemented behind a PBX, excessive cumulative losses in signal levels going out and returning through the PBX can greatly affect the intelligibility of voice-band transmission and reception. Similar problems may occur with losses through a CO in dial-up implementations.
|
Note SLIC-2, DID-2 and UTC-2 circuit cards offer jumper-selectable output level options (0
dBm or -3 dBm) on a per port basis. The UTC-2 also supports jumper-selectable 600 and 900 impedance on a per port basis. Refer to the card-specific technical description in the Cisco VCO/4K Card Technical Descriptions for jumper settings. |
Out-of-service lines/trunks can cause a degradation in system performance because calls must wait for available lines/trunks before processing can be completed. Calls made through dial-up stations may be blocked at the CO because no dial-up trunk is available to complete the routing to the VCO/4K.
|
Note Reversed tip and ring connections prevent the VCO/4K from seizing an outgoing trunk. Tip and ring reversal on an incoming trunk causes calls to be dropped immediately. |
The best way to assure early detection of line/trunk problems is to run frequent checks of call completing trunks. Do this at the MDF/Digital Cross Connect with a handset. Dial-up service can also be checked, but the procedure is more complicated and would require progressively busying out each dial-up trunk.
Another method of checking trunk status is to use the Card Display function under the Diagnostics menu to monitor individual trunk cards. If a port on an interface card does not appear to be processing calls, use the Port Display to look at the specific circuit. Use the Set Up Paths Utility to verify voice path to a suspected port.
Permanent signal conditions inform the host that a line or trunk has not released within 30 seconds of a release by the VCO/4K system. Refer to the Cisco VCO/4K Standard Programming Reference or Cisco VCO/4K Extended Programming Reference ($D2) for information on the Permanent Signal Condition (PSC) report.
Port supervision errors, which occur when tip and ring is reversed on outgoing trunks or when there is no response to an outgoing seizure, are also reported to the host.
If the problem is on the system side of the MDF/Digital Cross Connector, check interconnecting cables, the midplane adapter, and the current status of the interface card to which the faulty line/trunk is connected. If an entire group of lines/trunks is out-of-service, suspect a bad interface card.
If the problem is on the network or direct connect station side of the MDF/Digital Cross Connect, the responsibility for repairing the circuit depends on where the system is located and who is supporting line/trunk services. The faulty circuit number should be recorded and a repair history maintained.
Problems with the host will be due to a failure of the application running on the host or a failure of the host computer.
Because the VCO/4K functions as a server to a host computer, fault isolation must also take into account the state of the host computer and its application software at the time a fault is discovered. Troubleshooting thus requires knowledge of the host computer system, the diagnostic capabilities of the application software, the error logging and diagnostic capabilities of the VCO/4K, and basic telephone network test and service procedures.
Application developers must provide service technicians with details of the diagnostic capabilities of the host computer system and its application package.
Hardware or software problems occurring at the host can manifest themselves in the following ways:
Cisco Systems encourages VCO/4K application developers to incorporate diagnostic capabilities into their application programs. Refer to the Cisco VCO/4K System Administrator's Guide, the Cisco VCO/4K Standard Programming Reference or Cisco VCO/4K Extended Programming Reference (sections on Supervision & Call Progress Tone Detection and Conferencing) for additional details on the performance monitoring features of the VCO/4K system software.
The host application must be able to generate its own error messages. This is particularly true whenever the host receives an alarm report from the VCO/4K. Such alarms are usually the result of a failure (in call processing or communications) detected by the host application software. A detailed error message should indicate why the alarm was triggered so that a service technician can quickly isolate and remedy the cause.
Troubleshooting the host computer system is covered in more detail in "SS7 Subsystem Troubleshooting."
Problems with peripheral equipment reduces your ability to access the VCO/4K system, but the likelihood of a peripheral malfunction causing the VCO/4K system to drop calls is remote. Problems can be related to the following:
The principal causes of these problems are improper installation, improper cabling, and/or loss of setup parameters (some peripherals may loose parameter settings if power is lost). The Cisco VCO/4K Installation Guide specifies the cabling and setup parameters required for interface with the VCO/4K. Users must enter peripheral operating parameters in the system database through the Peripheral Configuration utility (refer to the Cisco VCO/4K System Administrator's Guide for instructions). These parameters must match the setup parameters defined at the peripheral (refer to the OEM documentation supplied with the peripheral for setup instructions).
VDTs usually experience keyboard and monitor problems because of frequent use. Printer mechanisms wear out over time, and modems can be damaged by line surges over power or CO connections.
"Host Communications Troubleshooting" contains more detailed information on troubleshooting peripheral equipment.
Loss of input power results in failure of the VCO/4K system. Intermittent power surges and sags, as well as induced noise, can produce the following problems that lead to call loss:
For more information, see the "VCO/4K Chassis Power" section.
The causes of failure within the VCO subsystem generally fall in the following categories:
The Diagnostics Menu offers several options that allow the administrator to do the following:
For a complete description of these functions and usage instructions, refer to the Cisco VCO/4K System Administrator's Guide.
Status LEDs indicate the operational status of individual circuit cards and subsystems. Refer to the Cisco VCO/4K Card Technical Descriptions and the SS7 Supplements for the LED patterns for all VCO/4K system components.
|
Note The operational status of LEDs on peripheral and specialized telecommunications equipment varies according to manufacturer. Technicians are advised to review OEM manuals for detailed information. |
The VCO subsystem is typically the core of the troubleshooting effort on a VCO/4K system. "VCO Subsystem Troubleshooting" contains detailed procedures for troubleshooting the VCO subsystem.
The VCO subsystem generic software incorporates a variety of diagnostic tools to help isolate the possible causes of a problem. These tools include error logs, status LEDs, alarm conditions, and administrative maintenance and diagnostic routines.
The diagnostic tools supported by the generic software must be complemented by diagnostic routines incorporated into the host application software. The VCO subsystem command set includes support for the development of host-controlled diagnostics, including the ability to remove ports from service, monitor card status, and initiate alarms. Thus, the host can trigger events in the VCO/4K that can have the effect of placing portions of the system out of service. Replacing cards and performing other corrective maintenance procedures does not cure a fault caused by the host application.
Card downloads must be carefully coordinated with the Generic software. Mismatches here are a common cause of problems. See "VCO Subsystem Troubleshooting" for more information.
During initial system power up (cold reset), the software downloads are broadcast simultaneously to each card type (SPC and ICC cards do not receive a broadcast download; they receive downloads one at a time). The system is restored to operation after all downloads have been completed. If an individual downloadable circuit card is removed and replaced, it is selectively downloaded when its power-up sequence is completed before being activated.
VCO/4K systems support an alarm condition scheme consistent with the alarm requirements described in Bellcore specification OTGR: Network Maintenance: Network Element.
System-wide alarm conditions are divided into four severity levels—fatal, critical, major, and minor. Fatal alarms cause a system switchover (in redundant systems) or a system reset (in nonredundant systems).
Critical, major, and minor alarm conditions require the user to take action to resolve the problem. Recovery from a major alarm might require component replacement and a controller reset, thus placing the system out of service. Minor alarms might require software and/or hardware changes before the condition is eliminated and the alarm is reset.
The host can set two additional auxiliary alarms by sending a Set/Reset Host Alarms command ($C0 03). Refer to the Cisco VCO/4K Standard Programming Reference or Cisco VCO/4K Extended Programming Reference.
Alarm condition indicators appear:
The Alarm Arbiter Card (AAC) supports local and remote indications of system-wide problems by a combination of highly visible LEDs and external alarm contacts (when connected).
An option on the System Alarm Display allows users to disable these audible alarms. There is also a switch on the AAC card to disable the alarms.
|
Note The Audible Cutoff (Y/N) option on the System Alarms Display disables the Major Alarm LED indicator on the AAC as well as the external audible alarms. It does not clear the alarm condition. |
The Cisco VCO/4K Card Technical Descriptions describes major and minor alarm conditions for individual circuit cards and subsystems.
The following system administration screens provide indications of system alarms:
The Cisco VCO/4K System Administrator's Guide and the Cisco VCO/4K Installation Guide provide detailed procedures for booting the system from hard or floppy disk. The following subsections describe the maintenance implications of a system reset.
Resets are not required to service the Combined Controller (where the floppy drive resides) and/or the Storage/Control I/O module (where the hard drive resides), or to replace an NBC3.
An enhanced redundancy feature enables the standby controller to process the new setup redundancy information. Both the active and standby controllers consistently track all ports in a stable or setup state, as well as conference calls.
A standby controller can be serviced while the active controller maintains system operation.
|
Note To avoid an inadvertent reset or switchover between controllers, the Select switch on the AAC should be set to the active controller side—not in the AUTO position. Return the Select switch to the AUTO position after you have completed servicing one side. |
Automatic synchronization utilities copy and restore files from the active to the standby controller prior to restoring the standby controller to service. You can reboot standby controllers from hard disk or floppy disk without disrupting system operation.
Posted: Sat Sep 28 18:11:30 PDT 2002
All contents are Copyright © 1992--2002 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.
