Troubleshooting CES Connections and Network Clocking

CES Overview

Performing Basic Checks

CES Example Network

Initial Troubleshooting of CES

Checking the CES Circuit Status

Checking CES Circuit Configuration

Troubleshooting CES Using debug Commands

Troubleshooting Network Clocking

Overview of Network Clocking

Network Clock Module LEDs

Checking the Network Clock Source Configuration

Checking the CES Interface Clocking Configuration

Troubleshooting CES Connections and Network Clocking

This chapter provides troubleshooting information for connectivity problems in circuit emulation service (CES) environments and network clocking. For more information on CES, refer to the "Circuit Emulation Services and Voice over ATM" chapter in the Guide to ATM Technology , and "Configuring Circuit Emulation Services" chapter in the ATM Switch Router Software Configuration Guide .

Before you begin, make sure that all physical port connections are working correctly. For information on troubleshooting interfaces, refer to "Troubleshooting Switch Router ATM Interface Connections."

This chapter contains the following sections:

CES Overview

CES allows circuits to be transparently extended across an ATM network. CES is typically used to transport voice or video between the ATM switch router and non-ATM telephony devices, such as PBXs and TDMs, or video devices. Voice and video, unlike data traffic, are very sensitive to delay and delay variance. CES uses constant bit rate (CBR) virtual circuits (VCs), which guarantees acceptable delay and delay variation and thus satisfies the requirements of voice and video traffic.

Performing Basic Checks

This procedure outlines the steps for performing basic interface checks of the CES circuit configuration. Always check the following when a CES circuit fails to function:

For hard PVCs, do the VPI and VCI numbers match those assigned by the service provider?
For shaped VP tunnels, is the configured transmission rate within the range contracted with the service provider?

At the ATM switch router, use the following commands to check the CES configuration:

Command	Purpose
show ces circuit interface cbr card/subcard/port circuit-id	Confirms the configuration on the CES interface.
show atm vp interface atm card/subcard/port vpi	Confirms the configuration of the shaped VP tunnel.
show dcu leds	Confirms the status of the CES port LEDs.

Follow these steps to troubleshoot the CES configuration:

Step 1 Use the show ces circuit command to display the VPI and VCI configuration:

Switch# show ces circuit interface cbr 3/1/0 1

Circuit: Name CBR3/1/0:1, Circuit-state ADMIN_UP / oper-state UP Interface CBR3 
Port Clocking network-derived, aal1 Clocking Method CESIWF_AAL1_CLOCK_SYNC
Channel in use on this port: 1
Channels used by this circuit: 1
Cell-Rate: 172, Bit-Rate 64000
cas OFF, cell_header 0x100 (vci = 16)
Configured CDV 2000 usecs, Measured CDV unavailable
De-jitter: UnderFlow unavailable, OverFlow unavaliable
ErrTolerance 8, idleCircuitdetect OFF, onHookIdleCode 0x0
state: VcLoc, maxQueueDepth       81, startDequeueDepth       64
Partial Fill:       47, Structured Data Transfer 1
HardPVC       
src: CBR3/1/0 vpi 0, vci 16
  
Dst: ATM1/1/1 vpi 1, vci 101

Step 2 Check the Dst field to confirm that the VPI and VCI values match those assigned by the service provider. If not, reconfigure the CBR interface using the ces pvc command.

Step 3 Use the show atm vp command to display the connection traffic table index configuration for the shaped VP tunnel:

Switch# show atm vp interface atm 1/1/1 1 

 
Interface: ATM1/1/1, Type: oc3suni 
VPI = 1
Status: SHAPED TUNNEL
Time-since-last-status-change: 13:59:23
Connection-type: PVP 
Cast-type: point-to-point
Usage-Parameter-Control (UPC): pass
Wrr weight: 2
Number of OAM-configured connections: 0
OAM-configuration: disabled
OAM-states:  Not-applicable
Threshold Group: 1, Cells queued: 0
Rx cells: 0, Tx cells: 0
Tx Clp0:0,  Tx Clp1: 0
Rx Clp0:0,  Rx Clp1: 0
Rx Upc Violations:0, Rx cell drops:0
Rx Clp0 q full drops:0, Rx Clp1 qthresh drops:0
  
Rx connection-traffic-table-index: 10
  
Rx service-category: CBR (Constant Bit Rate)
  
Rx pcr-clp01: 4000
Rx scr-clp01: none
Rx mcr-clp01: none
Rx      cdvt: 1024 (from default for interface)
Rx       mbs: none
  
Tx connection-traffic-table-index: 10
  
Tx service-category: CBR (Constant Bit Rate)
  
Tx pcr-clp01: 4000
Tx scr-clp01: none
Tx mcr-clp01: none
Tx      cdvt: none
Tx       mbs: none

Step 4 Check the Rx service-category and Tx service-category fields for the CBR service.

Step 5 Check the Rx pcr-clp01 and Tx pcr-clp01 fields to ensure that the peak cell rate (PCR) is within the range contracted with the service provider.

Step 6 Use the show dcu leds command to display the status of the CES port LEDs:

NewLs1010# show dcu leds 
CBR3/1/0 [20]:  idle
CBR3/1/1 [21]:  idle
CBR3/1/2 [22]:  Red (loss of signal and loss of cells)
CBR3/1/3 [23]:  Red (loss of cells)

Step 7 If the port LED status is RED, do the following:

Check the cable for damage.
Check the length of the cable. It should not be more than 1000 feet or 304.8 meters long.
Check the interface configuration.

For detailed interface configuration information about CES, refer to the "Configuring Circuit Emulation Services" chapter in the ATM Switch Router Software Configuration Guide .

CES Example Network

In the example network in Figure 9-1, the ATM switch routers in the administration building and the remote sales building are each connected to a PBX and the WAN:

AdminFl1Ls1—ATM switch router with CES interface located in the administration building
SalesFl1Ls1—ATM switch router with CES interface located in the remote sales building

Figure 9-1: CES Example Network

This network example is used to describe the CES troubleshooting examples in the rest of this chapter.

For detailed configuration information about CES, refer to the "Configuring Circuit Emulation Services" chapter in the ATM Switch Router Software Configuration Guide .

Initial Troubleshooting of CES

This section describes initial troubleshooting steps that you should perform when beginning to troubleshoot a CES connection.

At the ATM switch router, use the following commands to check the CES configuration:

Command	Purpose
show ces status	Confirms the status of the CES circuits.
show ces circuit	Confirms the configuration of the CES PVCs.
show ces circuit interface cbr card/subcard/port circuit-id	Confirms the configuration on the CES interface.
show interfaces cbr card/subcard/port	Confirms the status of the CES interface.

Checking the CES Circuit Status

Use the following command to confirm that the configured CES circuit is up:

Step 1 Use the show ces status command to check the status of the CES circuit.

AdminFl1Ls1# show ces status

   Interface      IF      Admin        Port  Channels in
     Name       Status   Status        Type      use    
------------- -------- --------- ----------- -----------
     CBR3/1/0       UP        UP          T1 1-3,7,20-22,24
     CBR3/1/1     DOWN        UP          T1 1-24
     CBR3/1/2     DOWN        UP          T1 24
     CBR3/1/3       UP        UP          T1 10-13
AdminFl1Ls1#

Step 2 Check the IF and Admin Status fields to confirm that they are up.

If the interface is down, check for the following:

Disconnected or faulty cabling—Check cables.
Hardware failure—Swap hardware.

If administratively down, the interface has been administratively taken down. Use the no shutdown interface configuration command to reenable the interface.

For detailed interface configuration information about CES, refer to the "Configuring Circuit Emulation Services" chapter in the ATM Switch Router Software Configuration Guide .

Checking CES Circuit Configuration

Follow these steps to confirm that CES circuits are configured correctly:

Step 1 Use the show ces interface command to confirm the configuration of the circuit:

Switch# show ces circuit interface cbr 3/1/0 0

  
Circuit: Name CBR-PVC-A, Circuit-state ADMIN_UP / oper-state UP Interface CBR3/ 
  
Port Clocking network-derived, aal1 Clocking Method CESIWF_AAL1_CLOCK_SYNC
Channel in use on this port: 1-31
Channels used by this circuit: 1-31
Cell-Rate: 5447, Bit-Rate 2048000
cas OFF, cell_header 0x100 (vci = 16)
Configured CDV 2000 usecs, Measured CDV 1769 usecs
  
De-jitter: UnderFlow 42, OverFlow 0
ErrTolerance 8, idleCircuitdetect OFF, onHookIdleCode 0x0
state: VcAlarm, maxQueueDepth      823, startDequeueDepth      435
Partial Fill:       47, Structured Data Transfer 0
Passive SoftVC
  
Src: atm addr 47.0091.8100.0000.00e0.f75d.0401.4000.0c81.9030.10 vpi 0, vci 16
Switch#

Step 2 Check the oper-state field to confirm that it is up.

If down, verify that the aal1 Clocking Method field for each end of the circuit shows the same configuration.

Step 3 Check the Underflow and Overflow fields to confirm that the network clocking is synchronized.

Buffer overflows and underflows indicate a slight clocking difference between the devices. Buffer overflows occur when the transmitting device is faster than the receiving device; such a condition results in frame drops. Buffer underflows occur when the transmitting device is slower than the receiving device; such a condition results in frame resends. Check with your service provider to reduce the cell delay variation (CVD).

Step 4 Check the Src and Dst fields for the correct addresses.

Use the show atm status, show ces status, show atm address, and show ces address commands to confirm the source and destination address configuration.

For detailed interface configuration information about CES, refer to the "Configuring Circuit Emulation Services" chapter in the ATM Switch Router Software Configuration Guide . For detailed physical interface troubleshooting information, see the "Troubleshooting CES T1 and CES E1 Interfaces" section.

Follow these steps to confirm the CES interface configuration on the ATM switch router:

Step 1 Use the show ces circuit command to confirm the connection of the CES interfaces.

Switch# show ces circuit

Interface  Circuit  Circuit-Type     X-interface   X-vpi   X-vci Status
 CBR3/1/0    0       Passive SoftVC  ATM-P3/1/3      0      3088  UP
 CBR3/1/1    0       Passive SoftVC  ATM-P3/1/3      0      2064  UP
 CBR3/1/2    0       Active SoftVC   ATM-P3/1/3      0      1040  UP
 CBR3/1/3    0       Active SoftVC   ATM-P3/1/3      0        16  UP
Switch#

Step 2 Check the Circuit-Type field. It should contain the correct type for the circuit.

Step 3 Check the X-interface field. It should contain the correct destination interface for the circuit.

Step 4 Check the Status field. It should read "UP."

If the rest of the fields for the interface are correct but the status is "DOWN," then check the X-interface status using the show interfaces command. If the interface is administratively down, use the no shutdown command to reenable it.

If you determine that the physical interface is configured incorrectly, refer to the "Configuring Interfaces" chapter in the ATM Switch Router Software Configuration Guide .

For detailed interface configuration information about CES, refer to the "Configuring Circuit Emulation Services" chapter in the ATM Switch Router Software Configuration Guide .

Troubleshooting CES Using debug Commands

This section describes debug commands that you can use to troubleshoot CES circuits on an ATM switch router.

Use the following commands to debug CES connections on an ATM switch router:

Command	Purpose
debug ces-iwf connection	Debugs CES circuit connection events.
debug ces-iwf createloc cbr card/subcard/port [vpi] {on \| off}	Enables cell loss for debugging purposes.
debug ces-iwf dcu	Debugs CBR-DCU internal events.
debug ces-iwf internal	Debugs CES internal events.
no debug all	Turns off all debugging.

For detailed interface configuration information, refer to the "Configuring Circuit Emulation Services" chapter in the ATM Switch Router Software Configuration Guide .

Troubleshooting Network Clocking

This section describes how to troubleshoot problems with the network clocking configuration. For more information on network clocking, refer to the "Network Clock Synchronization" chapter in the Guide to ATM Technology and the "Initially Configuring the ATM Switch Router" chapter in the
ATM Switch Router Software Configuration Guide .

Overview of Network Clocking

Network clocking facilities generate or derive a clock signal and distribute it throughout a network to ensure synchronized network operation. This is important in delay-sensitive data types, such as voice and video, because these types of data must be received and transmitted at the same rate at every step, or hop, in a connection. If network synchronization is lost, data might be lost due to buffer overflow or underflow; cyclic redundancy check (CRC) errors might also occur.

Table 9-1 provides a summary of network clocking features.

Table 9-1: Network Clocking Feature Summary

Platform	Up/Down Detection	Loss of Synchronization Detection	Phase Adjustment Cutover	Stratum 3 Clock	BITS¹ Port	Clock Source Preference
Catalyst 8540 MSR with network clock module	Yes	Yes	Yes	Yes	Yes	Best
Catalyst 8510 MSR	Yes	Yes	Yes	No	No	Medium
LightStream 1010 with FC-PFQ	Yes	Yes	Yes	No	No	Medium
Catalyst 8540 MSR without network clock module	Yes	No	No	No	No	Poor
LightStream 1010 with FC-PCQ	Yes	No	No	No	No	Poor

¹BITS = Building Integrated Timing Supply

Network Clock Module LEDs

The network clock module faceplate LEDs provide status information for the BITS ports and the alarm port. The LEDs are described in Table 9-2.

Table 9-2: Network Clock Module LED Descriptions

LED	Status	Description
POWER	Green Off	The switch is powered on and the processor is functioning. The switch is powered off.
POWER	Green Off
STATUS	Green Red Orange	The clock module is the primary network clock source. The processor has crashed. The clock module is operating in standby mode.
MAJOR ALARM	Red Off	A major alarm condition has occurred. No major alarm reported.
MINOR ALARM	Red Off	A minor alarm condition has occurred. No minor alarm reported.
CRITICAL ALARM	Red Off	Not supported. No critical alarm reported.

The following are major alarm conditions:

A switchover from the primary clock source to the default clock source occurred.
A switchover from the secondary clock source to the default clock source occurred.
A loss of all references to the network clock source occurred while the network clock source was set to free running.
A route processor failure caused the network clock module to fail.
The network clock module is in holdover mode.

The following are minor alarm conditions:

A switchover from the primary clock source to the secondary clock source occurred.
A loss of a single reference to the network clock source occurred while the network clock source was set to free running.

Checking the Network Clock Source Configuration

Use the following commands to troubleshoot the network clock source configuration:

Command	Purpose
show network-clocks	Confirms the network clock configuration.
debug ports netclock	Debugs network clock events.
no debug all	Turns off all debugging.

Follow these steps to troubleshoot the network clocking configuration on an ATM switch that does not have the network clock module installed:

Step 1 The network clock type configured on the switch must match the current clock source. Use the show network-clocks command to display the network clocking configuration.

Switch# show network-clocks

  
clock configuration is NON-Revertive
  
Priority 1 clock source: ATM0/0/0(down)
Priority 2 clock source: No clock
Priority 3 clock source: No clock
Priority 4 clock source: No clock
Priority 5 clock source: System clock
  
Current clock source:System clock, priority:5

Step 2 Check the clock configuration field to confirm that the clock switchover mode is configured correctly. If it is not, use the network-clock-select command to correctly configure the clock switchover mode.

Step 3 Check the clock source fields to confirm that the desired clock sources are configured. Use the network-clock-select command to configure the clock sources. See Table 9-1 for a list of network clock source features.

Step 4 Check the status of the clock sources. If the clock source is listed as "down," check the interface status using the procedure described in "Troubleshooting Switch Router ATM Interface Connections."

Note Once the clock problem is solved, reconfigure the network clock using the network-clock-select command to make sure that the primary clock becomes the present clock source. The no shutdown command does not affect the network clock source status.

Follow these steps to troubleshoot the network clocking configuration on a switch that uses the network clock module:

Step 1 Use the show network-clocks command to display the network clocking configuration.

Switch# show network-clocks

Network clocking information:
---------------------------------------
  
Source switchover mode:    non-revertive
Netclkd state:             Active
Source selection method:   provisioned
NCLKM hardware status:     installed & usable
NCLKM status:              software enabled
  
Primary   clock source:    ATM11/1/2 Unlockable
Secondary clock source:    ATM12/0/0 one shot triggered
  
Present   clock source:    ATM12/0/0 (2) Locked

Step 2 Check the Source switchover mode field to confirm that the clock switchover mode is configured correctly. If it is not, use the network-clock-select command to correctly configure the network clocking mode.

Step 4 Check the status of the primary and secondary clock sources. The status follows the clock source name.

If the status is "Unlockable," check the network clock configuration on all the switches in the network for possible clock loops. A clock loop occurs when two switches derive clocking information from the same interface.
If the status is "one shot triggered," the clock source switchover mode is non-revertive and the primary clock source has gone down once. When the primary clock source comes back up, it does not become the present clock source. Correct this situation by manually setting the primary clock source as the present clock source, or the source switchover mode to nonrevertive, by using the network-clock-select command.
If the status is blank, compare the Primary clock source field with the Present clock source field. If the present clock source does not match the primary clock source, check the interface status using the procedures described in "Troubleshooting Switch Router ATM Interface Connections."

Note Once the clock problem is resolved, reconfigure the network clock using the network-clock-select command to make sure that the primary clock becomes the present clock source. The no shutdown command does not affect the network clock source status.

Follow these steps to troubleshoot the network clocking configuration on a switch that uses the Building Integrated Timing Supply (BITS) port on the network clock module.

Step 1 Use the debug ports netclock and show network-clocks commands to display the configuration and status of the BITS ports.

Switch# debug ports netclock

Rhino network clocks debugging is on
Switch# show network-clocks

Network clocking information:
---------------------------------------
Source switchover mode:   non-revertive
Netclkd state:            Active
Source selection method:  provisioned
NCLKM hardware status:    installed & usable
NCLKM status:             software enabled
  
Primary   clock source:   BITS 0 in T1 mode 
Secondary clock source:   ATM11/1/2  one shot triggered
Present   clock source:   ATM11/1/2 (2) Locked
  
bits 0 state             :down (32827388) 
bits 0 admin state       :up
  
bits 1 state             :down (32827388)
bits 1 admin state       :up
do_not_switch  flag      :0
other_holdover flag      :0
p_one_shot     flag      :0
s_one_shot     flag      :0
p_l_state                :Reset
s_l_state                :Reset
other_priority           :0
other_type               :0
ncdp in use              :0
Hello        tx seq no   :0
Hello        rx seq no   :0
Clock update tx seq no   :0
Clock update rx seq no   :0
Hello timer              :running:1,
Ref lock timer           :running:1, time_left:580
<information deleted>

Step 2 Check the state field for the BITS port being used. If it is down, check the cable for damage.

Step 3 Check the BITS port LED. If it is not on, check the interface mode type. The mode type is either T1 or E1. T1 is the default mode. Use the network-clock-select command to change the BITS interface mode type.

For detailed interface configuration information on network clocking, refer to the "Initially Configuring the ATM Switch Router," and Chapter 18, "Configuring Circuit Emulation Services" chapter in the ATM Switch Router Software Configuration Guide .

Checking the CES Interface Clocking Configuration

The clocking configuration of a CES interface might affect the traffic flow on the circuit. If quality of the transmission has degraded, follow these steps to troubleshoot the CES interface clocking for unstructured services:

Step 1 Check the Underflow and Overflow fields. Use the show ces interface command to display the network clocking configuration:

Switch# show ces interface cbr 3/1/0

Interface:      CBR3/1/0        Port-type:E1-120ohms-DCU
IF Status:      UP              Admin Status: UP
Channels in use on this port: 1
LineType: E1_LT        LineCoding: HDB3  LoopConfig: Payload     
SignalMode: NoSignalling   XmtClockSrc: network-derived
  
DataFormat: Structured     AAL1 Clocking Mode: Synchronous  LineLength: 330_440 
e1InternationalBits 0x3, e1NationalBits 0x1F, e1MultiFrameBits 0xB
LineState:  RcvAIS LoopbackState
Errors in the Current Interval:
  PCVs        0 LCVs        0   ESs         0   SESs        0   SEFSs        0
  UASs        0 CSSs        0   LESs        0   BESs        0   DMs          0
Errors in the last 24Hrs:
  PCVs        0 LCVs        0   ESs         0   SESs        0   SEFSs        0
  UASs        0 CSSs        0   LESs        0   BESs        0   DMs          0
Input  Counters: 0 cells, 0 bytes
Output Counters: 0 cells, 0 bytes

Step 2 Check the AAL1 Clocking Mode field. Use the ces aal1 clock command to modify the clocking mode to adaptive. Adaptive clocking does not require an external clock source. If the problem ceases when the clocking mode is changed from synchronous to SRTS, the reference clock is the problem.

Table of Contents

Troubleshooting CES Connections and Network Clocking