CPE History and Troubleshooting

This chapter describes how to use device information for troubleshooting. Among the features available for this purpose are:

Device History

This section describes the Device History feature, which provides a detailed history of significant events that occur in a device-provisioning lifecycle. This feature can be helpful for troubleshooting.

You can view device history through the API or through the administrator user interface.

•

To retrieve the history of a specific device, invoke the following API:
com.cisco.provisioning.cpe.api.ipdevice.history getHistory (DeviceID deviceID)

Table 8-1 details the specific record types supported by the BAC Device History feature:

Table 8-1 Supported Device History Records
Record Type	Description	Message Format
AddedAutomatically	Recorded when a previously unregistered device is automatically added to BAC after contacting the DPE.	time: Device record was automatically created.
AddedAutomatically	Example: `Tue Jul 11 18:41:42 EDT 2006: Device record was automatically created.`
AddedViaAPI	Recorded when a device is added to BAC by using the client API.	time: Device record was added via API by user username.
AddedViaAPI	Example: Tue Jul 11 18:41:42 EDT 2006: Device record was added via API by user "admin".
UpdatedViaAPI	Recorded when a device record stored at the RDU was modified using the client API.	time: Device record was updated reason by user username.
UpdatedViaAPI	Example: Tue Jul 11 18:41:42 EDT 2006: Device record was updated via API command "IPDevice.changeClassOfService" by user "admin".
FirstContact	Recorded when a device first contacts BAC.	time: Device made first contact with BAC.
FirstContact	Example: Tue Jul 11 18:41:42 EDT 2006: Device made first contact with BAC.
CPEDiscoveredData Updated	Recorded when a device reports new values for a device parameter, which are tracked by the RDU as discovered data. For details on discovered parameters, see Discovering CPE Parameters.	time: Recorded updates of number parameter values discovered from device.
CPEDiscoveredData Updated	Example: `Tue Jul 11 18:41:42 EDT 2006: Recorded updates of 2 parameter values discovered from device.`
ConfigGenRequested	Recorded when device instruction regeneration is initiated for a device. The resulting instructions are sent to all DPEs in the device's provisioning group.	time: Device policy instructions regeneration initiated as a result of reason by user username.
ConfigGenRequested	Example: `Tue Jul 11 18:41:42 EDT 2006: Device policy instructions regeneration initiated as a result of execution of API Command "IPDevice.performOperation" by user "admin".`
ConfigUpdated	Recorded when BAC activates a new configuration on a device.	time: Device was configured according to configuration version: configuration revision.
ConfigUpdated	Example: `Tue Jul 11 18:41:42 EDT 2006: Device was configured according to configuration version: 215438bb.`
UpdatedConfig Available	Recorded when new configuration instructions for the device have been sent from the RDU to the DPEs.	time: Configuration policy for device was updated; new version: configuration revision.
UpdatedConfig Available	Example: `Tue Jul 11 18:41:42 EDT 2006: Configuration policy for device was updated; new version: 215438bb.`
ReportedNewConfig	Recorded when a device reports using a new configuration via the ParameterKey contained in the Inform message.	time: Device reported new configuration version: configuration revision.
ReportedNewConfig	Example: `Tue Jul 11 18:41:42 EDT 2006: Device reported new configuration version: 215438bb.`
NewOperationQueued	Recorded when a new device operation is initiated via the client API or the administrator user interface and is queued by BAC for execution on a device.	time: On-connect operation operation name with ID operation ID was queued by user username.
NewOperationQueued	Example: `Tue Jul 11 18:41:42 EDT 2006: On-connect operation "SetParamValues" with ID "15e2ccd:10c5f4b2ad0:80000024" was queued by user "admin".`
OperationExecuted	Recorded when an operation initiated via the client API or the administrator user interface is successfully executed on a device.	time: Operation mode operation name with ID operation id was executed on the device.
OperationExecuted	Example: `Tue Jul 11 18:41:42 EDT 2006: On-connect operation "SetParamValues" with ID "15e2ccd:10c5f4b2ad0:80000024" was executed on the device.`
OperationRemoved	Recorded when a device operation queued within BAC is removed.	time: On-connect operation operation name with ID operation id was removed.
OperationRemoved	Example: `Tue Jul 11 18:41:42 EDT 2006: On-connect operation "GetParamValues" with ID "15e2ccd:10c5f4b2ad0:80000025" was removed.`
ReportedNewFirmware	Recorded when a device reports by using new firmware via the Inform message.	time: Device reported new firmware/software version: firmware/software version.
ReportedNewFirmware	Example: `Tue Jul 11 18:41:42 EDT 2006: Device reported new firmware/software version: 6d9bfec2.`
UpdatedFirmwareRules Available	Recorded when new firmware rules for the device have been sent from the RDU to the DPEs.	time: Firmware rules for device were updated; new version: firmware rules revision.
UpdatedFirmwareRules Available	Example: `Tue Jul 11 18:41:42 EDT 2006: Firmware rules for device were updated; new version: 6d9bfec2.`

Configuring Device History

You use the Device History feature to record significant events in a device-provisioning lifecycle. This section describes how to enable or disable this feature.

Enabling Device History

Device History is, by default, enabled. To enable or disable this feature from the API, change the SERVER_DEVICE_HISTORY_ENABLE_KEY property.

Note Every time Device History is enabled or disabled, a message is logged in audit.log. Device history is also stored in the troubleshooting log.

Step 1

Choose Configuration on the Primary Navigation bar or Main Menu page.

Step 4

On the defaults page, against Device History, click the Enabled radio button.

Note

To disable the device history feature, click the Disabled radio button. When Device History is disabled, no new updates to any existing device or details of any new device are recorded in the device history. However, existing events are retained.

Viewing Device History

Step 1

From the Devices page, locate the device whose history you want to view. You can use one of the search types for this purpose.

Step 2

Click the View Details icon (

) corresponding to the correct device.

Step 3

The Device Details page appears. Against View Device History Details, click the View Details icon.

You can retrieve device history by using the API command IPDevice.getDeviceHistory().

Configuring Device History Size

To configure the maximum number of device history entries from the administrator user interface, choose Configuration > Defaults > System Defaults. Enter a value in the field corresponding to Maximum Device History Entries. The default number of entries is 40.

To configure this number from the API, set the SERVER_MAX_DEVICE_HISTORY_SIZE property.

If you reduce the value of the Maximum Device History Entries, the existing records which exceed the limit do not get purged until a new device history is recorded. At that point, the device's history is updated according to the limit on maximum device history entries that the administrator sets. If the maximum number has been increased, the older entries are retained. If the maximum number has been decreased, the oldest entries are deleted.

Device History Records

The device history records are logged in a rolling list. When the list reaches the limit that the administrator specifies, the oldest entry is removed. However, the following records are never deleted:

To optimize BAC performance that extensive recording of device activity might impact, you can enable or disable specific event types with the following system properties from the API:

Note

These properties are, by default, disabled. These options impact device history only if you have enabled the Device History feature.

Records the NewOperationQueued, OperationRemoved, and OperationExecuted history for on-connect device operations.

Note

If you disable the SERVER_DEVICE_HISTORY_IMMEDIATE_OP_ENABLE_KEY property and enable the SERVER_DEVICE_HISTORY_PROXY_OP_ENABLE_KEY property, the immediate operation is disabled and the on-connect operation is enabled.

Step 3

Click on the radio buttons corresponding to the operation you want to enable or disable:

•

To enable recording the history of immediate operations, click the Enabled radio button corresponding to Immediate Operation History.

•

To enable recording the history of on-connect operations, click the Enabled radio button corresponding to On-Connect Operation History.

•

To enable recording the history of configuration generation, click the Enabled radio button corresponding to Instruction Generation History.

Device Faults

In large installations, devices with recurring faults can cause bottlenecks and affect performance. Recurring faults can result from a device malfunction or misconfiguration of BAC. You can use BAC to detect recurring faults occurring at the RDU and the DPE servers through the Recurring Device Faults feature.

A recurrent fault is one that occurs many times over a short period of time, or one with high chances of occurring repeatedly. For instance, a fault may result while attempting to configure a device based on the BAC configuration policy because a device is missing a parameter. Such a fault qualifies as recurring even if it happens only once, because this operation is executed on every device contact. However, if a one-time device operation is initiated from the API and results in a fault, this fault is returned to the API client in the operation response and is not considered to be recurring.

•

A configuration template at the RDU may refer to a property that was deleted from the system. Every time the RDU tries to generate instructions for devices by using this template, an error occurs. This error qualifies the device as the one causing a recurring fault and is reported to the user.

•

A device sends an invalid or incomplete Inform message because of a bug in the device firmware. The same Inform is generated on every device contact. This problem also qualifies as a recurring fault.

This feature provides details about the last fault for given devices as well as aggregated fault statistics for the system as a whole, the RDU server, and each DPE server.

The RDU and the DPE maintain a list of recurring faults. Each fault contains the date and time of occurrence, the ID of the device, and a fault description.

Faults automatically expire and are purged from the system as soon as their lifetime expires. After a fault is purged from the system, the statistics are adjusted accordingly.

At the RDU:
Failures during instruction generation.
Failures during extension execution.
At the DPE:
Violations of the CPE WAN Management Protocol.
Failures during DataSync instruction processing, which is responsible for discovering data from a device and updating the RDU.
Failures during FirmwareRules instruction processing, which is responsible for executing firmware rules configured in the RDU firmware rules templates.
Failures during ConfigSync instruction processing, which is responsible for updating device configuration according to RDU configuration templates.
Failures during UnknownDevice instruction processing, which is responsible for processing unknown or unregistered devices.

Note BAC limits the number of devices returned in the device fault list on the administrator user interface to 1,000. If the number of devices with faults exceeds 1,000, the operator will see a message stating that more devices have faults that just those on the screen.

To avoid impacting performance, BAC does not store fault information on the disk. If you restart the server, you will lose fault data that was maintained by that server in its memory. However, if faults repeat again, they are reported.

Retrieving Device Faults

Device fault information is available via the API and the administrator user interface.

•

Configuration.getRDUDetails—Returns the list of devices with faults at the RDU.

•

Configuration.getDPEDetails—Returns the list of devices with faults at the DPE.

•

IPDevice.getDetails—Returns information about faults related to the device.

•

Servers > RDU—Displays device fault statistics at the RDU level and aggregated for all DPEs. Statistics appear for the following periods of time by default: 1, 3, 12, and 72 hours.

•

Servers > Provisioning Groups > Manage Provisioning Groups > View Provisioning Group Details—Displays device fault statistics for each DPE in a provisioning group.

•

Servers > DPEs > Manage Device Provisioning Engines > View Device Provisioning Engines Details—Displays the number of devices with faults, if any. Against Device with Faults, click the View Details icon. The Device Details page appears, detailing device fault statistics at the DPE level.

Note

The Device with Faults option, along with the View Details icon, appears on the View Device Provisioning Engines Details page only if there are devices with faults.

•

Devices > Manage Devices > Device Details—Displays the last fault, if any, for the selected device, as described in Figure 8-1.

BAC maintains details of the most recent device with fault at each server. On account of DPE redundancy, a specific device over time will contact multiple DPEs, and may make it to a fault list on one or each of those DPEs. The same device may also have recurring faults at the RDU. BAC provides an aggregated view of device faults at all servers. However, on each server no more than one fault per device is tracked at any given time, and faults data is removed from memory when it expires.

Device Troubleshooting

You can use this feature to collect highly detailed troubleshooting information about one or multiple specific devices. Troubleshooting information includes all server interactions related to a given device or a group of devices. This information includes administrator user interface operations, RDU API operations, DPE interactions with the CPE, and inter-server DPE-to-RDU interactions.

You can enable or disable troubleshooting for one or a number of specific devices without turning logging on, and without searching through log files for specific device information.

BAC maintains a list of devices, based on the device identifier, for which detailed troubleshooting information is collected. Troubleshooting information is stored centrally at the RDU and is maintained on a per-device basis; the DPEs do not store this data. Rather, the DPE forwards this information to the RDU which writes it to the device log file, troubleshooting.log, in the BPR_DATA/rdu/logs directory when it receives this information.

If the connection from the DPE to the RDU is lost, any new troubleshooting events occurring on the DPE are discarded. The logging of troubleshooting information resumes only after the RDU-DPE connection is restored.

The troubleshooting.log file is different from other log files such as rdu.log, dpe.log, and audit.log. The troubleshooting.log file only logs detailed troubleshooting information relating to a specific set of devices in the troubleshooting mode.

Note The tracking feature is turned off until one or more devices are added to the troubleshooting group.

When you enable or disable troubleshooting on a specific device, the change takes place immediately at all servers (RDU and DPEs); you do not have to reboot the RDU or the DPEs. The log files on the respective servers lists the current list of devices in the troubleshooting mode.

Configuring Device Troubleshooting

The Device Troubleshooting feature is disabled until one or more devices are put into troubleshooting mode. This section describes how you can enable or disable troubleshooting for a device from the administrator user interface. It also describes how to view a list of devices in troubleshooting mode and how to view troubleshooting log for a given device.

You can configure the maximum number of devices in troubleshooting mode to prevent inadvertently putting too many devices in this mode, and thus, diminishing server performance. By default, this number is set at 100. To configure the maximum number of the devices allowed in troubleshooting mode from the administrator user interface, click the Systems Defaults page via the Configuration > Defaults tabs. Enter a value in the Maximum Troubleshooting Device Count field.

Enabling Troubleshooting for a Device

To enable troubleshooting for a device, the device must be preregistered in the BAC RDU. If the device is not yet preregistered, add the device from the Manage Devices page by clicking the Add button. For information on adding devices, see Adding Device Records, page 16-11.

To enable troubleshooting for a device that already resides in the RDU database:

Step 1

From the Manage Devices page, click the Search Type drop-down list, and select the Device Identifier Option Search option. You can use the wildcard function for this search. (See Table 16-1.) Click Search.

Step 2

A list of devices appears. Click the Identifier link corresponding to the device that you want to track.

Step 3

The Modify Device page appears, listing various device parameters. Click the Enabled radio button corresponding to the Troubleshooting parameter.

To verify if a given device is enabled for troubleshooting, you can access the Device Details page, and view status against Troubleshooting.

Disabling Troubleshooting for a Device

Step 1

From the Devices tab, locate the device that you want to delete. You can use one of the search types for this purpose.

Step 2

Click the Identifier link corresponding to the device that you want to delete from the troubleshooting list.

Step 3

The Modify Device page appears. Against the Troubleshooting parameter, click the Disabled radio button.

Viewing List of Devices in Troubleshooting Mode

When you enable troubleshooting for a device, the device is automatically added to a special device group which contains a list of devices in troubleshooting mode. The group type is system and the group name is troubleshooting. You can access the list of devices in this group from the API or the administrator user interface.

Step 1

From the Manage Devices page, click the Search Type drop-down list, and select Group Search.

Step 2

From the Group (Group Type) drop-down list, select the troubleshooting (system) option to view all the devices in troubleshooting mode.

Note

An alternative way to view the list of devices in troubleshooting mode is to consult the RDU log (rdu.log) and the DPE log (dpe.log). The list of devices is logged whenever the server is started and whenever there is a change in the list of devices enabled for troubleshooting.

The devices enabled for troubleshooting appear in the log files with the log level of 5-notification. For details on log files, see Logging, page 19-2.

Viewing Device Troubleshooting Log

You can view the troubleshooting log file of a specific device in one of two ways.

Step 1

When a list of the devices in troubleshooting mode appears, click the View Details icon corresponding to a specific device.

Step 2

The Device Details page appears. Against View Troubleshooting Log, click the View Details icon.

Step 1

From the Manage Details page under the Devices tab, click the Search Type drop-down list, and select the Device Identifier Option Search option. You can use the wildcard character (*) for this search.

Step 3

A list of devices appears. Click the View Details icon corresponding to the device whose log file you want to check.

Step 4

The Device Details page appears. Against View Troubleshooting Log, click the View Details icon.

•

BAC-TROUBLESHOOTINGINPUT—Details of messages received by BAC servers are marked in grey.

•

BAC-TROUBLESHOOTINGOUTPUT—Details of messages sent by BAC servers are marked in green.

Table Of Contents