Troubleshooting Broadband Access Center

This chapter provides details on how to troubleshoot with Broadband Access Center (BAC). This chapter describes:

Troubleshooting Checklist

Table 19-1 Troubleshooting Checklist
Procedure		Refer to ...
1. Check if the BAC processes are up on all systems on which BAC components are installed.		Using BAC Process Watchdog from the Command Line, page 9-2
2. Check the BAC component logs for indications of high-severity errors. These include the information logged for: –RDU –DPE		RDU Logs DPE Logs
3. View server uptime from the administrator user interface to confirm that the servers are not bouncing.		Viewing Servers, page 16-21
4. View the RDU and DPE service performance statistics from the administrator's user interface. Observe any abnormal numbers, such as extended transaction times.		Viewing Servers, page 16-21
5. Check the syslog alerts log.		Syslog Alert Messages, page 11-1
6. Check the operating system and hardware resources, such as: –Disk space –CPU time –Memory		Solaris documentation for specific commands.
7. If troubleshooting a specific device, view the history of the device configuration from the administrator user interface.		Viewing Device History, page 16-12
8. If troubleshooting a specific device, view the device instructions that are cached at the DPE.		The show device-config command described in the Cisco Broadband Access Center DPE CLI Reference, 3.0.
9. Configure individual device troubleshooting from the administrator user interface and, after a period of time, inspect the troubleshooting log.		Configuring Device Troubleshooting
10. View device fault data for the system, the RDU, the DPE, or a specific device.		Device Faults
11. Configure a higher level of logging on the RDU or the appropriate DPE for detailed logging information.		The RDU Log Level Tool The log level command as described in the Cisco Broadband Access Center DPE CLI Reference, 3.0.

Logging

Logging of events is performed at both the DPE and RDU, and in some unique situations, DPE events are logged at the RDU to give them higher visibility. Log files are located in their own log directories and can be examined using any text file viewer. The files can be compressed to allow them to be easily e-mailed to the TAC or system integrators for troubleshooting and fault resolution.

Log Levels and Structures

The log file structure is described here, and illustrated in Example 19-1, and includes:

•

Date and Time—This is the date on which a message is logged. This information also identifies the applicable time zone.

•

Security Level—The logging system defines seven levels of severity (log levels as described in Table 19-2) that are used to identify the urgency with which you might want to address log issues. The process of configuring log levels is described in Configuring Log Levels:

Table 19-2 Logging Levels
Log Level	Description
0-Emergency	System unstable. Sets the logging function to save all emergency messages.
1-Alert	Immediate action needed. Sets the logging function to save all activities that need immediate action and those of a more severe nature.
2-Critical	Critical conditions exist. Sets the logging function to save all error messages and those of a more severe nature
3-Error	Error conditions exist. Sets the logging function to save all error messages and those of a more severe nature.
4-Warning	Warning conditions exist. Sets the logging function to save all warning messages and those of a more severe nature.
5-Notification	A normal, but significant, condition exists. Sets the logging function to save all notification messages and those of a more severe nature
6-Information	Informational messages. Sets the logging function to save all logging messages available
Note Another level known as 7-DEBUG is used exclusively by Cisco for debugging purposes. Do not use this level except at the direction of the Cisco TAC.


Domain Name	Data and Time	Facility	Sub- facility	Security Level	Msg ID	Message
BAC1:	2006 04 21 07:28:00 EDT:	BAC-	RDU-	6	0236:	BAC Regional Distribution Unit starting up
BAC1:	2006 04 21 07:28:00 EDT:	BAC-	RDU-	6	0566:	Initialized API defaults
BAC1:	2006 04 21 07:28:00 EDT:	BAC-	RDU-	6	0568:	Initialized server defaults
BAC1:	2006 04 21 07:28:00 EDT:	BAC-	RDU-	6	0569:	Created default admin user
BAC1:	2006 04 21 07:28:00 EDT:	BAC-	RDU-	6	0574:	Loaded 6 license keys
BAC1:	2006 04 21 07:28:00 EDT:	BAC-	RDU-	6	0575:	Database initialization completed in 471 msec
BAC1:	2006 04 21 07:28:00 EDT:	BAC-	RDU-	6	0015:	Unable to locate manifest file
BAC1:	2006 04 21 07:28:00 EDT:	BAC-	RDU-	6	0280:	Command error

Configuring Log Levels

You can configure logging levels for both the RDU and the DPE to suit your specific requirements. For example, the logging level for the RDU could be set to Warning, and the level for the DPE could be set to Alert.

Log messages are written based on certain events taking place. Whenever an event takes place, the appropriate log message and level are assigned and, if that level is less than or equal to the configured level the message is written to the log. The message is not written to the log if the level is higher than the configured value.

For example, assume that the log level is set to 4-Warning. All events generating messages with a log level of 4 or less are written into the log file. If the log level is set to 6-Information, the log file will receive all messages. Consequently, configuring a higher log level results in a larger log file size.

To configure the log level on the DPE, using the log level command from the DPE command line. See the Cisco Broadband Access Center DPE CLI Reference, Release 3.0, for detailed information.

Rotating Log Files

All log files, except perfstat.log, are numbered and rolled over based on a configured maximum file size. The default maximum file size is 10 MB. (To configure the maximum file size from the API, use the ServerDefaultsKeys.SERVER_LOG_MAXSIZE property.) Once a log file touches the configured limit, the data is rolled over to another file. This file is renamed in the XXX.N.log format, where:

For example, once rdu.log reaches the 10 MB limit, it is renamed as rdu.1.log. With every 10-MB increase in file size, the latest file is renamed as rdu.2.log, rdu.3.log, and so on. So, the rdu.7.log file will contain data more recent than rdu.4.log. However, the latest log information is always stored in rdu.log.

In the case of the perfstat.log file, the file is renamed everyday. The file is rolled over in the perfstat.N.log format, where N is any value between 1 and 29. For example, perfstat.29.log will be the oldest log while perfstat.1.log will be the most recent renamed perfstat.log file.

BAC stores up to 10 log files at a given time. For a list of log files in the RDU and DPE servers, see RDU Logs, and DPE Logs, respectively.

RDU Logs

•

rdu.log—Records all RDU events according to the configured logging severity level. (See Setting the RDU Log Level, for instructions on setting the default log levels.) To view rdu.log, see Viewing the rdu.log File.

•

audit.log—Records all high-level changes to the BAC configuration or functionality including the user who made the change. To view audit.log, see Viewing the audit.log File.

•

troubleshooting.log—Records detailed device information for troubleshooting a specific device, or a group of devices without turning logging on, and without searching through log files for device- or group-specific information. To view troubleshooting.log from the administrator user interface, see Viewing Device Troubleshooting Log.

•

perfstats.log—Records device performance statistics to help troubleshoot issues related to system performance. For more information, see Monitoring Broadband Access Center, page 11-1.

Viewing the rdu.log File

You can use any text processor to view the rdu.log file. In addition, you can view the log file from the administrator user interface. To do this:

Step 2

The View Regional Distribution Unit Details page appears. Click the View Details icon (

) corresponding to RDU Log File.

Viewing the audit.log File

You can use any text processor to view the audit.log file. In addition, you can view the log file from the administrator user interface. To do this:

Step 2

The View Regional Distribution Unit Details page appears. Click the View Details icon corresponding to Audit Log File.

The RDU Log Level Tool

Use the RDU log level tool to change the current log level of the RDU from the command line, using the setLogLevel.sh command. This tool is located in the BPR_HOME/rdu/bin directory. Table 19-2 identifies the available log levels and the types of message written to the log file when enabled.

Cisco recommends that you keep the RDU logging level at the Warning level to help maintain a steady operations state. The Information level is recommended to be used with caution if you need to maintain steady state performance during debug operations. You should exercise caution when running with the Information level set because this creates a great number of log entries, which in itself can adversely impact performance.

Note The RDU process has to be up to execute the log level tool. Also, you must be a privileged user to run this tool by using the setLogLevel.sh command.

Using the RDU Log Level Tool

All examples assume that the user name for the RDU is admin, the password for the RDU is changeme, and the RDU server is up.

•

-[0..6]—Identifies the logging level to be used. For a list of available levels, see Table 19-2.

•

-default—Sets the RDU to the installation default level 5 (notification).

•

-debug— Sets an interactive mode to enable or disable tracing categories for RDU server.

Note

You should only enable the debug settings that the Cisco support staff recommends.

Setting the RDU Log Level

You can use this tool to change the logging level from one value to any other value.

The following example illustrates how to set the RDU logging level to the warning level, as indicated by the number 4 in the setLogLevel.sh command. The actual log level set is not important for the procedure, it can be interchanged as required.

Step 3

Enter the RDU username at the prompt. In this example, the default username (admin) is used.

Step 4

Enter the RDU password for the RDU at the prompt. In this example, the default password (changeme) is used.

This message appears to notify you that the log level has been changed. In this example, the level was 5, for notification, and is now 4, for warning.

Viewing the RDU's Current Log Level

You can use this tool to view the RDU log and determine which logging level is configured before attempting to change the value.

DPE Logs

•

dpe.log—Records all events having the configured default level. In situations where the DPE undergoes catastrophic failure, such as engaging in a series of system crashes, the catastrophic errors are also logged into the rdu.log file.

•

perfstats.log—Records device performance statistics to help troubleshoot issues related to system performance. For more information, see Monitoring Broadband Access Center, page 11-1.

Viewing the dpe.log File

You can use any text viewer to view the dpe.log file. In addition you can use the show log command, from the DPE CLI, to view the log file. Refer to the Cisco Broadband Access Center DPE CLI Reference, Release 3.0, for additional information.

You can also view the DPE log file using the BAC administrator user interface. To do this:

Step 2

Click the link corresponding to the DPE whose log file you want to view.

Step 3

The View Device Provisioning Engines Details page appears. To view the contents of the dpe.log file, click the View Details icon against DPE Log File in the Log Files area.

Table Of Contents