|
Table Of Contents
Monitoring Broadband Access Center
Monitoring Servers by Using SNMP
Using the snmpAgentCfgUtil.sh Tool
Using the Administrator User Interface
Monitoring Performance Statistics
Monitoring Broadband Access Center
This chapter describes how you can monitor the central RDU servers and the DPE servers in a Broadband Access Center (BAC) deployment. It describes:
• Monitoring Servers by Using SNMP
• Monitoring Performance Statistics
Syslog Alert Messages
BAC generates alerts through the Solaris syslog service. Syslog is a client-server protocol that manages the logging of information on Solaris. BAC syslog alerts are not a logging service; they provide a notification that a problem exists, but do not necessarily define the specific cause of the problem. You might find this information in the appropriate BAC log files.
Message Format
When BAC generates an alert message, the format is:
XXX-#-####: Message
•XXX—Identifies the facility code, which can include:
–RDU (regional distribution unit).
–DPE (device provisioning engine).
–AGENT (rduSnmpAgent or dpeSnmpAgent).
•#—Identifies the severity level in use. The three levels of alerts are:
–1, which is alert.
–3, which is error.
–6, which identifies informational messages.
•###—Identifies the numeric error code as described in the following sections.
•Message—Provides the alert text or message.
RDU Alerts
Table 11-1 identifies the RDU alerts.
Table 11-1 RDU Alerts
Alert DescriptionRDU-1-101: RDU ran out of disk space
Indicates that the storage partition that the RDU server uses ran out of space. After encountering this error, the RDU attempts to restart automatically, but will typically encounter the same error again until more storage space is available.
See BAC Support Tools and Advanced Concepts, for additional information on upgrading the disk.
RDU-1-103: RDU ran out of memory
Indicates that the RDU ran out of memory. After encountering this error, the RDU server restarts automatically.
RDU-1-111: Evaluation key for technology [technology_name] expired
Appears if an evaluation key for the technology specified expires. You must contact Cisco sales or TAC for a new license key.
RDU-1-115: You have used [percent]% of available [technology_name] licenses.
Identifies the quantity of licences used (in percentage) out of the total number of allowable licenses. Appears when you reach 80% of the license capacity.
BPR-RDU-4-1140: DNS took X seconds for lookup of address [10.0.0.1/test.com]; Check DNS configuration and health of servers
Indicates that BAC performance may be slow due to delayed response from the DNS. The alert is generated whenever IP Address look-up takes more than 60 seconds.
Note Whenever an RDU syslog alert is sent, additional details (if any) can be found in log file BPR_DATA/rdu/logs/rdu.log.
DPE Alerts
Whenever a DPE syslog alert is sent, you can find additional details in the DPE logs.
You can use the show log command to access the DPE logs. See Cisco Broadband Access Center DPE CLI Reference, Release 3.0, for additional information.
Some DPE errors are also propagated to the RDU server log files. You can find these in the BPR_DATA/rdu/logs/rdu.log file.
Table 11-2 identifies the DPE alerts.
Watchdog Agent Alerts
Whenever the watchdog process sends a syslog alert, you can find error details (if any) in the BPR_DATA/agent/logs/agent_console.log file and the log files corresponding to the specific component mentioned in the alert (if any). For example, if you receive an alert similar to The rdu unexpectedly terminated, you would check the RDU server log file (BPR_DATA/rdu/logs/rdu.log) for additional information. Table 11-3 identifies the watchdog agent alerts.
The component variable presented in the watchdog agent alerts list shown in Table 11-3 represents any of these component values:
•rdu
•dpe
•tomcat
•cli
•snmpAgent
Monitoring Servers by Using SNMP
BAC supports management of servers via SNMP. Specifically, an SNMP-based management system can be used to monitor BAC server state, license utilization information, server connections, and server-specific statistics.
SNMP Agent
The BAC SNMP agents support SNMP informs and traps, collectively called as notifications hereafter. You can configure the SNMP agent on the DPE by using snmp-server CLI commands, and on the RDU by using the snmpAgentCfgutil.sh tool.
See Using the snmpAgentCfgUtil.sh Tool, for additional information on the SNMP configuration command line tool, and the Cisco Broadband Access Center DPE CLI Reference, Release 3.0, for additional information on the DPE CLI.
MIB Support
BAC supports several different MIBs. These include:
•CISCO-BACC-DPE-MIB
•CISCO-BACC-RDU-MIB
•CISCO-BACC-SERVER-MIB
Table 11-4 summarizes the MIB support in BAC:
Table 11-4 BAC-Supported MIBs
Installation Component MIBs SupportedDPE
CISCO-BACC-SERVER-MIB
CISCO-BACC-DPE-MIB
RDU
CISCO-BACC-SERVER-MIB
CISCO-BACC-RDU-MIB
The RDU SNMP agent supports the CISCO-BACC-RDU-MIB, which defines managed objects for the RDU. This MIB defines statistics related to the state of the RDU and the statistics on the communication interface between the RDU and DPE.
The DPE SNMP agent supports the CISCO-BACC-DPE-MIB, which defines managed objects for the DPE. This MIB provides some basic DPE configuration and statistics information.
The SNMP agent supports the CISCO-BACC-SERVER-MIB. This MIB defines the managed objects that are common to all servers on BAC. This MIB supports the monitoring of multiple BAC servers when they are installed on the same device. The
ciscoBaccServerStateChanged
notification is generated every time a server state change occurs.
Note For a description of all objects, refer to the corresponding MIB files under the BPR_HOME/rdu/mibs directory.
Using the snmpAgentCfgUtil.sh Tool
You can use the snmpAgentCfgUtil.sh tool to manage the SNMP agent on a Solaris system.
By using this tool, which is located in the BPR_HOME/snmp/bin directory, you can add (or remove) your host to a list of other hosts that receive SNMP notifications, and start and stop the SNMP agent process.
Note The default port number of an SNMP agent that is running on a Solaris computer, is 8001.
You can use the snmpAgentCfgUtil.sh tool for:
• Adding an SNMP Agent Community
• Deleting an SNMP Agent Community
• Changing the SNMP Agent Location
• Displaying SNMP Agent Settings
Adding a Host
You use this command to add the host address to the list of hosts that receive SNMP notifications from the SNMP agent.
Syntax Description
snmpAgentCfgUtil.sh add host host-addr community community [udp-port port]
•host-addr—Specifies the IP address of the host that you want to add to the list of hosts
•community—Specifies the community (read or write) to use while sending SNMP notifications
•port—Identifies the UDP port used for sending the SNMP notifications
Examples
# ./snmpAgentCfgUtil.sh add host test.cisco.com community trapCommunity udp-port 162
OK
Please restart [stop and start] SNMP agent.
Note The changes that you introduce through this command do not take effect until you restart the SNMP agent by using the /etc/init.d/bprAgent restart snmpAgent command. For detailed information, see BAC Process Watchdog, page 9-1.
Deleting a Host
You use this command to remove a host from the list of those receiving SNMP notifications from the SNMP agent.
Syntax Description
snmpAgentCfgUtil.sh delete host host-addr
host-addr—Specifies the IP address of the host that you want to delete from the list of hosts.
Examples
# ./snmpAgentCfgUtil.sh delete host test.cisco.com
OK
Please restart [stop and start] SNMP agent.
Note The changes that you introduce through this command do not take effect until you restart the SNMP agent by using the /etc/init.d/bprAgent restart snmpAgent command. For detailed information, see BAC Process Watchdog, page 9-1.
Adding an SNMP Agent Community
You use this command to add an SNMP community string to restrict access to the SNMP agent. The SNMP community name is used as a shared secret, with SNMP managers accessing the BAC SNMP agent.
Syntax Description
snmpAgentCfgUtil.sh add community string [ro | rw]
•string—Identifies the SNMP community.
•ro—Assigns a read-only (ro) community string. Only get requests (queries) can be performed. The
ro
community string allows get requests, but no set operations. The network management system and the managed device must reference the same community string.•rw—Assigns a read-write (rw) community string. SNMP applications require read-write access for set operations. The
rw
community string enables write access to object identifier (OID) values.
Note The default ro and rw community strings are
bacread
andbacwrite
, respectively. Cisco recommends that you change these values before deploying BAC. To change them, add new community names and delete the default ones.Examples
# ./snmpAgentCfgUtil.sh add community fsda54 ro
OK
Please restart [stop and start] SNMP agent.
Note The changes that you introduce through this command do not take effect until you restart the SNMP agent by using the /etc/init.d/bprAgent restart snmpAgent command. For detailed information, see BAC Process Watchdog, page 9-1.
Deleting an SNMP Agent Community
You use this command to delete an SNMP community string to prevent access to the SNMP agent.
Syntax Description
snmpAgentCfgUtil.sh delete community string [ro | rw]
•string—Identifies the SNMP community
•ro—Assigns a read-only (ro) community string
•rw—Assigns a read-write (rw) community string
Note For additional information on the ro and rw community strings, see Adding an SNMP Agent Community.
Examples
# ./snmpAgentCfgUtil.sh delete community fsda54 ro
OK Please restart [stop and start] SNMP agent.
Note The changes that you introduce through this command do not take effect until you restart the SNMP agent by using the /etc/init.d/bprAgent restart snmpAgent command. For detailed information, see BAC Process Watchdog, page 9-1.
Starting the SNMP Agent
You use this command to start the SNMP agent process on any Solaris computer on which BAC is already installed.
Note You can also start the SNMP agent by invoking the BAC watchdog process agent by using the /etc/init.d/bprAgent start snmpAgent command. For more information, see Using BAC Process Watchdog from the Command Line, page 9-2.
Examples
# ./snmpAgentCfgUtil.sh start
Process snmpAgent has been started
Stopping the SNMP Agent
You use this command to stop the SNMP agent process on any Solaris computer on which BAC is already installed.
Note You can also stop the SNMP agent by invoking the BAC watchdog process agent by using the /etc/init.d/bprAgent stop snmpAgent command. For more information, see Using BAC Process Watchdog from the Command Line, page 9-2.
Examples
# ./snmpAgentCfgUtil.sh stop
Process snmpAgent has stopped
Configuring an SNMP Agent Listening Port
You use this command to specify the port number to which the SNMP agent will listen. The default port number that the RDU SNMP agent uses is 8001.
Syntax Description
snmpAgentCfgUtil.sh udp-port port
port— Identifies the port number to which the SNMP agent will listen.
Examples
# ./snmpAgentCfgUtil.sh udp-port 8001
OK
Please restart [stop and start] SNMP agent.
Changing the SNMP Agent Location
You use this command to enter a string of text that you use to indicate the location of the device running the SNMP agent. For example, you could use this string to identify the physical location of the device. You can enter any string up to 255 characters long.
Syntax Description
snmpAgentCfgUtil.sh location location
location—Specifies the character string identifying the agents location.
Examples
In this example, the physical location of the SNMP agent is in an equipment rack identified as equipment rack 5D:
# snmpAgentCfgUtil.sh location "equipment rack 5D"
Setting Up SNMP Contacts
You use this command to enter a string of text that you use to identify the contact person for the SNMP agent, together with information on how to contact this person. For example, you could use this string to identify a specific person including that person's telephone number. You can enter any string up to 255 characters long.
Syntax Description
snmpAgentCfgUtil.sh contact contact-info
contact-info— Specifies the character string identifying the individual to contact concerning the SNMP agent.
Examples
In this example, the contact name is Ace Duffy and the telephone extension is 1234:
# ./snmpAgentCfgUtil.sh contact "Ace Duffy - ext 1234"
Displaying SNMP Agent Settings
You use this command to display all current SNMP settings.
Syntax Description
snmpAgentCfgUtil.sh show
Examples
# ./snmpAgentCfgUtil.sh show
Location : Washington_1
Contact : John
Port Number : 8001
Notification Type : trap
Notification Recipient Table :
[ Host IP address, Community, UDP Port ]
[ 10.10.10.1, public, 162 ]
Access Control Table :
Read Only Communities
bacread
Read Write Communities
bacwrite
Specifying SNMP Notification Types
You use this command to specify which types of notifications (traps or informs) the SNMP agent will send. By default, the agent sends traps; although you can set this to send SNMP informs instead.
Syntax Description
snmpAgentCfgUtil.sh inform [retries retry_count timeout timeout] | trap
Where the parameter is the back-off timeout between retries.
Examples
snmpAgentCfgUtil.sh inform retries 3 timeout 1000
OK
Please restart [stop and start] SNMP agent.
Note Use the snmpAgentCfgUtil.sh show command to verify your configuration settings.
# ./snmpAgentCfgUtil.sh show
Location : <unknown>
Contact : <unknown>
Port Number : 8001
Notification Type : inform
Notification Retries : 3
Notification Timeout : 1000
Notification Recipient Table :
[ Host IP address, Community, UDP Port ]
Access Control Table :
Read Only Communities
bacread
Read Write Communities
bacwrite
Monitoring Server Status
This section describes how you can monitor the performance of the RDU and DPE servers in a BAC deployment. These servers are the central RDU server and the DPE servers.
You can check server statistics from the:
•Administrator user interface
•DPE CLI
•RDU and DPE log files using the administrator user interface or the DPE CLI.
Using the Administrator User Interface
To view server statistics available on the administrator user interface:
Step 1 On the Primary Navigation Bar, click the Server tab.
Step 2 The Secondary Navigation Bar displays your options: DPEs, Provisioning Group, RDU.
Click the:
•DPEs tab to monitor all DPEs currently registered in the BAC database.
•RDU tab to display RDU status and statistics.
Step 3 If you clicked:
•DPEs—The Manage Device Provisioning Engine page appears. Each DPE name on this page is a link to another page that shows the details for that DPE. Click this link to display the details page.
•RDU—The View Regional Distribution Unit Details page appears.
Using the DPE CLI
To monitor the status of the DPE server, run the show dpe command to check if the DPE is running and displays the state of the process and, if running, its operational statistics.
Note This command does not indicate if the DPE is running successfully, only that the process itself is currently executing. However, when the DPE is running, you can use statistics that this command prints to determine if the DPE is successfully servicing requests.
Example 11-1 show dpe Output
dpe# show dpe
BAC Agent is running
Process dpe is not running
This result occurs when the DPE is not running.
dpe# show dpe
BAC Agent is running
Process dpe is running
Version BAC 3.0 (SOL_CBAC3_0_L_000000000000).
Caching 1 device configs and 1 external files.
0 sessions succeed and 0 sessions failed.
0 file requests succeed and 0 file requests failed.
0 immediate proxy operations received: 0 succeed, and 0 failed.
Connection status is Ready.
Running for 4 hours 30 mins 16 secs.
This result occurs when the DPE is running.
Note For more information, refer to the Cisco Broadband Access Center DPE CLI Reference, Release 3.0.
Monitoring Performance Statistics
BAC provides a rich set of statistics to aid in troubleshooting system performance. The statistics are available across different major components, including the RDU, the Provisioning API Command Engine, and device operations.
You can enable the collection of performance statistics from the administrator user interface or from the DPE CLI.
•To enable or disable performance statistics on the RDU, from the user interface, choose Configuration > Defaults > System Defaults.
–To enable this feature, against Performance Statistics Collection, click the Enabled radio button.
–To disable this feature, against Performance Statistics Collection, click the Disabled radio button.
•To enable or disable performance statistics on the DPE, from the DPE CLI in the enabled mode, enter debug dpe statistics. To disable performance statistics from the CLI, use the no debug dpe statistics command.
Note Before using any debug command, ensure that DPE debugging is enabled by running the debug on command. For more information, refer to the Cisco Broadband Access Center DPE CLI Reference, Release 3.0.
After you enable the performance statistics feature, you can choose to view performance statistics from the perfstat.log file or analyze the data by using the runStatAnalyzer.sh tool.
You can also view CWMP statistics specifically by using the administrator user interface. Choose Servers > DPEs > Manage Device Provisioning Page > View Device Provisioning Engines Details. (See Figure 16-5.)
For details on performance statistics collection, see:
Understanding perfstat.log
You can monitor performance statistics by using the data recorded in the perfstat.log file, in which statistics data is logged at specific intervals; this time interval is 5 minutes. The perfstat.log file resides in separate directories for the RDU (BPR_DATA/rdu/logs/statistics) and the DPE (BPR_DATA/dpe/logs/statistics).
Each perfstat.log file stores data for a minimum of one day and a maximum of 30 days. Since you can turn on and turn off the performance statistics feature, the logs may not represent data for consecutive days.
The perfstat.log file is renamed every day by using the perfstat.N.log format, where N is any value between 1 and 29. For example, perfstat.29.log will be the oldest log while perfstat.1.log will be the most recent renamed perfstat.log file.
Note The data is stored in comma-separated vector format. The format of each statistic is yyyymmdd:hh:mm, component, interval-in-milliseconds, stat1-tag, stat1-value, stat2-tag, stat2-value, ... Stat1-tag and stat1-value specify the tag ID and the value of each statistic, respectively.
Using runStatAnalyzer.sh
You can use BAC to analyze and provide a summary of performance statistics by using the runStatAnalyzer.sh tool. To analyze collected performance statistics, run the runStatAnalyzer.sh tool from the:
•BPR_HOME/rdu/bin directory for the RDU.
•BPR_HOME/dpe/bin directory for the DPE.
Syntax Description
# runStatAnalyzer.sh [-d perfdata-dir] [-s start-time] [-e end-time] [-c component] [-f output-format] [-help] [-help components] [-help statistics [component]]
•perfdata-dir—Specifies the directory from which performance statistics are analyzed. This is the perfstatN.dat file located in the following default directories:
–BPR_HOME/rdu/logs/statistics for the RDU
–BPR_HOME/dpe/logs/statistics for the DPE
•start-time—Specifies the time from which collected data is to be analyzed. By default, all collected statistics are reported. Use this time format to specify start-time: yyyy-mm-dd:hh:mm.
•end-time—Specifies the time until which data collected data is to be analyzed. By default, all collected statistics are reported. Use this time format to specify end-time: yyyy-mm-dd:hh:mm.
•component—Specifies the BAC component for which you want to analyze statistics. You can choose to specify all components (by using the all option) or specify from the following list of supported components:
•output-format—Specifies the format of the output you want, which could be:
–summary—Provides output of the transaction rate summary. This is the default option.
Note The summarized transaction rate is calculated based on the 5-minute interval data recorded in perfstat.log.
–log—Provides output similar to a log message.
•-help—Provides usage information on the runStatAnalyzer.sh tool.
•-help components—Provides information on the BAC component for which you can analyze statistics.
•-help statistics component—Provides information on the statistics that each BAC component returns. You can choose to view help for all components (by using the all option) or for individual components: pace, rdu, ext, cwmp, httpfile, proxyreq.
Example 11-2 Log Output Through runStatAnalyzer.sh
# runStatAnalyzer.sh -s 2006-04-11:12:59 -e 2006-04-11:13:09 -c pace -f log
2006-04-11:12:59 PACE statistics last 5 minutes- In Queue 0; Dropped 0; Dropped-Full Queue 0; Batches Received 0; Internal Batches Received 0; Succeed 0; Failed 0; Processed 0; Processing avgTime 0 msec; Batch maxTime 0 msec; In Queue maxTime 0 msec; Processing maxTime 0 msec; CRS Completed 0
2006-04-11:13:04 PACE statistics last 5 minutes- In Queue 0; Dropped 0; Dropped-Full Queue 0; Batches Received 0; Internal Batches Received 0; Succeed 0; Failed 0; Processed 0; Processing avgTime 0 msec; Batch maxTime 0 msec; In Queue maxTime 0 msec; Processing maxTime 0 msec; CRS Completed 0
2006-04-11:13:09 PACE statistics last 5 minutes- In Queue 0; Dropped 0; Dropped-Full Queue 0; Batches Received 0; Internal Batches Received 0; Succeed 0; Failed 0; Processed 0; Processing avgTime 0 msec; Batch maxTime 0 msec; In Queue maxTime 0 msec; Processing maxTime 0 msec; CRS Completed 0
Note The number of statistics available varies on the component specified.
Example 11-3 Summary Output Through runStatAnalyzer.sh
# runStatAnalyzer.sh -s 2006-04-11:12:59 -e 2006-04-11:13:29 -c pace -f summary
2006-04-11:13:04 PACE statistics last 5 minutes- In Queue 0; Dropped 0; Dropped-Full Queue 0; Batches Received 0; Internal Batches Received 0; Succeed 0; Failed 0; Processed 0; Processing avgTime 0 msec; Batch maxTime 0 msec; In Queue maxTime 0 msec; Processing maxTime 0 msec; CRS Completed 0
2006-04-11:13:29 PACE statistics last 30 minutes- In Queue 0; Dropped 0; Dropped-Full Queue 0; Batches Received 0; Internal Batches Received 0; Succeed 0; Failed 0; Processed 0; Processing avgTime 0 msec; Batch maxTime 0 msec; In Queue maxTime 0 msec; Processing maxTime 0 msec; CRS Completed 0
Note Summarized data is visible only if a complete set of data is available for the given interval. For example, the summary output of a 30-minute summarized interval appears only if there is 30 minutes worth of data. Based on the data available, the summarized time intervals are 5 minutes, 30 minutes, 60 minutes, 3 hours, 6 hours, 12 hours, 24 hours, 7 days, 14 days, 21 days, and 30 days.
Posted: Thu Aug 31 23:33:50 PDT 2006
All contents are Copyright © 1992--2006 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.