Troubleshooting

This module describes how to troubleshoot the SM installation and daily operation.

Using the Troubleshooting

Each entry in this module consists of an error message, probable cause(s), and solution. Note that the same error message may appear in more than one section of this module.

When an unexpected error occurs during the system's installation or daily operation, search for the error message throughout this module (the message may appear in more than one place). When you find the error message, read the section below the message and try the recommended solution. If the message appears more than once, try to correct the most probable cause first.

General Errors

SM Not Running


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status Could not connect to SM.`
Probable cause:	The SM server has not been started.
Solution:	Use the following command to start/restart the SM server: `>p3sm --start`

SM in Failure Mode


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status SM is running. SM operational state is Failure Command terminated successfully`
Probable cause:	The SM server restarted three times in 30 minutes due to an internal error.
Solution:	This error can happen only in a cluster setup. Check the pcube user log and the Veritas Cluster Server log for the reason for the failure that caused the reboots. Act according to the problem in the logs. Additional operations that can be taken are: •Use the following command to extract a support file: `>p3sm --extract-support-file -f ../support.zip` •Send the support file to Cisco's customer support •Use the following command to start/restart the SM server and get out of the Failure state: `>p3sm --restart`

General Setup Errors

Cannot Run this Script-/etc/motd File Exists


Error message:	The following sequence of output appears when running one of the installation/upgrade scripts: `>./<script-name> <script-name>: Cannot run this script - /etc/motd file exists <script-name>: Please remove the file or move it to another location <script-name>: and run the script again` This error applies to the following scripts: •install-sm.sh •upgrade-sm.sh •cluster-upgrade.sh •install-vcs-agents.sh
Probable cause:	The /etc/motd file exists or has a size greater than 0.
Solution:	Remove the file or move it to another location.

install-sm.sh Script-User is not Root


Error message:	The following sequence of output appears (in response to the command ./install-sm.sh ): `>./install-sm.sh install-sm.sh: Starting SM installation sequence install-sm.sh: Error - this script must be run by root - exiting.`
Probable cause:	You started the installation sequence as user and not as superuser .
Solution:	Run the install-sm.sh script as superuser .

install-sm.sh Script-User pcube Exists


Error message:	The following sequence of output appears (in response to the command ./install-sm.sh ): `# ./install-sm.sh install-sm.sh: Starting SM installation sequence install-sm.sh: Error - pcube user exists and has home /export/home/pcube, not /opt/pcube - remove it or use -o - exiting. #`
Probable cause:	Your machine already has the user pcube .
Solution:	Run the installation using the -o option (overwrite), as follows: `# ./install-sm.sh -o`

install-tt.sh Script

Note A minimum of 1.5 GB of free hard disk space is required to install the TimesTen database.


Error message:	The following sequence of output appears (in response to the command install-tt.sh/export/home/pcube/lib/tt ): `>install-tt.sh /export/home/pcube/lib/tt install-tt.sh: Starting TimesTen P-Cube installation sequence install-tt.sh: Error - This script must be run by root - exiting.`
Probable cause:	You started the installation sequence as user and not as superuser .
Solution:	Run the install-tt.sh script as superuser . Note that the TimesTen directory name given, /lib/tt/ , must be relative to the (pcube) user directory. For example, if the user directory is pcube, install TimesTen in /export/home/pcube/lib/tt/.

install-dsn.sh Script

Note that the TimesTen directory name given, /var/tt/ , must be relative to the (pcube) user directory. For example, if the user directory is pcube, install TimesTen in /export/home/pcube/var/tt/.

TimesTen Database Setup Errors

Introduction

The TimesTen configuration consists of several configuration files. This section explains the purpose and scope of each of these files. When troubleshooting the TimesTen, you will be requested to edit these configuration files and reboot the machine or restart the SM. In most cases, the defaults applied by the SM installation procedure are satisfactory.

Note Changing the TimesTen configuration files should be done with extreme care, and it is best to consult Cisco technical support prior to making any changes.

System (Kernel) Configuration File

The kernel configuration file is a system configuration file, which affects system-wide configuration parameters:

The Subscriber Manager installation procedure configures this file to add extra semaphores and shared memory to the system. After editing this file, you have to reboot the machine for the changes to take effect.

If you are running other applications that require changes in this file's semaphore and shared memory values, take care that the TimesTen configuration does not override the other application's configuration, or vice versa. You should consult with the Cisco technical support for the proper values to use for the file configuration parameters.

Configuration File /var/TimesTen/sys.odbc.ini

The file /var/TimesTen/sys.odbc.ini is a TimesTen configuration file that configures system DSNs. Any user on the machine on which the system DSN is defined can use this file. The SM DSNs are system DSNs that are named PCube_SM_Repository and PCube_SM_Local_Repository, and which have the following system DSN configuration parameters:

•

PermSize—The size of the permanent memory region for the data store, in megabytes. You may increase PermSize but not decrease it.

The data stored in the permanent memory region includes tables and indexes that make up a TimesTen data store. The permanent data partition is written to the disk periodically.

•

TempSize—The size of the memory allocated to the temporary region, in megabytes.

Temporary data includes locks, cursors, compiled commands, and other structures needed for command execution and query evaluation. The temporary data partition is created when a data store is loaded into memory and is destroyed when the data store is unloaded.

•

SMPOptLevel—Optimizes the database operation on multi-processor machines. If the machine is a multi-processor platform, set parameter SMPOptLevel to 1 (default is 0).

Configuration File ~pcube/.odbc.ini

The file ~pcube/.odbc.ini is a TimesTen configuration file that configures user DSNs.

TimesTen DSN Configuration—Cannot Find Requested DSN


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 CLIENT]Cannot find the requested DSN (PCube_SM_Repository_CS) in ODBCINI`
Probable cause:	The TimesTen Client DSN is not configured correctly in file ~pcube/.odbc.ini.
Solution:	Ensure that file ~pcube/.odbc.ini contains the following: `[ODBC Data Sources] PCube_SM_Repository_CS=TimesTen 5.0 Client Driver`

TimesTen DSN Configuration—Data Source Name Not Found


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver]Data source name not found and no default driver specified`
Probable cause:	The TimesTen Client DSN is not configured correctly in file ~pcube/.odbc.ini.
Solution:	Ensure that file ~pcube/.odbc.ini contains the following: `[PCube_SM_Repository_CS] TTC_SERVER_DSN=PCube_SM_Repository` Ensure that file /var/TimesTen/sys.odbc.ini contains the following: `[ODBC Data Sources] PCube_SM_Repository=TimesTen 5.0 Driver [PCube_SM_Repository] Driver=__TTDIR__/TimesTen/pcubesm22/lib/lib tten.so DataStore=__VARDIR__/pcube_SM_Repository`

TimesTen Database Settings—Cannot Connect to Data Source


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not set up correctly: [TimesTen][TimesTen 5.0.35 CLIENT]Unable to connect to data source (DSN: pcube_SM_Repository_CS; Network Address: X.X.X.X; Port Number: XXX): This operation has Timed Out. Try increasing your ODBC timeout attribute or check to make sure the target TimesTen Server is running`
Probable cause:	The following causes are possible: •The address of the Server DS is incorrect. •The port of the Server DS is incorrect. •TimesTen is not active.
Solution:	The Service Control solutions for the above causes are: •(The address of the Server DS is incorrect.) Ensure that file ~pcube/.odbc.ini contains the following: `TTC_SERVER=127.0.0.1` •(The port of the Server DS is incorrect.) On a default installation, ensure that file ~pcube/.odbc.ini does not contain "TTC_SERVER_PORT". On a non-default installation, ensure that file ~pcube/.odbc.ini does contain "TTC_SERVER_PORT=Non-default-port". •(TimesTen is not active.) Run the following command: `~pcube/lib/tt/TimesTen/pcubesm22/bin/tt Status` If TimesTen is not working, re-install TimesTen. If the above solutions do not work, please refer to the TimesTen manual.

TimesTen Configuration Error—Not Enough Memory


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver][TimesTen]TT0836: Cannot create data store shared-memory segment, error 1455 -- file "db.c", lineno 6289, procedure "sbDbConnect()"`
Probable cause:	There is not enough memory for creating TimesTen's in-memory database.
Solution:	Do all of the following: •Ensure that the Unix machine has at least 1024 MB of memory installed. •Ensure that the configured memory size parameters ( PermSize and TempSize ) specified in file /var/TimesTen/sys.odbc.ini are less than the total amount of memory installed in the machine. •For Solaris, ensure that the maximum shared memory (parameter shmsys:shminfo_shmmax ) specified in file /etc/system is less than the total amount of memory installed in the machine. •For Red Hat, ensure that the maximum shared memory (parameter kernel.shmmax ) specified in file /etc/sysctl.conf is less than the total amount of memory installed in the Linux Machine.

TimesTen Configuration Error—Incorrect Memory Definitions


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status SM is running. SM operational state is XXX. Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver]Overflow in converting data store or log file size from megabytes to bytes, or in converting log buffer size from kilobytes to bytes`
Probable cause:	The memory definitions of DSN are incorrect.
Solution:	Ensure that the configured permanent memory size and log file size (parameters PermSize and LogSize ) are less than the total amount of memory and of disk space specified in file /var/TimesTen/sys.odbc.ini.

TimesTen Configuration Error—Cannot Create Semaphores


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver][TimesTen]TT0925: Cannot create data store semaphores (Invalid argument) - file "db.c", lineno 5124, procedure "sbDbCreate()", sqlState: 08001, errorCode: 925`
Probable cause:	TimesTen was unable to create the data store semaphores that are defined in the kernel configuration file (Solaris: /etc/system for Solaris; for Red Hat: /etc/sysctl.conf for Red Hat).
Solution:	Do all of the following: •Ensure that the machine has at least 1024 MB of memory installed. •Reboot the machine after the first time that TimesTen is installed. •Verify the contents of the system (kernel) configuration file: –For Solaris, ensure that file /etc/system contains the following: `semsys:seminfo_semmni = 20 semsys:seminfo_semmsl = 100 semsys:seminfo_semmns = 2000 semsys:seminfo_semmnu = 2000` –For Red Hat, ensure that file /etc/sysctl.conf contains the following: `---- Begin settings for TimesTen kernel.sem = "SEMMSL_250 SEMMNS_32000 SEMOPM_100 SEMMNI_100 ---- End of settings for TimesTen`

TimesTen Configuration Error—Cannot Read Data Store File


Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): `>p3sm --sm-status SM is running. SM operational state is XXX. Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver][TimesTen]TT0845: Cannot read data store file. OS-detected error: Error 0 -- file "db.c", lineno 6320, procedure "sbDbConnect()"`
Probable cause:	TimesTen was unable to read the data store file, probably due to an error during the installation. This error occurs when installing a TimesTen application on top of an existing TimesTen, without having first uninstalled the old TimesTen database.
Solution:	Do the following: 1. Remove the database, by using the SM p3dbCLU with the following commands: `>p3db -destroy-rep-db >p3db -destroy-local-db` 2. Uninstall TimesTen with the following commands: `>su Password: # ~pcube/lib/tt/TimesTen4.5/32/bin/setup. sh -uninstall` 3. Re-install TimesTen, either by running the SM install-tt.sh script or by using the installation files supplied by TimesTen.

TimesTen Configuration Error—Data Store Space Exhausted


Error message:	The following sequence of output appears in the SM log (while using the SM APIs): `java.io.IOException: Failure in putting subscriber 45977166__00:50:bf:97:c1:b2 : [TimesTen][TimesTen 5.0.35 ODBC Driver][TimesTen]TT0802: Data store space exhausted -- file "blk.c", lineno 1571, procedure "sbBlkAlloc"`
Probable cause:	The TimesTen database has already reached its maximum capacity, which caused the operation of adding a new subscriber to the database to fail.
Solution:	Usually, doing just one of the following is sufficient: •Reduce the number of the subscribers handled by the SM (of course, this solution is not always possible). •Configure the system to support a larger number of subscribers. Note that this solution may require editing one or more of the TimesTen configuration files discussed in Introduction as well as rebooting the machine. •Move the SM to a more powerful machine; this could be a faster CPU (or more CPUs), a larger disk, more RAM, etc. For help and guidance in implementing the last two solutions, please contact Cisco Technical Support..

Network Management Command Line Utility (p3net) Errors

First Connection—Operation Timed Out


Error message:	The following sequence of output appears (in response to the command p3net --connect): `>p3net --connect --ne-name=YYYY Error - failed to connect to element 'YYYY' Operation timed out: connect`
Probable cause:	The following causes are possible: •The IP address is incorrect. •The element YYYY is down.
Solution:	The Service Control solutions for the above causes are: •Ensure that the IP address is correct. •Ensure that the element YYYY is online and is connected via its management port.

Status Error—Connection Down


Error message:	The following sequence of output appears (in response to the command p3net --show-ne): `>p3net --show --ne-name=se0 Network Element Information: ============================ Name: YYY Description: testing element Host: X.X.X.X Ip: X.X.X.X Port: 14374 Status: Connection down (Failure in connecting to agent on host, Connection refused: connect) Type: SCE1000 Domain: smartNET.policy.unitTestSubscribers Subscriber Management: Not Active`
Probable cause:	The following causes are possible: •The IP address is incorrect. •The element YYYY is down.
Solution:	The Service Control solutions for the above causes are: •Ensure that the IP address is correct. •Ensure that the element YYYY is online and is connected via its management port. •Ensure that the PRPC adapter is online on the port that the status indicates.

Status Error—Subscriber Management Down


Error message:	The following sequence of output appears (in response to the command p3net --show-ne): `>p3net --show --ne-name=se0 Network Element Information: ============================ Name: YYY Description: testing element Host: X.X.X.X Ip: X.X.X.X Port: 14374 Status: Connection ready Type: SCE1000 Domain: smartNET.policy.unitTestSubscribers Subscriber Management: Not Active`
Probable cause:	The Subscriber Management field indicates whether the SM successfully performed SM-SCE subscriber synchronization. If the value of the field is Not Active , it is possible that the SM failed to synchronize the SCE.
Solution:	One possible solution is to force SM-SCE resynchronization by using the CLU command p3sm --resync.

Subscriber Database Command Line Utility (p3subsdb) Errors

CSV File Validation Error


Error message:	The following sequence of output appears (in response to the command p3subsdb --import): `>p3subsdb --import --file=/export/home/pcube/XXX.csv Error - Failed to validate the file XXX.csv See import-results.txt for detailed errors description. >cat import-results.txt x.csv:1: expected 2 items but got 4 items. 1 subscribers, 1 errors. NO APPLICATION INSTALLED, MAKE SURE TO INSTALL PQI BEFORE IMPORTING CM WITH TUNEABLES.`
Probable cause:	You tried to import a four-field csv file to the SM, but no application (SCA BB) was installed. For example, the following csv file for a SCA BB application contains four fields: `# CSV line format: subscriber-id, domain, mappings, package-id JerryS,subscribers,80.179.152.159,0 ElainB,subscribers,194.90.12.2,3` However, the default definition file that defines csv file parsing rules contains only two fields: name and ip mapping.
Solution:	Do one of the following: •Install an application (SCA BB) on the SM (for details, see the Cisco Service Control Application for Broadband User Guide ). •Import a csv file that has just two fields.

Cable Support Command Line Utility (p3cable) Errors

CSV File Import Error


Error message:	The following sequence of output appears (in response to command p3cable --import-cm): `>p3cable --import-cm --file=/export/home/pcube/XXX.csv Importing cable modems ... 0% Importing cable modems ... 100% Error - Errors during import from 'H:\work\Mng\dev\install\ems\bin\win32\x.cs v': Imported 1 CM(s). 1 Error(s). See cm-import-results.txt for detailed errors description. >cat cm-import-results.txt x.csv:1: expected 2 items but got 4 items. 1 cable modem(s); 1 error(s). NO APPLICATION INSTALLED, MAKE SURE TO INSTALL PQI BEFORE IMPORTING CM WITH TUNEABLES.`
Probable cause:	You tried to import a four-field csv file to the SM, but no application (SCA BB) was installed. For example, the following csv file for a SCA BB application contains four fields: `# CSV line format: subscriber-id, domain, mappings, package-id JerryS,subscribers,80.179.152.159,0 ElainB,subscribers,194.90.12.2,3` However, the default definition file that defines csv file parsing rules contains only two fields: name and ip mapping.
Solution:	Do one of the following: •Install an application (SCA BB) on the SM (for details, see the Cisco Service Control Application for Broadband User Guide ). •Import a csv file that has just two fields.

Configuration Errors

Network Management Errors


Error message (1):	`Error section <section name>: cannot contain white spaces.`
Probable cause:	[SCE.XXX] section cannot contain white spaces (SCE name cannot contain white space).
Solution:	Remove the white spaces.


Error message (2):	`Error in section <section name>: host <ip address>already exists in section <section name>`
Probable cause:	Configuration cannot contain two SCEs with the same IP address.
Solution:	Change the IP address of one of the SCEs.


Error message (3):	`Unknown NE <name>found in domain <domain name>section: it does not have [SCE.<name>] section`
Probable cause:	The section <domain name>includes, under the elements property, an SCE that is not defined in an [SCE.XXX] section.
Solution:	Add the missing [SCE.XXX] section to the file.


Error message (4):	`Duplicate NE <name>found in domain <domain name>section: it already appears in <domain name >domain section.`
Probable cause:	Same SCE cannot belong to more than one domain.
Solution:	Remove the SCE from all but one of the domains.

Domain Errors


Error message (1):	`Error in section <section name>: cannot contain white spaces.`
Probable cause:	[Domain.XXX] section cannot contain white spaces (Domain name cannot contain white space).
Solution:	Remove the white spaces.


Error message (2):	Error <alias name>value - alias name should not start with `CM_'.
Probable cause:	The alias name cannot start with CM_ because this is the prefix of hidden domain generated by the SM when working with CMs (see CPE as Subscriber in Cable Environment ).
Solution:	Use a different prefix for the alias.


Error message (3):	`Alias <alias name>already exists in [<domain name>] section`
Probable cause:	Each alias can appear in only one [Domain.XXX] section.
Solution:	Alias mentioned in the error message should be removed from all but one [Domain.XXX] section.


Error message (4):	`Unknown domain <domain name>found in [LEG-Domains Association]. It does not appear as a section.`
Probable cause:	Domain mentioned in given section does not have a [Domain.XXX] section.
Solution:	Domain mentioned in error message should be given a [Domain.XXX] section.


Error message (5):	`Invalid non-integer value: <value>for property '<property name>' in section [section name].`
Probable cause:	Properties in [Domain.XXX] sections do not have integer values.
Solution:	Properties mentioned in error message should be given integer values.


Error message (6):	`Error in section <domain name>: Property - <name>not found: <list of application properties>`
Probable cause:	Property defined in [Domain.XXX] section is not found in properties list defined by installed application.
Solution:	Delete the properties mentioned in the error message (or define them in the installed application).


Error message (7):	`New configuration was not applied due to the following warnings: Warning - Cannot remove domain <domain name>with <num of SCEs>SCEs. Note that all subscribers will be removed from domain db. Please use '--ignore-warnings' option to complete the operation.`
Probable cause:	This is actually a warning: its purpose is to warn that the user removed from the p3sm.cfg file a domain that contained SCEs (which probably contained subscribers, etc.), and that the user will lose all subscriber data relevant for that domain. This warning appears only after the CLU command p3sm --load-configis activated.
Solution:	To avoid this warning, use the --ignore-warnings option.

PRPC Errors


Error message:	`New configuration was not applied due to the following warnings: Warning - PRPC configuration was changed. Note: Reloading may take up to 5 seconds. Please use '--ignore-warnings' option to complete the operation.`
Probable cause:	This is actually a warning: it is displayed after the CLU command p3sm -load-configis activated when the PRPC configuration in the p3sm.cfg file has been changed.
Solution:	Use the --ignore-warnings option to complete the operation.

RADIUS Listener Errors


Error message:	`Duplicate NAS identifier <nasID>found in section [NAS name] : already exists in <other NAS name>`
Probable cause:	<nasID>is not unique.
Solution:	Change one of the <nasID>so that both are unique.

Common Validation Errors

The following configuration errors are relevant for all sections/parameters of the p3sm.cfg file.


Error message (1):	`Unknown property <property name>found in section [<section name>] in configuration file <file name>.`
Probable cause:	Property written in the p3sm.cfg file is unknown to the SM. Maybe the name is misspelled or the property belongs in a different section.
Solution:	Ensure that the name is spelled correctly and that the property resides in the correct section.


Error message (2):	`Unknown section [<section name>] found in configuration file {2}.`
Probable cause:	The section written in the p3sm.cfg file is unknown to the SM. Maybe the name is misspelled.
Solution:	Ensure that the section name is spelled correctly.


Error message (3):	`Error value <value>for property <property name>in section [<section name>]. Optional values: [<values range>]`
Probable cause:	Value of the property is invalid. The <values range>field contains the valid values.
Solution:	Specify any valid value for the property.


Error message (4):	`Missing mandatory property <property name>in section [<section name>].`
Probable cause:	The property <property name>is mandatory and must appear in the section <section name>.
Solution:	Set a value for the requested property in the specified section.


Error message (5):	`Error value <property value>for property <property name>in [<section name>] section. Valid format: [0..255].[0..255].[0..255].[0..255]`
Probable cause:	The value is an invalid IP address.
Solution:	Specify a valid IP address.


Error message (6):	`Error empty value for <property name>property in [<section name>] section - must have at least one character`
Probable cause:	Value of the <property name>is empty; for example, prop=
Solution:	Specify a non-empty value for the property.


Error message (7):	`Section <section name>added when already exists`
Probable cause:	The section with <section name>appears more than once. This error is most likely to occur for the [SCE.XXX] and [Domain.XXX] sections.
Solution:	Use the specified section name only once.

Table Of Contents