Cisco Service Control Management Suite Collection Manager User Guide

Wireless service providers are successfully rolling out 2.5G and 3G-based data services to their subscribers. These services are expected to significantly increase much needed average revenue per user (ARPU) for sustained business models and rapid rollout of new services.

These data services require new ways of offering services and new ways of billing these services to subscribers. The Cisco Service Control solutions enable:

DSL providers and ISPs targeting residential and business broadband customers must find new ways to get maximum leverage from their existing infrastructures, while differentiating their offerings with enhanced IP services.

Cisco products add a new layer of service intelligence and control to existing networks They:

Cable MSOs have successfully deployed high-speed cable modem services to millions of homes. Now they must move beyond providing commodity broadband access by introducing differentiated services and by implementing the service control necessary to fully manage service delivery through their broadband infrastructure. Cisco Service Control solutions enable:

Cisco provides complete network FCAPS (Fault, Configuration, Accounting, Performance, Security) Management.

Two interfaces are provided for network management:

Where the Cisco Service Control Application for Broadband (SCA BB) enforces different policies on different subscribers and tracks usage on an individual subscriber basis, the Cisco Service Control Management Suite (SCMS) Subscriber Manager (SM) may be used as middleware software for bridging between the OSS and the SCE platforms. Subscriber information is stored in the SM database and can be distributed between multiple platforms according to actual subscriber placement.

The SM provides subscriber awareness by mapping network IDs to subscriber IDs. It can obtain subscriber information using dedicated integration modules that integrate with AAA devices, such as Radius or DHCP servers.

Subscriber information may be obtained in one of two ways:

Service configuration management is the ability to configure the general service definitions of a service control application. A service configuration file containing settings for traffic classification, accounting and reporting, and control is created and applied to an SCE platform. SCA BB provides tools to automate the distribution of these configuration files to SCE platforms. This simple, standards-based approach makes it easy to manage multiple devices in a large network.

Service Control provides an easy-to-use GUI to edit and create these files and a complete set of APIs to automate their creation.

All analysis and data processing functions of the SCE platform result in the generation of Raw Data Records (RDRs). These RDRs are processed by the Cisco Service Control Management Suite Collection Manager. The Collection Manager software is an implementation of a collection system that receives RDRs from one or more SCE platforms. It collects these records and processes them in one of its adapters. Each adapter performs a specific action on the RDR.

RDRs contain a wide variety of information and statistics, depending on the configuration of the system. There are three main categories of RDRs:

As each incoming RDR arrives from an SCE platform, the RDR Server adds an arrival timestamp and the ID of the source SCE platform to it, and then sends the RDR to the Categorizer.

The Categorizer classifies each RDR according to its RDR tag. It decides the destination adapters for the RDR and through which Priority Queue it should be sent.

An RDR can be mapped to more than one adapter. A qualified technician defines the flow in a configuration file based on user requirements.

Each adapter has one or more Priority Queues; a persistent buffer is assigned to each Priority Queue.

A Priority Queue queues each RDR according to its priority level and stores it in a persistent buffer until the adapter processes it.

A persistent buffer is a non-volatile storage area that ensures that the system processes RDRs even in cases of hardware, software, or power failures.

The JDBC Adapter receives RDRs, processes them, and stores the records in a database.

This adapter is designed to be compatible with any database server that is JDBC-compliant, and transforms the records accordingly. The JDBC Adapter can be configured to use a database operating on a remote machine.

The JDBC Adapter is preconfigured to support the following databases:

The CSV Adapter receives RDRs, processes them, and writes the records to files on the disk in comma-separated value format. Using standard mechanisms such as FTP, a service provider’s OSS or a third-party billing system can retrieve these records to generate enhanced accounting and network traffic analysis records.

The TA Adapter receives Subscriber Usage RDRs, aggregates the data they contain, and outputs Top Reports to the database and aggregated daily statistics of all subscribers (not just the top consumers) to CSV files. Top Reports are lists of the top subscribers for different metrics (for example, the top 50 volume or session consumers in the last hour).

This adapter maintains a persistent saved state (saved to disk) to minimize any data loss in case of failure.

The TA Adapter, which uses the JDBC Adapter infrastructure, can be configured to use any JDBC-compliant database, either locally or remotely.

The RAG Adapter processes RDRs of one or more types and aggregates the data from predesignated field positions into buckets. The contents of the buckets are written to CSV files.

The HTTPC Adapter receives RDRs, processes them, and sends them to a Policy Server over HTTP.

The HTTPC Adapter can be configured to set the various HTTP requests according to the various Policy Server modes and the required action for a specific flow.

The HTTPC Adapter receives only two types of RDR: one to signal to the Policy Server that a flow has started, the other to signal that the flow has ended.

In bundled mode, the CM uses the Sybase Adaptive Server Enterprise database, which supports transaction-intensive enterprise applications, allows you to store and retrieve information online, and can warehouse information as needed.

The Sybase database is located on the same server as the other CM components. It uses a simple schema consisting of a group of small, simple tables. The JDBC Adapter sends converted RDRs to the database to be stored in these tables. Records can then be accessed using standard database query and reporting tools. (Cisco provides a template-based reporting tool that can generate reports on subscriber usage, network resource analysis, and traffic analysis; for information about the Service Control reporting tool, see the Cisco Service Control Application Reporter User Guide.)

Database maintenance is performed using operating system commands and scripts. The CM supports automatic purging of old records from the bundled database. By default, the report tables are automatically purged of every record that is more than two weeks old. The records are polled once every hour. Database maintenance can be configured using the dbperiodic.py utility script. For more information, see Managing the Periodic Deletion of Old Records.

Any JDBC-compliant database (for example, Oracle™ or MySQL) may be used with the CM in conjunction with the JDBC Adapter. In this case, the database can be local or remote. You should configure the JDBC Adapter to use this database, and also configure a database pack to supply the CM with the parameters of the database (such as its IP address and port). You should also supply a JDBC driver for the database, to be used by the adapter when connecting to it. For more details about configuring the CM to work with an external database, see Managing the Database and CSV Repository.

The CM distribution contains a script, check_prerequisites.sh, located in the install_scripts directory, that helps to determine whether a system meets the requirements for installing a CM or the bundled Sybase database.

The script checks overall readiness of the system for a CM or Sybase installation. The main prerequisites checked are:

Collection Manager 3.0.0 or later can be installed on any Sun SPARC Machine running Solaris that conforms to the requirements listed in the following sections.

forceload: sys/semsys
forceload: sys/shmsys
(For installations with bundled Sybase) If you are using database periodic delete, the scmscm user should be able to schedule and run cron jobs.

TZ=Japan

TZ=:Etc/GMT+5

Collection Manager 3.0.0 or later can be installed on any i386 running Red Hat Linux that conforms to the requirements listed in the following sections.

scmscm ALL= NOPASSWD: XXX/scripts/dbconf.sh

If you do not want to install Sybase (for example, when working in unbundled mode), go to Phase 2: Installing Collection Manager Software.

The script usage is as follows:

installsyb.sh --sybhome=SYBHOME { --datadir=DATADIR | --expert }

The following table describes the TCP/UDP ports on which the CM software and associated components (such as the Sybase database) listen. This table may help the network administrator understand the behavior of the software and its adherence to the security policy.

The ports listed are those on which the device listens constantly. You should allow access on these port numbers; otherwise, certain operations may fail.

Some operations (such as file transfer) cause a device to temporarily open ports other than those listed; however, these ports close automatically when the operation ends.

Since the database structures have changed significantly for release 3.0, a special procedure must be followed in order to preserve the data while upgrading.

The upgrade procedure is described in the following sections.

To set which servers (CM or Sybase) are activated at startup, use the on-boot.py script:

~scmscm/setup/on-boot.py --cm=flag --sybase=flag

Changes take effect the next time the system restarts.

Run the script with no parameters to see the current startup status of each component.

To shut down or bring up a configured adapter, or to list the currently running CM adapters, use the adapterconf.py script:

~scmscm/scripts/adapterconf.py --op=action [--adapter=adapter name]

An adapter can be defined to turn on when the CM starts by removing the remark character at the start of the appropriate line in the cm.conf file.

Example

The following example defines the RAG adapter to turn on when the CM starts.

adapter.4 = com.cisco.scmscm.adapters.rag.RAGAdapter

Example

The following example defines the CSV adapter to remain off when the CM starts.

#adapter.2 = com.cisco.scmscm.adapters.CSVAdapter

To shut down or start the CM database, or to show the operational status of the database, use the dbconf.sh script:

~scmscm/scripts/dbconf.sh --op=action

The script can only be used with a bundled database.

To drop a connection to a particular SCE, use the sceconf.py script:

~scmscm/scripts/sceconf.py --op=drop --ip=IP address

This script can be used only if the HTTP Adaptor of the CM is running.

This script is also used to display information about the SCE connection. (See Checking the SCE Connection.)

To make (or remove) an entry for monitor.sh, the periodic checker script, in the cron (periodic scheduler) subsystem, use the setup-monitor.sh script:

~scmscm/setup/monitor/setup-monitor.sh -a flag [-i flag]

$ ./setup-monitor.sh -a uninstall

The periodic checker script, monitor.sh, calls a series of sub-scripts that monitor different aspects of a running system:

~scmscm/setup/monitor/monitor.sh { -a | TEST NAME } [ -v ] [ -d ]

The script is not intended to be run from the command line, although you can do so. Test results are sent to the syslog subsystem and are logged in the file /var/log/messages.

To display the percentage of free space in the database report tables and the associated transaction log, use the dbfree.sh script:

~scmscm/scripts/dbfree.sh

The script can be used only with a bundled database.

To display the momentary total rate of reports entering the CM, use the rdr-rate.py script

~scmscm/scripts/rdr-rate.py

The output is a single floating-point number representing the total rate per second of incoming RDRs (from all sources) that have entered the CM in the past 5 seconds.

This script can be used only if the HTTP Adaptor of the CM is running.

To display information about the SCE connections, use the sceconf.py script:

~scmscm/scripts/sceconf.py --op=list

This script can be used only if the HTTP Adaptor of the CM is running.

The script is also used to drop a connection from a particular SCE. See Dropping an SCE Connection.

Example

The following example shows SCE connection output:

To verify that the Server is functioning correctly, use the alive.sh script:

~scmscm/setup/alive.sh

The script verifies that the following components are operational:

If any component is down, the script issues an error message.

To list all of the tables in the database, use the dbtables.sh script:

~scmscm/scripts/dbtables.sh

Where applicable, the number of lines in the table and the earliest and latest timestamps are displayed.

Actual content of the tables can be displayed using the Cisco Service Control Application (SCA) Reporter. For more information, see the Cisco Service Control Application Reporter User Guide.

Example

The following is a sample output of dbtables.sh:

To delete a single table or all current tables from the database, use the droptable.sh script:

~scmscm/scripts/droptable.sh [-f] tableParameter

To remove records from a database table based on the timestamps of the records, use the prunetable.sh script:

~scmscm/scripts/prunetable.sh [-f] num_days table_name

Periodic deletion of a table does not begin while a previous periodic deletion is still running. This prevents excessive load on the database, which would degrade insertion performance in the adapters.

When two or more tables are scheduled to be reduced at the same time, the tables are processed in the order in which they are listed in the periodic delete configuration file.

For ease of configuration, you can schedule periodic deletion for all tables consecutively on one schedule.

To create text file backups of all the tables in the database, use the sybback.sh script:

~scmscm/scripts/sybback.sh -d path_to_backup_directory

The script converts all tables to ASCII files and copies the files to a backup directory.

To restore a database from the backup file that was created by the sybback.sh script, use the sybrestore.sh script:

~scmscm/scripts/sybrestore.sh -d path_to_restore_directory

CSV files are stored in several subdirectories. Each subdirectory is given the number of a Raw Data Record (RDR) tag. (RDR tags define the type of the RDR.) Each RDR is stored in the subdirectory whose name matches its RDR tag number. For more information on RDR tags, see the Cisco Service Control Application for Broadband Reference Guide.

The CSV files are (automatically) sequentially numbered, with separate numbering in each directory. You can change the location of the parent directory by editing the cm.conf file located in the cm/config directory.

Use the csvconf.sh script, ~scmscm/scripts/csvconf.sh, to:

> ~scmscm/scripts/csvconf.sh --maxlines=10000

When a comma is contained within a field in a CSV file, an escape sequence is used to indicate that the comma does not mark the end of the field.

Three escape methods are supported:

The first two escape methods are compatible with Microsoft^® Excel. The Backslash method is not compatible with Excel, but is retained for backward compatibility.

By default, single quotation marks are used. You can change the escape method by modifying the value of the escapeMethod attribute. This attribute is located in the csvadapter.conf file in the CSVAdapter directory. The value must be one of: backslash, quote, or url.

Optionally, the CSV Adapter can escape nonprintable characters. Enabling this option incurs a performance penalty for the adapter; by default, the option is disabled.

When the option is enabled, each non-printable character, such as CR and LF, contained within a field is preceded by a backslash (\).

This option can be enabled in the csvadapter.conf file in the CSVAdapter directory. Changes in this file require a CM restart to take effect.

Before the VM templates are loaded and parsed by any CM components (for instance, a TA or JDBC Adapter, or a script), the parsing context is initialized with the following Java objects:

All application-related configuration is done in one file (tables.xml) that includes the following items:

This information is used primarily to populate the tables object in the template parsing context. See The tables Object.

The following sample illustrates definitions for the mapping between native types and SQL types, and utility macros such as the optcomma macro, which inserts a comma between successive elements of lists.

In the following code sample, note that the only required fields are the URL and connection options (for authentication).

Blank lines in the code separate the code into distinct fields for readability and to ease later configuration changes.

Any string can be parsed as a VTL template with the complete context in place. The result of the parsing is displayed on the standard output. To parse a string, call the adapter with the -parse flag. Here are a few examples (the responses are shown in bold):

$ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter -parse 'xxx'

xxx

$ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter -parse '$dbinfo.url'

jdbc:oracle:thin:@localhost:1521:apricot

$ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter -parse '$tools.getTableByName($tables, "LUR").sql.createTable'

create table RPT_LUR (
 TIME_STAMP   date
   ,RECORD_SOURCE   integer
   ,LINK_ID   integer
   ,GENERATOR_ID   integer
   ,SERVICE_ID   integer
   ,CONFIGURED_DURATION   integer
   ,DURATION   integer
   ,END_TIME   integer
   ,UPSTREAM_VOLUME   integer
   ,DOWNSTREAM_VOLUME   integer
   ,SESSIONS   integer
  )

To see a dump of all the contents of the tables and dbinfo structures as created by the templates, use the -debug flag. When this flag is used, a very detailed view of all the fields, properties, and options of these structures is printed to standard output.

The following sample of an update operation demonstrates how to programmatically change the value in the database table holding the Service Control Engine (SCE) time zone offset setting. The name of this table is usually JCONF_SE_TZ_OFFSET; since the table may be assigned another name, it is referred to here by its logical name TZ. See the listing in The tables.xml File.

To avoid the need to first check that the table exists and then update it, the table is dropped (ignoring the error status if it does not exist) and then recreated, and the proper values are inserted. Since the table contains a timestamp column, you must get the current date from the database. This operation is specific to each database vendor; therefore this example uses the preconfigured getdate operation that has been defined in the templates.

Note the use of the tools assignParams and getTableByName to generate the SQL.

When a result set is returned by an executed query, it is displayed to standard output in tabular form with appropriate column headers.

Suppose you have created several tablespaces and wish to distribute the CM tables among them. An easy way to do this is to specify the tablespace to be used for each table in the file tables.xml. For one table, the definition looks like this (note especially the code in bold):

This sample adds the required tablespaces (tspace1 and tspace2) for the index and for the table. There is no preconfigured meaning to the option tablespace in the CM; any new option name could have been used. Its meaning is derived from its subsequent use in the templates.

To create the table in the correct tablespace, modify create_table.vm as follows:

To create the index in its own tablespace, modify create_indexes.vm as follows:

To implement rolling partitioning for a particular table on a weekly basis, you can create a partitioned option for the table in the tables.xml file in a similar manner to the example in the previous section (Using Custom tablespaces). Then augment the create_table.vm code as follows (note especially the code in bold):

Since Oracle does not accept nonconstant expression for the time boundaries, the values must be hardwired for the time the tables are created.

Create a cron job to roll the partitions (delete an old partition and create a new one) on a weekly basis. This cron job runs a script that calls the command-line interface of the JDBC Adapter (as explained in Using the JDBC Framework in Scripts) to issue the appropriate alter table drop partition and alter table add partition SQL commands.