Raw Data Formatting: The RDR Formatter and NetFlow Exporting

Cisco Service Control is able to deliver gathered reporting data to an external application for collecting, aggregation, storage and processing over two protocols:

These two protocols can be used simultaneously in the same deployment. However, any specific destination (external collector) to which data is sent can be configured with only one protocol.

Information About the RDR Formatter and NetFlow Exporting Support

The RDR Formatter

The RDR formatter is used to gather the streams of Raw Data Records (RDRs) events passed from the application, format the data into external reporting protocol (RdrV1 or NetFlowV9), and send these reports to the appropriate destination(s). As the exporting of NetFlow traffic is done by the RDR Formatter, any of the configurations of the RDR Formatter affects the exporting of NetFlowV9 reports. For more information regarding RDR types and a description of their formats, see the Cisco Service Control Application for Broadband Reference Guide.

NetFlow

NetFlow reporting protocol is an industry standard for delivering gathered reporting data for external application for collecting, aggregation, storage and processing. The NetFlow protocol option integrates the Service Control solution with a wide range of existing data collectors and reporters.

NetFlow Terminology

A device (in this case, the RDR formatter component in the SCE platform) with NetFlow services enabled, responsible for exporting information using NetFlowV9 protocol.

A device that receives records from one or more exporters. It processes the received export packet(s) by parsing and storing the record information. Records can be optionally aggregated before being stored on the hard disk.

A packet originating at the exporter, carrying the records of the exporter to the NetFlow collector.

The first part of an export packet. the packet header provides basic information about the packet such as the NetFlow version, number of records contained within the packet, sequence numbering, and the observation domain source ID.

A generic term for a collection of flow records that have a similar structure. In an export packet, one or more flowsets follow the packet header. There are two different types of flowsets:

One or more template records that have been grouped together in an export packet

One or more records, of the same type, that are grouped together in an export packet. Each record is either a flow data record or an options data record previously defined by a template record or an options template record.

A data record that contains values of the flow parameters corresponding to a template record.

NetFlow Exporting Support

The RDR formatter supports the exporting of NetFlowV9 reports and is able to send export packets to the configured destinations. The packets contain template records and data records. The template records define the format of the following data records. Each export packet may contain both types of records, or only one type of records.

Each RDR type supported for NetFlowV9 exporting has a pre-defined mapping that allows the RDR formatter to convert it to a NetFlow V9 report and sent it over a NetFlow destination. The SCE platform maintains template records for several RDR types, with the structure of each NetFlow data record that corresponds to that RDR type. All NetFlow templates are pre-defined; users cannot create or edit the NetFlow templates.

Please note that if an RDR tag that is not supported for NetFlow exporting is configured to be sent over a NetFlow destination, this report will not be formatted and sent, and a special counter will be incremented along and a warning will be logged. (See the output of the show rdr-formatter statistics command - 'unsupported-tags'.)

Although template records and data records can be mixed in the same export packet, the template must precede any related data records. For this reason, templates are included in the first export packet, the first time a record is sent, and again at configured intervals (template refresh) to ensure that data records will be recognized and read correctly.

Data Destinations

About Data Destinations

The SCE platform can be configured with a maximum of eight destinations, three destinations per category. Each destination is defined by the following parameters:

The destination is assigned a priority for each category to which it is assigned.

The following figure illustrates the simplest data destination topology, with only one category and one destination.

The following figure illustrates a complex topology using two categories and four destinations. Each category can send data to three of the four destinations.

Categories

In certain installations, data must be sent to different collector servers according to their type. For instance, in the pre-paid environment, some types of data must be sent to the pre-paid collector to get a new quota, while others should be sent to the mediation system. In this case, the data types are divided into up to four groups, and each group, or category, is assigned to a particular destination or destinations. The categories are defined by the application running on the SCE platform.

The system supports up to four categories. Therefore, the destination must be configured regarding each category in use. Each destination may be assigned to more than one category and may be assigned the same or different priorities for each category.

It is also possible to remove a category from a destination, leaving only the desired category. If all categories are removed, the destination itself is deleted.

By default, the categories are referred to as Category 1 through Category 4. However, the user may define meaningful names for the categories. This generally reduces confusion and prevents errors.

Priority

The priority value is used to indicate whether the destination should be a destination for a given category.

Priority is related to the redundant forwarding mode, in that it indicates which is the primary active connection. Priority values have no effect in multicast forwarding mode.

Each destination is assigned a priority value for each category. The first destination that is configured is automatically assigned a priority of 100 (highest priority) for all categories, unless explicitly defined otherwise.

•

Two destinations may not have the same priority for one category. The priority values for destinations within a category must be unique to have any meaning.

•

If only one priority value is assigned to the destination, that priority is automatically assigned to all categories for that destination.

•

If only one category is assigned a priority value for a destination, no RDRs from the other categories will be sent to the specified destination.

•

Assign a high priority if data from the specified category should be sent to this destination. Assign a low priority if data from the specified category should less likely to be sent to this destination.

•

Redundant forwarding mode — Assign a high priority to the primary destination for the system/category. Assign a lower priority to the secondary destination for the system/category.

Setting DSCP for NetFlow

When using the NetFlowV9 protocol, priority can be defined by configuring a DSCP value to be assigned to the NetFlow packets. This DSCP value defines the DiffServ level of the NetFlow traffic to all destinations.

Forwarding Modes

When more than one destination is defined for a category, the system must decide which of these destinations is to receive the data. This is determined by the forwarding mode. There are three forwarding modes:

•

Redundancy — All records are sent only to the primary (active) connection. If the primary connection fails, the records will be sent to the connected destination with the next highest priority.

•

Multicast — All records are sent to all destinations. This feature may negatively affect performance in an installation with a high rate of data.

•

Simple load balancing — Each successive record is sent to a different destination, one destination after the other, in a round robin manner. It is the responsibility of the collectors to aggregate the records.

If one connection fails, the contents of the history buffer are sent to all connected destinations.

Note Some types of deployments using the NetFlow protocol require multicast forwarding mode. In a deployment where there are multiple destinations for at least one category, and at least one of those is a NetFlow destination, the multicast forwarding mode must be configured.

Protocol

These two protocols can be used simultaneously in the same deployment. However, each destination can support only one protocol.

Transport Type

How to Configure Data Destinations and Categories

How to Configure a Data Destination

About Configuring the Data Destinations

There are three general categories of CLI commands related to the configuration of data destinations:

•

General commands that apply to both the RDRv1 protocol and the NetFlow protocol

•

Commands that are relevant only to the RDR formatter (may affect NetFlow exporting as well)

•

Commands relevant only to the NetFlowV9 protocol and the NetFlow exporting support

Options

In order for the data records, either RDRs or NetFlow export packets, from the SCE platform to arrive at the correct location, the following parameters must be configured:

•

protocol — the protocol used for data sent to the destination (either RDRv1 or NetFlow; if no protocol is assigned the protocol is RdrV1)

•

transport — the transport type, TCP or UDP (optional, as this parameter is determined by the protocol)

A priority value may be assigned. Priority is important in the redundancy forwarding mode, but not crucial in multicast mode. Remember that in multicast mode, the existence of any priority value causes the destination to receive reports. The relationship between priorities and categories is addressed in How to Configure a Destination and Assign Categories.

Step 1

From the SCE(config)# prompt, type rdr-formatter destination ip-addressport portnumber [priority priority] protocol protocol[transport transport] , and press Enter.

Defines the destination. When no category is specified, as in the above example, the specified priority is assigned to all categories.

Configuring the Data Destinations: Examples

This example shows how to configure a simple system with only one destination, using the NetFlow protocol. With only one destination, it is not necessary to configure a priority value.

The following example shows how to configure two destinations in a system without using the categories.

The first destination will automatically be assigned a priority of 100, and therefore the priority does not need to be explicitly defined. For the second destination, the priority must be explicitly defined.

The same priority will automatically be assigned to both categories for each destination, but since the categories will be ignored, this is irrelevant.

SCE(config)# rdr-formatter destination 10.1.1.205 port 33000 protocol RdrV1 transport tcp 
SCE(config)# rdr-formatter destination 10.1.1.206 port 33000 priority 80 protocol RdrV1 
transport tcp

How to Configure the Data Categories

•

Configuring the destinations with the proper priorities for each category, as well as configuring all the other destination parameters, may be approached in several different ways, and may take some planning. Refer to the examples below for illustrations of some of the issues involved in configuring categories.

Step 1

SFrom the SCE(config)# prompt, type rdr-formatter category number category-numbername category-name, and press Enter.

Defines the name for the specified category number. This category name can then be used in any rdr-formattercommand instead of the category number.

How to Configure a Destination and Assign Categories

•

priority — the priority value assigned to this category for this destination (1-100)

•

protocol — the protocol used for data sent to the destination (either RDRv1 or NetFlow; if no protocol is assigned the protocol is RdrV1)

•

transport — the transport type, TCP or UDP (optional, as this parameter is determined by the protocol)

•

Note that within each category the priorities must be unique for each destination.

Step 1

From the SCE(config)# prompt, type rdr-formatter destination ip-addressport portnumbercategory [name category-name| number category-number] [priority priority] [ category [name category-name| number category-number] [priority priority]] [ category [name category-name| number category-number] [priority priority]] [ category [name category-name| number category-number] [priority priority]] protocol protocol[transport transport] , and press Enter.

The following example defines a name for one category, and then configures two destinations, assigning each to a different category (see diagram).

The data from category 1 goes to the first destination, so a high priority was assigned to that category in the first destination, and no priority in the second.

Since all data from category 2 (prepaid) goes to the second destination, the priority assigned to category 2 is assigned only to the second destination and not to the first.

Note that if there is a loss of connection to either destination, transmission of data of the relevant category is interrupted until the connection is re-established. There is no redundant connection defined for either category.

SCE(config)# rdr-formatter category number 2 name prepaid 
SCE(config)# rdr-formatter destination 10.1.1.205 port 33000 category number 1 priority 90 
protocol RdrV1 transport tcp 
SCE(config)# rdr-formatter destination 10.1.1.206 port 33000 category name prepaid 
protocol RdrV1 transport tcp

This example is similar to the above, but a low priority is assigned to the second category for each destination, rather than no priority. This allows each destination to function as a backup for the other in case of a problem with one of the connections (redundancy forwarding mode).

SCE(config)# rdr-formatter category number 2 name prepaid 
SCE(config)# rdr-formatter destination 10.1.1.205 port 33000 category name prepaid 
priority 90 category number 1 priority 25 protocol RdrV1 transport tcp 
SCE(config)# rdr-formatter destination 10.1.1.206 port 33000 category number 1 priority 80 
category name prepaid priority 20 protocol RdrV1 transport tcp

This example demonstrates two methods for assigning one category to the first destination only, while the other category uses the second destination as the primary destination, and the first destination as a secondary destination.

SCE(config)# rdr-formatter category number 2 name prepaid 
SCE(config)# rdr-formatter destination 10.1.1.205 port 33000 category name prepaid 
priority 90 category number 1 priority 10 protocol RdrV1 transport tcp 
SCE(config)# rdr-formatter destination 10.1.1.206 port 33000 category number 1 priority 95 
protocol RdrV1 transport tcp

In the following example, all priority values seem quite high. However, it is the relative values of priorities for a category that determine which destination is the primary destination.

SCE(config)# rdr-formatter category number 2 name prepaid 
SCE(config)# rdr-formatter destination 10.1.1.205 port 33000 priority 90 protocol RdrV1 
transport tcp 
SCE(config)# rdr-formatter destination 10.1.1.206 port 33000 priority 95 protocol RdrV1 
transport tcp 
SCE(config)# no rdr-formatter destination 10.1.1.206 port 33000 category name prepaid 
protocol RdrV1 transport tcp

This example illustrates a more complex configuration with one category (prepaid) assigned to one destination and the other (billing) being sent to both destinations, in multi-cast mode.

The forwarding mode is defined for the entire RDR formatter, not just one category. Since the category "prepaid" goes to only one destination, the forwarding mode is irrelevant. It is relevant, however to the "billing" category, since it goes to two different destinations.

SCE(config)# rdr-formatter forwarding-mode multi-cast 
SCE(config)# rdr-formatter category number 1 name billing 
SCE(config)# rdr-formatter category number 2 name prepaid 
SCE(config)# rdr-formatter destination 10.1.1.205 port 33000 priority 40 protocol 
NetFlowV9 transport udp 
SCE(config)# no rdr-formatter destination 10.1.1.205 port 33000 category name prepaid 
protocol NetFlowV9 transport udp 
SCE(config)# rdr-formatter destination 10.10.10.96 port 33000 category name billing 
priority 90 protocol NetFlowV9 transport udp 
SCE(config)# rdr-formatter destination 10.1.96.0 port 33000 category name prepaid priority 
80prepaid priority 80 protocol NetFlowV9 transport udp

Finally, the following example illustrates a configuration with three categories and three destinations, as follows:

•

Category 3: "Special Prepaid", NetFlow V9 protocol, goes to Destination 3, RDRv1 protocol goes to Destination 2

SCE(config)# rdr-formatter forwarding-mode multi-cast 
SCE(config)# rdr-formatter category number 1 name billing 
SCE(config)# rdr-formatter category number 2 name prepaid 
SCE(config)# rdr-formatter category number 3 name special-prepaid 
SCE(config)# rdr-formatter destination 10.1.1.205 port 33000 category name billing 
priority 90 category name prepaid priority 80 protocol RdrV1 transport tcp 
SCE(config)# rdr-formatter destination 10.10.10.96 port 33000 category name prepaid 
priority 90 category name special-prepaid priority 80 protocol RdrV1 transport tcp 
SCE(config)# rdr-formatter destination 10.1.1.206 port 33000 category name special-prepaid 
priority 90 protocol NetFlowV9 transport udp

How to Configure the Forwarding Mode

In a deployment where there are multiple destinations for at least one category, and at least one of those is a NetFlow destination, the multicast forwarding mode must be configured.

Options

•

redundancy — All records are sent only to the primary (active) connection. If the primary connection fails, the records will be sent to the connected destination with the next highest priority.

•

multicast — All records are sent to all destinations. This feature may negatively affect performance in an installation with a high rate of data.

•

load-balancing — Each successive record is sent to a different destination, one destination after the other, in a round robin manner. It is the responsibility of the collectors to aggregate the records.

Step 1

From the SCE(config)# prompt, type rdr-formatter forwarding-mode mode, and press Enter.

Configuring the Forwarding Mode: Example

How to Configure the RDR Formatter

Options

•

Setting the size of the RDR formatter history buffer. Note, however, that the size of the history buffer must be zero bytes (the default value). Other values may cause duplication of RDRs.

How to Enable the RDR Formatter

Step 1

From the SCE(config)# prompt, type service rdr-formatter , and press Enter.

How to Disable the RDR Formatter

Step 1

From the SCE(config)# prompt, type no service rdr-formatter , and press Enter.

How to Configure the Size of the RDR Formatter History Buffer

Options

Step 1

From the SCE(config)# prompt, type rdr-formatter history-size 0 , and press Enter.

How to Configure the NetFlow Exporting Support

Options

The following options are relevant specifically to NetFlow exporting support (within the RDR-Formatter):

•

Assigning a DSCP value to the NetFlow export packets to a specified destination for priority configuration.

•

Configuring the frequency of exporting the template records (template refresh interval)

How to Configure a DSCP Value for NetFlow

Options

•

dscp-value — DSCP value to be assigned to he NetFlow packets over all destinations (0-63 in HEX format)

Step 1

From the SCE(config)# prompt, type rdr-formatter protocol NetFlowV9 dscp dscp-value, and press Enter.

How to Configure the Template Refresh Interval

Options

•

timeout-value — the frequency of exporting the template records in seconds (1 - 86400 .)

Step 1

From the SCE(config)# prompt, type rdr-formatter destination ip-addressport port-numberprotocol NetFlowV9 template data timeout timeout-value, and press Enter.

How to Configure Dynamic Mapping of RDRs to Categories

About Dynamic Mapping of RDRs to Categories

Each RDR tag has a list of categories. The default category is the one that was assigned when application was loaded.

The configuration of categories to RDR tags is done by adding and removing mappings. A user can add a mapping of an RDR tag to a category and can remove a mapping, including the default mapping. If all categories are removed from a tag, the tag is ignored as long as it remains with no mapped categories.

The user must provide the RDR tag ID and the category number to add or remove. The configuration is saved as part of the application configuration.

How to Configuring Mappings

Options

•

tag-umber — The complete 32 bit value given as an hexadecimal number. The RDR tag must be already configured in the Formatter by the application.

•

category-number — Number of the category (1-4) to which to map the RDR tag.

How to Add a Mapping to a Category

Step 1

From the SCE(config)# prompt, type rdr-formatter rdr-mapping (tag-id tag-numbercategory-number category-number) and press Enter.

If the table already contains a mapping with the same tag and category number, an error is issued and nothing is done.

How to Remove a Mapping from a Category

Step 1

From the SCE(config)# prompt, type no rdr-formatter rdr-mapping (tag-id tag-numbercategory-number category-number) and press Enter.

How to Restore the Default Mapping for a Specified RDR Tag

Step 1

From the SCE(config)# prompt, type default rdr-formatter rdr-mapping tag-id tag-numberand press Enter.

How to Display Data Destination Configuration and Statistics

The following commands can be used to display the RDR formatter configuration and statistics:

Refer to the Cisco Service Control Engine CLI Command Reference for a complete description of the other show rdr-formatter commands.

How to the Display the Current RDR Formatter Configuration

The system can display the complete data destination configuration, or just specific parameters.

Displaying the RDR Formatter Configuration: Example

The following example shows how to display the current RDR formatter configuration.

SCE#show rdr-formatter 
Status: enabled 
Connection is: up 
Forwarding mode: redundancy 
Connection table: 
-------------------------------------------------------------------------| 
Collector   | Port|Status|           Priority per Category:              | 
IP Addres / |     |      |-----------------------------------------------| 
Host-Name   |     |      |Category1  |Category2  |Category3  |Category4  | 
-------------------------------------------------------------------------| 
10.1.1.205  |33000|Up    |100 primary|100 primary|100 primary|100 primary| 
10.1.1.206  |33000|Down  |60         |60         |60         |60         | 
10.12.12.12 |33000|Up    |40         |40         |40         |40         | 
-------------------------------------------------------------------------| 
RDR:    queued:    0 , sent:4460807,  thrown:   0,  format-mismatch:0 
UM:     queued:    0 , sent:      0,  thrown:   0 
Logger: queued:    0 , sent:     39,  thrown:   0 
Last time these counters were cleared: 20:23:05  IST  WED March 14 2007

How to the Display the Current RDR Formatter Statistics

Step 1

From the SCE>prompt, type show rdr-formatter statistics , and press Enter.

Displaying the Current RDR Formatter Statistics: Example

The following example shows how to display the current statistics in a deployment using both RDRv1 and NetFlow protocols.

SCE#show rdr-formatter statistics                    
RDR-formatter statistics: 
========================= 
Category 1: 
sent:               1794517 
in-queue:           0 
thrown:             0 
format-mismatch:    0 
unsupported-tags:   1701243 
rate:               2 RDRs per second 
max-rate:           64 RDRs per second 
Category 2: 
sent:               12040436 
in-queue:           0 
thrown:             0 
format-mismatch:    0 
unsupported-tags:   0 
rate:               12 RDRs per second 
max-rate:           453 RDRs per second 
Category 3: 
sent:               0 
in-queue:           0 
thrown:             0 
format-mismatch:    0 
unsupported-tags:   0 
rate:               0 RDRs per second 
max-rate:           0 RDRs per second 
Category 4: 
sent:               0 
in-queue:           0 
thrown:             0 
format-mismatch:    0 
unsupported-tags:   0 
rate:               0 RDRs per second 
max-rate:           0 RDRs per second 
Destination:    10.56.201.50 Port: 33000 Status: up     
Sent:   13835366 
Rate:   211  Max:   679 
Last connection establishment:  17 hours, 5 minutes, 14 seconds 
Destination:     10.56.204.7 Port: 33000 Status: up     
Sent:   12134054 
Rate:   183  Max:   595 
Sent Templates:         13732 
Sent Data Records:   12134054 
Refresh Timeout (Sec):      5 
Last connection establishment:  17 hours, 5 minutes, 15 seconds

How to Disable the Linecard from Sending RDRs

The silent command disables the linecard from issuing data records. Both RDRs and NetFlow export packets are suppressed.

Table Of Contents