Port Monitor (Essential SNMP)

12.6. Port Monitor

Most TCP/IP services use static ports to listen for incoming requests. Monitoring these ports allows you to see whether particular servers or services are responding or not. For example, you can tell whether your mail server is alive by periodically poking port 25, which is the port on which an SMTP server listens for requests. Some other ports to monitor are FTP (23), HTTP (80) and POP3 (110).[70] A freely available program called netcat can connect to and interact with a specific port on any device. We can write a wrapper for this program to watch a given port or service; if something happens outside of its normal operation, then we can send a trap. In this section, we'll develop a wrapper that checks the SMTP port (25) on our mail server. The program is very simple, but the results are outstanding!

[70]Check your services file for a listing of port numbers and their corresponding services. On Unix systems, this file is usually in the directory /etc; on Windows it is usually in a directory such as C:\WINNT \System32\drivers\etc, though its location may vary depending on the version of Windows you are using.

Before we start to write the program, let's establish what we want to do. Telnet to port 25 of your SMTP server. Once you're connected, you can issue the command HELO mydomain.com. This should give you a response of 250. After you get a response from the mail server, issue the QUIT command, which tells the server you are done. Your session should look something like this:

$ telnet mail.ora.com 25
220 smtp.oreilly.com ESMTP O'Reilly & Associates Sendmail 8.11.2 ready
HELO mydomain.com
250 OK
QUIT
221 closing connection

The netcat program needs to know what commands you want to send to the port you are monitoring. We will be sending only two commands to our mail server, so we'll create a file called input.txt that looks like this:

HELO mydomain.com
QUIT

Next, we should test this file and see what output we get from the server. The actual netcat executable is named nc; to test the file, run it like this:

$ /opt/OV/local/bin/netcat/nc -i 1 mailserver 25 < input.txt

This command produces the same results as the telnet session. You won't see the commands in your input.txt file echoed, but you should see the server's responses. Once you have verified that netcat works and gives the same response each time, save a copy of its output to the file mail_good. This file will be used to determine what a normal response from your mail server looks like. You can save the output to a file with the following command:

$ /opt/OV/local/bin/netcat/nc -i 1 mailserver 25 < input.txt > mail_good

An alternate approach is to search for the line numbered 250 in the mail server's output. This code indicates that the server is up and running, though not necessarily processing mail correctly. In any case, searching for 250 shields you from variations in the server's response to your connection.

Here's a script called mail_poller.pl that automates the process. Edit the appropriate lines in this script to reflect your local environment. Once you have customized the script, you should be ready to go. There are no command-line arguments. The script generates an output file called mail_status that contains a 0 (zero) if the server is okay (i.e., if the output of netcat matches $GOOD_FILE); any number other than 0 indicates that an error has occurred:

#!/usr/local/bin/perl
# filename: mail_poller.pl

$HOME_LOC     = "/opt/OV/local/bin/netcat";
$NC_LOC       = "/opt/netcat/nc";
$DIFF_LOC     = "/bin/diff";
$ECHO_LOC     = "/bin/echo";

$MAIL_SERVER  = "mail.exampledomain.com";
$MAIL_PORT    =  25;
$INPUT_FILE   = "$HOME_LOC\/input.txt";
$GOOD_FILE    = "$HOME_LOC\/mail_good";
$CURRENT_FILE = "$HOME_LOC\/mail_current";
$EXIT_FILE    = "$HOME_LOC\/mail_status";

$DEBUG = 0;

print "$NC_LOC -i 1 -w 3 $MAIL_SERVER $MAIL_PORT 
    \< $INPUT_FILE \> $CURRENT_FILE\n" unless (!($DEBUG));

$NETCAT_RES = system "$NC_LOC -i 1 -w 3 $MAIL_SERVER $MAIL_PORT 
    \< $INPUT_FILE \> $CURRENT_FILE";
$NETCAT_RES = $NETCAT_RES / 256;

if ($NETCAT_RES)
{
    # We had a problem with netcat... maybe a timeout?
    system "$ECHO_LOC $NETCAT_RES > $EXIT_FILE";
    &cleanup;
}

$DIFF_RES = system "$DIFF_LOC $GOOD_FILE $CURRENT_FILE";
$DIFF_RES = $DIFF_RES / 256;

if ($DIFF_RES)
{
    # looks like things are different!
    system "$ECHO_LOC $DIFF_RES > $EXIT_FILE";
    &cleanup;
}
else
{
    # All systems go!
    system "$ECHO_LOC 0 > $EXIT_FILE";
    &cleanup;
}

sub cleanup
{
   unlink "$CURRENT_FILE";
   exit 0;
}

After you run the program, review the results in mail_status. If you can, try shutting down the mail server and running the script again. Your file should now contain a nonzero error status.

Once you have made sure the script works in your environment, you can insert an entry in crontab to execute this program at whatever interval you would like. In our environment, we use a 10-minute interval:

# Check the mail server and create a file that we can poll via OpenView
1,11,21,31,41,51 * * * * /opt/OV/local/bin/netcat/mail_poller.pl

Notice we staggered the polling so that we don't check on the hour, half hour, or quarter hour. Once cron has started updating mail_status regularly, you can use tools such as the extensible OpenView agent to check the file's contents. You can configure the agent to poll the file regularly and send the results to your management console. The entry in my /etc/SnmpAgent.d/snmpd.extend looks like this:

serviceInfo     OBJECT IDENTIFIER ::= { mauro 5 }

-- BEGIN - serviceInfo
--

serMailPort  OBJECT-TYPE
    SYNTAX  INTEGER
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
        "This file is updated via crontab. It uses netcat to check the
         port and push a value into this file.
         FILE-NAME: /opt/OV/local/bin/netcat/mail_status"
    ::= { serviceInfo 0 }

We discuss the syntax of this file in Chapter 11, "Extensible SNMP Agents". Basically, this entry just defines a MIB object in the serviceInfo tree, which is node 5 under my private-enterprise tree. In other words, this object's OID is mauro.serviceInfo.serMailPort (2789.5.0). The object can be read by any program that can issue an SNMP get operation. The DESCRIPTION, as we saw in Chapter 11, "Extensible SNMP Agents", specifies a filename from which the agent will read an integer value to use as the value of this object. This program can easily be modified to monitor any port on any number of machines. If you're ambitious, you might want to think about turning the serMailPort object into an array that reports the status of all your mail servers.

Our goal in this chapter hasn't been to provide you with scripts you can immediately place in your environment. More to the point, we have wanted to show you what's possible, and get you thinking about how you might be able to write scripts that provide elaborate custom monitoring features. If you're thinking creatively about what you can do with SNMP, we've succeeded.