Log Files and Auditing (Network Troubleshooting Tools)

11.2. Log Files and Auditing

A primary source of information on any system is its log files. Of course, log files are not unique to networking software. They are simply another aspect of general systems management that you must master.

Some applications manage their own log files. Web servers and accounting software are prime examples. Many of these applications have specific needs that aren't well matched to a more general approach. In dealing with these, you will have to consult the documentation and deal with each on a case-by-case basis. Fortunately, most Unix software is now designed to use a central logging service, syslog, which greatly simplifies management.

11.2.1. syslog

You are probably already familiar with syslog, a versatile logging tool written by Eric Allman. What is often overlooked is that syslog can be used across networks. You can log events from your Cisco router to your Unix server. There are even a number of Windows versions available. Here is a quick review of syslog.

An early and persistent criticism of Unix was that every application seemed to have its own set of log files hidden away in its own directories. syslog was designed to automate and standardize the process of maintaining system log files. The main program is the daemon syslogd, typically started as a separate process during system initialization. Messages can be sent to the daemon either through a set of library routines or by a user command, logger. logger is particularly useful for logging messages from scripts or for testing syslog, e.g., checking file permissions.

11.2.1.1. Configuring syslog

syslogd 's behavior is initialized through a configuration file, which by default is /etc/syslog.conf. An alternative file can be specified with the -f option when the daemon is started. If changes are made to the configuration file, syslogd must be restarted for the changes to take effect. The easiest way to do this is to send it a HUP signal using the kill command. For example:

bsd1# kill -HUP 127

where 127 is the PID for syslogd, found using the ps command. (Alternately, the PID is written to the file /var/run/syslogd.pid on some systems.)

The configuration file is a text file with two fields separated by tabs, not spaces! Blank lines are ignored. Lines beginning with # in the first column are comments. The first field is a selector, and the second is an action. The selector identifies the program or facility sending the message. It is composed of both a facility name and a security level. The facility names must be selected from a short list of facilities defined for the kernel. You should consult the manpage for syslogd for a complete list and description of facilities, as these vary from implementation to implementation. The security level is also taken from a predefined list: emerg, alert, crit, err, warning, notice, info, or debug. Their meanings are just what you might guess. emerg is the most severe. You can also use * for all or none for nothing. Multiple facilities can be combined on a single line if you separate them with commas. Multiple selectors must be separated with semicolons.

The Action field tells where to send the messages. Messages can be sent to files, including device files such as the console or printers, logged-in users, or remote hosts. Pathnames must be absolute, and the file must exit with the appropriate permissions. You should be circumspect in sending too much to the console. Otherwise, you may be overwhelmed by messages when you are using the console, particularly when you need the console the most. If you want multiple actions, you will need multiple lines in the configuration file.

Here are a few lines from a syslog.conf file that should help to clarify this:

mail.info                                       /var/log/maillog
cron.*                                          /var/log/cron
security.*                                      @loghost.netlab.lander.edu
*.notice;news.err                               root
*.err                                           /dev/console
*.emerg                                         *

The first line says that all informational messages from sendmail and other mail related programs should be appended to the file /var/log/maillog. The second line says all messages from cron, regardless of severity, should be appended to the file /var/log/cron. The next line says that all security messages should be sent to a remote system, loghost.netlab.lander.edu. Either a hostname or an IP address can be used. The fourth line says that all notice-level messages and any news error messages should be sent to root if root is logged on. The next to last line says that all error messages, including news error messages, should be displayed on the system console. Finally, the last line says emergency messages should be sent to all users. It is easy to get carried away with configuration files, so remember to keep yours simple.

One problem with syslog on some systems is that, by default, the log files are world readable. This is a potential security hole. For example, if you log mail transactions, any user can determine who is sending mail to whom -- not necessarily something you want.

11.2.1.2. Remote logging

For anything but the smallest of networks, you really should consider remote logging for two reasons. First, there is simply the issue of managing and checking everything on a number of different systems. If all your log files are on a single system, this task is much easier. Second, should a system become compromised, one of the first things crackers alter are the log files. With remote logging, future entries to log files may be stopped, but you should still have the initial entries for the actual break-in.

To do remote logging, you will need to make appropriate entries in the configuration files for two systems. On the system generating the message, you'll need to specify the address of the remote logging machine. On the system receiving the message, you'll need to specify a file for the messages. Consider the case in which the source machine is bsd1 and the destination is bsd2. In the configuration file for bsd1, you might have an entry like:

local7.*                                        @bsd2.netlab.lander.edu

bsd2 's configuration file might have an entry like:

local7.*                                        /var/log/bsd1

Naming the file for the remote system makes it much easier to keep messages straight. Of course, you'll need to create the file and enable bsd2 to receive remote messages from bsd1.

You can use the logger command to test your configuration. For example, you might use the following to generate a message:

bsd1# logger -p local7.debug "testing"

This is what the file looks like on bsd2:

bsd2# cat bsd1
Dec 26 14:22:08 bsd1 jsloan: testing

Notice that both a timestamp and the source of the message have been included in the file.

There are a number of problems with remote logging. You should be aware that syslog uses UDP. If the remote host is down, the messages will be lost. You will need to make sure that your firewalls pass appropriate syslog traffic. syslog messages are in clear text, so they can be captured and read. Also, it is very easy to forge a syslog message.

It is also possible to overwhelm a host with syslog messages. For this reason, some versions of syslog provide options to control whether information from a remote system is allowed. For example, with FreeBSD the -s option can be used to enter secure mode so logging requests are ignored. Alternately, the -a option can be used to control hosts from which messages are accepted. With some versions of Linux, the -r option is used to enable a system to receive messages over the network. While you will need to enable your central logging systems to receive messages, you should probably disable this on all other systems to avoid potential denial-of-service attacks. Be sure to consult the manpage for syslogd to find the particulars for your system.

Both Linux and FreeBSD have other enhancements that you may want to consider. If security is a major concern, you may want to investigate secure syslog (ssyslog) or modular syslog (msyslog). For greater functionality, you may also want to look at syslog-ng.

telnet stream  tcp     nowait  root    /usr/libexec/telnetd    telnetd

to:

telnet  stream  tcp     nowait  root    /usr/sbin/tcpd          telnetd

Now, the wrapper daemon tcpd is started initially instead of telnetd, the telnet daemon. You'll need to make similar changes for each service you want to control. If the service is not where tcpd expects it, you can give an absolute path as an argument to tcpd in the configuration file.

TIP: Actually, there is an alternative way of configuring tcpwrappers. You can leave the inetd configuration file alone, move each service to a new location, and replace the service at its default location with tcpd. I strongly discourage this approach as it can create maintenance problems, particularly when you upgrade your system.

As noted, tcpwrappers is typically used for two functions -- logging and access control.[40] Logging is done through syslog. The particular facility used will depend on how tcpwrappers is compiled. Typically, mail or local2 is used. You will need to edit /etc/syslog.conf and recompile tcpwrappers if you want to change how logging is recorded.

[40]tcpwrappers provides additional functionality not described here, such as login banners.

Access is typically controlled through the file /etc/hosts.allow, though some systems may also have an /etc/hosts.deny file. These files specify which systems can access which services. These are a few potential rules based on the example configuration:

ALL : localhost : allow
sendmail : nice.guy.example.com : allow
sendmail : .evil.cracker.example.com : deny
sendmail : ALL : allow

tcpwrappers uses a first match wins approach. The first rule allows all services from the local machine without further testing. The next three rules control the sendmail program. The first rule allows a specific host, nice.guy.example.com. All hosts on the domain .evil.cracker.example.com are blocked. (Note the leading dot.) Finally, all other hosts are permitted to use sendmail.

There are a number of other forms for rules that are permitted, but these are all pretty straightforward. The distribution comes with a very nice example file. But, should you have problems, tcpwrappers comes with two utilities for testing configuration files. tcpdchk looks for general syntax errors within the file. tcpdmatch can be used to check how tcpd will respond to a specific action. (Kudos to Venema for including these!)

The primary limitation to tcpwrappers is that, since it disappears after it starts the target service, its control is limited to the brief period while it is running. It provides no protection from attacks that begin after that point.

tcpwrappers is a ubiquitous program. In fact, it is installed by default on many Linux systems. Incidentally, some versions of inetd now have wrappers technology built-in. Be sure to review your documentation.

11.2. Log Files and Auditing

11.2.1. syslog

11.2.1.1. Configuring syslog

11.2.1.2. Remote logging

11.2.2. Log File Management

11.2.3. Other Approaches to Logging

11.2.3.1. tcpwrappers