[Chapter 12] 12.4 Security Monitoring

12.4 Security Monitoring

A key element of effective network security is security monitoring. Good security is an ongoing process, and following the security guidelines discussed above is just the beginning. You must also monitor the systems to detect unauthorized user activity, and to locate and close security holes. Over time a system will change - active accounts become inactive; file permissions are changed. You need to detect and fix these problems as they arise.

12.4.1 Know Your System

Network security is monitored by examining the files and logs of individual systems on the network. To detect unusual activity on a system, you must know what activity is normal. What processes are normally running? Who is usually logged in? Who commonly logs in after hours? You need to know this, and more, about your system in order to develop a "feel" for how things should be. Some common UNIX commands - ps and who - can help you learn what normal activity is for your system.

The ps command displays the status of currently running processes. Run ps regularly to gain a clear picture of what processes run on the system at different times of the day, and who runs them. The Linux ps -au command and the ps -ef Solaris command display the user and the command that initiated each process. This should be sufficient information to learn who runs what, and when they run it. If you notice something unusual, investigate it. Make sure you understand how your system is being used.

The who command provides information about who is currently logged into your system. It displays who is logged in, what device they are using, when they logged in and, if applicable, what remote host they logged in from. (The w command, a variation of who available on some systems, also displays the currently active process started by each user.) The who command helps you learn who is usually logged in, as well as what remote hosts they normally log in from. Investigate any variations from the norm.

If any of these routine checks gives you reason to suspect a security problem, examine the system for unusual or modified files, for files that you know should be there but aren't, and for unusual login activity. This close examination of the system can also be made using everyday UNIX commands. Not every command or file we discuss will be available on every system. But every system will have some tools that help you keep a close eye on how your system is being used.

12.4.2 Looking for Trouble

Intruders often leave behind files or shell scripts to help them re-enter the system or gain root access. Use the ls -a | grep '^\.' command to check for files with names that begin with a dot ( . ). Intruders particularly favor names such as .mail , .xx , ... (dot, dot, dot), .. (dot, dot, space), or ..^G (dot, dot, control-G).

If any files with names like these are found, suspect a break-in. (Remember that one directory named . and one directory named .. are in every directory except the root directory.) Examine the contents of any suspicious files and follow your normal incident-reporting procedures.

You should also examine certain key files if you suspect a security problem:

/etc/inetd.conf: Check the names of the programs started from the /etc/inetd.conf file. In particular, make sure that it does not start any shell programs (e.g., /bin/csh ). Also check the programs that are started by inetd to make sure the programs have not been modified. /etc/inetd.conf should not be world-writable.
r command security files: Check /etc/hosts.equiv , /etc/hosts.lpd , and the .rhosts file in each user's home directory to make sure they have not been improperly modified. In particular, look for any plus-sign (+) entries, and any entries for hosts outside of your local trusted network. These files should not be world-writable.
/etc/passwd: Make sure that the /etc/passwd file has not been modified. Look for new usernames, and changes to the UID or GID of any account. /etc/passwd should not be world-writable.
Files run by cron or at: Check all of the files run by cron or at , looking for new files or unexplained changes. Sometimes intruders use procedures run by cron or at to re-admit themselves to the system, even after they have been kicked off.
Executable files: Check all executable files, binaries, and shell files to make sure they have not been modified by the intruder. The master checklist, mentioned in the previous section, is helpful for this. Executable files should not be world-writable.

If you find or even suspect a problem, follow your reporting procedure and let people know about the problem. This is particularly important if you are connected to a local area network. A problem on your system could spread to other systems on the network.

12.4.2.1 Checking files

The find command is a powerful tool for detecting potential filesystem security problems because it can search the entire filesystem for files based on file permissions. Intruders often leave behind setuid programs to grant themselves root access. The following command searches for these files, recursively, starting from the root directory:

# 

find / -user root -perm -4000 -print

This find command starts searching at the root (/) for files owned by the user root ( -user root ) that have the setuid permission bit set ( -perm -4000 ). All matches found are displayed at the terminal ( -print ). If any filenames are displayed by find , closely examine the individual files to make sure that these permissions are correct. As a general rule, shell scripts should not have setuid permission.

You can use the find command to check for other problems that might open security holes for intruders. The other common problems that find checks for are world-writable files ( -perm -2 ), setgid files ( -perm -2000 ), and unowned files ( -nouser -o -nogroup ). World-writable and setgid files should be checked to make sure that these permissions are appropriate. As a general rule, files with names beginning with a dot (. ) should not be world-writable, and setgid permission, like setuid, should be avoided for shell scripts.

The process of scanning the filesystem can be automated with the Tripwire program. Tripwire is available from Purdue University at ftp://coast.cs.purdue.edu/pub/tools/unix/Tripwire . This package not only scans the filesystem for problems, it computes digital signatures to ensure that if any files are changed, the changes will be detected.

12.4.2.2 Checking login activity

Strange login activity, at odd times of the day or from unfamiliar locations, can indicate attempts by intruders to gain access to your system. We have already used the who command to check who is currently logged into the system. To check who has logged into the system in the past, use the last command.

The last command displays the contents of the wtmp file. [6] It is useful for learning normal login patterns and detecting abnormal login activity. The wtmp file keeps a historical record of who logged into the system, when they logged in, what remote site they logged in from, and when they logged out.

[6] This file is frequently stored in /usr/adm or /etc .

Figure 12.2 shows a single line of last command output. The figure highlights the fields that show the user who logged in, the device, the remote location from which the login originated (if applicable), the day, the date, the time logged in, the time logged out (if applicable), and the elapsed time.

Figure 12.2: Last command output

Simply typing last produces a large amount of output because every login stored in wtmp is displayed. To limit the output, specify a username or tty device on the command line. This limits the display to entries for the specified username or terminal. It is also useful to use grep to search last 's output for certain conditions. For example, the command below checks for logins that occur on Saturday or Sunday:

% 

last | grep 'S[au]' | more


craig     console                   Sun Dec 15 10:33   still logged in
reboot    ~                         Sat Dec 14 18:12
shutdown  ~                         Sat Dec 14 18:14
craig     ttyp3    modems.nuts.com  Sat Dec 14 17:11 - 17:43  (00:32)
craig     ttyp2    172.16.12.24     Sun Dec  8 21:47 - 21:52  (00:05)
	.
	.
--More--

The next example searches for root logins not originating from the console. If you didn't know who made the two logins reported in this example, be suspicious:

% 

last root | grep -v console


root      ttyp3    peanut.nuts.com   Tue Oct 29 13:12 - down   (00:03)
root      ftp      almond.nuts.com   Tue Sep 10 16:37 - 16:38  (00:00)

While the last command is a major source of information about previous login activity, it is not the only source. On some systems, the messages file records logins to the root account and failed logins. [7] Failed logins and root logins at odd times or from odd places are suspicious. The following grep command checks /usr/adm/messages for root login activity on a Linux system:

[7] Some systems, such as Solaris, don't log su activity and root logins in the messages file.

% 

grep -i login /usr/adm/messages


Nov 23 10:39:10 peanut login: ROOT LOGIN ON tty1
Nov 23 11:11:50 peanut login: 2 LOGIN FAILURES ON tty1, craig
Nov 23 11:25:11 peanut login: 2 LOGIN FAILURES ON tty1, root
Nov 23 11:25:16 peanut login: ROOT LOGIN ON tty1
Nov 23 11:28:15 peanut login: ROOT LOGIN ON tty1
Nov 24 22:31:40 peanut login: 2 LOGIN FAILURES ON tty1, craig
Nov 27 19:47:52 peanut login: 2 LOGIN FAILURES ON tty1, craig
Nov 29 11:10:36 peanut login: 2 LOGIN FAILURES ON tty1, craig
Dec  1 19:41:50 peanut login: 2 LOGIN FAILURES ON tty1, craig
Dec  9 22:05:27 peanut login: ROOT LOGIN ON tty1

Report any security problems that you detect, or even suspect. Don't be embarrassed to report a problem because it might turn out to be a false alarm. Don't keep quiet because you might get "blamed" for the security breach. Your silence will only help the intruder.

12.4.3 Automated Monitoring

Manually monitoring your system is time-consuming and prone to errors and omissions. Fortunately, several automated monitoring tools are available. The Web site http://ciac.llnl.gov/ciac/ToolsUnixSysMon.html lists many of them. Tripwire (mentioned earlier), Tiger, COPS, and SATAN are all popular monitoring tools. COPS and SATAN are described below.

12.4.3.1 COPS

COPS (Computer Oracle Password and Security) is a collection of programs that automate many of the computer monitoring procedures discussed in the previous sections. As with any monitoring system, COPS detects potential problems; it does not correct them. COPS does not replace personal monitoring by the system administrator, but it does provide additional tools to help the administrator perform monitoring tasks.

The tools in the COPS package check:

Permissions for files, directories, and devices
Contents of /etc/passwd and /etc/group files
Contents of /etc/hosts.equiv and ~/.rhosts files
Changes in SUID status

After completing these checks, COPS mails a report of the results to the system administrator.

COPS can be obtained at ftp://coast.cs.purdue.edu/pub/tools/unix . The tar file contains the source code and instructions for building COPS. Once COPS is built, edit the COPS shell file so that the variable SECURE points to the directory that contains the COPS programs, and the variable SECURE_USERS contains the email address of the person who should receive the COPS report. By default, the report is not mailed to anyone; it is written to a file. To force the report to be mailed to the SECURE_USERS, edit the COPS shell script by changing the MMAIL variable to MMAIL=YES.

The great advantage of COPS is that it is simple. COPS removes the hassles from security monitoring, making it more likely that these tasks will be performed. To run COPS, simply enter:

% 

cops

cops uses the system's hostname to create a directory within the directory defined by the SECURE variable. It writes the security report in this new directory in a file named after the current date. The format of the report's filename is year _ month _ day . For example, on peanut the home directory for the COPS programs is /usr/local/cops . If the current date is January 24, 1997, running the cops program creates the directory /usr/local/cops/peanut and writes the report into that directory with the file name 1997_Jan_24 . Here's a sample report:

peanut:/usr/local/cops/peanut> cat 1997_Jan_24
ATTENTION:
Security Report for Fri Jan 24 16:21:21 EST 1997
from host peanut

**** root.chk ****
**** dev.chk ****
Warning! NFS file system /home/craig exported with no restrictions!
**** is_able.chk ****
Warning! /usr/spool/uucp is _World_ writable!
Warning! /etc/securetty is _World_ readable!
**** rc.chk ****
**** cron.chk ****
**** group.chk ****
**** home.chk ****
Warning! User uucp's home directory /var/spool/uucppublic is mode 01777!
Warning! User nobody's home directory /dev/null is not a directory!
         (mode 020666)
Warning! User guest's home directory /dev/null is not a directory!
         (mode 020666)
**** passwd.chk ****
Warning! Password file, line 15, uid > 8 chars
         postmaster:*:14:12:postmaster:/var/spool/mail:/bin/bash
**** user.chk ****
**** misc.chk ****
**** ftp.chk ****
ftp-Warning! Incorrect permissions on "ls" in /home/ftp/bin!
ftp-Warning! Incorrect permissions on "passwd" in /home/ftp/etc!
ftp-Warning! Incorrect permissions on "group" in /home/ftp/etc!
**** pass.chk ****
**** kuang ****
**** bug.chk****

Look at each line in the report you receive. Some lines might indicate real problems, such as the first warning line in our sample report that indicates /home/craig is exported via NFS without proper access control. Other lines might indicate conditions that are not problems for your system. In our example, we decide to leave /etc/securetty with world-read permission. Read the file docs/warnings for an explanation of each warning message. Evaluate each line of the report and correct anything that needs correcting. Rerun COPS and examine the new report. It should report only the problems that you are willing to accept.

Once you're satisfied with your system's security, schedule COPS to run at regular intervals. New problems can be introduced into your system over time. It's better to have the COPS discover the problem than to have the "robbers" discover it!

12.4.3.2 SATAN

Another tool for testing the security of your system is the Security Administrator's Tool for Analyzing Networks (SATAN). SATAN's introduction was met by near hysteria in the popular press, largely because of the tool's name. Despite its name, SATAN is just another security tool.

SATAN does have some unique features. While COPS is intended for use on an individual system, SATAN is designed to test entire networks of systems. This is both a feature and a problem. If you are the administrator of your network, running SATAN allows you to check all of the systems on the network from one central system. If, however, you are responsible for only one system and you use SATAN to probe the other systems on your network, you will irritate all of the other system administrators on the network who will view the SATAN probes as attempted break-ins. Use SATAN only to test systems on your own network that you have officially recognized authority over.

Another feature of SATAN is that it uses your system's Web browser as the interface for viewing the security reports it generates. This is helpful if you have a large network of systems. The browser's ability to link together related documents allows SATAN to organize various hierarchies of security information. Use the browser to search for the most critical errors, the most troublesome subnets, or the most vulnerable hosts. The screenshot in Figure 12.3 shows a display of hosts listed in sequence from the one with the most security errors to the one with the least. Clicking on a hostname provides a specific report of the errors on that host.

Figure 12.3: SATAN interface

The information in Figure 12.3 comes from the foo.org database provided in the SATAN documentation set. Download the binary file satan.doc.tar.Z from ftp://ftp.win.tue.nl/pub/security/unix . Uncompress and untar the file and follow the simple instructions in the README file to build the documentation system. You can then play with SATAN without the danger of accidentally probing any of the systems on your network. If you like what you see, you can download the full product from the same location by getting the binary file satan.tar.Z .

For many sites, well-informed users and administrators, good password security, and good system monitoring provide adequate network security. But for some security-conscious sites, more may be desired. That "more" is usually some technique for limiting access between systems connected to the network, or for limiting access to the data the network carries. In the remainder of this chapter we look at various security techniques that limit access.