20.4 Integrity-Checking Tools

In this section, we'll look at a few integrity-checking tools that are currently available. This list is not comprehensive, but it is meant to be illustrative. Before you investigate using a third-party tool, however, you should check your documentation to see which tools are bundled in with your operating system.

20.4.1 BSD's mtree and Periodic Security Scans

BSD-derived operating systems come preconfigured with a set of security-checking scripts that are run automatically every night. Located in the directory /etc/periodic/security, these scripts perform a variety of functions, including the reporting of:

SUID files that have been newly created or removed, or that had their permissions changed
Changes in system mount points
New users created with a UID of 0
Users without a password
Firewall violations
Kernel messages
Failed logins

Many Linux distributions provide similar security-checking scripts to be run daily, weekly, and monthly.

In addition to the nightly security script, BSD systems contain a program called mtree that can create a database of file sizes, permissions, attributes, and cryptographic checksums. Once this database is built, the program can report any differences between the database and the files that are actually in the directory. Most BSD systems use the mtree program to create this database when the operating system is first installed, then periodically run the mtree program to report any files that have changed.

An example security output is shown in Example 20-1. In this example, it appears that two SUID programs were modified. If you were the manager of this system and those files were modified without your knowledge, you would want to investigate. It is possible that the change was made by a person who was authorized but simply forgot to tell you. On the other hand, it is possible that an attacker has installed a new version of procmail that has a Trojan Horse that will do something terrible when a codeword is sent to one of your users in an email message.

Example 20-1. An example run of the FreeBSD nightly security report

From root@r2.nitroba.com  Thu Aug 15 02:23:06 2002
Return-Path: <root@r2.nitroba.com>
Delivered-To: simsong@r2.nitroba.com
Received: by r2.nitroba.com (Postfix, from userid 0)
        id AA8A9E44327; Thu, 15 Aug 2002 02:23:05 -0400 (EDT)
To: root@r2.nitroba.com
Subject: r2.nitroba.com security run output
Message-Id: <20020815062305.AA8A9E44327@r2.nitroba.com>
Date: Thu, 15 Aug 2002 02:23:05 -0400 (EDT)
From: root@r2.nitroba.com (Nitroba Root)
Status: O
X-Status:
X-Keywords:
X-UID: 603
r2.nitroba.com setuid diffs:
< 1864963 -rwxr-sr-x 1 root mail 11444 May 2 07:43:42 2001
/raid4/usr/local/bin/lockfile
---
< 1864963 -rwxr-sr-x 1 root mail 12676 August 14 09:38:17 2002 /raid4/usr/local/bin/lockfile
4515c4515
---
< 1864963 -rwxr-sr-x 1 root mail 63560 May 2 07:43:42 2001 /raid4/usr/local/bin/procmail
---
< 1864963 -rwxr-sr-x 1 root mail 66460 August 14 09:38:17 2002 /raid4/usr/local/bin/procmail

Checking setuid files and devices:
r2.nitroba.com setuid diffs:

Checking for uids of 0:
root 0
shutdown 0
toor 0

Checking for passwordless accounts:

r2.nitroba.com denied packets:
> 00200      570        34200 deny ip from any to 127.0.0.0/8
> 00464       70         4200 deny ip from 64.242.140.13 to any
> 01002       29         1476 deny tcp from any to any 3306
> 01020        3          180 deny tcp from any to any 111
> 02001     4167       222584 deny tcp from any to any setup

r2.nitroba.com kernel log messages:
> Aug 14 09:50:34 <auth.notice> r2 su: simsong to root on /dev/ttyp1
> Aug 14 17:41:35 <daemon.err> r2 named[85]: /etc/namedb/named.conf:50: cannot redefine 
zone '15.7.64.in-addr.arpa' cl\
ass IN
> Aug 14 21:32:04 <auth.notice> r2 su: simsong to root on /dev/ttyp0

r2.nitroba.com login failures:

r2.nitroba.com refused connections:

-- End of security output --

20.4.2 Packaging Tools

Many modern versions of Unix include a "packaging" system with which precompiled applications can be distributed and automatically installed (see Chapter 17 for more information about these systems). Both Linux and BSD-based packaging systems include provisions for packages to be cryptographically signed and for the files in the package to be compared with those on the hard disk. For the reasons discussed earlier, this form of integrity checking can be defeated by some forms of attacks. Nevertheless, it is better than having no integrity checking at all!

20.4.2.1 Integrity checking with RPM under Linux

The RPM (or RPM Package Manager) system provides a mechanism for checking a package's installed files against the system's package database.

An RPM package contains all of the files needed for a given application, along with installation routines and cryptographic checksums. After a package is installed, information about the package, including file checksums, is stored in a database on the system. Using the RPM system, the following command will check the integrity of the autorpm package against the system database and report any discrepancies:

# rpm -V autorpm
S.5....T c /etc/autorpm.d/redhat-updates.conf

Each line of output describes a file that has changed from its entry in the database. In this case, the file /etc/autorpm.d/redhat-updates.conf has a different size (S), MD5 checksum (5), and modification time (T) from those expected because this configuration file (which is indicated by c) has been edited from its installed state. The command also checks file mode, device numbers for device files, link paths, and user/group ownership. The command rpm -Va will check all files of all installed RPM packages.

To use rpm as an effective integrity checker, you must ensure that the system package database has not been compromised. The package database is typically a set of Berkeley DB files stored in /var/lib/rpm. Any of the techniques discussed earlier can be used to verify the integrity of this database—making copies to a read-only medium or generating cryptographic signatures of the DB files are particularly suitable.

The RPM package database won't contain any information about software that's not installed through rpm, and thus isn't a complete solution for most systems. For example, although Linux distributions that use RPM usually distribute the kernel image and modules as RPMs, if you build a custom kernel, your kernel would not be installed through rpm unless you made the effort to learn to package your own software. But it's easy to use, and a good start for systems that use it, as well as another way to provide defense-in-depth.

20.4.2.2 Integrity checking with the BSD pkg_info command

The BSD pkg (package) system is similar in spirit to the RPM system. At a bare minimum, a package consists of a list of files and dependencies on other packages. Packages can be downloaded as binary code and installed as such, or they can be compiled from sources and installed.

The BSD pkg_info command lists all of the packages currently residing on the system; options can be given to this command to list all of the files in a given package. Alternatively, a file can be specified to the pkg_info command, and the command will report which package was responsible for installing that file. When provided with the -g option, this command will compare the checksums for the files that have been installed with the package database and report the files whose checksums no longer match.

Overall, the pkg_info command is useful for checking the consistency of subsystems that have been installed from the "ports" directories or installed as packages. Unfortunately, what it can't do is check the integrity of the underlying operating system, as the base operating system is not installed from packages.

20.4.3 Tripwire

Earlier, we described a method of generating a list of file attributes and message digests. The problem with this approach is that we don't really want that information for every file. For instance, we want to know if the owner or protection modes of /etc/passwd change, but we don't care about the size or checksum because we expect the contents to change. At the same time, we are very concerned if the contents of /bin/login are altered.

We would also like to be able to use different message digest algorithms. In some cases, we are concerned enough that we want to use three strong algorithms, even if they take a long time to make the comparison; after all, one of the algorithms might be broken soon.^[10] In other environments, a fast but less secure algorithm, used in conjunction with other methods, might be all that is necessary.

^[10] This is not so far-fetched. Partial attacks are possible on the MD4 message digest algorithm. In 2002, German cryptanalyst Hans Dobbertin discovered a weakness in the compression function used in MD5 that may make it possible for an attacker to generate two different files (e.g., working software and compromised software) that produce the same MD5 digest.

In an attempt to meet these needs^[11] the Tripwire package was written at Purdue by Gene Kim and Gene Spafford. Tripwire is a program that runs on most major versions of Unix (and several obscure versions). It reads a configuration file of files and directories to monitor, and then tracks changes to inode information and contents. The database is highly configurable, and allows the administrator to specify particular attributes to monitor, and particular message digest algorithms to use for each file.

^[11] And more. See the papers that come with the distribution.

Tripwire has been commercialized by Tripwire, Inc., a company founded by Gene Kim and W. Wyatt Starnes. Tripwire, the company, has created a management console for the program, ported it to Windows, and created a specialized version of Tripwire for network devices such as switches, routers, and firewalls. For information on the company and its commercial products, visit Tripwire's web site at http://www.tripwire.com.

In addition to Tripwire's commercial offerings, the company oversees the development of "Open Tripwire," a free version of the Tripwire system. This version of Tripwire is distributed under the GNU Public License (GPL). Information on it can be found at http://www.tripwire.org. Another GPL alternative with similar features is AIDE, which can be found at http://www.cs.tut.fi/~rammer/aide.html.

Today's Tripwire is a sophisticated system that uses policy files to define which parts of a system should be checked. These files are cryptographically signed and used by the Tripwire runtime system. Tripwire can be run in an automated fashion or on an as-needed basis.

Vendor-Supplied Fingerprints

If you're using Solaris, you can take advantage of Sun's Solaris Fingerprints Database. This database stores MD5 checksums for over 2 million files distributed with various versions of Solaris, as well as patches and unbundled Solaris software. To confirm that a file on your system has not been modified, download a clean copy of MD5 from Sun's web site, and generate a checksum for the file. Submit the checksum to the web form at http://sunsolve.sun.com/pub-cgi/fileFingerprints.pl, and the script will tell you if it matches any known Sun-distributed Solaris file. If it doesn't, and you expect it to, it may be compromised.

The Solaris Fingerprints Database has two limitations when compared to products like Tripwire. First, it's not convenient to check large numbers of files against the database through the Web; at the time of this writing, Sun was considering ways to distribute larger portions of the database to make this kind of checking more feasible. Second, it can't help you ensure the integrity of software that you compile yourself for your system or that isn't provided by Sun. But despite these limitations, the database is a truly positive effort by a vendor in the area of integrity management, and we hope that more vendors will follow suit.

In the example in the next section, we explain how to build and use the GPL version of Tripwire. The commercial and open source releases are similar to install and use, although in the commercial release some filenames differ; some additional tools are provided, and the policy and configuration files have more options for integrity checking and reporting. The commercial version also supports a central configuration and reporting station, which can be a big help in a large-scale environment.

20.4.3.1 Building Tripwire

To build the Tripwire package, you must follow these steps:

Download a copy from Tripwire, Inc. If you are using a BSD system, you can build Tripwire out of the /usr/ports/security/tripwire directory. If you wish, you can verify the source code's digital signature. RPMs are also available for some Linux distributions.
Read all the README files in the distribution. Be certain that you understand the topics discussed. Pay special attention to the details of customization for local considerations, including the adaptation for the local operating system. Compile and install the program according to the instructions that are provided.
Create a policy file. Tripwire comes with a basic policy file that will probably be fine for your needs, but it may require some customizations. For example, the Tripwire policy file specifies that a Kerberos 5 system installed in /usr/local/krb5 should be included in the checks. If you don't have Kerberos 5 installed in this directory, you may wish to remove these lines from the Tripwire configuration file, as you will otherwise get errors complaining that these directories cannot be opened whenever Tripwire runs. If a Tripwire policy file is available for your operating system, use it as your starting point.
Install the policy file once it has been is created. This process includes creating a Tripwire key for your site, creating a Tripwire key for your system, creating the configuration file from your policy file, and finally having Tripwire perform an initial scan of your system. All of these procedures are performed automatically by Tripwire's installation scripts, and you should follow them if you possibly can.
Finally, you may wish to copy the binary for Tripwire and the configuration file to a protected directory that is located on (normally) read-only storage. Doing so increases the chances that, if your system is compromised, you will be able to use your Tripwire system to determine which files have been changed.

In general, it is best to install Tripwire on a system that is known to be clean—ideally, a system that has had a clean reinstall of the software. In practice, this is not always possible. Caveat operator!

20.4.3.2 Running Tripwire

You run Tripwire from the protected version on a periodic basis to check for changes. You should occasionally run it manually, rather than only from cron. This step ensures that Tripwire is actually run and you will see the output.

Example 20-2 shows sample output from a Tripwire run. In this case, the run is the second time that the Tripwire program has run on this machine. Notice that Tripwire has detected that certain files (the MRTG output file, as well as DNS temporary files) have changed. The system has also detected that the Tripwire manpages have been removed and replaced with compressed versions of the files.

Example 20-2. A sample Tripwire report

r2# tripwire --check
Parsing policy file: /usr/local/etc/tripwire/tw.pol
*** Processing Unix File System ***
Performing integrity check...
### Warning: File system error.
### Filename: /.login
### No such file or directory
### Continuing...
### Warning: File system error.
### Filename: /kernel.GENERIC
### No such file or directory
### Continuing...
Wrote report file: /var/db/tripwire/report/r2.nitroba.com-20020817-182201.twr

Tripwire(R) 2.3.0 Integrity Check Report

Report generated by:          root
Report created on:            Sat Aug 17 18:22:01 2002
Database last updated on:     Never

=======================================================================
Report Summary:
=======================================================================

Host name:                    r2.nitroba.com
Host IP address:              64.7.15.234
Host ID:                      None
Policy file used:             /usr/local/etc/tripwire/tw.pol
Configuration file used:      /usr/local/etc/tripwire/tw.cfg
Database file used:           /var/db/tripwire/r2.nitroba.com.twd
Command line used:            tripwire --check 

=======================================================================
Rule Summary: 
=======================================================================

-----------------------------------------------------------------------
  Section: Unix File System
-----------------------------------------------------------------------

  Rule Name                       Severity Level    Added    Removed  Modified
  ---------                       --------------    -----    -------  -
  Invariant Directories           66                0        0        0        
  Sources                         100               0        0        0        
  Temporary directories           33                0        0        0        
* Tripwire Data Files             100               1        0        0        
* Local files                     66                8        8        6        
  Tripwire Binaries               100               0        0        0        
  Libraries, include files, and other system files
                                  100               0        0        0        
  System Administration Programs  100               0        0        0        
  User Utilities                  100               0        0        0        
  X11R6                           100               0        0        0        
  NIS                             100               0        0        0        
  (/var/yp)
* /etc                            100               0        0        2        
  Security Control                100               0        0        0        
  Root's home                     100               0        0        0        
  FreeBSD Kernel                  100               0        0        0        
  FreeBSD Modules                 100               0        0        0        
  /dev                            100               0        0        0        
  Linux Compatibility             100               0        0        0        
  (/compat)

Total objects scanned:  98571
Total violations found:  25

=======================================================================
Object Summary: 
=======================================================================

-----------------------------------------------------------------------
# Section: Unix File System
-----------------------------------------------------------------------

-----------------------------------------------------------------------
Rule Name: Local files (/usr/local/etc)
Severity Level: 66
-----------------------------------------------------------------------

Modified:
"/usr/local/etc/mrtg"
"/usr/local/etc/mrtg/mrtg.ok"
"/usr/local/etc/postfix/prng_exch"
"/usr/local/etc/tripwire"

-----------------------------------------------------------------------
Rule Name: Local files (/usr/local/man/man5)
Severity Level: 66
-----------------------------------------------------------------------

Added:

"/usr/local/man/man5/twpolicy.5.gz"
"/usr/local/man/man5/twfiles.5.gz"
"/usr/local/man/man5/twconfig.5.gz"

Removed:
"/usr/local/man/man5/twconfig.5"
"/usr/local/man/man5/twfiles.5"
"/usr/local/man/man5/twpolicy.5"

Modified:
"/usr/local/man/man5"

-----------------------------------------------------------------------
Rule Name: Local files (/usr/local/man/man8)
Severity Level: 66
-----------------------------------------------------------------------

Added:
"/usr/local/man/man8/siggen.8.gz"
"/usr/local/man/man8/tripwire.8.gz"
"/usr/local/man/man8/twadmin.8.gz"
"/usr/local/man/man8/twintro.8.gz"
"/usr/local/man/man8/twprint.8.gz"

Removed:
"/usr/local/man/man8/siggen.8"
"/usr/local/man/man8/tripwire.8"
"/usr/local/man/man8/twadmin.8"
"/usr/local/man/man8/twintro.8"
"/usr/local/man/man8/twprint.8"

Modified:
"/usr/local/man/man8"

-----------------------------------------------------------------------
Rule Name: Tripwire Data Files (/var/db/tripwire)
Severity Level: 100
-----------------------------------------------------------------------

Added:
"/var/db/tripwire/r2.nitroba.com.twd"

-----------------------------------------------------------------------
Rule Name: /etc (/etc)
Severity Level: 100
-----------------------------------------------------------------------

Modified:
"/etc/namedb/sand/sand.PHONESWEEP.COM.bak"
"/etc/namedb/sand/sand.SANDSTORM.NET.bak"

=======================================================================
Error Report: 
=======================================================================

-----------------------------------------------------------------------
  Section: Unix File System
-----------------------------------------------------------------------

 1.  File system error.
     Filename: /.login
     No such file or directory
 2.  File system error.
     Filename: /kernel.GENERIC
     No such file or directory

-----------------------------------------------------------------------
*** End of report ***

Tripwire 2.3 Portions copyright 2000 Tripwire, Inc. Tripwire is a registered trademark of
Tripwire, Inc. This software comes with ABSOLUTELY NO WARRANTY; for details use --version.
This is free software which may be redistributed or modified only under certain 
conditions; see COPYING for details. All rights reserved.
Integrity check complete.
r2#

Tripwire has many options and can be used for other things besides simple change detection. The papers and manpages provided in the distribution are quite detailed, and we recommend that you consult them for further information .