|
22.3 Cleaning Up After the IntruderThis section discusses in detail how to find out what an intruder may have done and how you should clean up afterwards. 22.3.1 Analyzing the Log FilesEven if you don't catch an intruder in the act, you still have a good chance of finding the intruder's tracks by routinely looking through the system logs. (For a detailed description of the Unix log files, see Chapter 21.) Remember: look for things out of the ordinary. For example:
On the other hand, if the intruder is sufficiently skillful and achieves superuser access on your machine, he may erase all evidence of the invasion. Simply because your system has no record of an intrusion in the log files, you can't assume that your system hasn't been attacked. Many intruders operate with little finesse: instead of carefully editing out a record of their attacks, they simply delete or corrupt the entire log file. This means that if you discover a log file deleted or containing corrupted information, there is a possibility that the computer has been successfully broken into. However, a break-in is not the only possible conclusion. Missing or corrupted logs might mean that one of your system administrators was careless; there might even be an automatic program in your system that erases the log files at periodic intervals. You may also discover that your system has been attacked if you notice unauthorized changes in system programs or in an individual user's files. This is another good reason for using something like the Tripwire tool to monitor your files for changes (see Chapter 20). If your system logs to a hardcopy terminal or another computer, you may wish to examine that log first because you know that it couldn't have been surreptitiously modified by an attacker coming in by the telephone or network. 22.3.2 Preserving the EvidenceIf you wish to prosecute your attacker (if you ever find the person) or sue them for damages, you will need to preserve some evidence that a crime has been committed. Even if you do not wish to take any legal action, you may find it useful to collect evidence of the attack so that you have the ability to reconstruct what happened. If you have a compromised system, a very powerful thing that you can do is to make a block-for-block image of the partitions on the compromised computer. These images can then be analyzed at a later time using The Coroner's Toolkit (written by Dan Farmer and Wietse Venema), TASK (an updated version of TCT by Brian Carrier), or a commercial product such as EnCase (Guidance Software). You can make a block-for-block image by either copying a disk's raw partitions into a disk file of a larger hard drive (such as a high-capacity removable Firewire drive) or by copying the blocks over a local area network to a suitable system. For example, suppose that this is the df output of the compromised system: # df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/ad0s1a 257838 80448 156764 34% /devfs
1 1 0 100% /dev
/dev/ad0s1d 257838 4 237208 0% /tmp
/dev/ad0s1f 7025318 1747156 4716138 27% /usr
/dev/ad0s1e 257838 5090 232122 2% /var
#
You should probably make a copy of the /dev/ad0s1a, /dev/ad0s1d, /dev/ad0s1f, and /dev/ad0s1e partitions, used for the /, /tmp, /usr, and /var partitions. You should probably also copy the /dev/ad0s1b partition, which is used for swapping. To make these copies with the dd command and send the contents to another computer, you could pipe the output from dd into ssh: # dd if=/dev/ad0s1a conv=noerror,sync | ssh myhost "cat > root.img" # dd if=/dev/ad0s1b conv=noerror,sync | ssh myhost "cat > swap.img" # dd if=/dev/ad0s1d conv=noerror,sync | ssh myhost "cat > tmp.img" # dd if=/dev/ad0s1f conv=noerror,sync | ssh myhost "cat > usr.img" # dd if=/dev/ad0s1e conv=noerror,sync | ssh myhost "cat > var.img" Making a block-for-block copy of a partition will freeze in place all of the files—including files that are regular files and files that were recently deleted but are still accessible. It will not, however, freeze memory, capture the current state of processes, or otherwise freeze the running system. For that you will need to take other measures such as using the lsof command or making selective copies of items in the /proc filesystem. Here are some other ideas:
There are commercial products that you may find useful to assist you in preserving evidence, including high-speed disk duplicators and network forensics analysis tools (NFATs) that record all packets entering and leaving an organization. If you have involved law enforcement authorities, try to speak with them before attempting to preserve any evidence on your own. 22.3.3 Assessing the DamageIf your intruder gained superuser access, or access to another privileged account such as mail, he may have modified your system to make it easier for him to break in again. In particular, your intruder may have:
If the intruder committed either of the last two misdeeds, he'll potentially have access to a legitimate account and will be able to get back in no matter what other precautions are taken. You'll have to change all of the passwords on the system. After a successful break-in, you must perform a careful audit to determine the extent of the damage. Depending on the nature of the break-in, you'll have to examine your entire system. You may also need to examine other systems on your local network, or possibly the entire network (including routers and other network devices).
22.3.3.1 New accountsAfter a break-in, scan the /etc/passwd file for newly created accounts. If you have made a backup copy of /etc/passwd, use diff to compare the two files. But don't let the automated check be a substitute for going through the /etc/passwd file by hand because the intruder might have also modified your copy of the file or the diff program. (This is the reason why it is advantageous to keep a second copy of the /etc/passwd file and all of your comparison tools on removable media such as a Zip disk.) Delete any accounts that have been created by an intruder. You may wish to make a paper record of each account before deleting it in case you wish to prosecute the intruder (assuming that you ever find the villain). Also, be sure to check that every line of the /etc/passwd file is in the proper format, and that no UID or password fields have been changed to unauthorized values. Remember: simply adding an extra colon to the /etc/passwd entry for root can do the same amount of damage as removing the superuser's password entirely! The following awk command will print /etc/passwd entries that do not have seven fields,[5] that specify the superuser, or that do not have a password. In this example, note that you may need to replace cat /etc/passwd with another command to display the file containing encrypted passwords, particularly if you use shadow passwords or a network-based authentication system like NIS or NIS+:
# cat /etc/passwd | awk -F: 'NF != 9|| $3 == 0 || $2 == "" { print $1 " " $2 " " $3}'
root xq7Xm0Tv 0
johnson f3V6Wv/u 0
sidney 104
#
This awk command sets the field separator to the colon (:), which matches the format of the /etc/passwd file. The awk command then prints out the first three fields (username, password, and UID) of any line in the /etc/passwd file that does not have seven fields, has a UID of 0, or has no password. In this example, the user johnson has had her UID changed to 0, making her account an alias for the superuser, and the user sidney has had his password removed. Here are the original lines from /etc/passwd: johnson:f3V6Wv/u:103:103:Glenda Johnson:/usr/home/johnson:/bin/tcsh sidney:8d3kf873:104:104:Sidney Smith:/usr/home/sidney:/bin/tcsh and here are the resulting modified lines: johnson:f3V6Wv/u:0:0:Glenda Johnson:/usr/home/johnson:/bin/tcsh sidney::104:104:Sidney Smith:/usr/home/sidney:/bin/tcsh This automated check is much more reliable than a visual inspection, but make sure that the script you use to run this automated check hasn't been corrupted by an attacker. One approach is to type the awk command each time you use it instead of embedding it in a shell script (or use Tripwire). 22.3.3.2 Changes in file contentsAn intruder who gains superuser privileges can change any file on your system. Although you should make a thorough inventory of your computer's entire filesystem, you should look specifically for any changes to the system that affect security. For example, an intruder may have inserted trap doors or logic bombs to do damage at a later point in time. One way to easily locate changes to system programs is to use the guidelines described in Chapter 6. 22.3.3.3 Changes in file and directory protectionsAfter a break-in, review the protection of every critical file on your system. Intruders who gain superuser privileges may change the protections of critical files to make it easier for them to regain superuser access in the future. For example, an intruder might have changed the mode of the /bin directory to 777 to make it easier to modify system software in the future, or altered the protections on /dev/kmem to modify system calls directly using a symbolic debugger. 22.3.3.4 New SUID and SGID filesIntruders who gain superuser access frequently create SUID and SGID files. After a break-in, scan your system to make sure that new SUID files have not been created. See the section Section 6.5.6 for information about how to do this. 22.3.3.5 Changes in .rhosts filesAn intruder may have created new .rhosts files in your users' home directories, or may have modified existing .rhosts files. (The .rhosts file allows other users on the network to log into your account without providing a password. For more information, see Chapter 12.) After a break-in, tell your users to check their .rhosts files to make sure that none of these files have been modified. Also, Chapter 16 contains a shell script that you can use to get a list of every .rhosts file on the system. After a break-in, you may wish to delete every .rhosts file rather than take the chance that a file modified by the attacker won't be caught by the account's rightful owner. After all, the .rhosts file is simply a convenience, and your legitimate users can recreate their .rhosts files as necessary.[6]
22.3.3.6 Changes to .ssh/authorized_keys filesMany security-conscious systems don't run an rlogin daemon. Instead, they run the more secure SSH daemon. Unfortunately, an intruder who gains access to your system can also create or modify the authorized_keys file (typically in the .ssh subdirectory of each user's home directory). By placing his public key into this file, the intruder (or anyone to whom he gives the private key) can connect to the user's account again. After a break-in, tell your users to check (or delete and recreate) their authorized_keys files as well. Or you may decide to take no chances and delete every authorized_keys file on your system. 22.3.3.7 Changes to the /etc/hosts.equiv fileA n intruder may have added more machines to your /etc/hosts.equiv file, so be sure to check for changes to this file. Also, check your /etc/netgroups and /etc/exports files (or equivalents) if you are running NIS or NFS. 22.3.3.8 Changes to startup filesAn intruder may have modified the contents of dot (.) files in your users' home directories. Instruct all of your users to check these files and report anything suspicious. You can remind your users to check the files by renaming them with names like login.old, cshrc.old, and profile.old. Be sure to check the versions of those files belonging to the root user, and also check the /etc/profile file. If you are using sendmail , the attacker may have created or modified the .forward files so that they run programs when mail is received. This aspect is especially critical on nonuser accounts such as ftp and uucp. If you know the precise time that the intruder was logged in, you can list all of the dot files in users' home directories, sort the list by time of day, and then check them for changes. A simple shell script to use is shown here: #!/bin/ksh
# Search for .files in home directories.
for user in $(cat-passwd | /bin/awk -F: 'length($6) > 0 {print $6}')
do
for name in $user/.*
do
[[ $name == $user/.. ]] && continue
[[ -f $name ]] && print "$name"
done
done | xargs ls -ltd
However, using timestamps may not detect all modifications, as discussed at the end of this chapter. The -c and -l options to the ls command should also be used to check for modifications to permission settings and determine if the mtime was altered to hide a modification. Another approach is to sweep the entire filesystem with a tool that will display a timeline of every file's mtime, ctime, and atime. You can configure this timeline for the time of intrusion. This may give you some clues as to what was done. For instance, if your compiler, loader, and libraries all show access times within a few seconds of each other, you can conclude that the intruder compiled something. Programs for analyzing these files are part of the Coroners Toolkit and the TASK forensics package. Another helpful program is mac-daddy, available from http://www.incident_response.org/.
22.3.3.9 Hidden files and directoriesThe intruder may have created a "hidden directory" on your computer, and may be using it as a repository for stolen information or for programs that break security. On older Unix systems, one common trick for creating a hidden directory was to unlink (as root) the ".." directory in a subdirectory and then create a new one. The contents of such a hidden directory were overlooked by programs such as find that search the filesystem for special files.[7] Modern versions of Unix, however, detect such hidden directories as inconsistencies when you run the fsck program. For this reason, be sure to run fsck on each filesystem as part of your routine security monitoring.
Intruders often hide their files in directories with names that are a little difficult to discover or enter on the command line. This way, a novice system administrator who discovers the hidden directory will probably not figure out how to access its contents. Filenames that are difficult to discover or enter include ".. " (dot dot space), along with control characters, backspaces, or other special characters. Some names can be entered in Unicode that display as familiar alphabetic characters but cannot be entered easily from the keyboard. You can often discover hidden directories easily because they cause results that differ from those of normal directories. For example: prose% ls -l drwxr-xr-x 1 orpheus 1024 Jul 17 11:55 foobar prose% cd foobar foobar: No such file or directory prose% In this case, the real name of the directory is foobar, with a space following the letter r. The easy way of entering filenames like this one is to use the shell's wildcard capability. The wildcard *ob* will match the directory foobar, no matter how many spaces or other characters it has in it, as long the letters o and b are adjacent: prose% ls -l drwxr-xr-x 1 orpheus 1024 Jul 17 11:55 foobar prose% cd *ob* prose% If you suspect that a filename has embedded control characters, you can use the cat -v command to determine what they are. For example: % ls -l total 1 -rw-r--r-- 1 john 21 Mar 10 23:38 bogus?file % echo * | cat -v bogus^Yfile % In this example, the file bogus?file actually has a Ctrl-Y character between the letters bogus and the letters file. Some versions of the ls command print control characters as question marks (?). To see what the control character actually was, however, you must send the raw filename to the cat command, which is accomplished with the shell echo. If you are using a recent version of the Korn shell, you can also list all the files in the local directory in a readable manner. This approach works even when your ls command has been replaced with an altered version. $ printf $'entry: %q\n' .* * entry: . entry: .. entry: $'..\n' entry: $'bogus\031file' entry: temp0001 entry: temp0002 $ Another powerful tool for viewing the contents of directories is the GNU Emacs dired mode. This mode gives you a graphical display of the files and subdirectories that a directory contains. You can initiate this mode by using the M-X dired command, or simply by using the C-X C-F find-file command and specifying a directory name instead of a filename. Once you are viewing a directory with dired, you can use it to step in and out of subdirectories and view files, even if the directory and filenames contain untypable characters. 22.3.3.10 Unowned filesSometimes attackers leave files in the filesystem that are not owned by any user or group—that is, the files have a UID or GID that does not correspond to any entries in the /etc/passwd and /etc/group files. This can happen if the attacker created an account and some files, and then deleted the account—leaving the files. Alternatively, the attacker might have been modifying the raw inodes on a disk and changed a UID by accident. You can search for these files with the find command, as shown in the following example: # find / -nouser -o -nogroup -print Remember: if you are using NFS, you should instead run the following find command on each server: # find / \( -local -o -prune \) -nouser -o -nogroup -print You might also notice unowned files on your system if you delete a user from the /etc/passwd file but leave a few of that user's files on the system. Unowned files can also result from loading a tar or dump tape from another system and specifying the option to set the owner of the restored files to the archived owner—generally something to avoid. It is a good idea to scan for unowned files on a regular basis, copy them to tape (in case they're ever needed), and then delete them from your system. 22.3.3.11 New network servicesMany intruders (and many attack scripts) will install network daemons that provide backdoor access to the compromised host at a later time, or that can be used to direct the host to act as a zombie in attacks against other hosts. Although these new services can sometimes be detected by examining ps or top output on the compromised host, those programs are also frequently modified. Other possibilities include using netstat, lsof, sockstat, or examining the process directories in the /proc filesystem if your system supports it, but all of these could be modified by an attacker.[8] You may be able to detect new daemons using nmap or another port-scanning tool from an uncompromised machine on the same network. (Of course, it's always safest to disconnect a compromised machine from your network while you're investigating it.)
22.3.4 Never Trust Anything Except HardcopyIf your system is compromised, don't trust anything on its disks. If you discover changes in files on your system that seem suspicious, don't believe anything that your system tells you because a good system cracker can change anything on the computer. This may seem extreme, but it is probably better to spend a little extra time restoring files and playing detective now than it would be to replay the entire incident when the intruder attacks again. Remember: an attacker who becomes the superuser on your computer can do anything to it—change any byte on the hard disk. The attacker can compile and install new versions of any system program—so there might be changes, but your standard utilities might not tell you about them. The attacker can patch the kernel that the computer is running, possibly disabling security features that you have previously enabled. The attacker can even open the raw disk devices for reading and writing. Essentially, an attacker who becomes the superuser can warp your system to her liking—if she has sufficient skill, motivation, and time. Often, she doesn't need (or have) great skill. Instead, she has access to rootkits put together by others with more skill. For example, suppose you discover a change in a file and do an ls -l or an ls -lt. The modification time you see printed for the file may not be the actual modification time of the file. There are at least five ways for an attacker to affect the time that is displayed by this command, all of which have been used in actual system attacks:
The only limit to the powers of an attacker who has gained superuser status is that the attacker cannot change something that has been indelibly printed on a piece of paper to which the attacker does not have access. For this reason, if you have a logging facility that logs whenever the date is changed, you should consider having the log made to a hardcopy terminal or to another computer. Then, be sure to examine this log on a regular basis. To further protect yourself, you should have a bootable copy of your operating system on a Zip disk, CD-ROM, or other removable storage device. This gives you a way of booting and examining your system with a set of tools that are known to be uncorrupted. Coupled with a database of message digests of unmodified files,[9] you should be able to find anything important that was modified on your system—provided that your message digests were generated from uncorrupted versions of your software. Remember that you cannot necessarily trust your backups, as you don't know when the intrusion started; use distribution media if possible.
The next step is to get a printed copy of all of the necessary logs that you may have available (e.g., console logs and printed copies of network logs) and examine these logs to try to get an idea of what the unauthorized intruder has done. You will also want to see if anything unusual has happened on the system since the time the intruder logged in. These logs may give you a hint as to which programs the intruder was running and which actions the intruder took. Be sure to initial and timestamp these printouts. 22.3.5 Resuming OperationThe next step in handling a break-in is to restore the system to a working state. How quickly you must be back in operation, and what you intend to do about the break-in over the long term, will determine when and how you perform this step. Generally, you have a few options about how to proceed from this point:
At a minimum, you should get whatever assurance you can that you restored everything damaged on the system, and fixed whatever it was that allowed the intruder in. Then, if you have been keeping good backups, you can restore the system to a working state. The difficulty of determining what failed and allowed an intruder in is complicated by the fact that there is usually little data in the logs to show what happened, and there are few things you can execute to reverse-engineer the break-in. Most break-ins seem to result from either bugs or, less commonly, compromised user passwords (suspect this especially if you find that the intruders have installed a sniffer on your system). If the break-in was the result of a bug, you may have difficulty determining what it is, especially if it is a new one that has not been widely exploited. Here are some things to try:
If you suspect that it is a bug in some system software, you can try contacting your vendor to see if you can get some assistance there. In most cases it helps if you have a maintenance contract or are a major customer. You might consult recent postings to the security groups on web sites or mailing lists. Often, current vulnerabilities are discussed at these locations in great detail. This is a mixed blessing because not only does this provide you with some valuable information to protect (or restore) your site, but it also often provides details to hacker wannabes who are looking for ways to break into systems. It is also the case that sometimes these sites contain information that is incorrect and even dangerous if implemented. Therefore, be very wary of what you read. Finally, you may wish to contact a FIRST team appropriate for your site. Teams in FIRST often have some insight into current break-ins, largely because they see so many reports of them. Contacting a representative from one of the teams may result in some good advice for things to check before you put your system back into operation. However, many teams have rules of operation that prevent them from giving too much explicit information about active vulnerabilities until the appropriate vendors have announced a fix. Thus, you may not be able to get complete information from this source. 22.3.6 Damage ControlIf you've already restored the system, what damage is there to control? Well, the aftermath, primarily. You need to follow through on any untoward consequences of the break-in. For instance, was proprietary information copied? If so, you need to notify your legal counsel and consider what to do. You should determine which of the following concerns need to be addressed:
The answers to the above questions will vary from situation to situation and incident to incident. We'll cover a few of them in more detail in succeeding chapters. |
|
|