[Chapter 24] 24.5 An Example

24.5 An Example

Suppose you're a system administrator and John Q. Random is there with you in your office. Suddenly, an alert window pops up on your display, triggered by a Swatch rule monitoring the syslog output. The syslog message has indicated that John Q. Random has logged in and has used the su command to become root .

The user must be an intruder - an intruder who has become root!

Fortunately, in one of the windows on your terminal you have a superuser shell. You decide that the best course of action is to bring your system to an immediate halt. To do so, you execute the commands:

# 
sync
 
# 
/etc/init 0

Alternatively, you can send a TERM signal to the init process:

# 
sync

# 
kill -TERM 1

This method is not the recommended procedure on System V systems, but is required on BSD systems.

Your decision to halt the system was based on the fact that you had no idea who this intruder was or what he was doing, and the fact that the intruder had become the superuser. After the intruder is the superuser, you don't know what parts of the operating system he is modifying, if any.

For example, the intruder may be replacing system programs and destroying log files. You decide that the best thing you can do is to shut the system down and go to a protected terminal where you know that no other intruder is going to be interfering with the system while you figure out what's going on.

The next step is to get a printed copy of all of the necessary logs that you may have available (e.g., console logs and printed copies of network logs), and to examine these logs to try to get an idea of what the unauthorized intruder has done. You will also want to see if anything unusual has happened on the system since the intruder logged in. These logs may give you a hint as to what programs the intruder was running and what actions the intruder took. Be sure to initial and timestamp these printouts.

Do not confine your examination to today's logs. If the intruder is now logged in as root , he may have also been on the system under another account name earlier. If your logs go back for a few days, examine the older versions as well. If they are on your backup tapes, consider retrieving them from the tapes.

If the break-in is something that you wish to pursue further - possibly with law enforcement - be sure to do a complete backup of the system to tape. This way, you'll have evidence in the form of the corrupted system. Also, save copies of the logs. Keep a written log of everything you've done and are about to do, and be sure to write the time of day along with each notation.

The next step is to determine how the intruder got in and then to make sure the intruder can't get in again. Examine the entire system. Check the permissions and the modes on all your files. Scan for new SUID or SGID files. Look for additions in /etc/passwd . If you have constructed checklists of your program directories, rerun them to look for any changes.

Remember: the intruder may not be an outsider! The majority of major incidents occur from inside the organization, either from someone with current access or someone who recently had legitimate access. When you perform your evaluation, don't forget to consider the case that the behavior you saw coming from a user account was not someone breaking a password and coming in from outside, but was someone on the inside who broke the password, or perhaps it was the real account owner herself!

Only after performing all these steps, and checking all this information, should you bring the system back up.

24.5.1 Never Trust Anything Except Hardcopy

If your system is compromised, don't trust anything that is online. If you discover changes in files on your system that seem suspicious, don't believe anything that your system tells you, because a good system cracker can change anything on the computer. This may seem extreme, but it is probably better to spend a little extra time restoring files and playing detective now than it would be to replay the entire incident when the intruder gets in again.

Remember, an attacker who becomes the superuser on your computer can do anything to it, change any byte on the hard disk. The attacker can compile and install new versions of any system program - so there might be changes, but your standard utilities might not tell you about them. The attacker can patch the kernel that the computer is running, possibly disabling security features that you have previously enabled. The attacker can even open the raw disk devices for reading and writing. Essentially, attackers who becomes the superuser can warp your system to their liking - if they have sufficient skill, motivation, and time. Often, they don't need (or have) great skill. Instead, they have access to toolkits put together by others with more skill.

For example, suppose you discover a change in a file and do an ls -l or an ls -lt. The modification time you see printed for the file may not be the actual modification time of the file. There are at least four ways for an attacker to modify the time that is displayed by this command, all of which have been used in actual system attacks:

The attacker could write a program that changes the modification time of the file using the utimes () system call.
The attacker could have altered the system clock by using the date command. The attacker could then modify your files and, finally, reset the date back again. This technique has the advantage for the attacker that the inode access and creation times also get set.
The attacker could write to the raw disk, changing saved values of any stored time.
The attacker could have modified the ls command to show a predetermined modification time whenever this file is examined.

The only limit to the powers of an attacker who has gained superuser status is that the attacker cannot change something that has been printed on a line printer or a hardcopy terminal. For this reason, if you have a logging facility that logs whenever the date is changed, you might consider having the log made to a hardcopy terminal or to another computer. Then, be sure to examine this log on a regular basis.

It is also the case that we recommend that you have a bootable copy of your operating system on a removable disk pack so, when needed, you can boot from a known good copy of the system and do your examination of the system with uncorrupted tools. Coupled with a database of message digests of unmodified files such as that produced by a tool such as Tripwire, you should be able to find anything that was modified on your system.