B.3 Controlling and Examining Processes
In addition to ps
and kill, Unix supports a large number of lesser
known tools for examining and controlling running processes. These
commands can be useful for programmers and system administrators;
they are also very helpful in analyzing the processes of an attacker
during and after a break-in. Some of the ways you can examine or
control processes include the following:
- gdb
-
You can attach to the running process with a debugger such as
gdb.
- gcore
-
You can use the gcore command to dump the
process memory map.
- lsof
-
You can use the lsof program to list the open
files in use by the program.
- /proc
-
You can examine the process directly using the
/proc process filesystem.
- pstree
-
You can see a tree of all processes with the
pstree command.
Not all of these tools are available on every version of Unix.
Strictly speaking, many of these tools will work with processes that
are either running or stopped. However, if you have a rogue process
on your system, you may wish to stop it with the SIGSTOP signal
before examining it.
|
One reason to be familiar with these tools is that many attackers
will modify a penetrated system in such a way that the system
ps command will no longer display processes
belonging to the attacker. These modifications are most often done
with programs that are collectively known as
rootkits.
Once a system has been modified with a rootkit, it can be very
difficult to detect the continued presence of an attacker. However,
few rootkits modify such programs as lsof or
pstree. Thus, if these tools show that a process
is present on your system but the ps command
does not, that is a good indication that your system has been
compromised.
|
|
B.3.1 gdb: Controlling a Process
On
many systems, you can use the gdb command to
"attach" to a running process. If
the process is running an executable that was linked with a full
symbol table, you will be able to use the debugger to examine the
process's variables and detailed call stack. Even if
you do not have an executable that was linked for debugging, you may
be able to use the debugger to determine what the process is doing.
adb
and dbx are
additional debuggers you can use to control processes.
B.3.2 gcore: Dumping Core
Under many versions of Unix, you can use the
gcore program to generate a core file of a
running process. A core file is a specially formatted image of the
memory being used by the process at the time the signal was caught.
|
Some versions of Unix name core files core.####,
in which #### is the PID of the process that
generated the core file, or name.core, in which
name is the name of the
program's executable. Others simply use the name
core.
|
|
Once you have a core file, you can examine it with
adb (a debugger),
dbx (another debugger), or
gdb (yet another debugger). By examining the
core file with a debugger, you can see which routines were executed,
register values, and more. If you simply want to get an idea of what
the process was doing, you can run strings (a
program that finds printable strings in a binary file) over the core
image to see which files it was referencing. If the process was
running a shell such as sh or
csh, the strings command
will display the shell's history.
Programs that you run may also dump core if they receive one of the
signals that causes a core dump. On systems without a
gcore program, you can send a SIGEMT or SIGSYS
signal to cause the program to dump core. This method will work only
if the process is currently in a directory where it can write, it has
not redefined the action to receive the signal, and the core will not
be larger than the core file limits imposed for the
process's UID. If you use this approach, you will
also be faced with the problem of finding where the process left the
core file!
|
Core files are big! You can fill your disk with a core file—be
sure to look at the memory size of a process via the
ps command before you try to get its core image.
|
|
B.3.3 lsof: Examining a Process
The
List of Open Files (lsof) command is now
provided as a standard part of many Unix systems; it is available as
a free download for still more systems.
As the name implies, this command examines the
kernel's table of file descriptors associated with
each process and displays the name of each file that is currently
opened. In addition to giving the name of each file being currently
referenced, lsof reveals the name of the
executable currently being run by the process and the filenames of
all mapped-in shared libraries. Besides this information, current
versions of lsof can report open TCP/IP
connections, as well as TCP and UDP sockets that are being listened
to.
When lsof is run by a user, the program
restricts its output to processes that are owned by that user. When
lsof is run by the superuser, the program
displays output for all processes on the system.
Here is an example of the output from the lsof
command:
[simsong@r2 ~] 304 % lsof
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tcsh 81776 simsong cwd VDIR 13,2 15360 12657945 /usr/home/simsong
tcsh 81776 simsong rtd VDIR 116,131072 1024 2 /
tcsh 81776 simsong txt VREG 116,131072 638988 6355 /bin/tcsh
tcsh 81776 simsong 15u VCHR 5,3 0t138415 7898 /dev/ttyp3
tcsh 81776 simsong 16u VCHR 5,3 0t138415 7898 /dev/ttyp3
tcsh 81776 simsong 17u VCHR 5,3 0t138415 7898 /dev/ttyp3
tcsh 81776 simsong 18u VCHR 5,3 0t138415 7898 /dev/ttyp3
tcsh 81776 simsong 19u VCHR 5,3 0t138415 7898 /dev/ttyp3
lsof 81991 simsong cwd VDIR 13,2 15360 12657945 /usr/home/simsong
lsof 81991 simsong rtd VDIR 116,131072 1024 2 /
lsof 81991 simsong txt VREG 13,2 106848 7618686 /usr/local/sbin/lsof
lsof 81991 simsong txt VREG 13,2 76752 1984040 /usr/libexec/ld-elf.
so.1
lsof 81991 simsong txt VREG 13,2 19232 634990 /usr/lib/libkvm.so.2
lsof 81991 simsong txt VREG 13,2 573888 634976 /usr/lib/libc.so.4
lsof 81991 simsong 0u VCHR 5,3 0t138415 7898 /dev/ttyp3
lsof 81991 simsong 1u VCHR 5,3 0t138415 7898 /dev/ttyp3
lsof 81991 simsong 2u VCHR 5,3 0t138415 7898 /dev/ttyp3
lsof 81991 simsong 3r VCHR 2,0 0t0 6880 /dev/mem
lsof 81991 simsong 4r VCHR 2,1 0xc28649c0 6872 /dev/kmem
[simsong@r2 ~] 305 %
The lsof program has too many options to list
here. There are significant security issues that arise from its
installation and use—specifically, lsof
lists the names of files throughout the filesystem, and this
information is cached in a file that is located in the home directory
of the person who runs the lsof command. The
lsof command can be compiled and installed with
various options that minimize the privacy exposure that can result
from these cache files. For details, consult the
lsof documentation.
B.3.4 /proc: Examining a Process Directly
/proc is the process filesystem. It allows
user programs to access aspects of a process through the filesystem
interface in a relatively transparent and straightforward fashion,
without having to open up the kernel's memory and
wade through memory structures. It also allows direct access to the
memory space of other processes, which is otherwise impossible or
very difficult.
B.3.5 pstree: Viewing the Process Tree
Every Unix process has an associated
parent process. Normally, this information is displayed as a form
similar to the display of the PPID field output by the
ps command. The pstree
command uses this information to draw a graph of all of the processes
currently running.
During a break-in, the process tree can be very useful for
understanding which processes were launched by the attacker and which
are innocent processes that happen to be running on the same system.
With the -u option, the
pstree command will show UID
transitions—that is, when one process has a child that is
executing under a separate UID. Another useful option is
-a, which shows the entire command line that was
executed. For a list of all the options, see the documentation.
Here is an example of the output of the pstree
program:
% pstree -u
init-+-arpwatch
|-cron
|-dhcpd
|-gdomap(nobody)
|-8*[getty]
|-httpd---11*[httpd(http)]
|-inetd-+-imapd(simsong)
| `-sslwrap
|-lpd
|-master-+-2*[bounce(postfix)]
| |-cleanup(postfix)
| |-flush(postfix)
| |-local(postfix)
| |-pickup(postfix)
| |-qmgr(postfix)
| |-2*[smtp(postfix)]
| |-smtpd(postfix)
| |-tlsmgr(postfix)
| `-trivial-rewrite(postfix)
|-mountd
|-moused
|-named
|-nfsd---4*[nfsd]
|-4*[nfsiod]
|-nmbd---nmbd
|-ntpd
|-portmap(daemon)
|-pwcheck
|-rpc.statd
|-rwhod(daemon)
|-setiathome(nobody)
|-slapd
|-smbd---smbd(beth)
|-snmpd
|-sshd-+-sshd---tcsh(simsong)---pstree
| |-sshd---tcsh(simsong)---tcsh(root)
| `-sshd---tcsh(simsong)
|-syslogd
`-usbd
The boldfaced line near the end of this output shows that
init (executing as root)
spawned an sshd process (executing as
root). This process forks a child (still
root-owned in this case) for each incoming
connection. When simsong logged into this
sshd connection, it started up a shell
(tcsh) owned by simsong,
and simsong has apparently managed to start a
root-privileged tcsh shell
(perhaps with /bin/su, but if you
don't expect simsong to have
the root password, this is cause for concern)!
|