 |  |
48.6. Beware of Sluggish Performance
Contrary to popular myth, systems don't just start to fail for no reason. If your system is starting to perform poorly, chances are it's because of something that's been initiated. In most cases, the cause has innocuous roots, such as a poorly designed script; however, sluggish performance could also mean an external attack. Regardless of the origin of the decreasing efficiency, you'll want to take steps to locate the problem and remove it before it takes your system down.
If you notice that your systems performance is degrading, there are several built-in utilities you can use to troubleshoot possible problems. Probably the most commonly used utility is ps (Section 24.5); however, there are other utilities that can provide useful information.
48.6.1. Check Processes
The first check to perform if you think that you have a destructive agent running on your machine is the processes currently in operation. You'll use the basic ps command to do this, after first checking to make sure that ps itself hasn't been replaced by a bogus program (check installation date, location, and size to see if the ps utility has been replaced).
Running the ps command with the flags -aux shows each user's processes, the CPU and memory usage, time started and command. Here's an example of output:
> ps -aux root 6910 0.0 0.1 2088 516 ?? IsJ 30Apr02 1:04.80 /usr/sbin/sshd root 6955 0.0 0.0 2600 384 ?? IsJ 30Apr02 0:06.67 /usr/local/sbin/xinetd -pidfile /var/run/xinetd.pid root 6970 0.0 0.0 624 0 #C1- IWJ - 0:00.00 /bin/sh /usr/virtual/share/ pkgs/installed/mysql-server/3.22.32/bin/ mysql 6994 0.0 0.0 11216 144 #C1- SJ 30Apr02 0:35.83 /usr/local/libexec/ mysqld --basedir=/usr/local --datadir=/var/db/my root 7003 0.0 0.3 10028 2616 ?? SsJ 30Apr02 3:33.55 /usr/local/www/bin/httpd -DSSL nobody 38060 0.0 0.3 10324 3116 ?? SJ 12:01PM 0:08.60 /usr/local/www/bin/httpd -DSSL nobody 38061 0.0 0.3 10332 2612 ?? SJ 12:01PM 0:08.23 /usr/local/www/bin/httpd -DSSL nobody 38062 0.0 0.3 11212 2656 ?? SJ 12:01PM 0:08.89 /usr/local/www/bin/httpd -DSSL nobody 38117 0.0 0.2 10352 2580 ?? SJ 12:01PM 0:09.37 /usr/local/www/bin/httpd -DSSL nobody 38314 0.0 0.2 10332 2596 ?? SJ 12:03PM 0:08.98 /usr/local/www/bin/httpd -DSSL root 62104 0.0 0.0 2112 400 ?? SJ 9:57AM 0:00.16 sshd: shelleyp@ttyp2 (sshd)
In this listing, several processes are being run by root, but all are normal processes and accounted for. In addition, several processes are being run by "nobody," which is the generic user used with HTTP web page access. Using additional ps flags displays additional information, including -e for environment and -f for command-line and environment information of swapped-out processes.
48.6.2. Checking Swap Space
If your system is under DoS attack, your swap space is a vulnerable point. This hard disk space is reserved for use by the operating system and to provide space for temporary files. If your system is sluggish and you suspect a possible DoS attack -- or just a badly behaving script that results in a lot of temporary files -- the first thing you should check is how much swap space you have.
The pstat utility can be used to check swap space when using the -s option on the command line:
pstat -s
The result will be a listing of swap areas by device with available and used swap space. If the percentage of used space is much higher than normal, you probably have a bad script or external interference. Additional utilities can help you determine which.
Within FreeBSD and other Unix systems, swapinfo returns the same information as pstat -s. If you're running a Mac OS X system, instead of pstat, you'll use the ls command and check the contents of /var/vm:
ls -l /var/vm -rw-------T 1 root wheel 000000000 Jun 4 12:56 swapfile0
Since the system wasn't under load, the swap space didn't have any contents at the time this command was run.
48.6.3. Check Network Connections
Another check you can run if your system is running sluggishly and you think you might be under attack is netstat. This command will return activity on Unix sockets as well as all of the active Internet connections, including referrals if the connection occurs through HTTP.
Here's an example of netstat output:
Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 burningbird.http a25253.upc-a.che.3617 TIME_WAIT tcp4 0 0 burningbird.http pm66.internetsee.4301 TIME_WAIT tcp4 0 0 burningbird.http strider.ccs.neu..4492 TIME_WAIT tcp4 0 0 burningbird.http strider.ccs.neu..4491 TIME_WAIT tcp4 0 0 burningbird.http strider.ccs.neu..4490 TIME_WAIT tcp4 0 0 burningbird.http mailgate.ltsbfou.57600 FIN_WAIT_2 tcp4 0 0 burningbird.http mailgate.ltsbfou.57595 FIN_WAIT_2 tcp4 0 20 burningbird.ssh adsl-64-168-24-1.1076 ESTABLISHED tcp4 0 0 burningbird.submission *.* LISTEN tcp4 0 0 burningbird.smtp *.* LISTEN tcp4 0 0 burningbird.domain *.* LISTEN tcp4 0 0 burningbird.http *.* LISTEN tcp4 0 0 burningbird.https *.* LISTEN tcp4 0 0 burningbird.pop3s *.* LISTEN tcp4 0 0 burningbird.ssh *.* LISTEN udp4 0 0 burningbird.domain *.* udp4 0 0 burningbird.syslog *.* Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr e5ed4cc0 stream 0 0 e5f0cbc0 0 0 0 /tmp/mysql.sock e5ed4d40 stream 0 0 0 0 0 0 e5e08380 dgram 0 0 0 e5ed4dc0 0 e5e083c0 e5e083c0 dgram 0 0 0 e5ed4dc0 0 e5ed4d80 e5ed4d80 dgram 0 0 0 e5ed4dc0 0 0 e5ed4dc0 dgram 0 0 e556c040 0 e5e08380 0 /var/run/log
Specifying netstat with the command line option -s provides a detailed report of per-protocol -- TCP, UDP, IP, and so on -- usage statistics.
The netstat program is helpful not only for determining if someone is trying to break into your system, but also for determining if your system is having basic communication problems.
48.6.4. Other Checks
You can use iostat to check I/O statistics on your various devices. For instance, to check to see what kind of activity is occurring on all devices every three seconds for nine runs, issue the following command:
# iostat -odICTw 2 -c 9
tty mlxd0 acd0 fd0 md0 cpu
tin tout blk xfr msps blk xfr msps blk xfr msps blk xfr msps us ni sy in id
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0
0 0 224 12 167 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0
0 0 568 36 55.8 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0
0 0 144 5 402 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0
0 0 112 7 287 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0
0 0 48 3 670 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0
0 0 240 15 134 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0
0 0 192 12 168 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0
0 0 96 6 335 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0 0 0The result allows you to compare I/O over a period of time. Note that in some systems, iostat may be io_stat, instead.
Another check is vmstat (vm_stat), which displays the virtual memory statistics for a machine. As with iostat, running the command several times over a period of time can show if there is unusual activity within virtual memory. For instance, if the free memory unexpectedly decreases, no known user is running a process, the occurrence of the free memory use is consistent (occurring at a set time of day), and no other system processes or cron jobs are known to be running, you probably have an intruding application running somewhere on the system. Other tests can then be used to help you determine what the application is.
-- SP
|  | |
| 48.5. What We Mean by DoS |  | 48.7. Intruder Detection |
Copyright © 2003 O'Reilly & Associates. All rights reserved.