6.1 Handle Deep Queues
To understand the potential problems with deep queues, first consider
how sendmail processes a single queue when its
QueueSortOrder option (QueueSortOrder) is set to the default of
sendmail is instructed to process a queue it
opens the queue directory for reading and reads that directory to
gather a list of qf files to process. Each
qf file sendmail finds is
opened for reading and scanned for important pieces of information.
The N line in each qf file,
for example, holds the number of times the message has been tried.
The P line holds each message's
After all messages have been opened, read, and closed, and after the
information from each has been saved internally,
sendmail sorts that information. The purpose of
the sort is to ensure that new mail is tried before old, and that
high-priority mail is tried before low-priority mail.
Under normal circumstances this process occurs quickly. But when
queues get abnormally deep, things can go wrong in unusual ways. In
the following, we show one way that sendmail
could be run on a major mail-sending machine:
The idea here is to create two mail-handling daemons. The first
handles inbound mail and, because this is a mail-sending machine, we
expect that this inbound daemon will perform little work. The second
daemon sends all mail it finds in its queue. It will
fork(2) a copy of itself once every 10 minutes,
and that copy will process all the messages in the queue. As
described earlier, each queued message is opened and read so that all
the messages can be sorted before delivery begins.
Because this hypothetical site is a major mail-sending site, we
expect a high rate for the number of sent messages. For the sake of
argument, let's say 30,000 messages need to be sent
Now suppose that one day a backhoe, a power failure, clumsy fingers,
or any of a thousand possible events causes this
site's only connection to the Internet to fail for
an hour. The site can neither look up host information with DNS, nor
can it connect to any remote sites. All the mail it tries to send
that hour fails, and instead of being removed from the queue, this
mail is left there so that it can be tried again later (presumably
after the problem is fixed).
An hour later, service is restored. First, the default:
causes a forked copy of sendmail to start
processing the queue. This time, however, the processing is not
swift. When a queue fills to 30,000 or more messages, the amount of
time it takes to preread the queue (to open and read every message),
increases to more than 20 minutes. And that 20 minutes is only for the
preread. During that 20 minutes no mail will be sent.
After that, things get worse. Ten minutes later a second
sendmail daemon is forked, and it, too, starts
to preread the queue. Now, instead of one
sendmail daemon opening and reading all messages
in a queue, we have two sendmail daemons doing
the same thing in parallel.
Contrary to what you might think, twice as much I/O on a disk is not
twice as fast. Disks are finite devices that perform a limited number
of disk-head moves per second
and can transmit only a fixed number of bytes per second. Because the
two sendmail daemons are 10 minutes out of step
with each other, each is reading and processing separate files.
Depending on the size of your in-memory disk cache, neither will
likely be able to take advantage of the efficiencies of such caching.
In short, two sendmail daemons processing a deep
queue in parallel is worse than a single
sendmail daemon processing that same queue
And if that weren't enough, another 10 minutes later
a third sendmail daemon starts to process the
By now the first sendmail daemon might have
finished its preread of the queue and might have actually begun to
send messages. But even if it has, now three
sendmail daemons are processing that single deep
queue and a curious thing starts to happen. Because the disk that
holds the queue is finite, the addition of a third
sendmail daemon slows the operation of the first
two. The second one, instead of taking 20 minutes to preread the
queue, will now take 30 minutes.
This means that every 10 minutes another
sendmail queue-processing daemon is added to the
mix. As each is added, each slows all the others that are already
running, and it isn't long before the load on the
machine starts to climb and the rate at which messages are delivered
falls at an alarming rate. In fact, when this sort of behavior hits a
very large-volume site, a sendmail
queue-processing daemon can start and seem to never finish.
Depending on the speed of your disk system, even limiting the number
of queue processors per queue might not save you from this sluggish
performance. Under V8.12 sendmail, for example,
you can limit the number of queue runners per queue with a queue
group (Section 11.4) definition such as this in your
mc configuration file:
QUEUE_GROUP(`fastq', `P=/q/fastq*, I=10m, R=10')
Here, the fastq group uses the queue disks mounted
as /q/fastq*, processes those disks once per 10
minutes (the I=10m), and limits itself to 10 queue
runners maximum (the R=10) across all the disks.
If there are few fastq* queue disks, and if they
fill to more than 30,000 messages each, they too can become sluggish,
even with only 10 runners processing them. In fact, with sufficient
filled queue depth, as few as two simultaneous queue runners can
seriously affect performance.
In extreme situations such as this, one alternative is to use
persistent queue runners (Section 11.8.3). With
persistent queue runners, you maintain a single queue runner that
alone reads the queue. After that single queue runner has read the
queue, it forks multiple child queue runners to process the queue,
with each child sharing the parent's queue
Here, the -qp causes one or more persistent queue
runners to launch. One is launched for each queue group, and will
persist to run, sleeping 10 minutes between each reading of the
queue. When it awakes, it gathers a list of queue files and launches
multiple child processes to handle that list. After the last child
has finished delivery and exited, the parent sleeps again.
Even with queue groups and persistent queue runners, you are
encouraged to spread queues across many directories and across many
disks and controllers. This increases parallelism and dramatically
lessens the likelihood that any given queue will overfill.
6.1.1 Recover from a Full Queue
When a queue directory is exceptionally full, you will likely notice
the problem only when performance on your queue-handling machine
becomes unusually sluggish. By that time, however, a drastic measure,
such as rebooting the server, might be the only cure. Clearly, early
detection is desirable.
Early signs that a queue is filling can be seen in the logging
messages that sendmail produces. You can develop
scripts that watch for lines such as these:
Dec 13 10:27:53 your.domain sendmail: grew WorkList for /var/spool/mqueue to
Dec 13 10:29:05 your.domain sendmail: grew WorkList for /var/spool/mqueue to
Dec 13 10:34:31 your.domain sendmail: grew WorkList for /var/spool/mqueue to
... etc., to:
Dec 13 12:40:22 your.domain sendmail: grew WorkList for /var/spool/mqueue to
Dec 13 12:42:50 your.domain sendmail: grew WorkList for /var/spool/mqueue to
Here, the WorkList refers to the number of
messages preread so far. By searching for unusual sizes, you can
determine when a queue is about to overfill.
Another technique is to run the mailq command
(Section 15.1.2) to observe the total number of
messages queued across all queues:
% mailq -OMaxQueueRunSize=0 | tail -1 V8.7 through V8.11
Total Requests: 34190
% mailq -bP V8.12 and above
Total requests: 34190
For V8.7 through V8.11, the MaxQueueRunSize=0
allows mailq to run swiftly, regardless of how
deep the queue or queues might be. Without that option, and with deep
queues, mailq would be just as slow as the
sluggish queue runs, but beginning with V8.12, the
-bP command-line switch does the same thing more
No matter how you detect the problem, the solution will be the same.
First, you need to kill all the competing
sendmail queue-processing daemons. There are a
wide number of ways to do this. The most common is to use
ps(1) to gather PID numbers and then kill each
queue-processing daemon individually. No matter how you kill the
queue-processing daemons, be sure to kill them all. If you
don't, you might find the problem surfacing again
before you have had a chance to fix the problem.
The best way to flush a full queue is with a command line something
# /usr/sbin/sendmail -OQueueSortOrder=filename -q10m -d99.100
# /usr/sbin/sendmail -OQueueSortOrder=random -q10m -d99.100 V8.12 and above
Here, the -d99.100 tells
sendmail to run in the foreground (so that you
can kill it easily when done). The -q10m causes a
queue-processing daemon to be launched once each 10 minutes (just
like before). You need this because one daemon can seem to hang when
delivering mail to a slow host. By running parallel daemons, you
avoid this pitfall.
Sorting by filename or random
(Section 11.7) causes sendmail
to skip the opening and reading of each queued message. Instead, it
only looks at the filename for its sorting or randomizing order. On
the downside, this prevents sendmail from
grouping messages for optimum delivery. On the upside, this reduces
the time to preread a huge queue from 20 or so minutes to less than 2
The QueueSortOrder=random (See this section) is just like the
QueueSortOrder=filename shown earlier, except that
it randomizes the list before beginning delivery. This method is
preferred, but is only available beginning with V8.12
After draining the full queue to a more manageable level, you can
discontinue this special process and rerun
sendmail in its normal manner.
If the full queue has to remain in service while the full state is
being solved, you can use the techniques in Section 11.9.1 to move that full queue out of the way so that
it can be processed in the background.