11.8 Cause Queues to Be Processed

The sendmail program offers two different methods for processing its queues. It can be told to process them periodically or to process them once and then exit.

11.8.1 Periodically with -q

The -q command-line switch is used both to cause queues to be processed and to specify the interval between queue runs.

A typical invocation of the sendmail daemon looks like this:

/usr/sbin/sendmail -bd -q1h

Here, the sendmail program is placed into listening mode with the -bd command-line switch. The -q1h command-line switch tells it to process the queue once each hour. Note that either switch puts sendmail into the background as a daemon. The -bd switch just allows sendmail to listen for incoming SMTP connections. Consider the following:

/usr/sbin/sendmail -bd
/usr/sbin/sendmail -q1h

This runs two daemons simultaneously. The first listens for incoming SMTP connections. The second processes the queues once per hour.

The time expression following the -q is constructed from an integer followed by a letter. The letters and the meaning of each are listed in Table 11-5. Integer and letter groups can be combined—for example, 5d12h means 5 days, 12 hours. If a letter is missing, the default is minutes.

At small sites, where mail messages are rarely queued, the time interval chosen can be small to ensure that all mail is delivered promptly. An interval of 15m (15 minutes) might be appropriate.

At many sites an interval of one hour is probably best. It is short enough to ensure that delays in delivery remain tolerable, yet long enough to ensure that queue processing does not overlap (see Section 11.8.3 for a way to run a persistent queue runner that avoids overlapping runs).

At large sites with huge amounts of mail and at sites that send a great deal of international mail, the interval has to be carefully tuned by observing how long it takes sendmail to process its queues and what causes that process to take a long time. Points to consider are the following:

• Network delays or delays at the receiving host can cause delivery to that host to time out. Timeouts are set with the Timeout option (Timeout).^[10] Each such timeout is logged at LOG_NOTICE with a message such as this:

  ^[10] Note that prior to V8 sendmail the r option set one timeout for all SMTP timeouts.

  timeout waiting for input from host during what

  Here, host is the name of the other host, and what specifies which timeout triggered the message (such as "client HELO" for to_helo). In general, timeouts should be large to ensure that mail to busy sites, and to large mailing lists, does not time out improperly. In observing queue processing, you might find that all messages but one process swiftly. That one, you might find, takes more than an hour because of a long SMTP timeout. A possible solution to this problem is to make all timeouts short so that most queue runs are processed quickly. Then, for example, the following command could be run a few times each night to specifically flush those long jobs:

  /usr/sbin/sendmail -OTimeout=2h -q

• A queue can take a long time to process because too many messages are being queued unnecessarily. Several options affect the placement of mail messages into the queue. The QueueLA option (QueueLA) tells sendmail to queue, rather than deliver, a message if the machine load is too high. Fewer messages will be queued if the value of that option is increased. The SuperSafe option (SuperSafe) tells sendmail to queue all messages for safety. If your machine "never" crashes, this might not be necessary. Or you might choose to turn off SuperSafe when sending short-lived notification mail, or when your queues are on a volatile filesystem, such as an async or tempfs filesystem. (RFC2824 recommends that you never turn off SuperSafe.) The HoldExpensive option (HoldExpensive) tells sendmail to queue messages to "expensive" delivery agents (those with the F=e flag set, F=e) rather than delivering them. If the queue is routinely filled with messages to expensive sites, you should reconsider your reasons for marking those sites as expensive.

• The queue can fill with messages because sendmail was run with the -odq or -odd command-line switch (see the DeliveryMode option, DeliveryMode). At sites that receive a great deal of UUCP mail for forwarding, the rmail(8) program is often set up to run sendmail in "queue-only" mode with the -odq command-line switch. If UUCP mail is clogging your normal mail services, you should consider queueing it to a separate queue directory. You can then process that other directory with a separate queue run of sendmail. (Use of separate queue directories is discussed in Section 11.9.)

• A slow machine can clog the queue. When a single machine is set up to handle the bulk of a site's mail, that machine should be as swift as possible. In general, a dedicated mail server should have a fast CPU with lots of memory. It should never allow users to log in to it, and might need to run its own name server daemon.

• On modern servers where a fast CPU with lots of memory is available, the bottleneck will likely be disk I/O. Equip the server with many disks spread over many controllers. Use multiple queue directories (Section 11.3) or queue groups (Section 11.4) to spread the I/O widely over those many disks.

11.8.2 From the Command Line

The -q command-line switch, invoked without a time interval argument, is used to run sendmail in queue-processing mode. In this mode, sendmail processes queues once and then exits. This mode can be run interactively from the command line or in the background via cron(8).

Other command-line switches can be combined with -q to refine the way queues are processed. The -v (verbose) switch causes sendmail to print information about each message it is processing, and to process multiple queues sequentially. The -d (debugging) switch can be used to produce additional information about the queue. We'll discuss the -v switch as it applies to the queue later in this chapter. Those -d debugging switches appropriate to the queue can be found in Table 16-4 of Section 16.6.

V8 sendmail allows variations on -q: -qI allows you to specify a specific message identifier for processing; -qR allows you to specify specific recipient addresses for processing; and -qS allows you to specify specific sender addresses for processing.^[11]

^[11] IDA and pre-V8 SunOS sendmail offer three command-line switches for processing the queue. The -M switch allows you to specify a specific message for processing. The -R switch allows you to specify specific recipient addresses for processing. The -S switch allows you to specify specific sender addresses for processing.

11.8.2.1 Process the queue once: -q

The -q command-line switch, without an interval argument, tells sendmail to process the queue once, then exit. As such, this switch is a handy administrative tool. When the queue fills unexpectedly between queue runs of the daemon, for example, the -q command-line switch can be used to force an immediate queue run:

# /usr/sbin/sendmail -q

When multiple queues are run this way, they are all processed in parallel (Section 11.3.1.2).

On machines that do not run the sendmail daemon, the -q command-line switch can be used in conjunction with cron(8) to periodically process the queue. The following crontab(5) file entry, for example, causes sendmail to be run once per hour, at five minutes past the hour, to silently process its queues and exit:

5 * * * * /usr/sbin/sendmail -q >/dev/null 2>&1

When used in conjunction with other switches (shown next), the -q switch allows many queue problems to be conveniently handled.

11.8.2.2 Combine -v with -q

The -q switch without an argument prevents sendmail from running in the background and detaching from its controlling terminal. But it also runs silently. To see what is going on, use the -v command-line switch in combination with the -q:

% /usr/sbin/sendmail -v -q

The -v command-line switch causes sendmail to print a step-by-step description of what it is doing. When running multiple queues, it also causes them to be processed in sequence. To illustrate, consider the following output produced by using both the -v and -q command-line switches:

Running /var/spool/mqueue/dB9JBR106687 (sequence 1 of 2)
<adams@dc.gov>... Connecting to dc.gov via ddn...
Trying 123.45.67.8... Connection timed out during user open with DC.GOV
<adams@dc.gov>... Deferred: Host DC.GOV is down

Running /var/spool/mqueue/dB9JDWt06701 (sequence 2 of 2)
<help@irs.dc.gov>... Connecting to irs.dc.gov via ddn...
Trying 123.45.67.88...  connected.
220 irs.dc.gov Sendmail 5.57/3.0 ready at Mon, 27 Jan 92 09:16:38 -0400

Here, two queued messages are being processed. The first fails because of a connection timeout and is requeued for a later queue run. The second succeeds (we omit the full SMTP dialog). After its delivery is complete, it is removed from the queue.

11.8.2.3 Process by identifier/recipient/sender: -q[ISR]

With V8 sendmail you can process a subset of all queued messages. You can select which to process based on queue identifier, recipient address, or sender address:

-qIident       match any queue ID that contains ident 
-qRrecip       match any recipient address that contains recip 
-qSfrom        match any sender address that contains from 

The -qI variation is followed by a queue identifier such as dB9JDWt06701. The -qR is followed by the address of a recipient. The -qS is followed by the address of a sender. In all three variations there must be no space between the uppercase letter and the identifier or address.

These variations are used to limit the selection of queued files that are processed. For example:

% /usr/sbin/sendmail -qSroot -qRbiff@here

Here, the queue is processed once. Only messages from root are processed. Of those, only messages that have biff@here as one of the recipients are processed.

In all three variations a partial specification of queueid, recipient, or sender is viewed by V8 sendmail as a substring. For example:

-qSroot

matches mail from all of the following:

root
ben@groots.edu
ben@GROOTS.EDU

The last line further illustrates that the substring match is a case-insensitive one. The substring match is literal. Wildcard characters (such as *) and regular expressions (such as .*@.*edu) won't work and might confuse the shell from which you run sendmail.

Multiple specifications can be combined on the command line (as shown earlier), but they all AND together:

% /usr/sbin/sendmail -qI123 -qSroot -qR@host.edu

Here, the queue is processed only for messages with the number 123 anywhere in the queue identifier that are also from root and that are also addressed to anyone at host.edu.

You can use the mailq command to preview the effect of these switches. For example, the following command will list (but not send) the messages that would be processed by the previous command line:

% mailq -qI123 -qSroot -qR@host.edu

11.8.2.4 Process by negated identifier/recipient/sender (V8.12 and above)

Beginning with V8.12 sendmail, you can prefix any of the I, S, or R specifications to -q with an ! character. The presence of an ! character prefix instructs sendmail to invert the logic of that particular test. For example:

% mailq -q\!Sroot -qR@host.edu

Here, we wish to process the queue for any message addressed to anyone at host.edu, just as we did in the previous section. But this time, we want to further limit that processing by including only messages with a sender that is not (the !) from root. Note that we prefix the ! with a backslash to protect it from the csh or tcsh shells (the backslash is not necessary for the Bourn shell and its derivatives).

In summary, these specifications for how to limit the queue can be mixed and matched, specified and negated, in any combination that works for you:

-qIident       match any queue ID that contains ident 
-q!Iident      match any queue ID that does not contain ident 
-qRrecip       match any recipient address that contains recip 
-q!Rrecip      match any recipient address that does not contain recip 
-qSfrom        match any sender address that contains from 
-q!Sfrom       match any sender address that does not contain from 

Only the ! character can be used to negate. Any other character will be interpreted as an argument to -q. The ! prefix must not follow the I, R, or S. If it follows, it will be interpreted as part of the expression to match.

11.8.2.5 Process by queue group with -qG (V8.12 and above)

Beginning with V8.12 sendmail, you can use the -qG command-line switch to process queues based on selected queue groups. It is used like this:

-qGgroupname

Here, only mail queued in the qroupname directories will be processed. If the name specified is of an unknown group, the following error will print and log, and the queue run will fail:

Queue group groupname unknown

This command-line switch can be used in combination with all the other queue processing switches. Consider, for example, the following:

% /usr/sbin/sendmail -qR@hostA.domain -qGslow

Here, sendmail will deliver queued messages only to users at hostA.domain, if those messages were queued in the slow queue group.

Multiple -qG command-line switches cannot be used at the same time. If you combine them, the following error will print and be logged:

Cannot use multiple -qG options

Unlike the -q[IRS] switches discussed earlier, the -qG command-line switch cannot be negated:

% /usr/sbin/sendmail -q!Gmqueue
Cannot use -q!G

11.8.2.6 Process the queue via ESMTP ETRN

The ESMTP ETRN command, based on RFC1985, causes V8.8 and above sendmail to asynchronously process its queue in a manner similar to the -qR command-line switch (Section 11.8.2.3). This command allows dial-on-demand sites to make an SMTP connection and to force the other side to process and send any mail that is queued for them. The form of this ESMTP command looks like this:

ETRN host

If host is missing, this error message will be returned:

550 Parameter required

Otherwise, the queue will be processed just as if the following command-line argument were given:

-qR@host

In both cases a qf file will be processed if it has host anywhere in the host part (following the @) of one of its R lines. The only difference here is that the former (the ETRN) operates asynchronously. That is, sendmail forks a copy of itself, and the forked child processes the queue.

Beginning with V8.12 sendmail, you can cause sendmail to process the queues by queue group. To do this, just replace the hostname with the name of the queue group, and prefix it with a literal # character:

ETRN #groupname

If the groupname has been defined, the queues for that queue group will be processed. Otherwise, the following error will be returned:

459 4.5.4 Queue badname unknown

One way to use ETRN is with a perl(1) script supplied with the sendmail source. See the file:

contrib/etrn.pl

You might have to change the first line of this file to get it to work, depending on where you installed perl(1) on your system. To run this program, just give it the name of your MX server:

% contrib/etrn.pl  your.mx.server  

The etrn.pl script will connect to that server, and it will send an ETRN command to that server for each host you list with a Cw or Fw command in your configuration file. The etrn.pl script is also its own manual page, which you can read with a command such as this:

% nroff -man contrib/etrn.pl | more

11.8.3 Persistent Queue Runners with -qp

V8.12 sendmail introduced persistent queue runners as a solution to some of the problems caused by periodic queue runners. Periodic queue runners are the result of a normal -qinterval command-line switch, or a Runners= queue group equate. Either causes:

• sendmail to fork one or more queue runners to process a queue or queue group each interval

• Every queue runner to open and read all the files in the queue to gather a list of envelopes to deliver

Persistent queue runners avoid these problems because a single process is dedicated to a queue, a queue group, or a grouping of queue groups (called a "workgroup"). A persistent queue runner is launched just like the periodic command-line queue runner, but with the addition of a p character:

-qpinterval

The p causes one or more persistent queue runners to be launched, one per queue group. One will be launched to handle your default queue group, and one more will be launched to handle each queue group defined by a QUEUE_GROUP mc option. Depending on the number of queue directories in each, these can be combined into a single workgroup. When you have many queue groups, you can end up with multiple workgroups controlling persistent runners.

Each persistent queue runner will sleep for interval. When it awakes, it reads all the files in the queues that belong to its workgroup, and sorts all the envelopes it finds into the proper order needed for delivery. After it has finished ordering the envelopes, it launches one or more regular queue runners to perform delivery using that already processed list. This can significantly reduce the disk I/O compared to that needed by periodic queue runners.

When the last of the regular queue runners has finished processing (and exited), the persistent queue runner goes back to sleep for interval.

In general, persistent queue runners are valuable only at sites that normally have queues that are very full. When a queue is normally near empty, persistent queue runners can introduce unforeseen delays. Note that a persistent queue runner will sleep again only when all of its regular queue runners have finished. One regular queue runner, delivering to a very slow site, can appear to hang, and so can cause the persistent queue runner to also appear to hang. Subsequent queue runs will be delayed until the hung site times out, allowing the persistent queue runner to sleep interval again.

At large sites, such delays will eventually smooth out due to the normal distribution of slow jobs. At small sites, such delays might be noticed and objected to. In general, persistent queue runners should be reserved for sites with full queues.

If interval is omitted, the default interval becomes one second:

-qp

When the default interval is used (by omitting the interval), the persistent queue runner will sleep one second between queue runs, unless the prior queue run was empty, in which instance it will sleep five seconds. If you choose the default interval, we recommend you also set the MinQueueAge option (MinQueueAge).

If interval is specified as zero, the effect is the same as if it were omitted. If interval is negative, the following error is logged and printed and sendmail exits:

Invalid -q value

If interval is nonnumeric (if you specify O when you mean zero) the following error is logged and printed, and sendmail exits:

Invalid time unit `O'

The process that was given the -qp command-line switch is the controlling process. It could be the listening daemon (if -bd or -bD were also used), or it could be a queue processing daemon (if only -qp and other queue processing limiters were specified). The controlling process has two special properties:

• To restart the persistent queue runners, you must instead restart the sendmail controlling process. You do that with a SIGHUP signal (as normal). If you try to signal the individual persistent queue runners, they will restart but with a penalty (each can be restarted this way only 10 times; see later in this section).

• If a persistent queue runner fails and exits, the controlling process will launch a new persistent queue runner.

If a persistent queue runner core dumps, the following will be logged and that queue runner will not be restarted:

persistent queue runner=number core dumped, signal=signal

If a persistent queue runner exits because of a caught signal, the following is logged and that queue runner is restarted:

persistent queue runner=number died, signal=signal

If a persistent queue runner is restarted because of a SIGHUP, the following is logged:

restart queue runner=number due to signal signal

If the -dno_persistent_restart debugging command-line switch is specified, a failed persistent queue runner will not be restarted, and the following error will be logged:

persistent queue runner=number, exited

A persistent queue runner will not be restarted if it has already been restarted 10 times. Instead, the following error will be logged and that persistent queue runner marked as bad:

ERROR: persistent queue runner=number restarted too many times, queue runner lost

If this happens, examine your logs. Some nonmail-related process might be signaling your persistent queue runners, or you might have bad memory, or you might have made a mistake when building sendmail and should rebuild it, or you might have a junior system administrator who does not know how to correctly restart sendmail.

Persistent queue runners look like executing periodic queue runners in process listings:

root   22958   476  ?  S   08:43   0:00 sendmail: accepting connections
root   22947   512  ?  S   08:32   0:00 sendmail: running queue: /var/spool/mqueues/
q.1/df

Here, the first line shows the controlling process, and the second line shows a persistent queue runner. Note that, even though the second entry says "running," it might not be.