11.4 Queue Groups (V8.12 and Above)

As of V8.12 sendmail it is possible to group queues according to selected criteria, and then to process each group with custom settings. This versatile ability is enabled and tuned with:

• The QUEUE_GROUP mc configuration command, which defines queue groups and sets their group properties

• The queuegroup mc feature, which allows you to select queue groups based on recipient hosts via the access database

• More sophisticated queue group selections, which you can make by writing your own rule sets

You can best tune queue groups by first understanding their limitations. We cover these topics in this section, but first we need to briefly discuss the default queue group.

11.4.1 The Default Queue Group

Prior to V8.12 sendmail there were no queue groups. Instead, every -q command and every queue option (such as QueueDirectory) applied to all the queue directories you had.^[5]

^[5] Unless you ran a separate queue-processing daemon for each set of queues. Then you could call them queue groups.

Beginning with V8.12, sendmail offers a way to define both multiple queue directories and a way to group them by function or specialty. For compatibility with old versions, a special queue group named mqueue is the default queue group. It takes on all the properties of every -q command, and every queue option, just like before.

When you later declare particular queue groups (as we show in the next section), those additional groups take all their properties from the default group, unless you override a particular property with a specific equate. Those equates and the command-line arguments or options they override are shown in Table 11-2.

For example, the following declares two different queue directories:

define(`QUEUE_DIR', `/var/spool/mqueue')
QUEUE_GROUP(`regularmail', `')
QUEUE_GROUP(`slowmail', `P=/var/spool/mqueue/slowqueue')

The first line declares the queue used by the default group (always known as mqueue). Any other queue groups that are declared (such as regularmail) will use that same directory unless the directory is overridden by the P= equate, as shown in the third line. That is, the default queue group's queue directory and everything else that is set for the default queue group is inherited by the regularmail group. For the slowmail queue group, however, everything but the queue directory is inherited. (See Section 11.4.2.5 for a description of the P= equate, and for the reason queue group directories must be subdirectories under QUEUE_DIR.)

11.4.2 The Q Configuration Command

Queue groups are declared with the Q configuration command. That command can take a wide range of appearances, but in all guises it takes the name of the queue group and then a sequence of equates:

Qgroupname, equates

The name of the queue group (here groupname) must follow the Q with no intervening spaces. If spaces are present, an error such as the following is printed and logged, and that Q line is ignored:

file.cf: line line number: queue : `=' expected 

The equates are optional, but if they are present they must follow the queue group's name and a comma or whitespace, or both:

Qgroupname, equates

The equates are formed by selecting one of the keywords shown in the leftmost column of Table 11-2, and following it with an equal sign and the value you wish to assign to that key-letter. Note that only the first letter is looked at by sendmail, so you can use the shorthand shown in parentheses if you wish. Also note that the first letter is case-sensitive—that is, R and r are different.

For example, both of the following declare a queue directory (the Path= and P=), and a queue-processing interval of 10 minutes (the Interval= and I=):

Qslowmail, Path=/disk1/mail/slowqueues, Interval=10m
Qslowmail, P=/disk1/mail/slowqueues, I=10m

A comma separates one equate from another. The comma can be optionally surrounded by whitespace characters (spaces and tabs). If the value following the comma is missing, an appropriate error will be printed and logged.

If an equate other than those shown in the table is used, an error such as the following is printed and logged, and that Q line is ignored:

file.cf: line line number: Qgroupname: unknown queue equate bad equate here 

11.4.2.1 The Flags= (F=) queue-group equate

The F= queue-group equate is used to set flags for the queue group. Currently there is only one flag, the f flag, which tells sendmail to fork multiple times to process the queue group in parallel (the exact opposite of the -qf command-line switch, which tells sendmail to not fork multiple times, but instead to run the queues serially in the foreground).

When this F= flag is specified, sendmail forks one queue processor for each queue directory in the group. But note that the sendmail program will only fork up to the total number of parallel processors set by the R= queue group equate. If that limit is fewer than the number of queues, the remaining queues are handled during the next queue run, in round-robin fashion.

When the fast processing of a queue group is required, we recommend you specify this F=f queue group flag. If speed is not of concern, you can reduce the system impact by omitting this flag. But if you omit it and then specify multiple runners with the R= queue group equate, the following message will print and be logged:

Warning: Q=queuegroup name:R=number: multiple queue runners specified
       but flag 'f' is not set

As a performance compromise, some parallelism can be attained and system impact reduced by setting this flag and limiting the number of runners specified with the R= queue group equate.

11.4.2.2 The Interval= (I=) queue-group equate

The I= queue-group equate specifies the time interval at which the queues in the queue group should be processed. The default interval is set by the -qinterval command-line switch, but can be overridden for a queue group using this I= queue group equate:

I=interval

The interval following the I= is constructed from an integer and a letter. The letters and the meaning of each are listed in Table 11-3. Integer and letter groups can be combined—for example, 5d12h means 5 days, 12 hours.

If the trailing letter is missing, the units default to minutes, thus the following defines an interval of 1 hour, 12 minutes:

Interval=1h12

In general, the use of a trailing letter is recommended for clarity, and to avoid problems in the future should sendmail defaults change.

11.4.2.3 The Jobs= (J=) queue-group equate

When a queue processor starts to process a queue directory, it first gathers a list of all the envelopes in that directory. It then sorts, or randomizes that list, and processes the envelopes in the resulting order. If no limit is imposed, all the envelopes will be processed before the queue run is complete.

The default limit, if there is one, is defined by the MaxQueueRunSize option (MaxQueueRunSize). But a separate limit that will override the default can be set for a queue group using this J= equate. If the default is non-zero and if this equate specifies zero, the default queues will have the default limit imposed but this group will have none. This J= queue-group equate is used like this:

Jobs=number

If number is zero or negative, no limit is imposed. If number is positive, that will be the maximum number of envelopes processed.

11.4.2.4 The Nice= (N=) queue-group equate

The niceness of a process determines its priority to be run. The larger the nice value, the lower the priority. The default nice value varies from one version of Unix to another. In all cases, however, they generally begin with the same nice value, so all processes generally get an equal chance to run.

With sendmail, the niceness of its queue processors is set by the NiceQueueRun option (NiceQueueRun). If that option specifies a positive value, the priority is reduced. If that option specifies a negative value, that priority is increased. In general, queue processors should run at a lower priority so as to minimize the adverse impact on other processes. On dedicated mail-sending machines, you might wish to increase the priority.

Each queue group inherits its nice value from the NiceQueueRun option, unless this N= queue-group equate is specified. This N= equate is used like this:

Nice=10   increase niceness by 10, lower priority
Nice=0    no change
Nice=-10  same as zero
Nice=b    same as zero

If the number is missing, nonnumeric, or negative, the niceness change is zero (no change). Otherwise, the niceness is increased (the priority is lowered) by the amount specified.

11.4.2.5 The Path= (P=) queue-group equate

The default location and name of the queue directory or directories is set by the QueueDirectory option (QueueDirectory). That option defines the default directory (for the default queue group mqueue) and the base path for all the other queue directories. The P= queue-group equate does not override the default (as the other equates do), but instead augments it.

The path specified by the P= queue-group equate must be a full (absolute) path, and must contain the name of a subdirectory or subdirectories of the default path. To illustrate, consider the following mc file declarations:

define(`QUEUE_DIR',`/var/spool/mqueues/q.*')        the default
QUEUE_GROUP(`aolmail', `P=/var/spool/mqueues/aolmail')  good, a subdirectory
QUEUE_GROUP(`bobmail', `P=/var/spool/mqueues/bob.*')good, a subdirectory
QUEUE_GROUP(`hotmail', `P=hotmail')                 bad, not a full path
QUEUE_GROUP(`slow', `P=/var/spool/slowqueue')       bad, not a subdirectory

Here, the fist line defines the default queues, which all begin with the characters q. and live under the path /var/spool/mqueues.

The second line correctly sets the queue for the aolmail queue group. The base path, /var/spool/mqueues, is the same for both the default and this group. Note that queue group directories can also specify multiple queues (as with the /var/spool/mqueues/bob.* in the third line).

The forth line shows that the path specified with P= must not be a relative pathname. If it is, sendmail will print and log the following error and exit:

QueuePath hotmailnot absolute

The last line shows that the path specified with P= must not use a base path different from the default. If it does, the following error will print and log, and sendmail will exit:

QueuePath  /var/spool/slowqueuenot subpath of QueueDirectory/var/spool/mqueues: No such 
file or directory

Note, however, that symbolic links under the default queue path are OK. That is, you can declare the last line in the previous example like the following, and then simply make the path you specify a symbolic link to the real directory somewhere else:

define(`QUEUE_DIR',`/var/spool/mqueues/q.*')        the default
QUEUE_GROUP(`slow', `P=/var/spool/mqueues/slowqueue')
                                              
                                a symbolic link to /var/spool/slowqueue

Note, however, that the path pointed to by the symbolic line must be as trusted as the default path, with narrow ownerships and permission (QueueFileMode).

11.4.2.6 The recipients= (r=) queue-group equate

The MaxRecipientsPerMessage option (MaxRecipientsPerMessage) sets the default limit for the number of recipients allowed per envelope. If there are more recipients than that limit in an envelope, sendmail will split the envelope into two or more envelopes, each with the limit or fewer recipients. If the MaxRecipientsPerMessage option is zero, no limit is imposed.

The r= queue-group equate allows you to override the default for each queue group. If the default allows unlimited recipients, or a large limit, you can use a smaller setting for your queue group. Or, if the default is too small, you can enlarge it. You use the r= equate like this:

recipients=99    set the limit to 99 recipients
recipients=0     set unlimited recipients
recipients=-99   same as r=0
recipients=none  same as r=0

Note that a zero or negative expression sets the limit to unlimited. A nonnumeric expression, such as in the last line, also sets the limit to zero (unlimited).

11.4.2.7 The Runners= (R=) queue-group equate

The Runners= (R=) queue-group equate tells sendmail how many queue processors to launch each queue-processing interval. The queues are serviced in round-robin order. So, for example, if your queue group has three queues, and you set R= to 1, 2, 3, and 4, respectively, you will see the runs shown inTable 11-4.

The Runners= queue-group equate is declared like the following:

Runners=12 12 per queue run
Runners=0  no limit, so one per queue each queue run
Runners=none   the same as R=0

If the number of queue-group runners specified by this equate is more than the number of queue children allowed by the MaxQueueChildren option (MaxQueueChildren), the number of queue-group runners is reduced to that amount, and the following error is logged and printed:

Q=queuegroup: R=number exceeds MaxQueueChildren=limit, set to MaxQueueChildren 

If the MaxQueueChildren option is set to zero, there is no limit to how many queue-group runners you can declare.

11.4.3 How to Declare Queue Groups with the m4 Technique

You declare queue groups inside your mc configuration file with the QUEUE_GROUP mc configuration macro. As you have seen in the previous sections, it is used like this:

QUEUE_GROUP(`group name', `equates') 

The queue group name can contain any characters except a comma or a whitespace character (a space or a tab).^[6] It must not be surrounded (inside the quotes) with whitespace characters.

^[6] However, we recommend that you use only letters, the dash character (hyphen), and the underscore character. Other characters might become illegal in future releases of sendmail.

The equates form the second argument to the QUEUE_GROUP mc configuration macro. The equates are described in Section 11.4.2.

To illustrate, consider the following QUEUE_GROUP mc configuration macro declaration:

QUEUE_GROUP(`slowmail', `P=/var/spool/mqueues/slowqueue')

Here, the name of the queue group is set to slowmail. The second argument is a single equate, the P= queue-group equate, which defines the queue directory or directories to be used by this queue group.

If you want to define which queue group to use for certain delivery agents, you can use the Q= delivery agent equate (Q=) as set, for example, with the LOCAL_MAILER_QGRP mc macro. For example, the following tells sendmail to queue all local mail in the /queues/lq queue directory:

QUEUE_DIR(`/queue')
QUEUE_GROUP(`localgroup', `P=/queue/lq')
define(`LOCAL_MAILER_QGRP', `localgroup')must be before MAILER(local)
MAILER(`local')

In the first line we set the default queue directory. In the second line we define the queue group localgroup, and set its queue directory to be /queue/lq. In the third line we declare that the Q= equate for the local delivery agent will be:

Q=localgroup

The fourth line declares support for the local delivery agent. Note that the definition of LOCAL_MAILER_QGRP must precede the MAILER(local), otherwise that definition will be silently ignored.

Those four lines cause all mail for local users to be queued in the /queue/lq directory. Note that you can dedicate queue groups for other delivery agents. See Q= for a full description of this process.

11.4.4 The FEATURE(queuegroup) and the access Database

The easiest way to select queue groups based on recipient addresses or recipient domains is by using the queuegroup feature. It is declared in your mc configuration file like this:

FEATURE(`queuegroup')
FEATURE(`queuegroup', `default group')

The first line causes the queue group to default to mqueue if a queue group in the access database is missing or nonexistent. The second line allows you to set a different default queue group. For example, consider the following lines from an mc file:

QUEUEGROUP(`localgroup', `/queue/lq')
FEATURE(`queuegroup', `localgroup')

This causes sendmail to use the group named localgroup instead of mqueue as the default if a queue group in the access database is missing or nonexistent.

Once you have enabled the queuegroup feature, the next step is to add lines such as the following to the source file for your access database:

QGRP:slow-poke.com      slowgroup
QGRP:root@notify.com    fastgroup
QGRP:your.domain        localgroup

Each line that selects queue groups must begin with the literal expression:

QGRP:

This prefix tells sendmail that you wish to map recipient addresses or domains to queue groups.

The first line causes mail to the slow-poke.com domain to use the queue group called slowgroup. This shows that you can list just a domain in the lefthand column and it will work just as expected.

The second line causes mail to the specific recipient root@notify.com to use the queue group named fastgroup. This line demonstrates that mail to an individual can be used in the lefthand column.

The third line illustrates your local domain, which shows that mail to your domain, your.domain, will use the queue group named localgroup.

If you omit the name of the queue group (not recommended), you will need to use the -e command-line switch with makemap to create the database. When you omit the name of the queue group the default queue group is used:

QGRP:another.your.domain
                            
                 queue group name missing (not recommended)

Here, if you defined a default queue group when you declared the queuegroup feature, that group will be selected. Otherwise, the group mqueue will be selected for this domain.

11.4.5 Rule Set Queue Group Selection

Normally, the access database, described earlier, is the easiest way to select queue groups. There might be times, however, when selecting by recipient address or domain is not sufficient. Should such a situation arise, you could set up your own rule sets. But be forewarned that if you do, the queuegroup feature cannot be used. If you try to use both, you will get the following warning every time sendmail starts to run:

WARNING: Ruleset queuegroup has multiple definitions

The first step in declaring your own rules to select queue groups is to declare a special rule set called queuegroup. You do that in your mc configuration file using the LOCAL_RULESETS macro:

LOCAL_RULESETS
Squeuegroup
           your rules here

The way this rule set works is simple. Any queue group for a recipient address that a rule selects is returned following the $# operator. For example, consider the following:

R $* <@some.domain>          $# somegroup

Here, mail bound for any user at some.domain will be queued in the somegroup queue group.

Normally queuegroup rule sets are used to select queue groups based on the recipient. If you wish to select based on the sender, you can do so using rules something like the following:

LOCAL_RULESETS
Squeuegroup
R $*                     $: $>canonify $&f
R $+ <@ lists.domain.>   $# lists

First, we fetch the sender address using $&f, and pass it through the canonify rule set 3 to focus on the host part. The second rule matches any user at the domain lists.domain, and selects the lists queue group.

Because there are no more rules following the second one, this rule set returns without selecting a queue group. If the queuegroup rule set fails to select a queue group, the default queue group (mqueue) is used.

Other possible uses for the queuegroup rule set might include:

• Queue inbound messages on a disk different from that used for outbound messages.

• Queue mail to suspect users in a queue that is not automatically processed so that the mail can be manually screened before delivery.

• Queue expendable mail, such as short-lived notification mail (e.g., "tea is served"), on a volatile disk that is erased when the machine is rebooted.

• Queue low-priority mail in a queue different from that used for high-priority mail.

Note that there are limitations on the use of this queuegroup rule set. First, this rule set is called directly from inside sendmail, so you should not call it from inside your own rules (if you do, the selected queue group will be ignored). And second, the queuegroup feature also uses this rule set, so you cannot share it with that feature.^[7]

^[7] You can copy the rules created by that feature and paste them into your own. However, that is not recommended because the copied rules might change with new releases of sendmail, and then the old copied rules will fail.

11.4.6 Queue Group Limitations

As you saw in Section 11.4.1, the default queue group (mqueue) is defined by options and the command line. If any given Q configuration command is missing a given equate, that queue group inherits that property as defined by the default queue group. There are, however, properties for the default queue group which have no equivalent equates. These properties are inherited by all queue groups and cannot be overridden with a queue-group equate. They are:

DeliveryMode option

If the DeliveryMode option (DeliveryMode) is set to queueonly or deferred, all mail will be queued rather than delivered. This affects all queue groups.

FastSplit option

This FastSplit option (FastSplit), when non-zero, prevents MX lookups prior to splitting an envelope and limits the number of envelopes that can be delivered on the initial attempt. This option, regardless of its value, affects all queue groups.

MaxQueueChildren option

The MaxQueueChildren option (MaxQueueChildren), when non-zero, limits the number of queue processors that can simultaneously run across all queues. If this is fewer than the total queue runners across all queue groups, it limits the run to this setting. Any queue groups that are not run are handled in the next run in round-robin order. There is no way to limit some queue groups and not limit others.

MinQueueAge option

Messages in a queue are processed no more often than the interval set by this MinQueueAge option (MinQueueAge). This limit is imposed even if a queue is processed more often. This limit is global and affects all queue groups.

-qI, -qR, and -qS command-line switches

The -qI command-line switch restricts a queue run to the messages that match the queue identifier specified. The -qR command-line switch restricts a queue run to the messages that match the recipient address pattern specified. The -qS command-line switch restricts a queue run to the messages that match the sender address pattern specified. Unless the -qG command-line switch is also used to limit the queue group, these limits are imposed across all queue groups.

QueueFactor, QueueLA, and RecipientFactor options

The QueueFactor (QueueFactor), QueueLA (QueueLA), and RecipientFactor (RecipientFactor) options are used to calculate the point at which sendmail should queue a message instead of delivering it. This calculation affects all queue groups.

QueueFileMode option

Beginning with V8.10 sendmail, the QueueFileMode option (QueueFileMode) defines the mode (permissions) of all queue files. This setting affects all queue files across all queue groups.

Timeout.queuereturn and Timeout.queuewarn options

The Timeout.queuereturn option (See this section) defines the maximum time interval that a message can remain in the queue before it is bounced because of a deferred delivery. The Timeout.queuewarn option (See this section) defines the interval at which a message, still in the queue, will result in a first and only warning message being sent to the sender. Both of these intervals globally affect all queue groups.