Load Balancing (Apache: The Definitive Guide, 3rd Edition)

12.6. Load Balancing

This section deals with the problems of running a high-volume web site on a number of physical servers. These problems are roughly:

Connecting the servers together.
Tuning individual servers to get the best out of the hardware and Apache.
Spreading the load among a number of servers with mod_backhand.
Spreading your data over the servers with Splash so that failure of one database machine does not crash the whole site.
Collecting log files in one place with rsync (see http://www.rsync.org/ ) — if you choose not to do your logging in the database.

12.6.1. Spreading the Load

The simplest and, in many ways, the best way to deal with an underpowered web site is to throw hardware at it. PCs are the cheapest way to buy MegaFlops, and TCP/IP connects them together nicely. All that's needed to make a server farm is something to balance the load around the PCs, keeping them all evenly up to the collar, like a well-driven team of horses.

There are expensive solutions: Cisco's LocalDirector, LinuxDirector, ServerIrons, and a host of others.

12.6.2. mod_backhand

The cheap solution is mod_backhand, distributed on the same licence as Apache. It originated in the Center for Networking and Distributed Systems at Johns Hopkins University.

Its function is to keep track of the resources of individual machines running Apache and connected in a cluster. It then diverts incoming requests to the machines with the largest available resources. There is a small overhead in the redirection, but overall, the cluster works much better.

In the simplest arrangement, a single server has the site's IP number and farms the requests out to the other servers, which are set up identically (apart from IP addresses) and with identical mod_backhand directives. The machines communicate with each other (once a second, by default, but this can be changed), exchanging information on the resources each currently has available. On the basis of this information, the machine that catches a request can forward it to the machine best able to deal with it. Naturally, there is a computing cost to this, but it is small and predictable.

mod_backhand works like a proxy server, but one that knows the capabilities of its proxies and how that capability varies from moment to moment.

It is possible to vary this setup so that different machines do different things — for instance, you might have some 64-bit processors (DEC Alphas, for example) which could specialize in running CGI scripts. PCs, however, are used to serve images.

A more complex setup is to use multiple servers fielding the incoming requests and handing them off to each other. There are essentially two ways of handling this. The first is to use standard load-balancing hardware to distribute the requests among the servers, and then using mod_backhand to redistribute them more intelligently. An alternative is to use round-robin DNS — that is, to give each machine a different IP address, but to have the server name resolve to all of the addresses. This has the advantage that you avoid the expense of the load balancer (and the problems of single points of failure, too), but the problem is that if a server dies, there's no easy way to handle the fact its IP address is no longer being serviced. One answer to this problem is Wackamole, also from CNDS, which builds on the rather marvelous Spread toolkit to ensure that every IP address is always in service on some machine.

This is all very fine and good, and the idea of mod_backhand — choosing a lightly loaded server to service a request on the fly — clearly seems a good one. But there are problems. The main one is deciding on the server. The operating system provides loading information in the form of a one-minute rolling average of the length of the run queue updated every five seconds. Since a busy site could get 5,000 hits before the next update, it is clear that just choosing the most lightly loaded server each time will overwhelm it. The granularity of this data is much too coarse. Consequently, mod_backhand has a number of methods for picking a reasonably lightly loaded server. Just which method is best involves a lot of real-world experimentation, and the jury is still out.

12.6.3. Installation of mod_backhand

Download the usual gzipped tarball from http://www.backhand.org/mod_backhand/download/mod_backhand.tar.gz. Surprisingly, it is less than 100KB long and arrives in a flash. Make it a source directory next to Apache's — we put it in /usr/wrc.mod_backhand. Ungzipping and detarring produces a subdirectory — /usr/wrc.mod_backhand/mod_backhand-1.0.1 with the usual source files in it.

The module is so simple it does not need the paraphernalia of configuration files. Just make sure you have a path to the Apache directory by running ls:

ls ../../apache/apache_x.x.x

When it shows the contents of the Apache directory, turn it into:

./precompile ../../apache/apache_x.x.x

This will produce a commentary on the reconfiguration of Apache:

Copying source into apache tree...
Copying sample cgi script and logo into htdocs directory...
Adding libs to Apache's Configure...
Adding to Apache's Configuration.tmpl...
Setting extra shared libraries for FreeBSD (-lm)
Modifying httpd.conf-dist...
Updating Makefile.tmpl...

Now change to the apache source directory:
    ../../apache/apache_1.3.9
And do a ./configure...

If you want to enable backhand (why would you have done this if you didn't?)
then add:  --enable-module=backhand --enable-shared=backhand
to your apache configure command.  For example, I use:

   ./configure --prefix=/var/backhand --enable-module=so \
     --enable-module=rewrite --enable-shared=rewrite \
     --enable-module=speling --enable-shared=speling \
     --enable-module=info --enable-shared=info \
     --enable-module=include --enable-shared=include \
     --enable-module=status --enable-shared=status \
     --enable-module=backhand --enable-shared=backhand

For those who prefer the semimanual route to making Apache, edit Configuration to include the line:

SharedModule modules/backhand/mod_backhand.cso

then run ./Configure and make.

This will make it possible to run mod_backhand as a DSO. The shiny new httpd needs to be moved onto your path — perhaps in /usr/local/bin.

This process, perhaps surprisingly, writes a demonstration set of Directives and Candidacy functions into the file .../apache_x.x.x/conf/httpd.conf-dist. The intention is good, but the data may not be all that fresh. For instance, when we did it, the file included byCPU (see later), which is now deprecated. We suggest you review it in light of what is upcoming in the next section and the latest mod_backhand documentation.

12.6.4. Directives

mod_backhand has seven Apache directives of its own:

Backhand

Backhand <candidacy function>
Default none 
Directory

This directive invokes one of the built-in mod_backhand candidacy functions — see later.

BackhandFromSO

BackhandFromSO <path to .so file> <name of function>
<argument>
Default none 
Directory

This directive invokes a DSO version of the candidacy function. At the time of writing the only one available was by Hostname (see later). The distribution includes the "C" source byHostname.c, which one could use as a prototype to write new functions. For example:

BackhandFromSO libexec/byHostname.so byHostname www

would eliminate all hostnames that do not include www.

UnixSocketDir

UnixSocketDir <Apache user home directory>
Default none 
Server

This directive gives mod_backhand a directory where it can write a file containing the performance details of this server — known as the "Arriba". Since mod_backhand has the permissions of Apache, this directory needs to be writable by webuser/webgroup — or whatever user/group you have configured Apache to run as. You might want to create a subdirectory /backhand beneath the Apache user's home directory, for example.

MulticastStats

MulticastStats <dest addr>:<port>[,ttl]M
ulticastStats <myip addr> <dest addr>:<port>[,ttl]
Default none 
Server

mod_backhand announces the status of its machine to others in the cluster by broadcasting or multicasting them periodically. By default, it broadcasts to the broadcast address of its own network (i.e., the one the server is listening on), but you may want it to send elsewhere. For example, you may have two networks, an Internet facing one that receives requests and a backend network for distributing them among the servers. In this case you probably want to configure mod_backhand to broadcast on the backend network. You are also likely to want to accept redirected requests on the backend network, so you'd also use the second form of the command to specify a different IP address for your server. For example, suppose your machine's Internet-facing interface is number 193.2.3.4, but your backend interface is 10.0.0.4 with a /24 netmask. Then you'd want to have this in your Config file:

MulticastStats 10.0.0.4 10.0.0.255:4445

The first form of the command (with only a destination address) is likely to be used when you are using multicast for the statistics instead of broadcast.

Incidentally, mod_backhand listens on all ports on which it is configured to broadcast — obviously, you should choose a UDP port not used for anything else.

AcceptStats

AcceptStats <ip address>[/<mask>]
Default none 
Server

This directive determines from where statistics will be accepted, which can be useful if you are running multiple clusters on a single network or to avoid accidentally picking up stuff that looks like statistics from the wrong network. It simply takes an IP address and netmask. So to correspond to the MulticastStats example given above, you would configure the following:

AcceptStats 10.0.0.0/24

If you need to listen on more than one network (or subnet), then you can use multiple AcceptStats directives. Note that this directive does not include a port number; so to avoid confusion, it would probably be best to use the same port on all networks that share media.

HTTPRedirectToIP

HTTPRedirectToIP 
Default none  
Directory

mod_backhand normally proxies to the other servers if it chooses not to handle the request itself. If HTTPRedirectToIP is used, then it will instead redirect the client, using an IP address rather than a DNS name.

HTTPRedirectToName

HTTPRedirectToName [format string]
Default [ServerName for the chosen Apache server] 
Directory

Like HTTPRedirectToIP, this tells mod_backhand to redirect instead of proxying. However, in this case it redirects to a DNS name constructed from the ServerName and the contents of the Host: header in the request. By default, it is the ServerName, but for complex setups hosting multiple servers on the same server farm, more cunning may be required to end up at the right virtual host on the right machine. So, the format string can be used to control the construction of the DNS name to which you're redirected. We can do no better than to reproduce mod_backhand's documentation:

The format string is just like C format string except that it only has two insertion tokens: %#S and %#H (where # is a number).

%-#S is the server name with the right # parts chopped off. If your server name is www-1.jersey.domain.com, %-3S will yield www-1.

%#S is the server name with only the # left parts preserved. If your server name is www-1.jersey.domain.com, %2S will yield www-1.jersey.

%-#H is the Host: with only the right # parts preserved. If the Host: is www.client.com, %-2S will yield client.com.

%#H will be the Host: with the left # parts chopped off. If the Host: is www.client.com, %1H will yield client.com.

For example, if you run a hosting company hosting.com and you have 5 machines named www[1-5].sanfran.hosting.com. You host www.client1.com and www.client2.com. You also add appropriate DNS names for www[1-5].sanfran.client[12].com.

Backhand HTTPRedirectToName %-2S.%-2H

This will redirect requests to www.client#.com to one of the www[1-5].sanfran.client#.com.

BackhandSelfRedirect

BackhandSelfRedirect <On|Off>
Default Off  
Directory

A common way to run Apache when heavily loaded is to have two instances of Apache running on the same server: one serving static content and doing load balancing and the second running CGIs, typically with mod_perl or some other built-in scripting module. The reason you do this is that each instance of Apache with mod_perl tends to consume a lot of memory, so you only want them to run when they need to. So, normally one sets them up on a different IP address and carefully arranges only the CGI URLs to go to that server (or uses mod_proxy to reverse proxy some URLs to that server). If you are running mod_backhand, though, you can allow it to redirect to another server on the same host. If BackhandSelfRedirect is off and the candidacy functions indicate that the host itself is the best candidate, then mod_backhand will simply "fall through" and allow the rest of Apache to handle the request. However, if BackhandSelfRedirect is on, then it will redirect to itself as if it were another host, thus invoking the "heavyweight" instance. Note that this requires you to set up the MulticastStats directive to use the interface the mod_perl (or whatever) instance to which it's bound, rather than the one to which the "lightweight" instance is bound.

BackhandLogLevel

BackhandLogLevel <+|-><mbcs|dcsn|net><all|1|2|3|4>
Default Off  
Directory

The details seem undocumented, but to get copious error messages in the error log, use this (note the commas):

BackhandLogLevel +net1, +dcsnall

To turn logging off, either don't use the directive at all or use:

BackhandLogLevel -mbscall, -netall, -dcsnall

BackhandModeratorPIDFile

BackhandModeratorPIDFile filename
Default none
Server

If present, this directive specifies a file in which the PID of the "moderator" process will be put. The moderator is the process that generates and receives statistics.

12.6.5. Candidacy Functions

These built-in candidacy functions — that help to select one server to deal with the incoming requests — follow the Backhand directives (see earlier):

byAge

byAge [time in seconds]
Default: 20 
Directory

This function steps around machines that are busy, have crashed, or are locked up: it eliminates servers that have not reported their resources for the "time in seconds".

byLoad

byLoad [bias - a floating point number]
Default none 
Directory

The byLoad function produces a list of servers sorted by load. The bias argument, a floating-point number, lets you prefer the server that originally catches the request by offsetting the extra cost of forwarding it. In other words, it may pay to let the first server cope with the request, even if it is not quite the least loaded. Sensible values would be in the region of 0 to 1.0.

byBusyChildren

byBusyChildren [bias - an integer]
Default none  
Directory

This orders by the number of busy Apache children. The bias is subtracted from the current server's number of children to allow the current server to service the request even if it isn't quite the busiest.

byCPU

byCPU 
Default  
Directory

The byCPU function has the same effect as byLoad but makes its decision on the basis of CPU loading. The FAQ says, "This is mostly useless", and who will argue with that? This function is of historical interest only.

byLogWindow

byLogWindow 
Default none 
Directory

The byLogWindow function eliminates the first log base 2 of the n servers listed: if there are 17 servers, it eliminates all after the first 4.

byRandom

byRandom 
Default none 
Directory

The byRandom function reorders the list of servers using a pseudorandom method.

byCost

byCost 
Default none  
Directory

The byCost function calculates the computing cost (mostly memory use, it seems) of redirection to each server and chooses the cheapest. The logic of the function is explained at http://www.cnds.jhu.edu/pub/papers/dss99.ps.

bySession

bySession cookie
Default off 
Directory

This chooses the server based on the value of a cookie, which should be the IP address of the server to choose. Note that mod_backhand does not set the cookie — it's up to you to arrange that (presumably in a CGI script). This is obviously handy for situations where there's a state associated with the client that is only available on the server to which it first connected.

addPrediction

AddPrediction 
Default none  
Directory

If this function is still available, it is strongly deprecated. We only mention it to advise you not to use it.

byHostname

byHostname <regexp> 
Default none 
Directory

This function needs to be run by BackhandFromSO (see earlier). It eliminates servers whose names do not pass the <regexp> regular expression. For example:

BackhandFromSO libexec/byHostname.so byHostname www

would eliminate all hostnames that do not include www.

12.6.6. The Config File

To avoid an obscure bug, make sure that Apache's User and Group directives are above this block:

LoadModule backhand_module libexec/mod_backhand.so
UnixSocketDir @@ServerRoot@@/backhand
# this multicast is actually broadcast because 128 < 224
# so no time to live parameter needed - ',1' restericts to the local networks
# MulticastStats 128.220.221.255:4445
MulticastStats 225.220.221.20:4445,1
AcceptStats 128.220.221.0/24

<Location "/backhand/">
  SetHandler backhand-handler
</Location>

The SetHandler directive produces the mod_backhand status page at the location specified — this shows the current servers, loads, etc.

The Candidacy functions should appear in a Directory or Location block. A sample scheme might be:

<Directory cgi-bin>
BackhandbyAge 6
BackhandFromSO libexec/byHostname.so byHostname (sun|alpha)
Backhand byRandom
BackHand byLogWindow
Backhand byLoad
</Directory>

This would do the following:

Eliminate all servers not heard from for six seconds
Choose servers who names were sub or alpha — to handle heavy CGI requests
Randomize the list of servers
Take a sample of the random list
Sort these servers in ascending order of load
Take the server at the top of the list

12.6.7. Example Site

Normally, we would construct an example site to illustrate our points, but in the case of mod_backhand, it's rather difficult to do so without using several machines. So, instead, our example will be from a live site that one of the authors (BL) runs, FreeBMD, which is a world-wide volunteer effort to transcribe the Birth, Marriage, and Death Index for England and Wales, currently comprising over 3,000 volunteers. You can see FreeBMD at http://www.freebmd.org.uk/ if you are interested. At the time of writing, FreeBMD was load-balanced across three machines, each with 250 GB of RAID disk, 2 GB of RAM, and around 25 million records in a MySQL database. Users upload and modify files on the machines, from which the database is built, and for that reason the configuration is nontrivial: the files must live on a "master" machine to maintain consistency easily. This means that part of the site has to be load-balanced. Anyway, we will present the configuration file for one of these machines with interleaved comments following the line(s) to which they refer.

HostnameLookups off

This speeds up logging.

User webserv
Group webserv

Just the usual deal, setting a user for the web server.

ServerName liberty.thebunker.net

The three machines are called liberty, fraternity, and equality — clearly, this line is different on each machine.

CoreDumpDirectory /tmp

For diagnostic purposes, we may need to see core dumps: Note that /tmp would not be a good choice on a shared machine — since it is available to all and might leak information. There can also be a security hole allowing people to overwrite arbitrary files using soft links.

UnixSocketDir /var/backhand

This is backhand's internal socket.

MulticastStats 239.255.0.0:10000,1

Since this site shares its network with other servers in the hosting facility (http://www.thebunker.net/) in which it lives, we decided to use multicast for the statistics. Note the TTL of 1, limiting them to the local network.

AcceptStats 213.129.65.176
AcceptStats 213.129.65.177
AcceptStats 213.129.65.178
AcceptStats 213.129.65.179
AcceptStats 213.129.65.180
AcceptStats 213.129.65.181

The three machines each have two IP addresses: one fixed and one administered by Wackamole (see earlier). The fixed address is useful for administration and also for functions that have to be pinned to a single machine. Since we don't know which of these will turn out to be the source address for backhand statistics, we mention them both.

NameVirtualHost *:80

The web servers also host a couple of related projects — FreeCEN, FreeREG, and FreeUKGEN — so we used name-based virtual hosting for them.

Listen *:80

Set up the listening port on all IPs.

MinSpareServers 1
MaxSpareServers 1
StartServers 1

Well, this is what happens if you let other people configure your webserver! Configuring the min and max spare servers to be the same is very bad, because it causes Apache to have to kill and restart child processes constantly and will lead to a somewhat unresponsive site. We'd recommend something more along the lines of a Min of 10 and a Max of 25. StartServers matters somewhat less, but it's useful to avoid horrendous loads at startup. This is, in fact, terrible practice, but we thought we'd leave it in as an object lesson.

MaxClients 100

Limit the total number of children to 100. Usually, this limit is determined by how much RAM you have, and the size of the Apache children.

MaxRequestsPerChild 10000

After 10,000 requests, restart the child. This is useful when running mod_perl to limit the total memory consumption, which otherwise tends to climb without limit.

LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\" \
"%{BackhandProxyRequest}n\" \"%{ProxiedFrom}n\""

This provides extra logging so we can see what backhand is up to.

Port 80

This is probably redundant, but it doesn't hurt.

ServerRoot /home/apache

Again, redundant but harmless.

TransferLog /home/apache/logs/access.log
ErrorLog /home/apache/logs/error.log

The "main" logs should hardly be used, since all the actual hosts are in VirtualHost sections.

PidFile /home/apache/logs/httpd.pid
LockFile /home/apache/logs/lockfile.lock

Again, probably redundant, but harmless.

<VirtualHost *:80>
        Port 80
        ServerName freebmd.rootsweb.com
        ServerAlias www.freebmd.org.uk www3.freebmd.org.uk

Finally, our first virtual host. Note that all of this will be the same on each host, except www3.freebmd.org.uk, which will be www1 or 2 on the others.

        DocumentRoot /home/apache/hosts/freebmd/html
        ServerAdmin register@freebmd.rootsweb.com
        TransferLog "| /home/apache/bin/rotatelogs 
                       /home/apache/logs/freebmd/access_log.liberty 86400"
        ErrorLog "| /home/apache/bin/rotatelogs 
                    /home/apache/logs/freebmd/error_log.liberty 86400"

Note that we rotate the logs — since this server gets many hits per second, that's a good thing to do before you are confronted with a 10 GB log file!

        SetEnv BMD_USER_DIR /home/apache/hosts/freebmd/users
        SetEnv AUDITLOG /home/apache/logs/freebmd/auditlog
        SetEnv CORRECTIONSLOG /home/apache/logs/freebmd/correctionslog
        SetEnv MASTER_DOMAIN www1.freebmd.org.uk
        SetEnv MY_DOMAIN www3.freebmd.org.uk

These are used to communicate local configurations to various scripts. Some of them exist because of differences between development and live environments, and some exist because of differences between the various platforms.

        AddType text/html .shtml
        AddHandler server-parsed .shtml
        DirectoryIndex index.shtml index.html

Set up server-parsed HTML, and allow for directory indexes using that.

        ScriptAlias /cgi /home/apache/hosts/freebmd/cgi
        ScriptAlias /admin-cgi /home/apache/hosts/freebmd/admin-cgi
        ScriptAlias /special-cgi /home/apache/hosts/freebmd/admin-cgi
        ScriptAlias /join /home/apache/hosts/freebmd/cgi/bmd-add-user.pl

The various different CGIs, some of which are secure below.

        Alias /scans /home/FreeBMD-scans
        Alias /logs /home/apache/logs/freebmd
        Alias /GUS /raid/freebmd/GUS/Live-GUS
        Alias /motd /home/apache/hosts/freebmd/motd
        Alias /icons /home/apache/hosts/freebmd/backhand-icons

And some aliases to keep everything sane.

        <Location /special-cgi>
                AllowOverride none
                AuthUserFile /home/apache/auth/freebmd/special_users
                AuthType Basic
                AuthName "Live FreeBMD - Liberty Special Administration Site"
                require valid-user
                SetEnv Administrator 1
        </Location>

special-cgi needs authentication before you can use it, and is also particular to this machine.

        <Location />
                Backhand byAge
                Backhand byLoad .5
        </Location>

This achieves load balance. byAge means we won't attempt to use servers that are no longer talking to us, and byLoad means use the least loaded machine — except we prefer ourselves if our load is within .5 of the minimum, to avoid silly proxying based on tiny load average differences. We're also looking into using byBusyChildren, which is probably more sensitive than byLoad, and we are also considering writing a backhand module to allow us to proxy by database load instead.

        <LocationMatch /cgi/(show-file|bmd-user-admin|bmd-add-user|bmd-bulk-add|
                       bmd-challenge|bmd-forgotten|bmd-synd|check-range|
                       list-synd|show-synd-info|submitter)\.pl>
                BackHand off
        </LocationMatch>

        <LocationMatch /(special-cgi|admin-cgi)/>
                BackHand off
        </LocationMatch>

        <LocationMatch /join>
                BackHand off
        </LocationMatch>

These scripts should not be load-balanced.

        <LocationMatch /cgi/bmd-files.pl>
                BackhandFromSO libexec/byHostname.so byHostname (equality)
        </LocationMatch>

This script should always go to equality.

        <LocationMatch /(freebmd|freereg|freecen|search)wusage>
                BackhandFromSO libexec/byHostname.so byHostname (fraternity)
        </LocationMatch>

And these should always go to fraternity.

        <Location /backhand>
                SetHandler backhand-handler
        </Location>

This sets the backhand status page up.

</VirtualHost>

For simplicity, we've left out the configuration for the other virtual hosts. They don't do anything any more interesting, anyway.