Advanced map tricks (Managing NFS and NIS, 2nd Edition)

9.5. Advanced map tricks

The automounter has several features that complement the "normal" NFS mount options. It can mount replicated filesystems from one of several potential servers, and it can perform hierarchical mounts of all of a server's directories when any one of them is referenced. This section starts with a discussion of these advanced automounter features, then explains how to get better performance out of the automounter by converting direct map entries into indirect maps and by using the automounter's subdirectory mount feature.

9.5.1. Replicated servers

Multiple location support in the automounter implements a simple network load-balancing scheme for replicated filesystems. At first glance, this seems to be a bit of overkill; after all, you don't need or want replication for read-write filesystems. However, serving large, read-only filesystems such as the manpages may add to an NFS server's request load. Having multiple servers share this load improves performance by reducing the total load placed on the most heavily used servers. Ideally, you want clients that are "close" to each server to mount its filesystems, reducing the amount of traffic that must go through bridges or routers.

For example, if you have four NFS servers that each export the manpages, the best client mounting scheme is probably not to have one-quarter of the clients mount /usr/man from each server. Instead, clients should mount the manpages from the server that is closest to them. Replicated filesystems are included in automounter maps simply by listing all possible servers in the map:

/usr/man        -ro     wahoo:/usr/man mahimahi:/usr/man \ 
                        thud:/usr/man onaga:/usr/man

The backslash at the end of the first line continues this indirect map entry onto the next line. If more than one server:directory pair is listed in an automounter map, the automounter pings all servers by sending a request to the null procedure of all NFS servers. From the set that responds, the automounter picks one that is "closest" by comparing the address of the servers with that of the clients. Ties are broken by using the server that responded to the ping first. The selected server is used by the automounter to serve the mount point.

There is also an element of load balancing at work here: if one of the /usr/man servers is so heavily loaded with other NFS traffic that it cannot reply to the ping before another server on the same net, then the client will choose the other server to handle its mount request. Solaris 2.6 introduced the feature of client-side failover, which was discussed in Section 6.5, "Replication". While it doesn't explicitly implement load balancing, if, after the mount, one server becomes overloaded enough, a client will find the server to be unresponsive and will dynamically switch to another server. Keep in mind the following:

If the ro mount option is not present, or if the soft option is present, client-side failover is not enabled, and in that situation, once a client performs a mount from a server, it continues to use that server until it unmounts the filesystem.
If the list of servers providing the filesystem changes, once the filesystem is mounted, with or without failover, the client cannot choose a different server before unmounting its first choice.

You can use the first-answer feature of replicated map entries to solve the multihomed host problem presented in Section 16.5.7, "Multihomed servers". Let's say that you have an NFS server on four networks, with hostnames boris, boris-bb2, boris-bb3, and boris-bb4 on those networks. Mounting all filesystems from boris makes the multihomed host perform loopback packet routing, but using the "right" hostname requires knowing which name is on your network. Building an automounter map with replicated entries solves this problem by letting the automounter find the fastest route to boris:

natasha    -rw,hard       boris:/export/home/boris \ 
                              boris-bb2:/export/home/boris \ 
                              boris-bb3:/export/home/boris \ 
                              boris-bb4:/export/home/boris

This would be an entry in the auto_home map. Since the server pathnames are the same, you can use a shorter form of the replicated map entry, putting all of the server names in a comma-separated list:

natasha  -rw,hard   boris,boris-bb2,boris-bb3,boris-bb4:/home/boris

The network interface on boris that is closest to the client will respond first, and each NFS client of boris will mount /home/natasha from the best network interface. Note that the replicated mount points don't refer to multiple filesystems, but rather multiple names for the same filesystem. The automounter just provides a neat way of managing all of them in a single place. Because /export/home/natasha is mounted read-write, client-side failover is not enabled. This is somewhat unfortunate since this is the one situation where client-side failover of a writable filesystem is safe: the filesystem is the same, because the physical host is the same. But the client has no way of knowing that.

When the automounter pings the remote servers, it's performing the equivalent of:

rpcinfo -u hostname nfs

for each listed server. If you see a larger number of null procedure calls than usual in the output of nfsstat on the NFS server, it might indicate that your automounter mounts of replicated filesystems are being performed repeatedly. The null calls do not require any disk accesses to service, but they can consume network bandwidth on the server; if the number of null calls becomes excessive it may be due to client machines continually mounting and unmounting replicated filesystems. Changing the value of the -t option to automount (as discussed previously in Section 9.2.4, "Tuning timeout values") reduces the frequency of mounting and unmounting.

You can also examine the /etc/rmtab file on the server to see how frequently its clients are mounting and unmounting automounted filesystems. When a filesystem is mounted, an entry is added to the /etc/rmtab file. When it gets unmounted, the entry isn't deleted from the file -- it is commented out by making the first character in the line a pound sign (#):

#epeche:/usr/share/man 
#haos:/usr/share/man 
#epeche:/usr/share/man 
depeche:/usr/share/man 
chaos:/usr/share/man

In this example, client depeche has mounted /usr/share/man three times, and client chaos has mounted that filesystem twice. This gives you client information to go along with the null NFS RPC counts provided by nfsstat -- you can tell which clients have been repeatedly mounting and unmounting a filesystem. Watch the size of the /etc/rmtab file over time; if it grows regularly and contains multiple entries for the same clients and filesystems, then you may want to change the automounter timeout value on those clients.

9.5.2. Hierarchical mounts

In addition to handling multiple servers for the same filesystem, the automounter can mount multiple trees from the same server in a hierarchy of mount points. Hierarchical mounts are simply a special form of indirect maps.

9.5.2.1. The -hosts map

The most widely used hierarchical mount is the builtin -hosts map, which mounts all exported filesystems from a named host.

The -hosts map references only the hosts database; the map semantics are built into the automounter. It is usually mounted on /net indicating that it contains filesystems from the entire network. The following line would appear in the master map:

/net -hosts -nobrowse

Except when using the enhanced autofs automounter, a user can then force mounts of all filesystems from a server by referencing the server's name as a subdirectory of /net:

% showmount -e wahoo 
/export1      (everyone) 
/export2  honeymoon 
/export3  honeymoon 
% cd /net/wahoo 
% ls -l
total 3
drwxrwxr-x  22 root     staff        512 Aug 12 16:02 export1
drwxrwxr-x   8 root     staff        512 Feb 18  1999 export2
drwxrwxr-x   9 root     staff        512 Sep  8 16:19 export3

When the automounter has to mount a filesystem on /net, it sends a request to the server asking for all exported filesystems. The automounter sorts the filesystems by pathname length, ensuring that subdirectories of exported filesystems appear later in the list than their parents.[14] The original automounter would then mount each item in the sorted list.

[14]If a directory pathname has a length of x characters, then any of its subdirectory's pathnames have length > x. Sorting by pathname length puts a parent directory ahead of all paths to its subdirectories.

The enhanced autofs automounter will lazily mount each exported filesystem as soon as a process does something significant such as changing its current working directory to an exported filesystem:

% cd /net/wahoo 
% ls -l 
total 3
dr-xr-xr-x   1 root     root           1 Sep 28 14:54 export1
dr-xr-xr-x   1 root     root           1 Sep 28 14:54 export2
dr-xr-xr-x   1 root     root           1 Sep 28 14:54 export3
% cd export1
% cd ..
% ls -l
total 3
drwxrwxr-x  22 root     root         512 Aug 12 16:02 export1
dr-xr-xr-x   1 root     root           1 Sep 28 14:54 export2
dr-xr-xr-x   1 root     root           1 Sep 28 14:54 export3

The act of doing the cd export1causes the automounter to perform an NFS mount over the /net/wahoo/export1 autofs vnode. Thus, users cannot casually force the client to mount each filesystem unless they do something like:

% ls /net/wahoo/*

This command invocation tells ls to read each directory of each exported filesystem of wahoo. The autofs filesystem considers an invocation of its readdir entry point to be a significant operation worthy of triggering an NFS mount.

There are a number of caveats for using the -hosts map with automounters that don't support lazy mounting of hierarchies:

By including the entire hosts database, the hosts map references servers that are both local and on remote networks; a casual reference to a remote server causes an NFS mount to occur through a router or gateway.
If the server itself is slow, or has a large number of filesystems (diskless client servers), then the -hosts map has a definite performance impact.
Unmounts of the filesystems are done from the bottom up, in the reverse order of the mounts. If a higher-level mount point is busy, then an unmount of the entire hierarchy fails. When the automounter fails to unmount a higher-level mount point, it must remount the filesystems it just unmounted. It walks back down the hierarchy from the busy mount point, mounting each filesystem. The remote server's filesystems are mounted on an all-or-nothing basis.
Earlier in this section, we said that the "most widely used hierarchical mount is the builtin -hosts map." If you are not careful, it can be the most widely used map, period. The reason why this is not good is that -hosts is location-dependent. Once your users get used to accessing resources like /net/wahoo/tools, instead of accessing /tools, it becomes difficult to move the resource to a different physical location. It is best to discourage use of /net. One way to do so is to respond rapidly to requests to modify existing maps, or add new maps, and also, bury the physical location several directories deep on the server that holds the resource. Users will prefer pathnames like /tools/debugger over /net/wahoo/export/software/tools/debugger.

These caveats don't apply to the enhanced autofs automounter. However, by default it does support browsing. Thus a new caveat is that if a network has lots of hosts, then users that do:

% ls /net

will trigger lots of network traffic as the automounter gets the list of hosts from NIS. Thus, you should use the -nobrowse option on the -hosts map.

TIP: Users sometimes complain that they cannot see a new filesystem exported from a server. This is because a /net mount from the server was in effect before the filesystem was exported, and the automounter has to timeout the mount before unmounting and remounting. Rather than waiting for that to happen, a simple workaround is to tell your users to access the server under /net with a name that differs by capitalizing one letter of the hostname. This works because hostnames are case-insensitive, yet Unix pathnames are case-sensitive. So, for example, if /net/wahoo was in effect before wahoo:/export4 was exported, then simply accessing /net/Wahoo will allow you to access export4 as well as the pre-existing export1, export2, and export3.

9.5.2.2. Hierarchical mounts in non -hosts maps

Let's return to our /tools example. Recall that /tools has:

/tools/deskset 
/tools/sting 
/tools/news 
/tools/bugview

and is an indirect automounter map for the /tools directory, called auto_tools:

deskset         -ro,intr mahimahi:/tools2/deskset 
sting                    mahimahi:/tools2/sting 
news                     thud:/tools3/news 
bugview                  jetstar:/usr/bugview

/tools/deskset contains several subdirectories, one of which is wonderworks-v1.0. You recently get a Version 2.0 of Wonderworks, and you find that it requires more disk space than what mahimahi:/tools2/deskset has available. You have several choices here:

Create a new map entry into auto_tools called deskset2 for the new version of wonderworks. The problem with this is that your users expect to look in /tools/deskset, and not /tools/deskset2 for the desktop productivity tools.
Move the deskset directory from mahimahi to a server with a large partition. The problem is that this will impact existing users that have mahimahi:/tools2/deskset mounted.
Create a hierarchical mount for the deskset map entry such that /tools/deskset/wonderworks-v2.0 is mounted from somewhere else. This solution has none of the disadvantages of the previous choices.

To do the last choice requires the following steps:

Create a mount point for wonderworks-v2.0 on server mahimahi:
```
On mahimahi:
# mkdir /tools/deskset/wonderworks-v2.0
```
Create a directory on another server (e.g., wahoo:/export/tools/deskset/wonderworks-v2.0) with sufficient disk space, and copy the wonderworks-v2.0 package to it. If necessary, export the directory via a new entry in /etc/dfs/dfstab and the shareall command.

Change the deskset entry in the auto_tools map to:

deskset         / -ro,intr mahimahi:/tools2/deskset \
	                /wonderworks-v2.0 -ro,intr mahimahi:/tools2/deskset

Now when the user accesses /tools/deskset, he or she will be able reference both /tools/wonderworks-v1.0 and /tools/wonderworks-v1.0.

As the example suggests, the syntax of a hierarchical mount's map entry is:

key-name subdirectory1 [ -mount-options ] server-filesystem-1 [ subdirectory2 [ -mount-options ] server-filesystem-2 ] ...

where a server-filesystem is one of:

server_name:pathname
server_name-i:pathname-i,server_name-ii:pathname-ii [,...]
server_name-i,server_name-ii [,...]:pathname

9.5.3. Conversion of direct maps

Direct mounts are useful for handling nonuniform naming schemes, but they may cause a number of performance problems if several direct mount points are included in a directory that is frequently searched. You can usually get better performance out of the automounter by converting direct maps into indirect maps. Instead of putting direct map mount points in the client filesystem, create symbolic links that point to a staging area managed by an indirect map.

Again, an example helps to explain the conversion process. Consider replacing a direct map for /usr/local with an indirect map auto_stage. To convert the direct map into an indirect map, we first create a symbolic link /usr/local that points to a staging area that we'll let the automounter manage:

Original direct map 
/usr/local      mahimahi:/local/$ARCH 
# ln -s /stage/local /usr/local 

New entry in auto_master map
/stage          auto_stage      -ro 

New indirect map auto_stage containing
local   -ro     mahimahi:/local/$ARCH

Note that /usr/local didn't exist before we made the link, since it was managed by the automounter. Also, we don't have to create the /stage staging directory, since it is an indirect map mount point.

The symbolic link points to a subdirectory of the mount point managed by the indirect map auto_stage. With the direct map, any reference to /usr/local is directed to the /stage mount point, which causes the automounter to mount the appropriate architecture-specific directory. This makes /usr/local look like a link to the mount.

Let's say a user now accesses /usr/local/bin/emacs. The client kernel follows /usr/local down to the symbolic link, which points to the /stage/local automounter mount point. The automounter picks up the reference to /stage as a reference to the auto_stage map, and it uses the next component -- local -- as a key in the map. This causes mahimahi:/local/$ARCH to be automounted. If you have several direct mount points, they can all be converted into links sharing a single auto_stage map.

9.5.4. Multiple indirection

So far the only map we've seen that refers to other maps is the auto_master map. Let's collect all of the indirect maps we've added to auto_master in this chapter:

# Directory     Map             Mount Options 
/home           auto_home       -nobrowse
/net            -hosts          -nobrowse
/tools          auto_tools      -ro 
/source         auto_source     -rw
/stage          auto_stage      -ro

One problem with this approach is that the top-level root ( / ) directory is beginning to get cluttered. Of course, one could simply add another component to the mount directory. If we want to put everything under /auto, then we could change indirect map entries of the master map to:

# Directory     Map             Mount Options 
/auto/home      auto_home       -nobrowse
/auto/net       -hosts          -nobrowse
/auto/tools     auto_tools      -ro 
/auto/source    auto_source     -rw
/auto/stage     auto_stage      -ro

If you are using the autofs automounter, then there is a more elegant approach: simply treat each indirect map as a map entry in new indirect map called auto_auto. To do this, the master map would look like:

# Directory     Map             Mount Options
/auto           auto_auto
/-              auto_direct

The auto_auto map is an indirect map. Like all other indirect maps, its first field has to be a directory relative to /auto, its second field has to be a set of mount options, and its third field has to be the name of the thing we are mounting. Here is what auto_auto looks like:

# Directory    Options                     Map being mounted 
home           -fstype=autofs,nobrowse     auto_home
net            -fstype=autofs,nobrowse     -hosts
tools          -fstype=autofs,ro           auto_tools 
source         -fstype=autofs,rw           auto_source
stage          -fstype=autofs,ro           auto_stage

The second and third fields in auto_auto are basically swapped from what they would be in auto_master. The difference is the presence of the fstype option. This option is needed to unambiguously tell the autofs automounter that this is not map entry referring to an NFS-mounted filesystem.

There is no limit on multiple indirection. This fact allows you to create sensible hierarchies that can be extended ad infinitum. Let's return to the auto_source example, which contains:

sunos5.6        -ro     srcserv:/source/sunos5.6 
sunos5.7        -ro     srcserv:/source/sunos5.7 
nfs             -ro     bigguy:/source/nfs_internals

You've decided to add Linux, BSD, FreeBSD, and System V sources to this map, and you have multiple versions of each. Rather than having a map of contain entries called sunos5.6, sunos5.7, linux1.0, linux2.0, bsd4.3, bsd4.4, sysVr3, sysVr4, etc., you decide that you want a hierarchy that branches first on the name of the operating system and then on the release. So you change auto_source to:

bsd             -fstype=autofs         auto_bsd
linux           -fstype=autofs         auto_linux
nfs             -ro                    bigguy:/source/nfs_internals
sunos           -fstype=atofs          auto_sunos
sysv            -fstype=atofs          auto_sysv

The auto_bsd map might contain:

4.1c        -ro     ancient:/export/source/bsd4.1c
4.2         -ro     ancient:/export/source/bsd4.2
4.3         -ro     ancient:/export/source/bsd4.3
4.4         -ro     srcsrv:/source/bsd4.4

This should be enough to get the idea; for brevity, we won't expand on what the other maps might look like.

Note that the auto_source map example contains both entries with fstype=autofs, and an nfs entry referring to bigguy:/source/nfs_internals.

By the way, you probably will want to leave the -hosts and auto_home maps at /net and /home. The reason is that lots of software assumes these mount points exist. So you would want auto_master to look like:

# Directory     Map             Mount Options
/auto           auto_auto
/home           auto_home       -nobrowse
/net            -hosts          -nobrowse
/-              auto_direct

9.5.5. Executable indirect maps

The autofs automounter contains another feature known as executable maps. If permissions on an indirect map file are marked as executable, then the autofs automounter assumes it is an executable program or shell script, and executes it, passing the key as the first and only argument to the program or script. The program or script must then display an indirect map entry, which can be hierarchal. For example, suppose /etc/auto_master has:

# Directory     Map             Mount Options
/auto           auto_auto
/home           auto_home       -nobrowse
/net            -hosts          -nobrowse
/net2           /etc/auto_exec
/-              auto_direct

Examine /etc/auto_exec:

% ls -l /etc/auto_exec
-rwxr-xr-x   1 root      sys         76 Oct 26 09:58 /etc/auto_exec
% cat /etc/auto_exec
#!/bin/sh
/usr/sbin/showmount -e $1 | \
awk 'NR > 1 {print $1       "'$1':"$1 " \\"}' | sort

This script takes the key value as if it is a hostname, and asks the NFS server, via the showmount command, which filesystems are exported. The output of showmount is then formatted by the awk command to produce a hierarchical map entry. You can test the script manually by doing:

% /etc/auto_exec foo
/export1        foo:/export1 \
/export2        foo:/export2 \

Thus, the script implements functionality similar to /net, with one difference. Note that the -nobrowse mount option isn't included in the /net2 entry of auto_master. This is because executable maps can't be browsed. There doesn't seem to be any reason why the enhanced autofs automounter couldn't have been implemented to support it, perhaps by having a browse= option that referred to yet another program or script to do the browsing.

If, for some reason, the executable program or script cannot resolve the key to a map entry, then it should display zero bytes of output to standard output. Any output displayed to standard error will be logged by the automounter onto the system console.

TIP: Make sure that if you have an automounter map file with the executable permission bit set that you actually want it to be executed.