Mount point constructions (Managing NFS and NIS, 2nd Edition)

18.7. Mount point constructions

The choice of a mount point naming scheme can have a significant impact on NFS server usage. Two common but inefficient constructions are stepping-stone mounts and server-resident symbolic links. In each case, the client must first query the NFS server owning the intermediate mount point (or symbolic link) before directing a request to the correct target server.

A stepping-stone mount exists when you mount one NFS filesystem on top of another directory, which is itself part of an NFS-mounted filesystem from a different server. For example:

# mount mahimahi:/usr           /usr 
# mount wahoo:/usr/local        /usr/local 
# mount poi:/usr/local/bin      /usr/local/bin

To perform a name lookup on /usr/local/bin/emacs, the NFS client performs directory searches and file attribute queries on all three NFS servers, when the only "interesting" server is poi. It's best to mount all of the subdirectories of /usr and /usr/local from a single fileserver, so that you don't send RPC requests to other fileservers simply because they own the intermediate components in the pathname. Stepping-stone mounts are frequently created for consistent naming schemes, but they add to the load of "small" RPC calls handled by all NFS servers.

Symbolic links are also useful for imposing symmetric naming conventions across multiple filesystems but they impose an unnecessary load on an NFS server that is regularly called upon to resolve the links (if the NFS client does not perform symbolic link caching). NFS pathnames are resolved a component at a time, so any symbolic links encountered in a pathname must be resolved by the host owning them.

For example, consider a /usr/local that is composed of links to various subdirectories on other servers:

# mount wahoo:/usr/local /usr/local 
# cd /usr/local 
# ls -l 
lrwxrwxrwx 1 root 16 May 17 19:12 bin -> /net/poi/bin 
lrwxrwxrwx 1 root 16 May 17 19:12 lib -> /net/mahimahi/lib 
lrwxrwxrwx 1 root 16 May 17 19:12 man -> /net/irie/man

Each reference to any file in /usr/local must first go through the server wahoo to get the appropriate symbolic link resolved. Once the link is read, the client machine can then look up the directory entry in the correct subdirectory of /net. Every request that requires looking up a pathname now requires two server requests instead of just one. Solaris, as well as other modern NFS implementations reduce this penalty by caching symbolic links. This helps the client avoid unnecessary trips to the intermediate server to resolve readlink requests.

Use nfsstat -s to examine the number of symbolic link resolutions performed on each server:

% nfsstat -ns 
Server nfs:
calls       badcalls    
221628      769         
Version 2: (774 calls)
null        getattr     setattr     root        lookup      readlink    
8 1%        0 0%        0 0%        0 0%        762 98%     0 0%        
read        wrcache     write       create      remove      rename      
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%        
link        symlink     mkdir       rmdir       readdir     statfs      
0 0%        0 0%        0 0%        0 0%        0 0%        4 0%        
Version 3: (219984 calls)
null        getattr     setattr     lookup      access      readlink    
1023 0%     73495 33%   4383 1%     31493 14%   26672 12%   46299 21%      
read        write       create      mkdir       symlink     mknod       
11606 5%    7618 3%     1892 0%     64 0%       37 0%       0 0%        
remove      rmdir       rename      link        readdir     readdirplus 
3183 1%     5 0%        308 0%      1145 0%     456 0%      1138 0%     
fsstat      fsinfo      pathconf    commit      
7076 3%     109 0%      178 0%       1804 0%

If the total percentage of readlink calls is more than 10% of the total number of lookup calls on all NFS servers, there is a symbolic link fairly high up in a frequently traversed path component. You should look at the total number of lookup and readlink calls on all servers, since the readlink is counted by the server that owns the link while the lookup is directed to the target of the symbolic link.

If you have one or more symbolic links that are creating a pathname lookup bottleneck on the server, remove the links (on the server) and replace them with a client-side NFS mount of the link's target. In the previous example, mounting the /net subdirectories directly in /usr/local would cut the number of /usr/local-related operations in half. The performance improvement derived from this change may be substantial when symbolic links are not cached, since every readlink call requires the server to read the link from disk. Stepping-stone mounts, although far from ideal, are faster than an equivalent configuration built from symbolic links when the clients do not cache symbolic link lookups.

Most filesystem naming problems can be resolved more easily and with far fewer performance penalties by using the automounter, as described in Chapter 9, "The Automounter".