Chapter 14. NFS Diagnostic Tools

14.1. NFS administration tools

NFS administration problems can be of different types. You can experience problems mounting a filesystem from a server due to export misconfiguration, problems with file permissions, missing information, out-of-date information, or severe performance constraints. The output of the NFS tools described in this chapter will serve as input for the performance analysis and tuning procedures in Chapter 17, "Network Performance Analysis".

Mount information is maintained in three files, as shown in Table 14-1.

Table 14-1. Mount information files

File	Host	Contents
/etc/dfs/sharetab	server	Currently exported filesystems
/etc/rmtab	server	host:directory name pairs for clients of this server
/etc/mnttab	client	Currently mounted filesystems

An NFS server is interested in the filesystems (and directories within those filesystem) it has exported and what clients have mounted filesystems from it. The /etc/dfs/sharetab file contains a list of the current exported filesystems and under normal conditions, it reflects the contents of the /etc/dfs/dfstab file line-for-line.

The existence of /etc/dfs/dfstab usually determines whether a machine becomes an NFS server and runs the mountd and nfsd daemons. During the boot process, the server checks for this file and executes the shareall script which, in turn, exports all filesystems specified in /etc/dfs/dfstab. The mountd and nfsd daemons will be started if at least one filesystem was successfully exported via NFS. An excerpt of the /etc/init.d/nfs.server boot script is shown here:

startnfsd=0
if [ -f /etc/dfs/dfstab ]; then
	 /usr/sbin/shareall -F nfs
	 if /usr/bin/grep -s nfs /etc/dfs/sharetab >/dev/null; then
           startnfsd=1
     fi
fi

if [ $startnfsd -ne 0 ]; then
	 /usr/lib/nfs/mountd
     /usr/lib/nfs/nfsd -a 16
fi

The dynamically managed file of exported filesystems, /etc/dfs/sharetab, is truncated to zero length during the boot process. This takes place in the nfs.server boot script, although the truncation code is not shown in this example. Once mountd is running, the contents of /etc/dfs/sharetab determine the mount operations that will be permitted by mountd.

/etc/dfs/sharetab is maintained by the share utility, so the modification time of /etc/dfs/sharetab indicates the last time filesystem export information was updated. If a client is unable to mount a filesystem even though the filesystem is named in the server's /etc/dfs/dfstab file, verify that the filesystem appears in the server's /etc/dfs/sharetab file by using share with no arguments:

server% share 
-               /export/home1   rw   "Cool folks"
-               /export/home2   root=mahimahi:thud   ""  

If the sharetab file is out-of-date, then re-running share on the server should make the filesystem available. Note that there's really no difference between cat /etc/dfs/sharetab and share with no arguments. Except for formatting differences, the output is the same.

When mountd accepts a mount request from a client, it notes the directory name passed in the mount request and the client hostname in /etc/rmtab. Entries in rmtab are long-lived; they remain in the file until the client performs an explicit umount of the filesystem. This file is not purged when a server reboots because the NFS mounts themselves are persistent across server failures.

Before an NFS client shuts down, it should try to unmount its remote filesystems. Clients that mount NFS filesystems, but never unmount them before shutting down, leave stale information in the server's rmtab file.

In an extreme case, changing a hostname without performing a umountall before taking the host down makes permanent entries in the server's rmtab file. Old information in /etc/rmtab has an annoying effect on shutdown, which uses the remote mount table to warn clients of the host that it is about to be rebooted. shutdown actually asks the mountd daemon for the current version of the remote mount table, but mountd loads its initial version of the table from the /etc/rmtab file. If the rmtab file is not accurate, then uninterested clients may be notified, or shutdown may attempt to find hosts that are no longer on the network. The out-of-date rmtab file won't cause the shutdown procedure to hang, but it will produce confusing messages. The contents of the rmtab file should only be used as a hint; mission-critical processing should never depend on its contents. For instance, it would be a very bad idea for a server to skip backups of filesystems listed in rmtab on the simple assumption that they are currently in use by NFS clients. There are multiple reasons why this file can be out-of-date.

The showmount command is used to review server-side mount information. It has three invocations:

showmount -a [server]    Prints client:directory pairs for server's clients. 
showmount -d [server]    Simply prints directory names mounted by server's clients.
showmount -e [server]     Prints the list of shared filesystems.

For example:

% showmount -a 
bears:/export/home1 
bears:/export/home2/wahoo 
honeymoon:/export/home2/wahoo
131.40.52.44:/export/home1
131.40.52.44:/export/home2
 
% showmount -d mahimahi 
/export/home1 
/export/home2

% showmount -e mahimahi
/export/home1       (everyone)
/export/home2       (everyone)

In the first example, an unknown host, indicated by the presence of an IP address instead of a hostname, has mounted filesystems from the local host. If the IP address is valid on the local network, then the host's name and IP address are mismatched in the name service hosts file or in the client's /etc/hosts file. However, this could also indicate a breach of security, particularly if the host is on another network or the host number is known to be unallocated.

Finally, the client can review its currently mounted filesystems using df, getting a brief look at the mount points and corresponding remote filesystem information:

df               Shows current mount information. 
df -F fstype     Looks at filesystems of type fstype only. 
df directory      Locates mount point for directory.

For example:

% df -k -F nfs 
filesystem            kbytes    used   avail capacity  Mounted on 
onaga:/export/onaga   585325  483295   43497    92%    /home/onaga 
thud:/export/thu      427520  364635   20133    95%    /home/thud 
mahimahi:/export/mahimahi 
                      371967  265490   69280    79%    /home/mahimahi

The -k option is used to report the total space allocated in the filesystem in kilobytes. When df is used to locate the mount point for a directory, it resolves symbolic links and determines the filesystem mounted at the link's target:

% ls -l /usr/local/bin 
lrwxrwxrwx  1 root         16 Jun  8 14:51 /usr/local/bin -> /tools/local/bin 
% df -k /usr/local/bin 
filesystem            kbytes    used   avail capacity  Mounted on 
mahimahi:/tools/local 217871  153022   43061    78%    /tools/local

df may produce confusing or conflicting results in heterogeneous environments. Not all systems agree on what the bytes used and bytes available fields should represent; in most cases they are the number of usable bytes available to the user left on the filesystem. Other systems may include the 10% space buffer included in the filesystem and overstate the amount of free space on the filesystem.

Detailed mount information is maintained in the /etc/mnttab file on the local host. Along with host (or device) names and mount points, mnttab lists the mount options used on the filesystem. mnttab shows the current state of the system, while /etc/vfstab only shows the filesystems to be mounted "by default." Invoking mount with no options prints the contents of mnttab ; supplying the -p option produces a listing that is suitable for inclusion in the /etc/vfstab file:

% mount 
/proc on /proc read/write/setuid on Wed Jul 26 01:33:02 2000
/ on /dev/dsk/c0t0d0s0 read/write/setuid/largefiles on Wed Jul 26 01:33:02 2000
/usr on /dev/dsk/c0t0d0s6 read/write/setuid/largefiles on Wed Jul 26 01:33:02 2000
/dev/fd on fd read/write/setuid on Wed Jul 26 01:33:02 2000
/export/home on /dev/dsk/c0t0d0s7 setuid/read/write/largefiles on Wed Jul 26 01:33:04 2000
/tmp on swap read/write on Wed Jul 26 01:33:04 2000
/home/labiaga on berlin:/export/home11/labiaga intr/nosuid/noquota/remote on Thu Jul 27 17:39:59 2000
/mnt on paris:/export/home/rome read/write/remote on Thu Jul 27 17:41:07 2000
 
% mount -p 
/proc - /proc proc - no rw,suid
/dev/dsk/c0t0d0s0 - / ufs - no rw,suid,largefiles
/dev/dsk/c0t0d0s6 - /usr ufs - no rw,suid,largefiles
fd - /dev/fd fd - no rw,suid
/dev/dsk/c0t0d0s7 - /export/home ufs - no suid,rw,largefiles
swap - /tmp tmpfs - no rw
berlin:/export/home11/labiaga - /home/labiaga nfs - no intr,nosuid,noquota
paris:/export/home/rome - /mnt nfs - no rw

Although you can take the output of the mount -p command and include the NFS mounts in the client's /etc/vfstab file, it is not recommended. Chapter 9, "The Automounter" describes the many reasons why dynamic mounts are preferred. However, if static cross-mounting is required, use the background (bg) option to avoid deadlock during server reboots when two servers cross-mount filesystems from each other and reboot at the same time.