Publicly available diagnostics (Managing NFS and NIS, 2nd Edition)

14.4. Publicly available diagnostics

Only a handful of publicly available NFS diagnostic tools exist at the time of this writing. The ethereal/tethereal network analyzer introduced in Chapter 13, "Network Diagnostic and Administrative Tools" provides detailed information for diagnosis of NFS problems at the protocol level. The NFSWATCH utility is mainly used to monitor NFS traffic over the network. The nfsbug and SATAN utilities are used to report potential security problems on NFS servers.

14.4.1. ethereal / tethereal

As described in Chapter 13, "Network Diagnostic and Administrative Tools", ethereal/tethereal can be used to capture network traffic and decode it to a great level of detail. Since ethereal/tethereal can decode NFS Version 2 and NFS Version 3 packets, it can be used to debug NFS communication, permissions, performance, and data corruption problems. It is very similar in functionality to snoop. It provides powerful filtering and is available for a diverse set of platforms where snoop is not.

Consider the example presented in the previous snoop section, where the NFS client rome attempts to access the contents of the filesystems exported by the server zeus through the /net automounter path:

rome% ls -la /net/zeus/export
total 5
dr-xr-xr-x   3 root     root           3 Jul 31 22:51 .
dr-xr-xr-x   2 root     root           2 Jul 31 22:40 ..
drwxr-xr-x   3 root     other        512 Jul 28 16:48 eng
dr-xr-xr-x   1 root     root           1 Jul 31 22:51 home
rome% ls /net/zeus/export/home
/net/zeus/export/home: Permission denied

The network traffic is captured into the /tmp/ethereal.cap file concurrently with the operation. Note that only traffic between rome and zeus is captured:

rome# tethereal -w /tmp/ethereal.cap host rome and host zeus
46 ^C
rome# tethereal -r /tmp/ethereal.cap
  1   0.000000      rome -> zeus     PORTMAP V2 GETPORT Call XID 0x398fd3ea
  2   0.003138      zeus -> rome     PORTMAP V2 GETPORT Reply XID 0x398fd3ea
  3   0.003328      rome -> zeus     NFS V3 NULL Call XID 0x398fd3eb
  4   0.004613      zeus -> rome     NFS V3 NULL Reply XID 0x398fd3eb
  5   0.005823      rome -> zeus     PORTMAP V2 GETPORT Call XID 0x398fca35
  6   0.008871      zeus -> rome     PORTMAP V2 GETPORT Reply XID 0x398fca35
  7   0.009823      rome -> zeus     TCP 49699 > 33168 [SYN] Seq=1251769928 Ack=0 
Win=24820 Len=0
  8   0.011067      zeus -> rome     TCP 33168 > 49699 [SYN, ACK] Seq=3939269366 
Ack=1251769929 Win=24820 Len=0
  9   0.011100      rome -> zeus     TCP 49699 > 33168 [ACK] Seq=1251769929 
Ack=3939269367 Win=24820 Len=0
 10   0.011339      rome -> zeus     MOUNT V1 EXPORT Call XID 0x398f20d9
 11   0.012102      zeus -> rome     TCP 33168 > 49699 [ACK] Seq=3939269367 
Ack=1251769973 Win=24776 Len=0
 12   0.018302      zeus -> rome     MOUNT V1 EXPORT Reply XID 0x398f20d9
 13   0.018332      rome -> zeus     TCP 49699 > 33168 [ACK] Seq=1251769973 
Ack=3939269463 Win=24820 Len=0
 14   0.018588      rome -> zeus     TCP 49699 > 33168 [FIN, ACK] Seq=1251769973 
Ack=3939269463 Win=24820 Len=0
 15   0.019245      zeus -> rome     TCP 33168 > 49699 [ACK] Seq=3939269463 
Ack=1251769974 Win=24820 Len=0
 16   0.020104      zeus -> rome      TCP 33168 > 49699 [FIN, ACK] Seq=3939269463 
Ack=1251769974 Win=24820 Len=0
 17   0.020143      rome -> zeus      TCP 49699 > 33168 [ACK] Seq=1251769974 
Ack=3939269464 Win=24820 Len=0
 18   0.020661      rome -> zeus      PORTMAP V2 GETPORT Call XID 0x398f0440
 19   0.024550      zeus -> rome      PORTMAP V2 GETPORT Reply XID 0x398f0440
 20   0.024731      rome -> zeus      MOUNT V3 NULL Call XID 0x398f0441
 21   0.026323      zeus -> rome      MOUNT V3 NULL Reply XID 0x398f0441
 22   0.026881      rome -> zeus      MOUNT V3 MNT Call XID 0x398f0442
 23   0.179757      zeus -> rome      MOUNT V3 MNT Reply XID 0x398f0442

The explanation given in the snoop section describing each packet applies to the tethereal capture file as well. The main difference is that listing the XID next to the operation type is less intuitive than expanding the arguments to the call as performed by snoop. We suspect this will be addressed in the future. You can see that the reason for failure is not obvious by just looking at this output format.

Fortunately, tethereal has extensive filtering capabilities and we can request all mount operations that failed. Using the mount.status filter, we determine that packet 23 returned a failure. We can then print the protocol tree for packet 23 alone and verify that indeed it failed with ERR_ACCESS:

rome# tethereal -r /tmp/ethereal.cap -R "mount.status != 0"
 23   0.179757      zeus -> rome      MOUNT V3 MNT Reply XID 0x398f0442
rome# tethereal -r /tmp/ethereal.cap -V -R "frame.number == 23"
...
Remote Procedure Call
    XID: 0x398f0442 (965674050)
    Message Type: Reply (1)
    Program: MOUNT (100005)
    Program Version: 3
    Procedure: MNT (1)
    Reply State: accepted (0)
    Verifier
        Flavor: AUTH_NULL (0)
        Length: 0
    Accept State: RPC executed successfully (0)
Mount Service
    Program Version: 3
    Procedure: MNT (1)
    Status: ERR_ACCESS (13)

For simplicity, only the RPC and Mount portions of the packet are shown. The RPC header decodes the transaction ID, message type indicating this to be a reply, program, and version number as well as the procedure invoked. The credential verifier is also decoded indicating that the server used no verifier in its reply (since the call did not specify it to begin with). A nice feature of snoop, that tethereal does not yet have, is the ability to indicate the frame for which this is a reply.

As expected, the status field of the mount service reply reports an error. Packet 12 contains the results of the export information request:

rome# tethereal -r /tmp/ethereal.cap -V -R "frame.number == 12"
...
Remote Procedure Call
    Last Fragment: Yes
    Fragment Length: 92
    XID: 0x398f20d9 (965681369)
    Message Type: Reply (1)
    Program: MOUNT (100005)
    Program Version: 1
    Procedure: EXPORT (5)
    Reply State: accepted (0)
    Verifier
        Flavor: AUTH_NULL (0)
        Length: 0
    Accept State: RPC executed successfully (0)
Mount Service
    Program Version: 1
    Procedure: EXPORT (5)
    Data (68 bytes)

   0  0000 0001 0000 000b 2f65 7870 6f72 742f   ......../export/ 
  10  656e 6700 0000 0000 0000 0001 0000 000c   eng............. 
  20  2f65 7870 6f72 742f 686f 6d65 0000 0001   /export/home.... 
  30  0000 0006 7665 726f 6e61 0000 0000 0000   ....verona...... 
  40  0000 0000                                 ....

The Data field of the Mount packet shows a hex dump of the export list. The interpreted text value is in the far right column. We can see how the export list is encoded into the packet as a set of exported directories , each followed by the list of hosts (or group of hosts) that they give access to.

14.4.2. Useful filters

Read filters help you remove the noise from a packet trace and let you see only the packets that interest you. If a packet meets the requirements expressed in the read filter, then it is printed. Read filters let you compare the fields within a protocol against a specific value, compare fields against other fields, and check the existence of specified fields or protocols altogether. One of the main strengths of tethereal is its powerful filters. You are encouraged to learn more about them from the tethereal documentation. The following list includes some of the read filters you are most likely to use when analyzing NFS-related traffic:

nfs

Displays NFS traffic regardless of the version. Note that MOUNT, NLM, and Portmapper traffic is not captured. Useful once the mount has already occurred. The following example displays all NFS protocol traffic involving the host rome:

# tethereal -R "nfs and ip.addr == rome"

nfs.status

Displays all replies to successful NFS calls when nfs.status == 0 or the replies to unsuccessful NFS calls otherwise. The originating call can be obtained using the rpc.xid filter. The following example displays all NFS failures:

# tethereal -R "nfs.status != 0"

rpc

Displays all RPC traffic regardless of the program number. The following example displays all RPC traffic on the wire:

# tethereal -R "rpc"

rpc.xid

Displays the RPC call or reply matching a given Transaction ID. This is useful when the call packet is available and the matching reply is needed, or viceversa. The following example finds the RPC call and reply with transaction ID equal to 0x398f0441:

# tethereal -R "rpc.xid == 0x398f0441"

tcp.port == 111 or udp.port == 111

Displays rpcbind and portmapper traffic. Useful during filesystem mount negotiation. The following example displays all rpcbind traffic on the network:

# tethereal -R "tcp.port == 111 or udp.port == 111"

rpc.program, rpc.programversion, rpc.procedure

Use rpc.program == 100005 to capture MOUNT protocol related traffic. Useful during the mount process. The following example displays all MOUNT protocol traffic between the hosts zeus and rome:

# tethereal -R "rpc.program == 100005 and ip.addr == zeus \
and ip.addr ==
rome"

Use rpc.program == 100021 to capture NLM traffic. Useful for tracking lock manager-related traffic. The following example displays all NFS Version 3 Network Lock Manager traffic between hosts zeus and rome. Note that NLM v4 is used for NFS Version 3:

# tethereal -R "rpc.program == 100021 and rpc.programversion == 4 \
and ip.addr == rome and ip.addr == zeus"

14.4.3. NFSWATCH

NFSWATCH was developed by David Curry of Purdue University in the late 1980s, with some improvements to the basic framework provided by Jeff Mogul of Digital Equipment Corporation (now Compaq). It is mainly used to monitor NFS activity on a given server, or NFS activity on the local network. NFSWATCH gathers its data by monitoring the network interface of the system where it is invoked.

NFSWATCH 4.3 is the most recent version at the time of this writing, and only supports NFS Version 2 over UDP. You should be aware that at the time of this writing, a bug in the tool causes NFS Version 3 traffic to the server to incorrectly increment the NFS Version 2 counters. This is due to the fact that the tool does not check the NFS version number of the packet received.

Regardless of its current limitations, NFSWATCH is still a very useful tool whose main features are worth mentioning:

The tool categorizes the incoming network traffic and continuously updates the statistics on the display. You can also instruct the tool to create a more detailed log file of the network traffic.
It allows you to log statistics for every NFS operation, for every exported filesystem, for files for which you specify particular interest, or for NFS clients that access your server.
It reports usage of NFS clients and users of the filesystems.
It can be run interactively or remotely (via rsh), or it can be scheduled to run from cron.
Total runtime can be specified for unsupervised traffic monitoring.

NFSWATCH is available at ftp://gatekeeper.dec.com/pub/net/ip/nfs/nfswatch4.3.tar.gz. The following example shows a sample log file of an NFSWATCH run on server zeus for a period of five seconds:

# NFSwatch log file
#    Packets from: all hosts
#    Packets to:   zeus
#
# begin
#
Date: Tue Aug  1 16:31:22 2000
Cycle Time: 5
Elapsed Time: 
#
# total packets      network  to host  dropped
#
Interval Packets:       2371     2371        0
Total Packets:          2371     2371        0
#
# packet counters              int      pct    total
#
ND Read:                         0       0%        0
ND Write:                        0       0%        0
NFS Read:                      166       7%      166
NFS Write:                     346      15%      346
NFS Mount:                       0       0%        0
YP/NIS/NIS+:                     0       0%        0
RPC Authorization:               0       0%        0
Other RPC Packets:            1844      78%     1844
TCP Packets:                     2       0%        2
UDP Packets:                  2358      99%     2358
ICMP Packets:                    1       0%        1
Routing Control:                 2       0%        2
Address Resolution:             10       0%       10
Reverse Addr Resol:              0       0%        0
Ethernet/FDDI Bdcst:            13       1%       13
Other Packets:                   0       0%        0
#
# nfs counters                 int      pct    total
#
/export/home:                  512     100%      512        (0/0/5/0/12/0/154/0/
335/2/0/0/0/0/3/1/0/0)
#
# file counters                int      pct    total
#
#
# nfs procs 
#
 Procedure          int   pct    total  completed  ave.resp  var.resp  max.resp
  CREATE              2    0%        2
  GETATTR             0    0%        0
  GETROOT             0    0%        0
  LINK                0    0%        0
  LOOKUP             12    2%       12
  MKDIR               3    1%        3
  NULLPROC            0    0%        0
  READ              154   30%      154
  READDIR             0    0%        0
  READLINK            0    0%        0
  REMOVE              0    0%        0
  RENAME              0    0%        0
  RMDIR               1    0%        1
  SETATTR             5    1%        5
  STATFS              0    0%        0
  SYMLINK             0    0%        0
  WCACHE              0    0%        0
  WRITE             335   65%      335

The NFSWATCH log shows the distribution of NFS READ, NFS WRITE, NFS MOUNT, NIS, and RPC AUTHORIZATION packets among others. The NFS counters section indicates the total number NFS operations per filesystem exported (one in this case) during the interval. The operation distribution denoted by (0/0/5/0/12/0/154/0/335/2/0/0/0/0/3/1/0/0) indicates that the majority of the operations occurred in the middle of the interval. The packet counters and nfs procs indicate that there were close to twice as many writes as reads. The low lookup count leads us to believe that most writes occurred to the same file.

14.4.4. nfsbug

The nfsbug utility was written by Leendert van Doorn in the mid-1990s to test hosts for well-known NFS problems and bugs. nfsbug is available at http://www.cs.vu.nl/~leendert. Use it to identify (and consequently correct) the following problems:

Find worldwide exportable filesystems. This is a common occurrence in large organizations with hundreds or thousands of NFS clients. System administrators choose to export filesystems to all clients instead of grouping the hosts into netgroups and exporting the filesystems only to the netgroups that really need access to the filesystems.
Determine the effectiveness of the export list.
Determine if filesystems can be mounted through the portmapper.
Attempt to guess filehandles and obtain access to filesystems not exported to the test client.
Exercise the system for well-known bugs.[34]
[34]According to Leendert's web page, the tool has not been updated in recent years, although he still plans to get to it at some point.

14.4.5. SATAN

SATAN is a tool used to find well-known security holes in Unix systems. SATAN stands for Security Administrator's Tool for Analyzing Networks. At the time of this writing, none of the problems SATAN probes for are new. Each one has already been discussed in CERT bulletins and each can be countered either by installing the appropriate patch or fixing a system configuration flaw. SATAN is available at http://www.fish.com/satan.

SATAN was written by Dan Farmer and Wietse Venema and first released for general availability in April of 1995. The tool is intended to help system administrators identify several common network-related security problems, hopefully before someone else has a chance to exploit them. The tool provides a description of the problem, explains the consequences if no action is taken, and indicates how to correct the problem. Note that the tool itself will not exploit the security hole. At the time of this writing, SATAN can identify and fix the following problems related to NFS and NIS:

NFS filesystems exported to arbitrary hosts
NFS filesystems exported to unprivileged programs
NFS filesystems exported via the portmapper
NIS password file access from arbitrary hosts

An extensive discussion of SATAN and its features can be found in Martin Freiss' book titled Protecting Networks with SATAN (O'Reilly & Associates).