14.4. Publicly available diagnosticsOnly a handful of publicly available NFS diagnostic tools exist at the time of this writing. The ethereal/tethereal network analyzer introduced in Chapter 13, "Network Diagnostic and Administrative Tools" provides detailed information for diagnosis of NFS problems at the protocol level. The NFSWATCH utility is mainly used to monitor NFS traffic over the network. The nfsbug and SATAN utilities are used to report potential security problems on NFS servers.
14.4.1. ethereal / tetherealAs described in Chapter 13, "Network Diagnostic and Administrative Tools", ethereal/tethereal can be used to capture network traffic and decode it to a great level of detail. Since ethereal/tethereal can decode NFS Version 2 and NFS Version 3 packets, it can be used to debug NFS communication, permissions, performance, and data corruption problems. It is very similar in functionality to snoop. It provides powerful filtering and is available for a diverse set of platforms where snoop is not. Consider the example presented in the previous snoop section, where the NFS client rome attempts to access the contents of the filesystems exported by the server zeus through the /net automounter path:
The network traffic is captured into the /tmp/ethereal.cap file concurrently with the operation. Note that only traffic between rome and zeus is captured:rome% ls -la /net/zeus/export total 5 dr-xr-xr-x 3 root root 3 Jul 31 22:51 . dr-xr-xr-x 2 root root 2 Jul 31 22:40 .. drwxr-xr-x 3 root other 512 Jul 28 16:48 eng dr-xr-xr-x 1 root root 1 Jul 31 22:51 home rome% ls /net/zeus/export/home /net/zeus/export/home: Permission denied
The explanation given in the snoop section describing each packet applies to the tethereal capture file as well. The main difference is that listing the XID next to the operation type is less intuitive than expanding the arguments to the call as performed by snoop. We suspect this will be addressed in the future. You can see that the reason for failure is not obvious by just looking at this output format. Fortunately, tethereal has extensive filtering capabilities and we can request all mount operations that failed. Using the mount.status filter, we determine that packet 23 returned a failure. We can then print the protocol tree for packet 23 alone and verify that indeed it failed with ERR_ACCESS:rome# tethereal -w /tmp/ethereal.cap host rome and host zeus 46 ^C rome# tethereal -r /tmp/ethereal.cap 1 0.000000 rome -> zeus PORTMAP V2 GETPORT Call XID 0x398fd3ea 2 0.003138 zeus -> rome PORTMAP V2 GETPORT Reply XID 0x398fd3ea 3 0.003328 rome -> zeus NFS V3 NULL Call XID 0x398fd3eb 4 0.004613 zeus -> rome NFS V3 NULL Reply XID 0x398fd3eb 5 0.005823 rome -> zeus PORTMAP V2 GETPORT Call XID 0x398fca35 6 0.008871 zeus -> rome PORTMAP V2 GETPORT Reply XID 0x398fca35 7 0.009823 rome -> zeus TCP 49699 > 33168 [SYN] Seq=1251769928 Ack=0 Win=24820 Len=0 8 0.011067 zeus -> rome TCP 33168 > 49699 [SYN, ACK] Seq=3939269366 Ack=1251769929 Win=24820 Len=0 9 0.011100 rome -> zeus TCP 49699 > 33168 [ACK] Seq=1251769929 Ack=3939269367 Win=24820 Len=0 10 0.011339 rome -> zeus MOUNT V1 EXPORT Call XID 0x398f20d9 11 0.012102 zeus -> rome TCP 33168 > 49699 [ACK] Seq=3939269367 Ack=1251769973 Win=24776 Len=0 12 0.018302 zeus -> rome MOUNT V1 EXPORT Reply XID 0x398f20d9 13 0.018332 rome -> zeus TCP 49699 > 33168 [ACK] Seq=1251769973 Ack=3939269463 Win=24820 Len=0 14 0.018588 rome -> zeus TCP 49699 > 33168 [FIN, ACK] Seq=1251769973 Ack=3939269463 Win=24820 Len=0 15 0.019245 zeus -> rome TCP 33168 > 49699 [ACK] Seq=3939269463 Ack=1251769974 Win=24820 Len=0 16 0.020104 zeus -> rome TCP 33168 > 49699 [FIN, ACK] Seq=3939269463 Ack=1251769974 Win=24820 Len=0 17 0.020143 rome -> zeus TCP 49699 > 33168 [ACK] Seq=1251769974 Ack=3939269464 Win=24820 Len=0 18 0.020661 rome -> zeus PORTMAP V2 GETPORT Call XID 0x398f0440 19 0.024550 zeus -> rome PORTMAP V2 GETPORT Reply XID 0x398f0440 20 0.024731 rome -> zeus MOUNT V3 NULL Call XID 0x398f0441 21 0.026323 zeus -> rome MOUNT V3 NULL Reply XID 0x398f0441 22 0.026881 rome -> zeus MOUNT V3 MNT Call XID 0x398f0442 23 0.179757 zeus -> rome MOUNT V3 MNT Reply XID 0x398f0442
For simplicity, only the RPC and Mount portions of the packet are shown. The RPC header decodes the transaction ID, message type indicating this to be a reply, program, and version number as well as the procedure invoked. The credential verifier is also decoded indicating that the server used no verifier in its reply (since the call did not specify it to begin with). A nice feature of snoop, that tethereal does not yet have, is the ability to indicate the frame for which this is a reply. As expected, the status field of the mount service reply reports an error. Packet 12 contains the results of the export information request:rome# tethereal -r /tmp/ethereal.cap -R "mount.status != 0" 23 0.179757 zeus -> rome MOUNT V3 MNT Reply XID 0x398f0442 rome# tethereal -r /tmp/ethereal.cap -V -R "frame.number == 23" ... Remote Procedure Call XID: 0x398f0442 (965674050) Message Type: Reply (1) Program: MOUNT (100005) Program Version: 3 Procedure: MNT (1) Reply State: accepted (0) Verifier Flavor: AUTH_NULL (0) Length: 0 Accept State: RPC executed successfully (0) Mount Service Program Version: 3 Procedure: MNT (1) Status: ERR_ACCESS (13)
The Data field of the Mount packet shows a hex dump of the export list. The interpreted text value is in the far right column. We can see how the export list is encoded into the packet as a set of exported directories , each followed by the list of hosts (or group of hosts) that they give access to.rome# tethereal -r /tmp/ethereal.cap -V -R "frame.number == 12" ... Remote Procedure Call Last Fragment: Yes Fragment Length: 92 XID: 0x398f20d9 (965681369) Message Type: Reply (1) Program: MOUNT (100005) Program Version: 1 Procedure: EXPORT (5) Reply State: accepted (0) Verifier Flavor: AUTH_NULL (0) Length: 0 Accept State: RPC executed successfully (0) Mount Service Program Version: 1 Procedure: EXPORT (5) Data (68 bytes) 0 0000 0001 0000 000b 2f65 7870 6f72 742f ......../export/ 10 656e 6700 0000 0000 0000 0001 0000 000c eng............. 20 2f65 7870 6f72 742f 686f 6d65 0000 0001 /export/home.... 30 0000 0006 7665 726f 6e61 0000 0000 0000 ....verona...... 40 0000 0000 ....
14.4.2. Useful filtersRead filters help you remove the noise from a packet trace and let you see only the packets that interest you. If a packet meets the requirements expressed in the read filter, then it is printed. Read filters let you compare the fields within a protocol against a specific value, compare fields against other fields, and check the existence of specified fields or protocols altogether. One of the main strengths of tethereal is its powerful filters. You are encouraged to learn more about them from the tethereal documentation. The following list includes some of the read filters you are most likely to use when analyzing NFS-related traffic:
14.4.3. NFSWATCHNFSWATCH was developed by David Curry of Purdue University in the late 1980s, with some improvements to the basic framework provided by Jeff Mogul of Digital Equipment Corporation (now Compaq). It is mainly used to monitor NFS activity on a given server, or NFS activity on the local network. NFSWATCH gathers its data by monitoring the network interface of the system where it is invoked. NFSWATCH 4.3 is the most recent version at the time of this writing, and only supports NFS Version 2 over UDP. You should be aware that at the time of this writing, a bug in the tool causes NFS Version 3 traffic to the server to incorrectly increment the NFS Version 2 counters. This is due to the fact that the tool does not check the NFS version number of the packet received. Regardless of its current limitations, NFSWATCH is still a very useful tool whose main features are worth mentioning:
The NFSWATCH log shows the distribution of NFS READ, NFS WRITE, NFS MOUNT, NIS, and RPC AUTHORIZATION packets among others. The NFS counters section indicates the total number NFS operations per filesystem exported (one in this case) during the interval. The operation distribution denoted by (0/0/5/0/12/0/154/0/335/2/0/0/0/0/3/1/0/0) indicates that the majority of the operations occurred in the middle of the interval. The packet counters and nfs procs indicate that there were close to twice as many writes as reads. The low lookup count leads us to believe that most writes occurred to the same file.# NFSwatch log file # Packets from: all hosts # Packets to: zeus # # begin # Date: Tue Aug 1 16:31:22 2000 Cycle Time: 5 Elapsed Time: # # total packets network to host dropped # Interval Packets: 2371 2371 0 Total Packets: 2371 2371 0 # # packet counters int pct total # ND Read: 0 0% 0 ND Write: 0 0% 0 NFS Read: 166 7% 166 NFS Write: 346 15% 346 NFS Mount: 0 0% 0 YP/NIS/NIS+: 0 0% 0 RPC Authorization: 0 0% 0 Other RPC Packets: 1844 78% 1844 TCP Packets: 2 0% 2 UDP Packets: 2358 99% 2358 ICMP Packets: 1 0% 1 Routing Control: 2 0% 2 Address Resolution: 10 0% 10 Reverse Addr Resol: 0 0% 0 Ethernet/FDDI Bdcst: 13 1% 13 Other Packets: 0 0% 0 # # nfs counters int pct total # /export/home: 512 100% 512 (0/0/5/0/12/0/154/0/ 335/2/0/0/0/0/3/1/0/0) # # file counters int pct total # # # nfs procs # Procedure int pct total completed ave.resp var.resp max.resp CREATE 2 0% 2 GETATTR 0 0% 0 GETROOT 0 0% 0 LINK 0 0% 0 LOOKUP 12 2% 12 MKDIR 3 1% 3 NULLPROC 0 0% 0 READ 154 30% 154 READDIR 0 0% 0 READLINK 0 0% 0 REMOVE 0 0% 0 RENAME 0 0% 0 RMDIR 1 0% 1 SETATTR 5 1% 5 STATFS 0 0% 0 SYMLINK 0 0% 0 WCACHE 0 0% 0 WRITE 335 65% 335
14.4.4. nfsbugThe nfsbug utility was written by Leendert van Doorn in the mid-1990s to test hosts for well-known NFS problems and bugs. nfsbug is available at http://www.cs.vu.nl/~leendert. Use it to identify (and consequently correct) the following problems:
14.4.5. SATANSATAN is a tool used to find well-known security holes in Unix systems. SATAN stands for Security Administrator's Tool for Analyzing Networks. At the time of this writing, none of the problems SATAN probes for are new. Each one has already been discussed in CERT bulletins and each can be countered either by installing the appropriate patch or fixing a system configuration flaw. SATAN is available at http://www.fish.com/satan. SATAN was written by Dan Farmer and Wietse Venema and first released for general availability in April of 1995. The tool is intended to help system administrators identify several common network-related security problems, hopefully before someone else has a chance to exploit them. The tool provides a description of the problem, explains the consequences if no action is taken, and indicates how to correct the problem. Note that the tool itself will not exploit the security hole. At the time of this writing, SATAN can identify and fix the following problems related to NFS and NIS:
Copyright © 2002 O'Reilly & Associates. All rights reserved.