14.3. snoopNetwork analyzers are ultimately the most useful tools available when it comes to debugging NFS problems. The snoop network analyzer bundled with Solaris was introduced in Section 13.5, "Network analyzers". This section presents an example of how to use snoop to resolve NFS-related problems. Consider the case where the NFS client rome attempts to access the contents of the filesystems exported by the server zeus through the /net automounter path:
The client is not able to open the contents of the directory /net/zeus/export/home, although the directory gives read and execute permissions to all users:rome% ls -la /net/zeus/export total 5 dr-xr-xr-x 3 root root 3 Jul 31 22:51 . dr-xr-xr-x 2 root root 2 Jul 31 22:40 .. drwxr-xr-x 3 root other 512 Jul 28 16:48 eng dr-xr-xr-x 1 root root 1 Jul 31 22:51 home rome% ls /net/zeus/export/home /net/zeus/export/home: Permission denied
The df command shows the -hosts automap mounted on the path of interest. This means that the NFS filesystem rome:/export/home has not yet been mounted. To investigate the problem further, snoop is invoked while the problematic ls command is rerun:rome% df -k /net/zeus/export/home filesystem kbytes used avail capacity Mounted on -hosts 0 0 0 0% /net/zeus/export/home
Packet 1 shows the client rome requesting the port number of the NFS service (RPC program number 100003, Version 3, over the UDP protocol) from the server's rpcbind (portmapper). Packet 2 shows the server's reply indicating nfsd is running on port 2049. Packet 3 shows the automounter's call to the server's nfsd daemon to verify that it is indeed running. The server's successful reply is shown in packet 4. Packet 5 shows the client's request for the port number for RPC program number 100005, Version 1, over TCP (the RPC MOUNT program). The server replies with packet 6 with port=33168. Packets 7 through 9 are TCP hand shaking between our NFS client and the server's mountd. Packet 10 shows the client's call to the server's mountd daemon (which implements the MOUNT program) currently running on port 33168. The client is requesting the list of exported entries. The server replies with packet 12 including the names of the two entries exported. Packets 18 and 19 are similar to packets 5 and 6, except that this time the client is asking for the port number of the MOUNT program version 3 running over UDP. Packet 20 and 21 show the client verifying that version 3 of the MOUNT service is up and running on the server. Finally, the client issues the Mount /export/home request to the server in packet 22, requesting the filehandle of the /export/home path. The server's mountd daemon checks its export list, and determines that the host rome is not present in it and replies to the client with a "Permission Denied" error in packet 23. The analysis indicates that the "Permission Denied" error returned to the ls command came from the MOUNT request made to the server, not from problems with directory mode bits on the client. Having gathered this information, we study the exported list on the server and quickly notice that the filesystem /export/home is exported only to the host verona:rome# snoop -i /tmp/snoop.cap rome zeus 1 0.00000 rome -> zeus PORTMAP C GETPORT prog=100003 (NFS) vers=3 proto=UDP 2 0.00314 zeus -> rome PORTMAP R GETPORT port=2049 3 0.00019 rome -> zeus NFS C NULL3 4 0.00110 zeus -> rome NFS R NULL3 5 0.00124 rome -> zeus PORTMAP C GETPORT prog=100005 (MOUNT) vers=1 proto=TCP 6 0.00283 zeus -> rome PORTMAP R GETPORT port=33168 7 0.00094 rome -> zeus TCP D=33168 S=49659 Syn Seq=1331963017 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460> 8 0.00142 zeus -> rome TCP D=49659 S=33168 Syn Ack=1331963018 Seq=4025012052 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460> 9 0.00003 rome -> zeus TCP D=33168 S=49659 Ack=4025012053 Seq=1331963018 Len=0 Win=24820 10 0.00024 rome -> zeus MOUNT1 C Get export list 11 0.00073 zeus -> rome TCP D=49659 S=33168 Ack=1331963062 Seq=4025012053 Len=0 Win=24776 12 0.00602 zeus -> rome MOUNT1 R Get export list 2 entries 13 0.00003 rome -> zeus TCP D=33168 S=49659 Ack=4025012173 Seq=1331963062 Len=0 Win=24820 14 0.00026 rome -> zeus TCP D=33168 S=49659 Fin Ack=4025012173 Seq=1331963062 Len=0 Win=24820 15 0.00065 zeus -> rome TCP D=49659 S=33168 Ack=1331963063 Seq=4025012173 Len=0 Win=24820 16 0.00079 zeus -> rome TCP D=49659 S=33168 Fin Ack=1331963063 Seq=4025012173 Len=0 Win=24820 17 0.00004 rome -> zeus TCP D=33168 S=49659 Ack=4025012174 Seq=1331963063 Len=0 Win=24820 18 0.00058 rome -> zeus PORTMAP C GETPORT prog=100005 (MOUNT) vers=3 proto=UDP 19 0.00412 zeus -> rome PORTMAP R GETPORT port=34582 20 0.00018 rome -> zeus MOUNT3 C Null 21 0.00134 zeus -> rome MOUNT3 R Null 22 0.00056 rome -> zeus MOUNT3 C Mount /export/home 23 0.23112 zeus -> rome MOUNT3 R Mount Permission denied
We could have obtained the same information by inspecting the contents of packet 12, which contains the export list requested during the transaction:rome$ showmount -e zeus export list for zeus: /export/eng (everyone) /export/home verona
For simplicity, only the RPC and NFS Mount portions of the packets are shown. Packet 10 is the request for the export list, packet 12 is the reply. Notice that every RPC packet contains the transaction ID (XID), the message type (call or reply), the status of the call, and the credentials. Notice that the RPC header includes the string "This is a reply to frame 10". This is not part of the network packet. Snoop keeps track of the XIDs it has processed and attempts to match calls with replies and retransmissions. This feature comes in very handy during debugging. The Mount portion of packet 12 shows the list of directories exported and the group of hosts to which they are exported. In this case, we can see that /export/home was only exported with access rights to the host verona. The problem can be fixed by adding the host rome to the export list on the server.rome# snoop -i /tmp/cap -v -p 10,12 ... Packet 10 arrived at 3:32:47.73 RPC: ----- SUN RPC Header ----- RPC: RPC: Record Mark: last fragment, length = 40 RPC: Transaction id = 965581102 RPC: Type = 0 (Call) RPC: RPC version = 2 RPC: Program = 100005 (MOUNT), version = 1, procedure = 5 RPC: Credentials: Flavor = 0 (None), len = 0 bytes RPC: Verifier : Flavor = 0 (None), len = 0 bytes RPC: MOUNT:----- NFS MOUNT ----- MOUNT: MOUNT:Proc = 5 (Return export list) MOUNT: ... Packet 12 arrived at 3:32:47.74 RPC: ----- SUN RPC Header ----- RPC: RPC: Record Mark: last fragment, length = 92 RPC: Transaction id = 965581102 RPC: Type = 1 (Reply) RPC: This is a reply to frame 10 RPC: Status = 0 (Accepted) RPC: Verifier : Flavor = 0 (None), len = 0 bytes RPC: Accept status = 0 (Success) RPC: MOUNT:----- NFS MOUNT ----- MOUNT: MOUNT:Proc = 5 (Return export list) MOUNT:Directory = /export/eng MOUNT:Directory = /export/home MOUNT: Group = verona MOUNT:
14.3.1. Useful filtersInformation is most useful when it can be organized into categories and noise can be filtered and ignored. snoop provides powerful capture filters that allow you to collect only the kind of information you are interested in. The following list of snoop filters is useful when capturing NFS-related traffic. snoop provides very nice NFS and RPC level debugging features. The logical and, or, and not operators can be used on filters to build more powerful composite filters. You are encouraged to review the snoop documentation to learn more about the multiple filters available.
Copyright © 2002 O'Reilly & Associates. All rights reserved.