Chapter 13. Network Diagnostic and Administrative Tools
Distributed computing architectures rely on a well-conditioned network and properly configured servers for their adequate performance and operation. NFS and NIS client performance degrades if your network is congested or your servers are unreliable. Retransmitted requests add to the noise level on the network or to the request backlog on the server, generally exacerbating any performance problems.
MAC and IP layer tools
Remote procedure call tools
Whenever you make a change, you run the risk of affecting more than just one machine. If you add a new NFS client, for example, you should consider all possible impacts on the computing environment: network bandwidth consumed by traffic to and from this node, or the incremental workload imposed on any servers used by the client. Similarly, when upgrading server resources you must identify those areas that are the tightest constraints: CPU speed, disk speed, or aggregate disk space. Adding another server to a network may not be as economical or beneficial as upgrading to faster disks, adding CPUs to an expandable server or offloading other tasks, such as web service to another host.
This portion of the book focuses on network analysis, debugging, and performance tuning. Its goal is to present the tools, procedures, and evaluation criteria used for analyzing network, NFS, or NIS problems. In addition to tuning and administration, these techniques can be used to evaluate proposals for expanding an existing network with additional clients or servers. Symptoms and causes of common problems will be examined in detail, but the overall focus is on developing techniques to be used on complex problems peculiar to your specific combination of hardware and software.
In this chapter, we present tools for examining the configuration and performance of individual network components, starting at the lowest level of basic point-to-point connectivity and working up to the RPC layer where the NFS- and NIS-specific issues come into play. The chapter includes examples relevant to problem diagnosis to define the methods for collecting and interpreting data about the network and its components. A healthy network is essential to the proper behavior of NFS and NIS. Developing network diagnostic skills is necessary for resolving problems that may only be apparent at the application level. NFS may behave poorly because of a saturated network or due to an overloaded server; a thorough examination of the problem requires checking each component involved. If you fail to understand the low-level operation of a facility, you are more likely to misinterpret performance or usage statistics provided for that facility. We cover the lower layers of the network protocols in detail so that you can see how they affect the performance and behavior of the application layer protocols like NFS and NIS.
As explained in Section 2.3, "Name service switch", NIS and DNS can be used concurrently to resolve hostnames on the same system, although throughout the remainder of this book, we assume that NIS is the only name service running. In some cases, we refer to local files that are used without NIS. However, examples and discussions refer to the most common NIS maps, as shown in Table 13-1.
Table 13-1. Common NIS maps and their nicknames
The /etc/inet/ipnodes file and ipnodes NIS map form a database that associates the names of nodes with their IP addresses. The IP addresses can be either IPv4 or IPv6 addresses. The ipnodes database was introduced in Solaris 8 to support IPv6-aware and IPv6-enabled applications that need to obtain IPv6 addresses. When these applications need IPv4 addresses, they first consult the ipnodes database. If the address is not found, they then consult the traditional hosts database. IPv6-unaware applications simply consult the hosts database, as they are unaware of IPv6 extensions. Although not a requirement, IPv4 addresses defined in the hosts database should be copied to the ipnodes database in order to prevent delays in name resolution, and to keep the ipnodes and hosts database in sync.
IPv6-aware applications can communicate with nodes that do not have an IPv4 address. This means that the application can handle the larger IPv6 addresses.
IPv6-enabled applications take advantage of some IPv6-specific feature. The enabled applications can still operate over IPv4, though in a degraded mode. IPv6-enabled applications are also IPv6-aware.
IPv6-unaware applications cannot handle IPv6 addresses; therefore, they cannot communicate with nodes that do not have an IPv4 address.Throughout this chapter we assume that your system is capable of using both IPv4 and IPv6, and uses the ipnodes database to obtain IP address mappings. If your system does not support IPv6 yet, then replace references to the ipnodes database with the hosts database. To reiterate, an unsuccessful lookup of an IPv4 address in the ipnodes database implies a subsequent lookup of the same address in the hosts database.
13.1. Broadcast addressesMany network problems stem from confusion or inconsistency in the way hosts form their IP broadcast addresses. Broadcast addresses are used when a packet must be sent to all machines on the local area network. For example, if your host needs to send a packet to another machine, it must know the remote machine's IP address and Ethernet address. It can determine the remote IP address by looking up the remote hostname in the NIS ipnodes map, but it may not have the corresponding Ethernet address. If this is the first time your machine is talking to this particular remote host, it won't have had an opportunity to locate or save the remote Ethernet address. The way to determine the remote machine's Ethernet address is to ask all of the hosts on the network if they have the information, using the Address Resolution Protocol (ARP). To broadcast this request to all hosts on the network, your host uses a special kind of destination address called a broadcast address. A normal (or unicast) address identifies only one host; a broadcast address identifies all hosts on the network.
To be an effective broadcast, the packet must reach all nodes on the local area network and be recognized as a broadcast packet by them. An improperly formed broadcast address, or one that other systems do not recognize as such, can be responsible for failures ranging from NIS clients that cannot find servers to storms of broadcast packets initiated by a single packet sent with the wrong broadcast address.
Like host addresses, broadcast addresses exist in both the MAC and IPv4 layers of the protocol stack. There are no broadcast addresses in IPv6; their function is superseded by multicast addresses. An IPv4 broadcast address is converted into a MAC broadcast address, just as a host-specific IP address is converted into a 48-bit Ethernet address. At the MAC layer, there is exactly one broadcast address; for Ethernet it is:
Multicast addresses are used to define subgroups of recipients of data. If a sender needs to contact a large number of hosts simultaneously, the sender can multicast a single message to all hosts listening on the given multicast address, instead of issuing multiple copies of the same message to every single host. The hosts listening on the multicast address do not need to be part of the same subnetwork as the sender.
Every node on the local network receives a packet having this destination MAC address. A host may ignore a broadcast if the request is for a service that it does not provide. A host processes every broadcast packet, at the very least deciding to discard it. Therefore, a high level of broadcast traffic hurts the performance of each host on the network.ff:ff:ff:ff:ff:ff
While the MAC layer broadcast address is very clearly defined, there is some variation in the form of IPv4 broadcast addresses. There are two distinct popular forms, mostly due to evolution of the networking code in Berkeley-based Unix systems. Examples of broadcast addresses of each form are shown for each IPv4 address class in Table 13-2. IPv4 address classes are described in Section 1.3.3, "IPv4 address classes".
The 4.2 BSD release of Unix introduced TCP/IP and required use of the zeros form of broadcast addresses. All derivatives of 4.2 BSD, including SunOS 3.x and early versions of Ultrix, retained this broadcast address requirement. In 4.3 BSD, the ones form of broadcast addresses was adopted, although the zeros form was still supported. Unix operating systems that are descendants of 4.3 BSD -- SunOS 4.x included -- support both one- and zero-filled broadcast addresses. Solaris supports only the ones form.
Table 13-2. Broadcast address forms
The ones form is the most widely accepted and is used in all examples in this book. Octets of the IPv4 address that specify the host number are filled in with 1-valued bits. A variation on the ones form is the zeros form, in which the host number is expressed as zero-valued octets. The all-ones form:
is a variation of the proper ones form address where the 255-valued octets occupy only the host number portion of the address.255.255.255.255
Confusion regarding the "proper" broadcast address stems from the interpretation of octet values 0 and 255 in IPv4 addresses. Zero-valued octets should be used as place holders when specifying a network number and imply "this" network, without any real implication for host numbers. For example, 18.104.22.168 means network number 129.7., but it does not necessarily name any hosts on the network.
Conversely, the one-filled octets are treated like wildcards and imply "any" host on the network. The network number is specified but the host number matches all hosts on that network. Using these connotations for octet values 0 and 255, the ones form of the broadcast address is "correct." There are cases in which the zeros form must be used for backwards compatibility with older operating system releases. Many systems were built using the zeros form of broadcast addresses.
The sole requirement in adopting a broadcast address form is to make the choice consistent across all machines on the network and compatible with your vendor's supported convention. Machines that expect a zeros-form broadcast address interpret a one-filled octet as part of a host number rather than a wildcard. Mixing broadcast address forms on the same network is the most common cause of broadcast storms, in which every confused node on the network transmits and retransmits replies to a broadcast address of a form complementary to the one it is using.
Broadcast addresses, muticast addresses, IP addresses, and other characteristics of the Ethernet interface are set with the ifconfig utility. Because ifconfig governs the lowest level interface of a node to the network, it is the logical place to begin the discussion of network tools.
Copyright © 2002 O'Reilly & Associates. All rights reserved.