Configuring LocalDirector Failover

Failover Overview

Failover Setup

Failover Command Summary

Implementing LocalDirector Failover

Redundant Power Planning

Failover Interface Tests

Failover Syslog Messages

show failover Command Output

Frequently Asked Questions

Configuring LocalDirector Failover

This chapter describes LocalDirector failover and contains the following sections:

Failover Overview

Two LocalDirector units can be configured to provide automatic backup capabilities for each other. This configuration is called LocalDirector failover.

In a LocalDirector failover configuration, one LocalDirector is the primary unit (default active) and the other LocalDirector is the backup unit (default standby). The primary unit performs normal network functions. The backup unit monitors the primary unit operation and is ready to take control if the primary unit fails.

Note You must use identical LocalDirector units as failover pairs. Each unit must have the same hardware platform configuration (model, interfaces, and software release).

When a failure occurs, the backup unit becomes the active unit and assumes control by exchanging Media Access Control (MAC) addresses with the primary unit. The units do not give up their primary or backup designations, however.

The active unit (whether primary or backup) uses the system IP and MAC addresses of the primary unit. The standby unit (whether primary or backup) uses the failover IP address and MAC address of the backup unit. Because the active unit always uses the same IP and MAC addresses, Address Resolution Protocol (ARP) entries do not need to change or time out anywhere on the network.

The standby unit monitors failover communications, active unit power status, interface line status, and received hello packets. A failure of any of these parameters on the active unit causes the standby unit to take control. A failure or change generates syslog messages regarding the cause of the failover.

To take a unit out of the "failed" state, power cycle the unit or use the failover reset command. The failover reset command also clears failover timers and counters for the LocalDirector unit. When a failed primary unit is brought back online, the primary unit does not automatically resume as the active unit (because the primary unit could immediately enter the failed state). However, if a failure is due to a lost signal on a network interface card, failover autorecovers when the network is available.

Use the failover active command to initiate a failover change from the standby unit. Use the no failover active command at the active unit to initiate a failover change. You can also use this feature to force an active unit offline for maintenance.

Note Enter configuration data at the active unit, since configuration replication from active unit to standby unit is automatic.

By default, the standby unit does not keep state information on each connection; all active connections are dropped and must be reestablished by the clients. However, if you configure stateful failover on the primary and secondary LocalDirector units, the standby LocalDirector not only has copies of the active LocalDirector configuration, but also has copies of the tables that show the active connections and their state. If the active unit fails, these connections are still valid, and users continue an active session with the server. Use the replicate command to configure stateful failover (the default replication port is interface 0) for each virtual server.

The configuration of the active unit is copied to the standby unit under the following conditions:

When the standby unit completes its initial bootup, the active unit replicates its entire configuration to the standby unit.
As commands are entered on the active unit, they are sent across to the standby unit. (The commands are sent through the failover cable.)
Using the write standby command on the active unit forces the entire configuration to the standby unit.

Note Do not make configuration changes on the standby LocalDirector.

Failover Setup

To set up a failover configuration, perform the following steps:

Note For failover cabling instructions, refer to the hardware installation guide that shipped with your LocalDirector.

Step 1 Power up the primary unit.

Step 2 Use the failover ip address command to set the IP address for the standby unit.

To take advantage of multiple IP addresses or dispatched mode, or allow the failover unit to be on a different network from that of the real servers, use the failover alias ip address command to set up an alias on the standby failover unit. A maximum of 256 aliases is allowed.

Step 3 If you want to configure stateful failover, use the replicate command. See the replicate command description in "Command Reference," for more information.

Note Turn off unused interfaces with the shutdown command.

Step 4 Write the primary configuration to a floppy disk or a TFTP server.

Note The standby unit does not write the configuration it receives from the primary unit into memory; the configuration needs to be backed up outside the LocalDirector pair to protect against memory failure on the primary unit.

Step 5 Power up the standby unit.

Step 6 Reboot the primary unit to start the configuration replication to the secondary unit and start failover monitoring.

Step 7 Check your configuration:

a. Use the show failover command to display the status of the failover cable and the unit that is active. See the section "show failover Command Output" later in this chapter for more information about this command.

b. Use the show ip address command to display the current IP address of the unit. If the unit is active, the system IP address appears; if the unit is standby, the failover IP address appears.

Failover Command Summary

Table 4-1 lists the commands that are used for failover configurations. For complete descriptions of these commands, see "Command Reference."

Table 4-1: Commands for Failover Configurations

Command	Description
failover	Sets up failover configuration.
failover active	Forces LocalDirector to active state.
failover alias ip address	Sets failover alias IP address.
failover hellotime	Sets failover hellotime
failover ip address	Sets failover IP address.
failover reset	Resets a failed LocalDirector.
replicate	Sets up stateful connections.
show failover	Shows status.
show ip	Displays IP address. For the active unit, the system IP address is displayed; for the standby unit, the failover IP address is displayed.

Implementing LocalDirector Failover

This section discusses implementing LocalDirector failover in a fault-tolerant network configuration. Many sites employ one or more LocalDirector units in situations where heavy traffic occurs and where redundant switches are used to route incoming traffic to multiple locations. Figure 4-1 shows a LocalDirector configuration that is fault-tolerant.

The configuration in Figure 4-1 produces the following results, given any component failure:

Router failure makes Hot Standby Router Protocol (HSRP) designate the other router active, and the switches identify the backup router as active, based on the activity showing up on its switch ports. The LocalDirector units stay in the nonfailover state, and traffic to the active LocalDirector is maintained by the Inter-Switch Link (ISL). Downstream traffic to the servers is sent as in the nonfailover state.
Switch failure causes the LocalDirector attached to that switch to produce failover if it is active because it perceives the interface as dead (no hello packets). If that LocalDirector is not active, nothing happens except that the switches identify the failed switch using ISL, and the switch attached to the active LocalDirector takes over the load.
LocalDirector failure causes failover in the LocalDirector units. The newly active LocalDirector starts sending and receiving traffic, and the switches move the MAC addresses used by the LocalDirector to other ports. The switches communicate this information through the ISL. If the switches are part of a spanning-tree group, they block these packets for twice the time period of the forward delay parameter.

Note that in the Figure 4-1 configuration, failure of a server-side switch removes access to the servers attached to it. This situation can be minimized by using servers with dual LAN ports, such as exist on some LAN cards designed for redundant links.

Figure 4-1: Fault-Tolerant Failover Configuration

Redundant Power Planning

In planning for redundant Web sites, it is wise to plan for power failures, so that equipment affected is backed up by other equipment that is not on the same power circuit.

It also makes sense to not provide power in such a way that multiple failovers occur at the same time; for example, having a gateway router and a switch served by the same power circuit or supply. This situation would mean that a switch and router would both try to converge their routes at the same time, which would cause problems in a heavily loaded network.

Failover Interface Tests

If there is a loss of network communication over an interface, failover begins a series of tests to determine which unit failed. These tests begin when hello messages are not heard for a number of consecutive 5-second intervals (the number is set by the failover hellotime command). Hello messages are sent over both network interfaces and the serial cable every 5 seconds.

The tests generate network traffic to determine which (if either) unit is failed. At the start of each test, each unit clears its received packet count for its interfaces. At the conclusion of each test, each unit checks whether it has received any traffic. If it has, the interface is considered operational. If one unit receives traffic for a test and the other unit does not, the unit that received no traffic is considered failed. If neither unit has received traffic, both units go to the next test.

Note If the failover IP address has not been set, failover does send hello messages over each interface, and the network activity, ARP, and broadcast ping tests are not performed.

The following lists the failover interface tests:

Link up or down test—This is a test of the network interface card. If an interface card is not plugged into an operational network, it is considered failed (for example, the hub or switch is failed, has a failed port, or a cable is unplugged).
Network activity test—This is a received network activity test. The unit counts all received packets for up to 5 seconds. If any packets are received at any time during this interval, the interface is considered operational and testing stops. If neither unit receives traffic, the ARP test begins.
ARP test—The ARP test consists of reading the ARP cache for the ten most recently acquired ARP entries. One at a time, the unit sends ARP requests to these servers, attempting to stimulate network traffic. After each request, each unit counts all received traffic for up to 5 seconds. If traffic is received, the interface is considered operational. If no traffic is received by either unit, an ARP request is sent to the next server. After the test tries the ten entries, if no traffic has been received, the ping broadcast test begins.
Broadcast ping test—The broadcast ping test consists of sending out a broadcast ping request. Each unit then counts all received packets for up to 5 seconds. If any packets are received at any time during this interval, the interface is considered operational and testing stops. If no traffic is received, the testing starts over again with the ARP test.

Failover Syslog Messages

Failover messages always have a syslog priority level of 2, which indicates a critical condition. All failover syslog messages are also sent as Simple Network Management Protocol (SNMP) syslog traps.

To receive SNMP syslog traps (SNMP failover traps), the SNMP agent must be configured to send SNMP traps to SNMP management stations, define a syslog host, and also compile the Cisco syslog MIB into your SNMP management station. See the snmp-server and syslog command descriptions in "Command Reference," for more information.

The syslog messages sent to record failover events are listed in the "Syslog Messages" section of Appendix A, "Troubleshooting for LocalDirector."

show failover Command Output

The following is the normal output for the show failover command. Note that the IP address used by each unit is displayed.

LD-430(config)# show failover

Failover On
Cable status:Normal
        This host:Secondary - Standby 
                Active time:0 weeks, 3 days, 23 hours, 2 minutes, 10 seconds
                Interface 4 (192.168.5.101):Normal 
                Interface 3 (192.168.5.101):Normal (Waiting)
                Interface 2 (192.168.5.101):Normal (Waiting)
                Interface 1 (192.168.5.101):Normal 
                Interface 0 (192.168.5.101):Normal (Waiting)
        Other host:Primary - Active 
                Active time:0 weeks, 0 days, 0 hours, 1 minutes, 55 seconds
                Interface 4 (192.168.5.100):Normal 
                Interface 3 (192.168.5.100):Normal (Waiting)
                Interface 2 (192.168.5.100):Normal (Waiting)
                Interface 1 (192.168.5.100):Normal 
                Interface 0 (192.168.5.100):Normal (Waiting)
LD-430(config)#

Failover does not start monitoring the network interfaces until it hears six hello packets on that interface. This happens in 30 to 60 seconds, in most cases.

If the failover pair is connected to a switch running spanning tree, the start of failover monitoring takes twice the forward delay time configured in the switch (typically 15 seconds) plus an additional 30 seconds. This delay occurs because the switch detects a temporary bridge loop at bootup (and immediately following a failover event). When this bridge loop is detected, the switch stops forwarding packets (including failover hello packets) until the forward delay times out. The switch then enters "listen" mode for a second forward delay time to listen for more bridge loops.

After twice the forward delay time, traffic should resume. LocalDirector remains in "waiting" mode until it hears six hello packets (one every 5 seconds for a total of 30 seconds). During this time, the LocalDirector passes traffic while suspending failover monitoring for hello packets. All other failover monitoring continues (power, interface, and failover cable hello).

Note You must set a failover IP address for failover to work. If you do not, monitoring of the interfaces remains in "waiting" mode. The show failover command displays 0.0.0.0 for an IP address that is not set.

The following example shows the output if failover has not started monitoring the network interfaces.

ld-prim(config)# show failover

	Failover On
Cable status: Normal
		This host: Primary - Active
			Active time: 6930 (sec)
			Interface 0 (192.168.89.1): Normal (Waiting)
			Interface 1 (192.168.89.1): Normal (Waiting)
		Other host: Secondary - Standby
			Active time: 15 (sec)
			Interface 0 (192.168.89.2): Normal (Waiting)
			Interface 1 (192.168.89.2): Normal (Waiting)

Note "Waiting" in this example means that monitoring of the network interfaces of the other unit has not yet started (and the site is not protected by the failover feature).

The following example shows that a failure has been detected. Note that interface 1 on the primary unit is the source of the failure. The units are back in waiting mode because of the failure. The failed unit has removed itself from the network (interfaces are down), and it is no longer sending hello packets on the network. The active unit remains in the waiting state until the failed unit is replaced and failover communications start again.

ld-prim(config)# show failover

	Failover On
Cable status: Normal
		This host: Primary - Standby (Failed)
			Active time: 7140 (sec)
			Interface 0 (192.168.89.2): Normal (Waiting)
			Interface 1 (192.168.89.2): Failed (Waiting)
		Other host: Secondary - Active
			Active time: 30 (sec)
			Interface 0 (192.168.89.1): Normal (Waiting)
			Interface 1 (192.168.89.1): Normal (Waiting)

Frequently Asked Questions

This section contains some frequently asked questions about the failover feature.

Can the failover feature work when the failover cable is not used?

: No, failover does not work without the cable. If you operate units without the failover cable, you are essentially running two separate LocalDirector units, which results in a bridge loop and floods the network. The failover cable is an essential part of failover.

How is startup initialization accomplished between two units?

: When a unit boots up, it defaults to Failover Off and Secondary unless the failover cable is present or failover has been saved in the configuration. The configuration from the active unit is also copied to the standby unit. If the cable is not present, the unit automatically becomes the active unit. If the cable is present, the unit that has the primary end of the failover cable plugged into it becomes the primary unit by default, unless the secondary unit is already active.

Can modems be used to extend the length of the failover cable?

: No, the cable cannot be extended using modems or other EIA/TIA-232 line extenders. Part of what the failover cable does is indicate the presence and power status of the other unit. When you place line extenders in this path, you are relaying the status of the line extender rather than the status of the other LocalDirector unit.

What happens when failover is triggered?

: A switch can be initiated by either unit. When a switch takes place, each unit changes state. The newly active unit assumes the IP address and MAC address of the previously active unit and begins accepting traffic for it. The new standby unit assumes the IP address and MAC address of the unit that was previously the standby unit.

Is the connection state shared between LocalDirector units?

: Yes, when the replicate command is used to configure stateful failover. If stateful failover is not configured, active connections are dropped when a failover switch occurs, and clients must reestablish the connections through the newly active unit. It is best to maintain state on connections with a longer connection time. Although it is possible to maintain state on connections that are short-lived, such as HTTP, it is not recommended.

How can both units be configured the same without the configuration being manually entered twice?

: The configuration is automatically replicated and can be forced with the write standby command.

What happens if a primary unit has a power failure?

: When the primary active LocalDirector experiences a power failure, the standby LocalDirector comes up in active mode. If the primary unit is powered up again, it becomes the standby unit.

What happens if an interface card is disconnected?

: When the primary active LocalDirector is failed by disconnecting the network interface (the cable is disconnected), the standby LocalDirector comes up in active mode as it should. When the interface is plugged back in, the unit automatically recovers; however, it does not take over as the active unit. It becomes the standby unit.

Does failover work in a switched environment?

: Yes, failover works in switched environments if you are running LocalDirector Version 1.6.3 or greater on both units.

What constitutes a failure?

Fault detection is based on the following:

Failover hello packets are received on each interface. If hello packets are not heard for a number of consecutive 5-second intervals (determined by the failover hellotime command), the interface is tested to determine which unit is at fault.
Cable errors. The cable is wired so that each unit can distinguish between a power failure in the other unit and an unplugged cable. If the standby unit detects that the active unit is powered down (or resets), it takes active control. If the cable is unplugged, a syslog message is generated but no switching occurs. An exception is at bootup, at which point an unplugged cable forces the unit active. If both units are powered up without the failover cable installed, they both become active, creating a duplicate IP address conflict on your network. The failover cable must be installed for failover to work correctly.
Failover communication. The two units share information every 5 seconds. If the standby unit does not hear from the active unit within a number of communication attempts (determined by the failover hellotime command) and the cable status is OK, the standby unit takes over as active.

How long does it take to detect a fault?
- Power failure (and cable failure) is detected within 5 seconds.
- Failover communication faults can be detected within 5 seconds. The failover hellotime command lets you set the detection rate.
- Network communication faults can be detected within 5 seconds. The failover hellotime command lets you set the detection rate.
What maintenance is required?

: Syslog messages are generated when any errors, configuration, or event state changes occur. Evaluate the failed unit and repair or replace it.

What LocalDirector software versions are compatible between LocalDirector failover units?

: Always use the same version of software on both LocalDirector units.

Note Use identical LocalDirector units as failover pairs. Make sure that the hardware platform and the number and type of interfaces on each unit are the same.

What is the recommended procedure for upgrading software in failover configurations?

: See Appendix B, "Upgrading LocalDirector Software," for failover upgrade instructions.

Table of Contents

Configuring LocalDirector Failover