Redundant Network Components

To eliminate single points of failure for networking, each subnet accessed by a cluster node is required to have redundant network interfaces. Redundant cables are also needed to protect against cable failures. Each interface card is connected to a different cable, and the cables themselves are connected by a component such as a hub or a bridge. This arrangement of physical cables connected to each other via a bridge or concentrator or switch is known as a bridged net.

IP addresses can be associated with interfaces on a bridged net. An interface that has an IP address associated with it is known as a primary interface, and an interface that does not have an IP address associated with it is known as a standby interface. Standby interfaces are those which are available for switching by Serviceguard if a failure occurs on the primary. When Serviceguard detects a primary interface failure, it will switch the IP addresses and any associated connections from the failed interface card to a healthy standby interface card.

Serviceguard supports a maximum of 30 network interfaces per node. For this purpose an interface is defined as anything represented as a LAN interface in the output of lanscan (1m), so the total of 30 can comprise any combination of physical LAN ports, VLAN ports, IPoIB interfaces and APA aggregates. (A node can have more than 30 such interfaces, but only 30 can be part of the cluster configuration.)

A selection of network configurations is described further in the following sections. See also “How the Network Manager Works ”. For detailed information about supported network configurations, consult Hewlett-Packard support.




	NOTE: Serial (RS232) lines are no longer supported for the cluster heartbeat. Fibre Channel, Token Ring and FDDI networks are no longer supported as heartbeat or data LANs.

Rules and Restrictions

A single subnet cannot be configured on different network interfaces (NICs) on the same node.
For IPv4 subnets, Serviceguard does not support different subnets on the same LAN interface.
- For IPv6, Serviceguard supports up to two subnets per LAN interface (site-local and global).
Serviceguard does support different subnets on the same bridged network (this applies at both the node and the cluster level).

Serviceguard does not support using networking tools such as ifconfig or the configuration file /etc/rc.config.d/netconf to add IP addresses to network interfaces that are configured into the Serviceguard cluster, unless those IP addresses themselves will be immediately configured into the cluster as stationary IP addresses.




	CAUTION: If you configure any address other than a stationary IP address on a Serviceguard network interface, it could collide with a relocatable package IP address assigned by Serviceguard. See “Stationary and Relocatable IP Addresses ”.

(Oracle VIPs are an exception to this rule; such configurations require the HP add-on product Serviceguard Extension for Oracle RAC).

Redundant Ethernet Configuration

The use of redundant network components is shown in Figure 2-1 “Redundant LANs ”, which is an Ethernet configuration.

Figure 2-1 Redundant LANs

In the figure, a two-node Serviceguard cluster has one bridged net configured with both a primary and a standby LAN card for the data/heartbeat subnet (subnetA). Another LAN card provides an optional dedicated heartbeat LAN. Note that the primary and standby LAN segments are connected by a hub to provide a redundant data/heartbeat subnet. Each node has its own IP address for this subnet. In case of a failure of a primary LAN card for the data/heartbeat subnet, Serviceguard will perform a local switch to the standby LAN card on the same node.

Redundant heartbeat is provided by the primary LAN and the dedicated LAN which are both carrying the heartbeat. In Figure 2-1 “Redundant LANs ”, local switching is not needed for the dedicated heartbeat LAN, since there is already a redundant path via the other subnet. In case of data congestion on the primary LAN, the dedicated heartbeat LAN will prevent a false diagnosis of heartbeat failure. Each node has its own IP address for the dedicated heartbeat LAN.




	NOTE: You should verify that network traffic is not too heavy on the heartbeat/ data LAN. If traffic is too heavy, this LAN might not perform adequately in transmitting heartbeats if the dedicated heartbeat LAN fails.

Cross-Subnet Configurations

As of Serviceguard A.11.18 it is possible to configure multiple subnets, joined by a router, both for the cluster heartbeat and for data, with some nodes using one subnet and some another.

A cross-subnet configuration allows:

Automatic package failover from a node on one subnet to a node on another
A cluster heartbeat that spans subnets.

Configuration Tasks

Cluster and package configuration tasks are affected as follows:

You must use the -w full option to cmquerycl to discover actual or potential nodes and subnets across routers.
You must configure two new parameters in the package configuration file to allow packages to fail over across subnets:
- ip_subnet_node - to indicate which nodes the subnet is configured on
- monitored_subnet_access - to indicate whether the subnet is configured on all nodes (FULL) or only some (PARTIAL)
(For legacy packages, see “Configuring Cross-Subnet Failover”.)
You should not use the wildcard (*) for node_name in the package configuration file, as this could allow the package to fail over across subnets when a node on the same subnet is eligible. Instead, list the nodes in order of preference.

Restrictions

The following restrictions apply:

All nodes in the cluster must belong to the same network domain (that is, the domain portion of the fully-qualified domain name must be the same).
The nodes must be fully connected at the IP level.
A minimum of two heartbeat paths must be configured for each cluster node.
There must be less than 200 milliseconds of latency in the heartbeat network.
Each heartbeat subnet on each node must be physically routed separately to the heartbeat subnet on another node; that is, each heartbeat path must be physically separate:
- The heartbeats must be statically routed; static route entries must be configured on each node to route the heartbeats through different paths.
- Failure of a single router must not affect both heartbeats at the same time.
Because Veritas Cluster File System from Symantec (CFS) requires link-level traffic communication (LLT) among the nodes, Serviceguard cannot be configured in cross-subnet configurations with CFS alone.
But CFS is supported in specific cross-subnet configurations with Serviceguard and HP add-on products such as Serviceguard Extension for Oracle RAC (SGeRAC); see the documentation listed below.
Each package subnet must be configured with a standby interface on the local bridged net. The standby interface can be shared between subnets.
Deploying applications in this environment requires careful consideration; see “Implications for Application Deployment”.
cmrunnode will fail if the “hostname LAN” is down on the node in question. (“Hostname LAN” refers to the public LAN on which the IP address that the node’s hostname resolves to is configured).
If a monitored_subnet is configured for PARTIAL monitored_subnet_access in a package’s configuration file, it must be configured on at least one of the nodes on the node_name list for that package. Conversely, if all of the subnets that are being monitored for this package are configured for PARTIAL access, each node on the node_name list must have at least one of these subnets configured.
- As in other configurations, a package will not start on a node unless the subnets configured on that node, and specified in the package configuration file as monitored subnets, are up.

For More Information

For more information on the details of configuring the cluster and packages in a cross-subnet context, see “Obtaining Cross-Subnet Information”, “About Cross-Subnet Failover” and (for legacy packages only) “Configuring Cross-Subnet Failover”.




	IMPORTANT: Although cross-subnet topology can be implemented on a single site, it is most commonly used by extended-distance clusters, and specifically site-aware disaster-tolerant clusters, which require HP add-on software. Design and configuration of such clusters are covered in the disaster-tolerant documentation delivered with Serviceguard. For more information, see the following documents at `http://www.docs.hp.com -> High Availability`: Understanding and Designing Serviceguard Disaster Tolerant Architectures Designing Disaster Tolerant HA Clusters Using Metrocluster and Continentalclusters Using Serviceguard Extension for RAC The white paper Configuration and Administration of Oracle 10g R2 RAC Database in HP Metrocluster The white paper Technical Considerations for Creating a Serviceguard Cluster that Spans Multiple IP Subnets

Replacing Failed Network Cards

Depending on the system configuration, it is possible to replace failed network cards while the cluster is running. The process is described under “Replacement of LAN Cards” in the chapter “Troubleshooting Your Cluster.” With some restrictions, you can also add and delete LAN interfaces to and from the cluster configuration while the cluster is running; see “Changing the Cluster Networking Configuration while the Cluster Is Running”.