Acknowledgments ................................................................................................................................ i
Introduction........................................................................................................................................ iii
1. How To Use This Manual .................................................................................................... iii
2. Document Conventions ........................................................................................................ iv
3. More to Come ...................................................................................................................... vi
3.1. Send in Your Feedback ......................................................................................... vi
4. Activate Your Subscription ................................................................................................. vii
4.1. Provide a Red Hat Login...................................................................................... vii
4.2. Provide Your Subscription Number ..................................................................... vii
4.3. Connect Your System..........................................................................................viii
I. Using the Red Hat Cluster Manager ............................................................................................ ix
1. Red Hat Cluster Manager Overview..................................................................................... 1
1.1. Red Hat Cluster Manager Features ........................................................................ 2
2. Hardware Installation and Operating System Configuration ................................................ 5
2.1. Choosing a Hardware Configuration ..................................................................... 5
2.2. Setting Up the Members ...................................................................................... 15
2.3. Installing and Configuring Red Hat Enterprise Linux ......................................... 17
2.4. Setting Up and Connecting the Cluster Hardware ............................................... 21
The Red Hat Cluster Manager software was originally based on the open source Kimberlite
(http://oss.missioncriticallinux.com/kimberlite/) cluster project, which was developed by Mission
Critical Linux, Inc.
Subsequent to its inception based on Kimberlite, developers at Red Hat have made a large number
of enhancements and modifications. The following is a non-comprehensive list highlighting some of
these enhancements.
• Packaging and integration into the Red Hat installation paradigm to simplify the end user’s experi-
ence.
• Addition of support for multiple cluster members.
• Addition of support for high availability NFS services.
• Addition of support for high availability Samba services.
• Addition of the Cluster Configuration Tool, a graphical configuration tool.
• Addition of the Cluster Status Tool, a graphical monitoring and administration tool.
• Addition of support for failover domains.
• Addition of support for Red Hat GFS, including the GFS Setup Druid and the LOCK_GULM
fencing driver.
• Addition of support for using watchdog timers as a data integrity provision.
• Addition of service monitoring which automatically restart a failed application.
• Rewrite of the service manager to facilitate additional cluster-wide operations.
• A set of miscellaneous bug fixes.
The Red Hat Cluster Manager software incorporates STONITH compliant power switch modules
from the Linux-HA project; refer to http://www.linux-ha.org/stonith/.
iiAcknowledgments
Introduction
The Red Hat Cluster Suite is a collection of technologies working together to provide data integrity
and the ability to maintain application availability in the event of a failure. Administrators can deploy
enterprise cluster solutions using a combination of hardware redundancy along with the failover and
load-balancing technologies in Red Hat Cluster Suite.
Red Hat Cluster Manager is a high-availability cluster solution specifically suited for database applications, network file servers, and World Wide Web (Web) servers with dynamic content. An Red Hat
Cluster Manager system features data integrity and application availability using redundant hardware,
shared disk storage, power management, and robust cluster communication and application failover
mechanisms.
Administrators can also deploy highly available applications services using Piranha, a load-balancing
and advanced routing cluster solution based on Linux Virtual Server (LVS) technology. Using
Piranha, administrators can build highly available e-commerce sites that feature complete
data integrity and service availability, in addition to load balancing capabilities. Refer to
Part II Configuring a Linux Virtual Server Cluster for more information.
This guide assumes that the user has an advanced working knowledge of Red Hat Enterprise Linux and
understands the concepts of server computing. For more information about using Red Hat Enterprise
Linux, refer to the following resources:
• Red Hat Enterprise Linux Installation Guide for information regarding installation.
• Red Hat Enterprise Linux Introduction to System Administration for introductory information for
new Red Hat Enterprise Linux system administrators.
• Red Hat Enterprise Linux System Administration Guide for more detailed information about con-
figuring Red Hat Enterprise Linux to suit your particular needs as a user.
• Red Hat Enterprise Linux Reference Guide provides detailed information suited for more experi-
enced users to refer to when needed, as opposed to step-by-step instructions.
• Red Hat Enterprise Linux Security Guide details the planning and the tools involved in creating a
secured computing environment for the data center, workplace, and home.
HTML, PDF, and RPM versions of the manuals are available on the Red Hat Enterprise Linux Documentation CD and online at:
http://www.redhat.com/docs/
1. How To Use This Manual
This manual contains information about setting up a Red Hat Cluster Manager system. These
tasks are described in Chapter 2 Hardware Installation and Operating System Configuration and
Chapter 3 Cluster Configuration.
For information about configuring Red Hat Cluster Manager cluster services, refer to
Chapter 4 Service Administration through Chapter 7 Setting Up Apache HTTP Server.
Part II Configuring a Linux Virtual Server Cluster describes how to achieve load balancing in an Red
Hat Enterprise Linux cluster by using the Linux Virtual Server.
For information about deploying a Red Hat Cluster Manager cluster in conjunction with Red Hat GFS
using the GFS Setup Druid, refer to Appendix C The GFS Setup Druid.
Appendix D Supplementary Hardware Informationcontainsdetailedconfiguration
informationonspecifichardwaredevicesandsharedstorageconfigurations.
Appendix E Supplementary Software Information contains background information on the cluster
software and other related information.
ivIntroduction
Appendix F Cluster Command-line Utilities provides usage and reference information on the
command-line utilities included with Red Hat Cluster Suite.
This guide assumes you have a thorough understanding of Red Hat Enterprise Linux system administration concepts and tasks. For detailed information on Red Hat Enterprise Linux system administration, refer to the Red Hat Enterprise Linux System Administration Guide. For reference information
on Red Hat Enterprise Linux, refer to the Red Hat Enterprise Linux Reference Guide.
2. Document Conventions
When you read this manual, certain words are represented in different fonts, typefaces, sizes, and
weights. This highlighting is systematic; different words are represented in the same style to indicate
their inclusion in a specific category. The types of words that are represented this way include the
following:
command
Linux commands (and other operating system commands, when used) are represented this way.
This style should indicate to you that you can type the word or phrase on the command line
and press [Enter] to invoke a command. Sometimes a command contains words that would be
displayed in a different style on their own (such as file names). In these cases, they are considered
to be part of the command, so the entire phrase is displayed as a command. For example:
Use the cat testfile command to view the contents of a file, named testfile, in the current
working directory.
file name
File names, directory names, paths, and RPM package names are represented this way. This style
should indicate that a particular file or directory exists by that name on your system. Examples:
The .bashrc file in your home directory contains bash shell definitions and aliases for your own
use.
The /etc/fstab file contains information about different system devices and file systems.
Install the webalizer RPM if you want to use a Web server log file analysis program.
application
This style indicates that the program is an end-user application (as opposed to system software).
For example:
Use Mozilla to browse the Web.
[key]
A key on the keyboard is shown in this style. For example:
To use [Tab] completion, type in a character and then press the [Tab] key. Your terminal displays
the list of files in the directory that start with that letter.
[key]-[combination]
A combination of keystrokes is represented in this way. For example:
The [Ctrl]-[Alt]-[Backspace] key combination exits your graphical session and return you to the
graphical login screen or the console.
Introductionv
text found on a GUI interface
A title, word, or phrase found on a GUI interface screen or window is shown in this style. Text
shown in this style is being used to identify a particular GUI screen or an element on a GUI
screen (such as text associated with a checkbox or field). Example:
Select the Require Password checkbox if you would like your screensaver to require a password
before stopping.
top level of a menu on a GUI screen or window
A word in this style indicates that the word is the top level of a pulldown menu. If you click on
the word on the GUI screen, the rest of the menu should appear. For example:
Under File on a GNOME terminal, the New Tab option allows you to open multiple shell
prompts in the same window.
If you need to type in a sequence of commands from a GUI menu, they are shown like the
following example:
Go to Main Menu Button (on the Panel) => Programming => Emacs to start the Emacs text
editor.
button on a GUI screen or window
This style indicates that the text can be found on a clickable button on a GUI screen. For example:
Click on the Back button to return to the webpage you last viewed.
computer output
Text in this style indicates text displayed to a shell prompt such as error messages and responses
to commands. For example:
The ls command displays the contents of a directory. For example:
The output returned in response to the command (in this case, the contents of the directory) is
shown in this style.
prompt
A prompt, which is a computer’s way of signifying that it is ready for you to input something, is
shown in this style. Examples:
$
#
[stephen@maturin stephen]$
leopard login:
user input
Text that the user has to type, either on the command line, or into a text box on a GUI screen, is
displayed in this style. In the following example, text is displayed in this style:
To boot your system into the text based installation program, you must type in the text command at the boot: prompt.
replaceable
Text used for examples, which is meant to be replaced with data provided by the user, is displayed
in this style. In the following example, <version-number> is displayed in this style:
viIntroduction
The directory for the kernel source is /usr/src/<version-number>/, where
<version-number> is the version of the kernel installed on this system.
Additionally, we use several different strategies to draw your attention to certain pieces of information.
In order of how critical the information is to your system, these items are marked as a note, tip,
important, caution, or warning. For example:
Note
Remember that Linux is case sensitive. In other words, a rose is not a ROSE is not a rOsE.
Tip
The directory /usr/share/doc/ contains additional documentation for packages installed on your
system.
Important
If you modify the DHCP configuration file, the changes do not take effect until you restart the DHCP
daemon.
Caution
Do not perform routine tasks as root — use a regular user account unless you need to use the root
account for system administration tasks.
Warning
Be careful to remove only the necessary Red Hat Enterprise Linux partitions. Removing other par titions could result in data loss or a corrupted system environment.
3. More to Come
This manual is part of Red Hat’s growing commitment to provide useful and timely support to Red
Hat Enterprise Linux users.
Introductionvii
3.1. Send in Your Feedback
If you spot a typo, or if you have thought of a way to make this manual better, we would love to
hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against the
component rh-cs.
Be sure to mention the manual’s identifier:
rh-cs(EN)-3-Print-RHI (2006-02-03T16:44)
By mentioning this manual’s identifier, we know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If you
have found an error, please include the section number and some of the surrounding text so we can
find it easily.
4. Activate Your Subscription
Before you can access service and software maintenance information, and the support documentation included in your subscription, you must activate your subscription by registering with Red Hat.
Registration includes these simple steps:
• Provide a Red Hat login
• Provide a subscription number
• Connect your system
The first time you boot your installation of Red Hat Enterprise Linux, you are prompted to register
with Red Hat using the Setup Agent. If you follow the prompts during the Setup Agent, you can
complete the registration steps and activate your subscription.
If you can not complete registration during the Setup Agent (which requires network access), you
can alternatively complete the Red Hat registration process online at http://www.redhat.com/register/.
4.1. Provide a Red Hat Login
If you do not have an existing Red Hat login, you can create one when prompted during the Setup
Agent or online at:
• Software updates, errata and maintenance via Red Hat Network
• Red Hat technical support resources, documentation, and Knowledgebase
If you have forgotten your Red Hat login, you can search for your Red Hat login online at:
https://rhn.redhat.com/help/forgot_password.pxt
viiiIntroduction
4.2. Provide Your Subscription Number
Your subscription number is located in the package that came with your order. If your package did not
include a subscription number, your subscription was activated for you and you can skip this step.
You can provide your subscription number when prompted during the Setup Agent or by visiting
http://www.redhat.com/register/.
4.3. Connect Your System
The Red Hat Network Registration Client helps you connect your system so that you can begin to get
updates and perform systems management. There are three ways to connect:
1. During the Setup Agent — Check the Send hardware information and Send system packagelist options when prompted.
2. After the Setup Agent has been completed — From the Main Menu, go to System Tools, then
select Red Hat Network.
3. After the Setup Agent has been completed — Enter the following command from the command
line as the root user:
• /usr/bin/up2date --register
I. Using the Red Hat Cluster Manager
Clustered systems provide reliability, scalability, and availability to critical production services. Using
the Red Hat Cluster Manager, administrators can create high availability clusters for filesharing, Web
servers, databases, and more. This part discusses the installation and configuration of cluster systems
using the recommended hardware and Red Hat Enterprise Linux.
This section is licensed under the GNU Free Documentation License. For details refer to the Copyright
page.
Table of Contents
1. Red Hat Cluster Manager Overview............................................................................................. 1
2. Hardware Installation and Operating System Configuration .................................................... 5
Red Hat Cluster Manager allows administrators to connect separate systems (called members or
nodes) together to create failover clusters that ensure application availability and data integrity under
several failure conditions. Administrators can use Red Hat Cluster Manager with database applications, file sharing services, web servers, and more.
To set up a failover cluster, you must connect the member systems (often referred to simply as mem-bers or nodes) to the cluster hardware, and configure the members into the cluster environment. The
foundation of a cluster is an advanced host membership algorithm. This algorithm ensures that the
cluster maintains complete data integrity at all times by using the following methods of inter-member
communication:
• Network connections between the cluster systems for heartbeat
• Shared state on shared disk storage to hold cluster status
To make an application and data highly available in a cluster, you must configure a service (such
as an application and shared disk storage) as a discrete, named group of properties and resources to
which you can assign an IP address to provide transparent client access. For example, you can set up
a service that provides clients with access to highly-available database application data.
You can associate a service with a failover domain, a subset of cluster members that are eligible to
run the service. In general, any eligible member can run the service and access the service data on
shared disk storage. However, each service can run on only one cluster member at a time, in order to
maintain data integrity. You can specify whether or not the members in a failover domain are ordered
by preference. You can also specify whether or not a service is restricted to run only on members of
its associated failover domain. (When associated with an unrestricted failover domain, a service can
be started on any cluster member in the event no member of the failover domain is available.)
You can set up an active-active configuration in which the members run different services simultaneously, or a hot-standby configuration in which a primary member runs all the services, and a backup
cluster system takes over only if the primary system fails.
Figure 1-1 shows an example of a cluster in an active-active configuration.
Figure 1-1. Example Cluster in Active-Active Configuration
2Chapter 1. Red Hat Cluster Manager Overview
If a hardware or software failure occurs, the cluster automatically restarts the failed member’s services
on the functional member. This service failover capability ensures that no data is lost, and there is little
disruption to users. When the failed member recovers, the cluster can re-balance the services across
the members.
In addition, you can cleanly stop the services running on a cluster system and then restart them on
another system. This service relocation capability allows you to maintain application and data availability when a cluster member requires maintenance.
1.1. Red Hat Cluster Manager Features
Cluster systems deployed with Red Hat Cluster Manager include the following features:
No-single-point-of-failure hardware configuration
Clusters can include a dual-controller RAID array, multiple network channels, and redundant
uninterruptible power supply (UPS) systems to ensure that no single failure results in application
down time or loss of data.
Alternately, a low-cost cluster can be set up to provide less availability than a no-single-pointof-failure cluster. For example, you can set up a cluster with a single-controller RAID array and
only a single Ethernet channel.
Certain low-cost alternatives, such as software RAID and multi-initiator parallel SCSI, are not
compatible or appropriate for use on the shared cluster storage. Refer to Section 2.1 Choosing a
Hardware Configuration, for more information.
Service configuration framework
Clusters allow you to easily configure individual services to make data and applications highly
available. To create a service, you specify the resources used in the service and properties for
the service, including the service name, application start, stop, and status script, disk partitions,
mount points, and the cluster members on which you prefer the service to run. After you add a
service, the cluster management software stores the information in a cluster configuration file on
shared storage, where the configuration data can be accessed by all cluster members.
The cluster provides an easy-to-use framework for database applications. For example, a database
service serves highly-available data to a database application. The application running on a cluster member provides network access to database client systems, such as Web servers. If the
service fails over to another member, the application can still access the shared database data. A
network-accessible database service is usually assigned an IP address, which is failed over along
with the service to maintain transparent access for clients.
The cluster service framework can be easily extended to other applications, as well.
Failover domains
By assigning a service to a restricted failover domain, you can limit the members that are eligible
to run a service in the event of a failover. (A service that is assigned to a restricted failover
domain cannot be started on a cluster member that is not included in that failover domain.) You
can order the members in a failover domain by preference to ensure that a particular member
runs the service (as long as that member is active). If a service is assigned to an unrestricted
failover domain, the service starts on any available cluster member (if none of the members of
the failover domain are available).
Data integrity assurance
To ensure data integrity, only one member can run a service and access service data at one
time. The use of power switches in the cluster hardware configuration enables a member to
power-cycle another member before restarting that member’s services during the failover process.
Chapter 1. Red Hat Cluster Manager Overview3
This prevents any two systems from simultaneously accessing the same data and corrupting it.
Although not required, it is recommended that power switches are used to guarantee data integrity
under all failure conditions. Watchdog timers are an optional variety of power control to ensure
correct operation of service failover.
Cluster administration user interface
The cluster administration interface facilitiates management tasks such as: creating, starting,
and stopping services; relocating services from one member to another; modifying the cluster
configuration (to add or remove services or resources); and monitoring the cluster members and
services.
Ethernet channel bonding
To monitor the health of the other members, each member monitors the health of the remote
power switch, if any, and issues heartbeat pings over network channels. With Ethernet channel
bonding, multiple Ethernet interfaces are configured to behave as one, reducing the risk of a
single-point-of-failure in the typical switched Ethernet connection between systems.
Shared storage for quorum information
Shared state information includes whether the member is active. Service state information includes whether the service is running and which member is running the service. Each member
checks to ensure that the status of the other members is up to date.
In a two-member cluster, each member periodically writes a timestamp and cluster state information to two shared cluster partitions located on shared disk storage. To ensure correct cluster
operation, if a member is unable to write to both the primary and shadow shared cluster partitions
at startup time, it is not allowed to join the cluster. In addition, if a member is not updating its
timestamp, and if heartbeats to the system fail, the member is removed from the cluster.
Figure 1-2 shows how members communicate in a cluster configuration. Note that the terminal
server used to access system consoles via serial ports is not a required cluster component.
Figure 1-2. Cluster Communication Mechanisms
4Chapter 1. Red Hat Cluster Manager Overview
Service failover capability
If a hardware or software failure occurs, the cluster takes the appropriate action to maintain
application availability and data integrity. For example, if a member completely fails, another
member (in the associated failover domain, if used, or in the cluster) restarts its services. Services
already running on this member are not disrupted.
When the failed member reboots and is able to write to the shared cluster partitions, it can rejoin
the cluster and run services. Depending on how the services are configured, the cluster can rebalance the services among the members.
Manual service relocation capability
In addition to automatic service failover, a cluster allows you to cleanly stop services on one
member and restart them on another member. You can perform planned maintenance on a member system while continuing to provide application and data availability.
Event logging facility
To ensure that problems are detected and resolved before they affect service availability, the cluster daemons log messages by using the conventional Linux syslog subsystem. You can customize
the severity level of the logged messages.
Application monitoring
The infrastructure in a cluster can optionally monitor the state and health of an application. In
this manner, should an application-specific failure occur, the cluster automatically restarts the
application. In response to the application failure, the application attempts to be restarted on the
member it was initially running on; failing that, it restarts on another cluster member. You can
specify which members are eligible to run a service by assigning a failover domain to the service.
Chapter 2.
Hardware Installation and Operating System
Configuration
To set up the hardware configuration and install Red Hat Enterprise Linux, follow these steps:
• Choose a cluster hardware configuration that meets the needs of applications and users; refer to
Section 2.1 Choosing a Hardware Configuration.
• Set up and connect the members and the optional console switch and network switch or hub; refer
to Section 2.2 Setting Up the Members.
• Install and configure Red Hat Enterprise Linux on the cluster members; refer to
Section 2.3 Installing and Configuring Red Hat Enterprise Linux .
• Set up the remaining cluster hardware components and connect them to the members; refer to
Section 2.4 Setting Up and Connecting the Cluster Hardware.
After setting up the hardware configuration and installing Red Hat Enterprise Linux, install the cluster
software.
Tip
Refer to the Red Hat Hardware Compatibility List available at http://hardware.redhat.com/hcl/ for a
list of compatible hardware. Perform a Quick Search for the term cluster to find results for power
switch and shared storage hardware certified for or compatible with Red Hat Cluster Manager. For
general system hardware compatibility searches, use manufacturer, brand, and/or model keywords
to check for compatibility with Red Hat Enterprise Linux.
2.1. Choosing a Hardware Configuration
The Red Hat Cluster Manager allows administrators to use commodity hardware to set up a cluster
configuration that meets the performance, availability, and data integrity needs of applications and
users. Cluster hardware ranges from low-cost minimum configurations that include only the components required for cluster operation, to high-end configurations that include redundant Ethernet
channels, hardware RAID, and power switches.
Regardless of configuration, the use of high-quality hardware in a cluster is recommended, as hardware malfunction is a primary cause of system down time.
Although all cluster configurations provide availability, some configurations protect against every
single point of failure. In addition, all cluster configurations provide data integrity, but some configurations protect data under every failure condition. Therefore, administrators must fully understand the
needs of their computing environment and also the availability and data integrity features of different
hardware configurations to choose the cluster hardware that meets the proper requirements.
When choosing a cluster hardware configuration, consider the following:
Performance requirements of applications and users
Choose a hardware configuration that provides adequate memory, CPU, and I/O resources. Be
sure that the configuration chosen can handle any future increases in workload as well.
6Chapter 2. Hardware Installation and Operating System Configuration
Cost restrictions
The hardware configuration chosen must meet budget requirements. For example, systems with
multiple I/O ports usually cost more than low-end systems with fewer expansion capabilities.
Availability requirements
If a computing environment requires the highest degree of availability, such as a production
environment, then a cluster hardware configuration that protects against all single points of failure, including disk, storage interconnect, Ethernet channel, and power failures is recommended.
Environments that can tolerate an interruption in availability, such as development environments, may not require as much protection. Refer to Section 2.4.3 Configuring UPS Systems and
Section 2.4.4 Configuring Shared Disk Storagefor more information about using redundant hardware for high availability.
Data integrity under all failure conditions requirement
Using power switches in a cluster configuration guarantees that service data is protected under
every failure condition. These devices enable a member to power cycle another member before
restarting its services during failover. Power switches protect against data corruption if an unresponsive (or hanging) member becomes responsive after its services have failed over and then
issues I/O to a disk that is also receiving I/O from the other member.
In addition, if a quorum daemon fails on a member, the member is no longer able to monitor the
shared cluster partitions. If you are not using power switches in the cluster, this error condition
may result in services being run on more than one member, which can cause data corruption.
Refer to Section 2.4.2 Configuring Power Switches for more information about the benefits of
using power switches in a cluster. It is recommended that production environments use power
switches or watchdog timers in the cluster configuration.
2.1.1. Shared Storage Requirements
The operation of the cluster depends on reliable, coordinated access to shared storage. In the event
of hardware failure, it is desirable to be able to disconnect one member from the shared storage for
repair without disrupting the other members. Shared storage is truly vital to the cluster configuration.
Testing has shown that it is difficult, if not impossible, to configure reliable multi-initiator parallel
SCSI configurations at data rates above 80MB/sec using standard SCSI adapters. Further tests have
shown that these configurations cannot support online repair because the bus does not work reliably
when the HBA terminators are disabled, and external terminators are used. For these reasons, multiinitiator SCSI configurations using standard adapters are not supported. Either single-initiator SCSI
bus adapters (connected to multi-ported storage) or Fibre Channel adapters are required.
The Red Hat Cluster Manager requires that all cluster members have simultaneous access to the shared
storage. Certain host RAID adapters are capable of providing this type of access to shared RAID units.
These products require extensive testing to ensure reliable operation, especially if the shared RAID
units are based on parallel SCSI buses. These products typically do not allow for online repair of
a failed member. Only host RAID adapters listed in the Red Hat Hardware Compatibility List are
supported.
The use of software RAID, or software Logical Volume Management (LVM), is not supported on
shared storage. This is because these products do not coordinate access from multiple hosts to shared
storage. Software RAID or LVM may be used on non-shared storage on cluster members (for example,
boot and system partitions, and other file systems which are not associated with any cluster services).
2.1.2. Minimum Hardware Requirements
A minimum hardware configuration includes only the hardware components that are required for
cluster operation, as follows:
Chapter 2. Hardware Installation and Operating System Configuration7
• Two servers to run cluster services
• Ethernet connection for sending heartbeat pings and for client network access
• Shared disk storage for the shared cluster partitions and service data
The hardware components described in Table 2-1 can be used to set up a minimum cluster configuration. This configuration does not guarantee data integrity under all failure conditions, because it does
not include power switches. Note that this is a sample configuration; it is possible to set up a minimum
configuration using other hardware.
Warning
The minimum cluster configuration is not a supported solution and should not be used in a production
environment, as it does not guarantee data integrity under all failure conditions.
HardwareDescription
Two serversEach member includes a network interface for client access and
for Ethernet connections and a SCSI adapter (termination
disabled) for the shared storage connection
Two network cables with RJ45
connectors
Network cables connect an Ethernet network interface on each
member to the network for client access and heartbeat pings.
RAID storage enclosureThe RAID storage enclosure contains one controller with at least
two host ports.
Two HD68 SCSI cablesEach cable connects one HBA to one port on the RAID
controller, creating two single-initiator SCSI buses.
Table 2-1. Example of Minimum Cluster Configuration
The minimum hardware configuration is the most cost-effective cluster configuration; however,
it includes multiple points of failure. For example, if the RAID controller fails, then all
cluster services become unavailable. When deploying the minimal hardware configuration,
software watchdog timers should be configured as a data integrity provision. Refer to
Section D.1.2.3 Configuring a Hardware Watchdog Timer for details.
To improve availability, protect against component failure, and guarantee data integrity under all failure conditions, the minimum configuration can be expanded, as described in Table 2-2.
ProblemSolution
Disk failureHardware RAID to replicate data across multiple disks
RAID controller failureDual RAID controllers to provide redundant access to
disk data
Heartbeat failureEthernet channel bonding and failover
Power source failureRedundant uninterruptible power supply (UPS) systems
Data corruption under all failure
Power switches or hardware-based watchdog timers
conditions
Table 2-2. Improving Availability and Guaranteeing Data Integrity
8Chapter 2. Hardware Installation and Operating System Configuration
A no single point of failure hardware configuration that guarantees data integrity under all failure
conditions can include the following components:
• At least two servers to run cluster services
• Ethernet connection between each member for heartbeat pings and for client network access
• Dual-controller RAID array to replicate shared partitions and service data
• Power switches to enable each member to power-cycle the other members during the failover pro-
cess
• Ethernet interfaces configured to use channel bonding
• At least two UPS systems for a highly-available source of power
The components described in Table 2-3 can be used to set up a no single point of failure cluster configuration that includes two single-initiator SCSI buses and power switches to guarantee data integrity
under all failure conditions. Note that this is a sample configuration; it is possible to set up a no single
point of failure configuration using other hardware.
HardwareDescription
Two servers (up to 8 supported) Each member includes the following hardware:
Two network interfaces for:
Point-to-point Ethernet connections
Client network access and Ethernet heartbeat pings
Three serial ports for:
Remote power switch connection
Connection to the terminal server
One Adaptec 29160 adapter (termination enabled) for the shared
disk storage connection.
One network switchA network switch enables the connection of multiple members to
a network.
One Cyclades terminal serverA terminal server allows for management of remote members
from a central location. (A terminal server is not required for
cluster operation.)
Four network cablesNetwork cables connect the terminal server and a network
interface on each member to the network switch.
Two RJ45 to DB9 crossover
cables
Two serial-attached power
switches
RJ45 to DB9 crossover cables connect a serial port on each
member to the Cyclades terminal server.
Power switches enable each member to power-cycle the other
member before restarting its services. The power cable for each
member is connected to its own power switch. Note that
serial-attach power switches are supported in two-member
clusters only.
Two null modem cablesNull modem cables connect a serial port on each member to the
power switch that provides power to the other member. This
connection enables each member to power-cycle the other
member.
FlashDisk RAID Disk Array
with dual controllers
Dual RAID controllers protect against disk and controller failure.
The RAID controllers provide simultaneous access to all the
logical units on the host ports.
Chapter 2. Hardware Installation and Operating System Configuration9
HardwareDescription
Two HD68 SCSI cablesHD68 cables connect each host bus adapter to a RAID enclosure
"in" port, creating two single-initiator SCSI buses.
Two terminatorsTerminators connected to each "out" port on the RAID enclosure
terminate both single-initiator SCSI buses.
Redundant UPS SystemsUPS systems provide a highly-available source of power. The
power cables for the power switches and the RAID enclosure are
connected to two UPS systems.
Table 2-3. Example of a No Single Point of Failure Configuration
Figure 2-1 shows an example of a no single point of failure hardware configuration that includes the
previously-described hardware, two single-initiator SCSI buses, and power switches to guarantee data
integrity under all error conditions. A "T" enclosed in a circle represents a SCSI terminator.
Figure 2-1. No Single Point of Failure Configuration Example
Cluster hardware configurations can also include other optional hardware components that are common in a computing environment. For example, a cluster can include a network switch or networkhub, which enables the connection of the members to a network. A cluster may also include a console
10Chapter 2. Hardware Installation and Operating System Configuration
switch, which facilitates the management of multiple members and eliminates the need for separate
monitors, mouses, and keyboards for each member.
One type of console switch is a terminal server, which enables connection to serial consoles and
management of many members from one remote location. As a low-cost alternative, you can use a
KVM (keyboard, video, and mouse) switch, which enables multiple members to share one keyboard,
monitor, and mouse. A KVM is suitable for configurations in which access to a graphical user interface
(GUI) to perform system management tasks is preferred.
When choosing a system, be sure that it provides the required PCI slots, network slots, and serial
ports. For example, a no single point of failure configuration requires multiple bonded Ethernet ports.
Refer to Section 2.2.1 Installing the Basic Cluster Hardware for more information.
2.1.3. Choosing the Type of Power Controller
The Red Hat Cluster Manager implementation consists of a generic power management layer and a set
of device-specific modules which accommodate a range of power management types. When selecting
the appropriate type of power controller to deploy in the cluster, it is important to recognize the
implications of specific device types. The following describes the types of supported power switches
followed by a summary table. For a more detailed description of the role a power switch plays to
ensure data integrity, refer to Section 2.4.2 Configuring Power Switches.
Serial-attached and network-attached power switches are separate devices which enable one cluster
member to power cycle another member. They resemble a power plug strip on which individual outlets
can be turned on and off under software control through either a serial or network cable. Networkattached power switches differ from serial-attached in that they connect to cluster members via an
Ethernet hub or switch, rather than direct connection to cluster members. A network-attached power
switch can not be directly attached to a cluster member using a crossover cable, as the power switch
would be unable to power cycle the other members.
Watchdog timers provide a means for failed members to remove themselves from the cluster prior
to another member taking over its services, rather than allowing one cluster member to power cycle
another. The normal operational mode for watchdog timers is that the cluster software must periodically reset a timer prior to its expiration. If the cluster software fails to reset the timer, the watchdog
triggers under the assumption that the member may have hung or otherwise failed. The healthy cluster
member allows a window of time to pass prior to concluding that another cluster member has failed
(by default, this window is 12 seconds). The watchdog timer interval must be less than the duration of
time for one cluster member to conclude that another has failed. In this manner, a healthy member can
assume, prior to taking over services, that the failed cluster member has safely removed itself from
the cluster (by rebooting) and is no longer a risk to data integrity. The underlying watchdog support
is included in the core Linux kernel. Red Hat Cluster Manager utilizes these watchdog features via its
standard APIs and configuration mechanism.
There are two types of watchdog timers: hardware-based and software-based. Hardware-based watchdog timers typically consist of system board components such as the Intel® i810 TCO chipset. This
circuitry has a high degree of independence from the main system CPU. This independence is beneficial in failure scenarios of a true system hang, as in this case it pulls down the system’s reset lead
resulting in a system reboot. Some PCI expansion cards provide watchdog features.
Software-based watchdog timers do not have any dedicated hardware. The implementation is a kernel
thread which is periodically run; if the timer duration has expired, the thread initiates a system reboot.
The vulnerability of the software watchdog timer is that under certain failure scenarios, such as system
hangs while interrupts are blocked, the kernel thread is not called. As a result, in such conditions it
cannot be definitively depended on for data integrity. This can cause the healthy cluster member to
take over services for a hung member which could cause data corruption under certain scenarios.
Finally, administrators can choose not to employ a power controller at all. When a power controller
is not in use, no provision exists for a cluster member to power cycle a failed member. Similarly, the
failed member cannot be guaranteed to reboot itself under all failure conditions.
Chapter 2. Hardware Installation and Operating System Configuration11
Important
Use of a power controller is strongly recommended as part of a production cluster environment.
Configuration of a cluster without a power controller is not supported.
Ultimately, the right type of power controller deployed in a cluster environment depends on the data
integrity requirements weighed against the cost and availability of external power switches.
Table 2-4 summarizes the types of supported power management modules and discusses their advantages and disadvantages individually.
TypeNotesProsCons
Serial-attached
power switches
(supported for
two-member
clusters only)
Network-attached
power switches
Hardware
Watchdog Timer
Software
Watchdog Timer
Two serial attached
power controllers are
used in a cluster (one per
member)
A single network
attached power
controller is required per
cluster (depending on
the number of
members); however, up
to three are supported
for each cluster member
Affords strong data
integrity guarantees
Offers acceptable data
integrity provisions
Affords strong data
integrity guarantees —
the power controller
itself is not a single
point of failure as there
are two in a cluster
Affords strong data
integrity guarantees and
can be used in clusters
with more than two
members
Obviates the need to
purchase external power
controller hardware
Obviates the need to
purchase external power
controller hardware;
works on any system
Requires purchase of
power controller
hardware and cables;
consumes serial ports;
can only be used in
two-member cluster
Requires purchase of
power controller
hardware — the power
controller itself can
become a single point of
failure (although they
are typically very
reliable devices)
Not all systems include
supported watchdog
hardware
Under some failure
scenarios, the software
watchdog is not
operational, opening a
small vulnerability
window
No power
controller
No power controller
function is in use
Obviates the need to
purchase external power
controller hardware;
Vulnerable to data
corruption under certain
failure scenarios
works on any system
Table 2-4. Power Switches
2.1.4. Cluster Hardware Components
Use the following tables to identify the hardware components required for the cluster configuration.
Table 2-5 includes the hardware required for the cluster members.
12Chapter 2. Hardware Installation and Operating System Configuration
HardwareQuantityDescriptionRequired
Cluster
members
eight
(maximum
supported)
Each member must provide enough PCI slots,
network slots, and serial ports for the cluster
hardware configuration. Because disk devices
Yes
must have the same name on each member, it is
recommended that the members have symmetric
I/O subsystems. It is also recommended that the
processor speed and amount of system memory
be adequate for the processes run on the cluster
members. Consult the Red Hat Enterprise Linux3 Release Notes for specifics. Refer to
Section 2.2.1 Installing the Basic Cluster Hardware
for more information.
Table 2-5. Cluster Member Hardware
Table 2-6 includes several different types of power switches.
A single cluster requires only one type of power switch.
HardwareQuantityDescriptionRequired
Serial power
switches
TwoIn a two-member cluster, use serial power
switches to enable each cluster member to
power-cycle the other member. Refer to
Section 2.4.2 Configuring Power Switches for
more information. Note, cluster members are
configured with either serial power switches
Strongly
recommended
for data
integrity under
all failure
conditions
(supported for two-member clusters only) or
network-attached power switches, but not both.
Null modem
cable
TwoNull modem cables connect a serial port on a
cluster member to a serial power switch. This
enables each member to power-cycle the other
Only if using
serial power
switches
member. Some power switches may require
different cables.
Mounting
bracket
OneSome power switches support rack mount
configurations and require a separate mounting
bracket.
Only for rack
mounting
power
switches
Network
power switch
One (depends
on member
count)
Network-attached power switches enable each
cluster member to power cycle all others. Refer
to Section 2.4.2 Configuring Power Switches for
more information.
Strongly
recommended
for data
integrity under
all failure
conditions
Watchdog
Timer
One per
member
Watchdog timers cause a failed cluster member
to remove itself from a cluster prior to a healthy
member taking over its services. Refer to
Section 2.4.2 Configuring Power Switches for
more information.
Recommended
for data
integrity on
systems which
provide
integrated
watchdog
hardware
Chapter 2. Hardware Installation and Operating System Configuration13
Table 2-6. Power Switch Hardware Table
Table 2-8 through Table 2-10 show a variety of hardware components for an administrator to choose
from. An individual cluster does not require all of the components listed in these tables.
HardwareQuantityDescriptionRequired
Network
interface
One for each
network
Each network connection requires a network
interface installed in a member.
Yes
connection
Network
switch or hub
Network cable One for each
OneA network switch or hub allows connection of
multiple members to a network.
A conventional network cable, such as a cable
network
interface
with an RJ45 connector, connects each network
interface to a network switch or a network hub.
No
Yes
Table 2-7. Network Hardware Table
HardwareQuantityDescriptionRequired
Host bus
adapter
One per
member
To connect to shared disk storage, install either
a parallel SCSI or a Fibre Channel host bus
Yes
adapter in a PCI slot in each cluster member.
For parallel SCSI, use a low voltage
differential (LVD) host bus adapter. Adapters
have either HD68 or VHDCI connectors.
Host-bus adapter based RAID cards are only
supported if they correctly support multi-host
operation. At the time of publication, there were
no fully tested host-bus adapter based RAID
cards.
External disk
storage
enclosure
At least oneUse Fibre Channel or single-initiator parallel
SCSI to connect the cluster members to a
single or dual-controller RAID array. To use
Yes
single-initiator buses, a RAID controller must
have multiple host ports and provide
simultaneous access to all the logical units on
the host ports. To use a dual-controller RAID
array, a logical unit must fail over from one
controller to the other in a way that is
transparent to the OS.
SCSI RAID arrays that provide simultaneous
access to all logical units on the host ports are
recommended.
To ensure symmetry of device IDs and LUNs,
many RAID arrays with dual redundant
controllers must be configured in an
active/passive mode.
Refer to
Section 2.4.4 Configuring Shared Disk Storage
for more information.
14Chapter 2. Hardware Installation and Operating System Configuration
HardwareQuantityDescriptionRequired
SCSI cableOne per
member
SCSI cables with 68 pins connect each host bus
adapter to a storage enclosure port. Cables have
either HD68 or VHDCI connectors. Cables vary
Only for
parallel SCSI
configurations
based on adapter type.
SCSI
terminator
As required by
hardware
configuration
For a RAID storage enclosure that uses "out"
ports (such as FlashDisk RAID Disk Array) and
is connected to single-initiator SCSI buses,
connect terminators to the "out" ports to
terminate the buses.
Only for
parallel SCSI
configurations
and only as
necessary for
termination
Fibre Channel
hub or switch
One or twoA Fibre Channel hub or switch may be required. Only for some
Fibre Channel
configurations
Fibre Channel
cable
As required by
hardware
configuration
A Fibre Channel cable connects a host bus
adapter to a storage enclosure port, a Fibre
Channel hub, or a Fibre Channel switch. If a hub
Only for Fibre
Channel
configurations
or switch is used, additional cables are needed to
connect the hub or switch to the storage adapter
ports.
Table 2-8. Shared Disk Storage Hardware Table
HardwareQuantityDescriptionRequired
Network
interface
Two for each
member
Each Ethernet connection requires
a network interface card installed
No
on all cluster members.
Network
crossover cable
One for each
channel
A network crossover cable
connects a network interface on
one member to a network interface
on other cluster members, creating
an Ethernet connection for
Only for a redundant
Ethernet connection (use
of channel-bonded
Ethernet connection is
preferred)
UPS systemOne or moreUninterruptible power supply (UPS) systems
protect against downtime if a power outage
occurs. UPS systems are highly recommended
Strongly
recommended
for availability
for cluster operation. Connect the power cables
for the shared storage enclosure and both power
switches to redundant UPS systems. Note, a
UPS system must be able to provide voltage for
an adequate period of time, and should be
connected to its own power circuit.
Table 2-10. UPS System Hardware Table
Loading...
+ 178 hidden pages
You need points to download manuals.
1 point = 1 manual.
You can buy points or you can get point for every manual you upload.