REDHAT CLUSTER SUITE User Manual

Red Hat Cluster Suite
Configuring and Managing a
Cluster
Red Hat Cluster Suite: Configuring and Managing a Cluster
Copyright © 2000-2004 Red Hat, Inc.Mission Critical Linux, Inc.K.M. Sorenson
Red Hat, Inc.
1801 Varsity Drive Raleigh NC 27606-2072 USA Phone: +1 919 754 3700 Phone: 888 733 4281 Fax: +1 919 754 3701 PO Box 13588 Research Triangle Park NC 27709 USA
rh-cs(EN)-3-Print-RHI (2006-02-03T16:44) For Part I Using the Red Hat Cluster Manager and Part III Appendixes, permission is granted to copy, distribute and/ormodify this document under the terms of the GNU Free DocumentationLicense, Version 1.1 or any later version published by theFree Software Foundation.A copy of the license is availableat http://www.gnu.org/licenses/fdl.html. The content described in this paragraph is copyrighted by © Mission Critical Linux, Inc. (2000), K.M. Sorenson (2000), and Red Hat, Inc. (2000-2003). This material in Part II Configuring a Linux Virtual Server Cluster may be distributed only subject to the terms and conditions set forth in the Open PublicationLicense, V1.0 or later (the latest version is presently available at http://www.opencontent.org/openpub/). Distribution of substantively modified versions of this material is prohibitedwithout
the explicit permissionof the copyright holder. Distribution of the work or derivative of the work in any standard(paper) book form for commercial purposes is prohibited unless prior permission is obtained from the copyright holder. The content described in this paragraph is copyrightedby © Red Hat, Inc. (2000-2003). Red Hat and the Red Hat "Shadow Man" logo are registered trademarks of Red Hat, Inc. in the United States and other countries. All other trademarks referencedherein are the property of their respective owners. The GPG fingerprint of the security@redhat.comkey is: CA 20 86 86 2B D6 9D FC 65 F6 EC C4 21 91 80 CD DB 42 A6 0E

Table of Contents

Acknowledgments ................................................................................................................................ i
Introduction........................................................................................................................................ iii
1. How To Use This Manual .................................................................................................... iii
2. Document Conventions ........................................................................................................ iv
3. More to Come ...................................................................................................................... vi
3.1. Send in Your Feedback ......................................................................................... vi
4. Activate Your Subscription ................................................................................................. vii
4.1. Provide a Red Hat Login...................................................................................... vii
4.2. Provide Your Subscription Number ..................................................................... vii
4.3. Connect Your System..........................................................................................viii
I. Using the Red Hat Cluster Manager ............................................................................................ ix
1. Red Hat Cluster Manager Overview..................................................................................... 1
1.1. Red Hat Cluster Manager Features ........................................................................ 2
2. Hardware Installation and Operating System Configuration ................................................ 5
2.1. Choosing a Hardware Configuration ..................................................................... 5
2.2. Setting Up the Members ...................................................................................... 15
2.3. Installing and Configuring Red Hat Enterprise Linux ......................................... 17
2.4. Setting Up and Connecting the Cluster Hardware ............................................... 21
3. Cluster Configuration .......................................................................................................... 35
3.1. Installing the Red Hat Cluster Suite Packages ..................................................... 35
3.2. Installation Notes for Red Hat Enterprise Linux 2.1 Users ................................. 37
3.3. The Cluster Configuration Tool ........................................................................ 37
3.4. Configuring the Cluster Software ........................................................................ 40
3.5. Editing the rawdevices File .............................................................................. 41
3.6. Configuring Cluster Daemons.............................................................................. 42
3.7. Adding and Deleting Members ............................................................................ 46
3.8. Configuring a Power Controller Connection ....................................................... 48
3.9. Configuring a Failover Domain ........................................................................... 49
3.10. Adding a Service to the Cluster ......................................................................... 51
3.11. Checking the Cluster Configuration................................................................... 54
3.12. Configuring syslogd Event Logging ............................................................... 56
4. Service Administration ....................................................................................................... 59
4.1. Configuring a Service .......................................................................................... 59
4.2. Displaying a Service Configuration ..................................................................... 62
4.3. Disabling a Service .............................................................................................. 63
4.4. Enabling a Service ............................................................................................... 63
4.5. Modifying a Service ............................................................................................. 63
4.6. Relocating a Service ............................................................................................ 64
4.7. Deleting a Service ................................................................................................ 64
4.8. Handling Failed Services ..................................................................................... 64
5. Database Services ............................................................................................................... 67
5.1. Setting Up an Oracle Service ............................................................................... 67
5.2. Tuning Oracle Service ......................................................................................... 72
5.3. Setting Up a MySQL Service .............................................................................. 73
6. Network File Sharing Services ........................................................................................... 77
6.1. Setting Up an NFS Service .................................................................................. 77
6.2. Using the NFS Druid .......................................................................................... 77
6.3. NFS Caveats......................................................................................................... 82
6.4. Importing the Contents of an NFS Exports File .................................................. 82
6.5. NFS Configuration: Active-Active Example ....................................................... 83
6.6. Setting Up a Samba Service ................................................................................. 84
6.7. Using the Samba Druid ........................................................................................ 86
6.8. Fields in the smb.conf.sharename File ........................................................... 90
7. Setting Up Apache HTTP Server ........................................................................................ 93
7.1. Apache HTTP Server Setup Overview ................................................................ 93
7.2. Configuring Shared Storage ................................................................................. 93
7.3. Installing and Configuring the Apache HTTP Server .......................................... 94
8. Cluster Administration ........................................................................................................ 97
8.1. Overview of the Cluster Status Tool.................................................................. 97
8.2. Displaying Cluster and Service Status ................................................................. 97
8.3. Starting and Stopping the Cluster Software ......................................................... 99
8.4. Modifying the Cluster Configuration................................................................. 100
8.5. Backing Up and Restoring the Cluster Database ............................................... 100
8.6. Modifying Cluster Event Logging ..................................................................... 101
8.7. Updating the Cluster Software........................................................................... 101
8.8. Changing the Cluster Name ............................................................................... 102
8.9. Disabling the Cluster Software .......................................................................... 102
8.10. Diagnosing and Correcting Problems in a Cluster ........................................... 102
II. Configuring a Linux Virtual Server Cluster ........................................................................... 107
9. Introduction to Linux Virtual Server................................................................................. 109
9.1. Technology Overview........................................................................................ 109
9.2. Basic Configurations .......................................................................................... 109
10. Linux Virtual Server Overview....................................................................................... 111
10.1. A Basic LVS Configuration ............................................................................. 111
10.2. A Three Tiered LVS Configuration.................................................................. 113
10.3. LVS Scheduling Overview ............................................................................... 114
10.4. Routing Methods.............................................................................................. 116
10.5. Persistence and Firewall Marks ....................................................................... 118
10.6. LVS Cluster — A Block Diagram ................................................................... 118
11. Initial LVS Configuration................................................................................................ 121
11.1. Configuring Services on the LVS Routers ....................................................... 121
11.2. Setting a Password for the Piranha Configuration Tool ............................... 122
11.3. Starting the Piranha Configuration Tool Service .......................................... 122
11.4. Limiting Access To the Piranha Configuration Tool.................................... 123
11.5. Turning on Packet Forwarding......................................................................... 124
11.6. Configuring Services on the Real Servers ....................................................... 124
12. Setting Up a Red Hat Enterprise Linux LVS Cluster...................................................... 125
12.1. The NAT LVS Cluster ...................................................................................... 125
12.2. Putting the Cluster Together ............................................................................ 127
12.3. Multi-port Services and LVS Clustering .......................................................... 128
12.4. FTP In an LVS Cluster ..................................................................................... 130
12.5. Saving Network Packet Filter Settings ............................................................ 132
13. Configuring the LVS Routers with Piranha Configuration Tool ................................. 133
13.1. Necessary Software.......................................................................................... 133
13.2. Logging Into the Piranha Configuration Tool .............................................. 133
13.3. CONTROL/MONITORING......................................................................... 134
13.4. GLOBAL SETTINGS ................................................................................... 135
13.5. REDUNDANCY............................................................................................. 137
13.6. VIRTUAL SERVERS .................................................................................... 139
13.7. Synchronizing Configuration Files .................................................................. 147
13.8. Starting the Cluster .......................................................................................... 148
III. Appendixes................................................................................................................................ 149
A. Using Red Hat Cluster Manager with Piranha ................................................................. 151
B. Using Red Hat GFS with Red Hat Cluster Suite.............................................................. 153
B.1. Terminology ...................................................................................................... 153
B.2. Changes to Red Hat Cluster .............................................................................. 154
B.3. Installation Scenarios ........................................................................................ 154
C. The GFS Setup Druid ....................................................................................................... 157
C.1. Cluster Name..................................................................................................... 157
C.2. LOCK_GULM parameters .................................................................................. 157
C.3. Choose Location for CCS Files ......................................................................... 158
C.4. Cluster Members ............................................................................................... 159
C.5. Saving Your Configuration and Next Steps....................................................... 161
D. Supplementary Hardware Information ............................................................................. 163
D.1. Setting Up Power Controllers ........................................................................... 163
D.2. SCSI Bus Configuration Requirements ............................................................ 165
D.3. SCSI Bus Termination ...................................................................................... 166
D.4. SCSI Bus Length............................................................................................... 166
D.5. SCSI Identification Numbers ............................................................................ 167
E. Supplementary Software Information .............................................................................. 169
E.1. Cluster Communication Mechanisms................................................................ 169
E.2. Failover and Recovery Scenarios ...................................................................... 170
E.3. Common Cluster Behaviors: General ................................................................ 170
E.4. Common Behaviors: Two Member Cluster with Disk-based Tie-breaker ........172
E.5. Common Behaviors: 2-4 Member Cluster with IP-based Tie-Breaker ............. 173
E.6. Common Behaviors: 3-5 Member Cluster ........................................................ 173
E.7. Common Behaviors: Cluster Service Daemons ................................................ 174
E.8. Common Behaviors: Miscellaneous .................................................................. 175
E.9. The cluster.xml File ..................................................................................... 175
F. Cluster Command-line Utilities ........................................................................................ 179
F.1. Using redhat-config-cluster-cmd ........................................................... 179
F.2. Using the shutil Utility ................................................................................... 180
F.3. Using the clusvcadm Utility ............................................................................ 180
F.4. Using the clufence Utility .............................................................................. 181
Index................................................................................................................................................. 183
Colophon.......................................................................................................................................... 191

Acknowledgments

The Red Hat Cluster Manager software was originally based on the open source Kimberlite (http://oss.missioncriticallinux.com/kimberlite/) cluster project, which was developed by Mission Critical Linux, Inc.
Subsequent to its inception based on Kimberlite, developers at Red Hat have made a large number of enhancements and modifications. The following is a non-comprehensive list highlighting some of these enhancements.
Packaging and integration into the Red Hat installation paradigm to simplify the end user’s experi-
ence.
Addition of support for multiple cluster members.
Addition of support for high availability NFS services.
Addition of support for high availability Samba services.
Addition of the Cluster Configuration Tool, a graphical configuration tool.
Addition of the Cluster Status Tool, a graphical monitoring and administration tool.
Addition of support for failover domains.
Addition of support for Red Hat GFS, including the GFS Setup Druid and the LOCK_GULM
fencing driver.
Addition of support for using watchdog timers as a data integrity provision.
Addition of service monitoring which automatically restart a failed application.
Rewrite of the service manager to facilitate additional cluster-wide operations.
A set of miscellaneous bug fixes.
The Red Hat Cluster Manager software incorporates STONITH compliant power switch modules from the Linux-HA project; refer to http://www.linux-ha.org/stonith/.
ii Acknowledgments

Introduction

The Red Hat Cluster Suite is a collection of technologies working together to provide data integrity and the ability to maintain application availability in the event of a failure. Administrators can deploy enterprise cluster solutions using a combination of hardware redundancy along with the failover and load-balancing technologies in Red Hat Cluster Suite.
Red Hat Cluster Manager is a high-availability cluster solution specifically suited for database appli­cations, network file servers, and World Wide Web (Web) servers with dynamic content. An Red Hat Cluster Manager system features data integrity and application availability using redundant hardware, shared disk storage, power management, and robust cluster communication and application failover mechanisms.
Administrators can also deploy highly available applications services using Piranha, a load-balancing and advanced routing cluster solution based on Linux Virtual Server (LVS) technology. Using Piranha, administrators can build highly available e-commerce sites that feature complete data integrity and service availability, in addition to load balancing capabilities. Refer to Part II Configuring a Linux Virtual Server Cluster for more information.
This guide assumes that the user has an advanced working knowledge of Red Hat Enterprise Linux and understands the concepts of server computing. For more information about using Red Hat Enterprise Linux, refer to the following resources:
Red Hat Enterprise Linux Installation Guide for information regarding installation.
Red Hat Enterprise Linux Introduction to System Administration for introductory information for
new Red Hat Enterprise Linux system administrators.
Red Hat Enterprise Linux System Administration Guide for more detailed information about con-
figuring Red Hat Enterprise Linux to suit your particular needs as a user.
Red Hat Enterprise Linux Reference Guide provides detailed information suited for more experi-
enced users to refer to when needed, as opposed to step-by-step instructions.
Red Hat Enterprise Linux Security Guide details the planning and the tools involved in creating a
secured computing environment for the data center, workplace, and home.
HTML, PDF, and RPM versions of the manuals are available on the Red Hat Enterprise Linux Docu­mentation CD and online at:
http://www.redhat.com/docs/

1. How To Use This Manual

This manual contains information about setting up a Red Hat Cluster Manager system. These tasks are described in Chapter 2 Hardware Installation and Operating System Configuration and Chapter 3 Cluster Configuration.
For information about configuring Red Hat Cluster Manager cluster services, refer to Chapter 4 Service Administration through Chapter 7 Setting Up Apache HTTP Server.
Part II Configuring a Linux Virtual Server Cluster describes how to achieve load balancing in an Red Hat Enterprise Linux cluster by using the Linux Virtual Server.
For information about deploying a Red Hat Cluster Manager cluster in conjunction with Red Hat GFS using the GFS Setup Druid, refer to Appendix C The GFS Setup Druid.
Appendix D Supplementary Hardware Information contains detailed configuration information on specific hardware devices and shared storage configurations. Appendix E Supplementary Software Information contains background information on the cluster software and other related information.
iv Introduction
Appendix F Cluster Command-line Utilities provides usage and reference information on the command-line utilities included with Red Hat Cluster Suite.
This guide assumes you have a thorough understanding of Red Hat Enterprise Linux system adminis­tration concepts and tasks. For detailed information on Red Hat Enterprise Linux system administra­tion, refer to the Red Hat Enterprise Linux System Administration Guide. For reference information on Red Hat Enterprise Linux, refer to the Red Hat Enterprise Linux Reference Guide.

2. Document Conventions

When you read this manual, certain words are represented in different fonts, typefaces, sizes, and weights. This highlighting is systematic; different words are represented in the same style to indicate their inclusion in a specific category. The types of words that are represented this way include the following:
command
Linux commands (and other operating system commands, when used) are represented this way. This style should indicate to you that you can type the word or phrase on the command line and press [Enter] to invoke a command. Sometimes a command contains words that would be displayed in a different style on their own (such as file names). In these cases, they are considered to be part of the command, so the entire phrase is displayed as a command. For example:
Use the cat testfile command to view the contents of a file, named testfile, in the current working directory.
file name
File names, directory names, paths, and RPM package names are represented this way. This style should indicate that a particular file or directory exists by that name on your system. Examples:
The .bashrc file in your home directory contains bash shell definitions and aliases for your own use.
The /etc/fstab file contains information about different system devices and file systems.
Install the webalizer RPM if you want to use a Web server log file analysis program.
application
This style indicates that the program is an end-user application (as opposed to system software). For example:
Use Mozilla to browse the Web.
[key]
A key on the keyboard is shown in this style. For example:
To use [Tab] completion, type in a character and then press the [Tab] key. Your terminal displays the list of files in the directory that start with that letter.
[key]-[combination]
A combination of keystrokes is represented in this way. For example:
The [Ctrl]-[Alt]-[Backspace] key combination exits your graphical session and return you to the graphical login screen or the console.
Introduction v
text found on a GUI interface
A title, word, or phrase found on a GUI interface screen or window is shown in this style. Text shown in this style is being used to identify a particular GUI screen or an element on a GUI screen (such as text associated with a checkbox or field). Example:
Select the Require Password checkbox if you would like your screensaver to require a password before stopping.
top level of a menu on a GUI screen or window
A word in this style indicates that the word is the top level of a pulldown menu. If you click on the word on the GUI screen, the rest of the menu should appear. For example:
Under File on a GNOME terminal, the New Tab option allows you to open multiple shell prompts in the same window.
If you need to type in a sequence of commands from a GUI menu, they are shown like the following example:
Go to Main Menu Button (on the Panel) => Programming => Emacs to start the Emacs text editor.
button on a GUI screen or window
This style indicates that the text can be found on a clickable button on a GUI screen. For example:
Click on the Back button to return to the webpage you last viewed.
computer output
Text in this style indicates text displayed to a shell prompt such as error messages and responses to commands. For example:
The ls command displays the contents of a directory. For example:
Desktop about.html logs paulwesterberg.png Mail backupfiles mail reports
The output returned in response to the command (in this case, the contents of the directory) is shown in this style.
prompt
A prompt, which is a computer’s way of signifying that it is ready for you to input something, is shown in this style. Examples:
$
#
[stephen@maturin stephen]$
leopard login:
user input
Text that the user has to type, either on the command line, or into a text box on a GUI screen, is displayed in this style. In the following example, text is displayed in this style:
To boot your system into the text based installation program, you must type in the text com­mand at the boot: prompt.
replaceable
Text used for examples, which is meant to be replaced with data provided by the user, is displayed in this style. In the following example, <version-number> is displayed in this style:
vi Introduction
The directory for the kernel source is /usr/src/<version-number>/, where <version-number> is the version of the kernel installed on this system.
Additionally, we use several different strategies to draw your attention to certain pieces of information. In order of how critical the information is to your system, these items are marked as a note, tip, important, caution, or warning. For example:
Note
Remember that Linux is case sensitive. In other words, a rose is not a ROSE is not a rOsE.
Tip
The directory /usr/share/doc/ contains additional documentation for packages installed on your system.
Important
If you modify the DHCP configuration file, the changes do not take effect until you restart the DHCP daemon.
Caution
Do not perform routine tasks as root — use a regular user account unless you need to use the root account for system administration tasks.
Warning
Be careful to remove only the necessary Red Hat Enterprise Linux partitions. Removing other par ti­tions could result in data loss or a corrupted system environment.

3. More to Come

This manual is part of Red Hat’s growing commitment to provide useful and timely support to Red Hat Enterprise Linux users.
Introduction vii

3.1. Send in Your Feedback

If you spot a typo, or if you have thought of a way to make this manual better, we would love to hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against the component rh-cs.
Be sure to mention the manual’s identifier:
rh-cs(EN)-3-Print-RHI (2006-02-03T16:44)
By mentioning this manual’s identifier, we know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If you have found an error, please include the section number and some of the surrounding text so we can find it easily.

4. Activate Your Subscription

Before you can access service and software maintenance information, and the support documenta­tion included in your subscription, you must activate your subscription by registering with Red Hat. Registration includes these simple steps:
Provide a Red Hat login
Provide a subscription number
Connect your system
The first time you boot your installation of Red Hat Enterprise Linux, you are prompted to register with Red Hat using the Setup Agent. If you follow the prompts during the Setup Agent, you can complete the registration steps and activate your subscription.
If you can not complete registration during the Setup Agent (which requires network access), you can alternatively complete the Red Hat registration process online at http://www.redhat.com/register/.

4.1. Provide a Red Hat Login

If you do not have an existing Red Hat login, you can create one when prompted during the Setup Agent or online at:
https://www.redhat.com/apps/activate/newlogin.html
A Red Hat login enables your access to:
Software updates, errata and maintenance via Red Hat Network
Red Hat technical support resources, documentation, and Knowledgebase
If you have forgotten your Red Hat login, you can search for your Red Hat login online at:
https://rhn.redhat.com/help/forgot_password.pxt
viii Introduction

4.2. Provide Your Subscription Number

Your subscription number is located in the package that came with your order. If your package did not include a subscription number, your subscription was activated for you and you can skip this step.
You can provide your subscription number when prompted during the Setup Agent or by visiting http://www.redhat.com/register/.

4.3. Connect Your System

The Red Hat Network Registration Client helps you connect your system so that you can begin to get updates and perform systems management. There are three ways to connect:
1. During the Setup Agent — Check the Send hardware information and Send system package list options when prompted.
2. After the Setup Agent has been completed — From the Main Menu, go to System Tools, then select Red Hat Network.
3. After the Setup Agent has been completed — Enter the following command from the command line as the root user:
/usr/bin/up2date --register

I. Using the Red Hat Cluster Manager

Clustered systems provide reliability, scalability, and availability to critical production services. Using the Red Hat Cluster Manager, administrators can create high availability clusters for filesharing, Web servers, databases, and more. This part discusses the installation and configuration of cluster systems using the recommended hardware and Red Hat Enterprise Linux.
This section is licensed under the GNU Free Documentation License. For details refer to the Copyright page.

Table of Contents

1. Red Hat Cluster Manager Overview............................................................................................. 1
2. Hardware Installation and Operating System Configuration .................................................... 5
3. Cluster Configuration ................................................................................................................... 35
4. Service Administration ................................................................................................................. 59
5. Database Services .......................................................................................................................... 67
6. Network File Sharing Services ..................................................................................................... 77
7. Setting Up Apache HTTP Server ................................................................................................ 93
8. Cluster Administration................................................................................................................. 97

Chapter 1.

Red Hat Cluster Manager Overview

Red Hat Cluster Manager allows administrators to connect separate systems (called members or nodes) together to create failover clusters that ensure application availability and data integrity under several failure conditions. Administrators can use Red Hat Cluster Manager with database applica­tions, file sharing services, web servers, and more.
To set up a failover cluster, you must connect the member systems (often referred to simply as mem- bers or nodes) to the cluster hardware, and configure the members into the cluster environment. The foundation of a cluster is an advanced host membership algorithm. This algorithm ensures that the cluster maintains complete data integrity at all times by using the following methods of inter-member communication:
Network connections between the cluster systems for heartbeat
Shared state on shared disk storage to hold cluster status
To make an application and data highly available in a cluster, you must configure a service (such as an application and shared disk storage) as a discrete, named group of properties and resources to which you can assign an IP address to provide transparent client access. For example, you can set up a service that provides clients with access to highly-available database application data.
You can associate a service with a failover domain, a subset of cluster members that are eligible to run the service. In general, any eligible member can run the service and access the service data on shared disk storage. However, each service can run on only one cluster member at a time, in order to maintain data integrity. You can specify whether or not the members in a failover domain are ordered by preference. You can also specify whether or not a service is restricted to run only on members of its associated failover domain. (When associated with an unrestricted failover domain, a service can be started on any cluster member in the event no member of the failover domain is available.)
You can set up an active-active configuration in which the members run different services simultane­ously, or a hot-standby configuration in which a primary member runs all the services, and a backup cluster system takes over only if the primary system fails.
Figure 1-1 shows an example of a cluster in an active-active configuration.
Figure 1-1. Example Cluster in Active-Active Configuration
2 Chapter 1. Red Hat Cluster Manager Overview
If a hardware or software failure occurs, the cluster automatically restarts the failed member’s services on the functional member. This service failover capability ensures that no data is lost, and there is little disruption to users. When the failed member recovers, the cluster can re-balance the services across the members.
In addition, you can cleanly stop the services running on a cluster system and then restart them on another system. This service relocation capability allows you to maintain application and data avail­ability when a cluster member requires maintenance.

1.1. Red Hat Cluster Manager Features

Cluster systems deployed with Red Hat Cluster Manager include the following features:
No-single-point-of-failure hardware configuration
Clusters can include a dual-controller RAID array, multiple network channels, and redundant uninterruptible power supply (UPS) systems to ensure that no single failure results in application down time or loss of data.
Alternately, a low-cost cluster can be set up to provide less availability than a no-single-point­of-failure cluster. For example, you can set up a cluster with a single-controller RAID array and only a single Ethernet channel.
Certain low-cost alternatives, such as software RAID and multi-initiator parallel SCSI, are not compatible or appropriate for use on the shared cluster storage. Refer to Section 2.1 Choosing a Hardware Configuration, for more information.
Service configuration framework
Clusters allow you to easily configure individual services to make data and applications highly available. To create a service, you specify the resources used in the service and properties for the service, including the service name, application start, stop, and status script, disk partitions, mount points, and the cluster members on which you prefer the service to run. After you add a service, the cluster management software stores the information in a cluster configuration file on shared storage, where the configuration data can be accessed by all cluster members.
The cluster provides an easy-to-use framework for database applications. For example, a database service serves highly-available data to a database application. The application running on a clus­ter member provides network access to database client systems, such as Web servers. If the service fails over to another member, the application can still access the shared database data. A network-accessible database service is usually assigned an IP address, which is failed over along with the service to maintain transparent access for clients.
The cluster service framework can be easily extended to other applications, as well.
Failover domains
By assigning a service to a restricted failover domain, you can limit the members that are eligible to run a service in the event of a failover. (A service that is assigned to a restricted failover domain cannot be started on a cluster member that is not included in that failover domain.) You can order the members in a failover domain by preference to ensure that a particular member runs the service (as long as that member is active). If a service is assigned to an unrestricted failover domain, the service starts on any available cluster member (if none of the members of the failover domain are available).
Data integrity assurance
To ensure data integrity, only one member can run a service and access service data at one time. The use of power switches in the cluster hardware configuration enables a member to power-cycle another member before restarting that member’s services during the failover process.
Chapter 1. Red Hat Cluster Manager Overview 3
This prevents any two systems from simultaneously accessing the same data and corrupting it. Although not required, it is recommended that power switches are used to guarantee data integrity under all failure conditions. Watchdog timers are an optional variety of power control to ensure correct operation of service failover.
Cluster administration user interface
The cluster administration interface facilitiates management tasks such as: creating, starting, and stopping services; relocating services from one member to another; modifying the cluster configuration (to add or remove services or resources); and monitoring the cluster members and services.
Ethernet channel bonding
To monitor the health of the other members, each member monitors the health of the remote power switch, if any, and issues heartbeat pings over network channels. With Ethernet channel bonding, multiple Ethernet interfaces are configured to behave as one, reducing the risk of a single-point-of-failure in the typical switched Ethernet connection between systems.
Shared storage for quorum information
Shared state information includes whether the member is active. Service state information in­cludes whether the service is running and which member is running the service. Each member checks to ensure that the status of the other members is up to date.
In a two-member cluster, each member periodically writes a timestamp and cluster state infor­mation to two shared cluster partitions located on shared disk storage. To ensure correct cluster operation, if a member is unable to write to both the primary and shadow shared cluster partitions at startup time, it is not allowed to join the cluster. In addition, if a member is not updating its timestamp, and if heartbeats to the system fail, the member is removed from the cluster.
Figure 1-2 shows how members communicate in a cluster configuration. Note that the terminal server used to access system consoles via serial ports is not a required cluster component.
Figure 1-2. Cluster Communication Mechanisms
4 Chapter 1. Red Hat Cluster Manager Overview
Service failover capability
If a hardware or software failure occurs, the cluster takes the appropriate action to maintain application availability and data integrity. For example, if a member completely fails, another member (in the associated failover domain, if used, or in the cluster) restarts its services. Services already running on this member are not disrupted.
When the failed member reboots and is able to write to the shared cluster partitions, it can rejoin the cluster and run services. Depending on how the services are configured, the cluster can re­balance the services among the members.
Manual service relocation capability
In addition to automatic service failover, a cluster allows you to cleanly stop services on one member and restart them on another member. You can perform planned maintenance on a mem­ber system while continuing to provide application and data availability.
Event logging facility
To ensure that problems are detected and resolved before they affect service availability, the clus­ter daemons log messages by using the conventional Linux syslog subsystem. You can customize the severity level of the logged messages.
Application monitoring
The infrastructure in a cluster can optionally monitor the state and health of an application. In this manner, should an application-specific failure occur, the cluster automatically restarts the application. In response to the application failure, the application attempts to be restarted on the member it was initially running on; failing that, it restarts on another cluster member. You can specify which members are eligible to run a service by assigning a failover domain to the service.

Chapter 2.

Hardware Installation and Operating System Configuration
To set up the hardware configuration and install Red Hat Enterprise Linux, follow these steps:
Choose a cluster hardware configuration that meets the needs of applications and users; refer to
Section 2.1 Choosing a Hardware Configuration.
Set up and connect the members and the optional console switch and network switch or hub; refer
to Section 2.2 Setting Up the Members.
Install and configure Red Hat Enterprise Linux on the cluster members; refer to
Section 2.3 Installing and Configuring Red Hat Enterprise Linux .
Set up the remaining cluster hardware components and connect them to the members; refer to
Section 2.4 Setting Up and Connecting the Cluster Hardware.
After setting up the hardware configuration and installing Red Hat Enterprise Linux, install the cluster software.
Tip
Refer to the Red Hat Hardware Compatibility List available at http://hardware.redhat.com/hcl/ for a list of compatible hardware. Perform a Quick Search for the term cluster to find results for power switch and shared storage hardware certified for or compatible with Red Hat Cluster Manager. For general system hardware compatibility searches, use manufacturer, brand, and/or model keywords to check for compatibility with Red Hat Enterprise Linux.
2.1. Choosing a Hardware Configuration
The Red Hat Cluster Manager allows administrators to use commodity hardware to set up a cluster configuration that meets the performance, availability, and data integrity needs of applications and users. Cluster hardware ranges from low-cost minimum configurations that include only the com­ponents required for cluster operation, to high-end configurations that include redundant Ethernet channels, hardware RAID, and power switches.
Regardless of configuration, the use of high-quality hardware in a cluster is recommended, as hard­ware malfunction is a primary cause of system down time.
Although all cluster configurations provide availability, some configurations protect against every single point of failure. In addition, all cluster configurations provide data integrity, but some configu­rations protect data under every failure condition. Therefore, administrators must fully understand the needs of their computing environment and also the availability and data integrity features of different hardware configurations to choose the cluster hardware that meets the proper requirements.
When choosing a cluster hardware configuration, consider the following:
Performance requirements of applications and users
Choose a hardware configuration that provides adequate memory, CPU, and I/O resources. Be sure that the configuration chosen can handle any future increases in workload as well.
6 Chapter 2. Hardware Installation and Operating System Configuration
Cost restrictions
The hardware configuration chosen must meet budget requirements. For example, systems with multiple I/O ports usually cost more than low-end systems with fewer expansion capabilities.
Availability requirements
If a computing environment requires the highest degree of availability, such as a production environment, then a cluster hardware configuration that protects against all single points of fail­ure, including disk, storage interconnect, Ethernet channel, and power failures is recommended. Environments that can tolerate an interruption in availability, such as development environ­ments, may not require as much protection. Refer to Section 2.4.3 Configuring UPS Systems and Section 2.4.4 Configuring Shared Disk Storagefor more information about using redundant hard­ware for high availability.
Data integrity under all failure conditions requirement
Using power switches in a cluster configuration guarantees that service data is protected under every failure condition. These devices enable a member to power cycle another member before restarting its services during failover. Power switches protect against data corruption if an unre­sponsive (or hanging) member becomes responsive after its services have failed over and then issues I/O to a disk that is also receiving I/O from the other member.
In addition, if a quorum daemon fails on a member, the member is no longer able to monitor the shared cluster partitions. If you are not using power switches in the cluster, this error condition may result in services being run on more than one member, which can cause data corruption. Refer to Section 2.4.2 Configuring Power Switches for more information about the benefits of using power switches in a cluster. It is recommended that production environments use power switches or watchdog timers in the cluster configuration.

2.1.1. Shared Storage Requirements

The operation of the cluster depends on reliable, coordinated access to shared storage. In the event of hardware failure, it is desirable to be able to disconnect one member from the shared storage for repair without disrupting the other members. Shared storage is truly vital to the cluster configuration.
Testing has shown that it is difficult, if not impossible, to configure reliable multi-initiator parallel SCSI configurations at data rates above 80MB/sec using standard SCSI adapters. Further tests have shown that these configurations cannot support online repair because the bus does not work reliably when the HBA terminators are disabled, and external terminators are used. For these reasons, multi­initiator SCSI configurations using standard adapters are not supported. Either single-initiator SCSI bus adapters (connected to multi-ported storage) or Fibre Channel adapters are required.
The Red Hat Cluster Manager requires that all cluster members have simultaneous access to the shared storage. Certain host RAID adapters are capable of providing this type of access to shared RAID units. These products require extensive testing to ensure reliable operation, especially if the shared RAID units are based on parallel SCSI buses. These products typically do not allow for online repair of a failed member. Only host RAID adapters listed in the Red Hat Hardware Compatibility List are supported.
The use of software RAID, or software Logical Volume Management (LVM), is not supported on shared storage. This is because these products do not coordinate access from multiple hosts to shared storage. Software RAID or LVM may be used on non-shared storage on cluster members (for example, boot and system partitions, and other file systems which are not associated with any cluster services).

2.1.2. Minimum Hardware Requirements

A minimum hardware configuration includes only the hardware components that are required for cluster operation, as follows:
Chapter 2. Hardware Installation and Operating System Configuration 7
Two servers to run cluster services
Ethernet connection for sending heartbeat pings and for client network access
Shared disk storage for the shared cluster partitions and service data
The hardware components described in Table 2-1 can be used to set up a minimum cluster configura­tion. This configuration does not guarantee data integrity under all failure conditions, because it does not include power switches. Note that this is a sample configuration; it is possible to set up a minimum configuration using other hardware.
Warning
The minimum cluster configuration is not a supported solution and should not be used in a production environment, as it does not guarantee data integrity under all failure conditions.
Hardware Description
Two servers Each member includes a network interface for client access and
for Ethernet connections and a SCSI adapter (termination disabled) for the shared storage connection
Two network cables with RJ45 connectors
Network cables connect an Ethernet network interface on each member to the network for client access and heartbeat pings.
RAID storage enclosure The RAID storage enclosure contains one controller with at least
two host ports.
Two HD68 SCSI cables Each cable connects one HBA to one port on the RAID
controller, creating two single-initiator SCSI buses.
Table 2-1. Example of Minimum Cluster Configuration
The minimum hardware configuration is the most cost-effective cluster configuration; however, it includes multiple points of failure. For example, if the RAID controller fails, then all cluster services become unavailable. When deploying the minimal hardware configuration, software watchdog timers should be configured as a data integrity provision. Refer to Section D.1.2.3 Configuring a Hardware Watchdog Timer for details.
To improve availability, protect against component failure, and guarantee data integrity under all fail­ure conditions, the minimum configuration can be expanded, as described in Table 2-2.
Problem Solution
Disk failure Hardware RAID to replicate data across multiple disks
RAID controller failure Dual RAID controllers to provide redundant access to
disk data
Heartbeat failure Ethernet channel bonding and failover
Power source failure Redundant uninterruptible power supply (UPS) systems
Data corruption under all failure
Power switches or hardware-based watchdog timers
conditions
Table 2-2. Improving Availability and Guaranteeing Data Integrity
8 Chapter 2. Hardware Installation and Operating System Configuration
A no single point of failure hardware configuration that guarantees data integrity under all failure conditions can include the following components:
At least two servers to run cluster services
Ethernet connection between each member for heartbeat pings and for client network access
Dual-controller RAID array to replicate shared partitions and service data
Power switches to enable each member to power-cycle the other members during the failover pro-
cess
Ethernet interfaces configured to use channel bonding
At least two UPS systems for a highly-available source of power
The components described in Table 2-3 can be used to set up a no single point of failure cluster con­figuration that includes two single-initiator SCSI buses and power switches to guarantee data integrity under all failure conditions. Note that this is a sample configuration; it is possible to set up a no single point of failure configuration using other hardware.
Hardware Description
Two servers (up to 8 supported) Each member includes the following hardware:
Two network interfaces for: Point-to-point Ethernet connections Client network access and Ethernet heartbeat pings Three serial ports for: Remote power switch connection
Connection to the terminal server One Adaptec 29160 adapter (termination enabled) for the shared disk storage connection.
One network switch A network switch enables the connection of multiple members to
a network.
One Cyclades terminal server A terminal server allows for management of remote members
from a central location. (A terminal server is not required for cluster operation.)
Four network cables Network cables connect the terminal server and a network
interface on each member to the network switch.
Two RJ45 to DB9 crossover cables
Two serial-attached power switches
RJ45 to DB9 crossover cables connect a serial port on each member to the Cyclades terminal server.
Power switches enable each member to power-cycle the other member before restarting its services. The power cable for each member is connected to its own power switch. Note that serial-attach power switches are supported in two-member clusters only.
Two null modem cables Null modem cables connect a serial port on each member to the
power switch that provides power to the other member. This connection enables each member to power-cycle the other member.
FlashDisk RAID Disk Array with dual controllers
Dual RAID controllers protect against disk and controller failure. The RAID controllers provide simultaneous access to all the logical units on the host ports.
Chapter 2. Hardware Installation and Operating System Configuration 9
Hardware Description
Two HD68 SCSI cables HD68 cables connect each host bus adapter to a RAID enclosure
"in" port, creating two single-initiator SCSI buses.
Two terminators Terminators connected to each "out" port on the RAID enclosure
terminate both single-initiator SCSI buses.
Redundant UPS Systems UPS systems provide a highly-available source of power. The
power cables for the power switches and the RAID enclosure are connected to two UPS systems.
Table 2-3. Example of a No Single Point of Failure Configuration
Figure 2-1 shows an example of a no single point of failure hardware configuration that includes the previously-described hardware, two single-initiator SCSI buses, and power switches to guarantee data integrity under all error conditions. A "T" enclosed in a circle represents a SCSI terminator.
Figure 2-1. No Single Point of Failure Configuration Example
Cluster hardware configurations can also include other optional hardware components that are com­mon in a computing environment. For example, a cluster can include a network switch or network hub, which enables the connection of the members to a network. A cluster may also include a console
10 Chapter 2. Hardware Installation and Operating System Configuration
switch, which facilitates the management of multiple members and eliminates the need for separate monitors, mouses, and keyboards for each member.
One type of console switch is a terminal server, which enables connection to serial consoles and management of many members from one remote location. As a low-cost alternative, you can use a KVM (keyboard, video, and mouse) switch, which enables multiple members to share one keyboard, monitor, and mouse. A KVM is suitable for configurations in which access to a graphical user interface (GUI) to perform system management tasks is preferred.
When choosing a system, be sure that it provides the required PCI slots, network slots, and serial ports. For example, a no single point of failure configuration requires multiple bonded Ethernet ports. Refer to Section 2.2.1 Installing the Basic Cluster Hardware for more information.

2.1.3. Choosing the Type of Power Controller

The Red Hat Cluster Manager implementation consists of a generic power management layer and a set of device-specific modules which accommodate a range of power management types. When selecting the appropriate type of power controller to deploy in the cluster, it is important to recognize the implications of specific device types. The following describes the types of supported power switches followed by a summary table. For a more detailed description of the role a power switch plays to ensure data integrity, refer to Section 2.4.2 Configuring Power Switches.
Serial-attached and network-attached power switches are separate devices which enable one cluster member to power cycle another member. They resemble a power plug strip on which individual outlets can be turned on and off under software control through either a serial or network cable. Network­attached power switches differ from serial-attached in that they connect to cluster members via an Ethernet hub or switch, rather than direct connection to cluster members. A network-attached power switch can not be directly attached to a cluster member using a crossover cable, as the power switch would be unable to power cycle the other members.
Watchdog timers provide a means for failed members to remove themselves from the cluster prior to another member taking over its services, rather than allowing one cluster member to power cycle another. The normal operational mode for watchdog timers is that the cluster software must periodi­cally reset a timer prior to its expiration. If the cluster software fails to reset the timer, the watchdog triggers under the assumption that the member may have hung or otherwise failed. The healthy cluster member allows a window of time to pass prior to concluding that another cluster member has failed (by default, this window is 12 seconds). The watchdog timer interval must be less than the duration of time for one cluster member to conclude that another has failed. In this manner, a healthy member can assume, prior to taking over services, that the failed cluster member has safely removed itself from the cluster (by rebooting) and is no longer a risk to data integrity. The underlying watchdog support is included in the core Linux kernel. Red Hat Cluster Manager utilizes these watchdog features via its standard APIs and configuration mechanism.
There are two types of watchdog timers: hardware-based and software-based. Hardware-based watch­dog timers typically consist of system board components such as the Intel® i810 TCO chipset. This circuitry has a high degree of independence from the main system CPU. This independence is bene­ficial in failure scenarios of a true system hang, as in this case it pulls down the system’s reset lead resulting in a system reboot. Some PCI expansion cards provide watchdog features.
Software-based watchdog timers do not have any dedicated hardware. The implementation is a kernel thread which is periodically run; if the timer duration has expired, the thread initiates a system reboot. The vulnerability of the software watchdog timer is that under certain failure scenarios, such as system hangs while interrupts are blocked, the kernel thread is not called. As a result, in such conditions it cannot be definitively depended on for data integrity. This can cause the healthy cluster member to take over services for a hung member which could cause data corruption under certain scenarios.
Finally, administrators can choose not to employ a power controller at all. When a power controller is not in use, no provision exists for a cluster member to power cycle a failed member. Similarly, the failed member cannot be guaranteed to reboot itself under all failure conditions.
Chapter 2. Hardware Installation and Operating System Configuration 11
Important
Use of a power controller is strongly recommended as part of a production cluster environment. Configuration of a cluster without a power controller is not supported.
Ultimately, the right type of power controller deployed in a cluster environment depends on the data integrity requirements weighed against the cost and availability of external power switches.
Table 2-4 summarizes the types of supported power management modules and discusses their advan­tages and disadvantages individually.
Type Notes Pros Cons
Serial-attached power switches (supported for two-member clusters only)
Network-attached power switches
Hardware Watchdog Timer
Software Watchdog Timer
Two serial attached power controllers are used in a cluster (one per member)
A single network attached power controller is required per cluster (depending on the number of members); however, up to three are supported for each cluster member
Affords strong data integrity guarantees
Offers acceptable data integrity provisions
Affords strong data integrity guarantees — the power controller itself is not a single point of failure as there are two in a cluster
Affords strong data integrity guarantees and can be used in clusters with more than two members
Obviates the need to purchase external power controller hardware
Obviates the need to purchase external power controller hardware; works on any system
Requires purchase of power controller hardware and cables; consumes serial ports; can only be used in two-member cluster
Requires purchase of power controller hardware — the power controller itself can become a single point of failure (although they are typically very reliable devices)
Not all systems include supported watchdog hardware
Under some failure scenarios, the software watchdog is not operational, opening a small vulnerability window
No power controller
No power controller function is in use
Obviates the need to purchase external power controller hardware;
Vulnerable to data corruption under certain failure scenarios
works on any system
Table 2-4. Power Switches

2.1.4. Cluster Hardware Components

Use the following tables to identify the hardware components required for the cluster configuration.
Table 2-5 includes the hardware required for the cluster members.
12 Chapter 2. Hardware Installation and Operating System Configuration
Hardware Quantity Description Required
Cluster members
eight (maximum supported)
Each member must provide enough PCI slots, network slots, and serial ports for the cluster hardware configuration. Because disk devices
Yes
must have the same name on each member, it is recommended that the members have symmetric I/O subsystems. It is also recommended that the processor speed and amount of system memory be adequate for the processes run on the cluster members. Consult the Red Hat Enterprise Linux 3 Release Notes for specifics. Refer to Section 2.2.1 Installing the Basic Cluster Hardware for more information.
Table 2-5. Cluster Member Hardware
Table 2-6 includes several different types of power switches.
A single cluster requires only one type of power switch.
Hardware Quantity Description Required
Serial power switches
Two In a two-member cluster, use serial power
switches to enable each cluster member to power-cycle the other member. Refer to Section 2.4.2 Configuring Power Switches for more information. Note, cluster members are configured with either serial power switches
Strongly recommended for data integrity under all failure
conditions (supported for two-member clusters only) or network-attached power switches, but not both.
Null modem cable
Two Null modem cables connect a serial port on a
cluster member to a serial power switch. This enables each member to power-cycle the other
Only if using
serial power
switches member. Some power switches may require different cables.
Mounting bracket
One Some power switches support rack mount
configurations and require a separate mounting bracket.
Only for rack
mounting
power
switches
Network power switch
One (depends on member count)
Network-attached power switches enable each cluster member to power cycle all others. Refer to Section 2.4.2 Configuring Power Switches for more information.
Strongly
recommended
for data
integrity under
all failure
conditions
Watchdog Timer
One per member
Watchdog timers cause a failed cluster member to remove itself from a cluster prior to a healthy member taking over its services. Refer to Section 2.4.2 Configuring Power Switches for more information.
Recommended
for data
integrity on
systems which
provide
integrated
watchdog
hardware
Chapter 2. Hardware Installation and Operating System Configuration 13
Table 2-6. Power Switch Hardware Table
Table 2-8 through Table 2-10 show a variety of hardware components for an administrator to choose from. An individual cluster does not require all of the components listed in these tables.
Hardware Quantity Description Required
Network interface
One for each network
Each network connection requires a network interface installed in a member.
Yes
connection
Network switch or hub
Network cable One for each
One A network switch or hub allows connection of
multiple members to a network.
A conventional network cable, such as a cable
network interface
with an RJ45 connector, connects each network interface to a network switch or a network hub.
No
Yes
Table 2-7. Network Hardware Table
Hardware Quantity Description Required
Host bus adapter
One per member
To connect to shared disk storage, install either a parallel SCSI or a Fibre Channel host bus
Yes
adapter in a PCI slot in each cluster member. For parallel SCSI, use a low voltage differential (LVD) host bus adapter. Adapters
have either HD68 or VHDCI connectors. Host-bus adapter based RAID cards are only supported if they correctly support multi-host operation. At the time of publication, there were no fully tested host-bus adapter based RAID cards.
External disk storage enclosure
At least one Use Fibre Channel or single-initiator parallel
SCSI to connect the cluster members to a
single or dual-controller RAID array. To use
Yes
single-initiator buses, a RAID controller must
have multiple host ports and provide
simultaneous access to all the logical units on
the host ports. To use a dual-controller RAID
array, a logical unit must fail over from one
controller to the other in a way that is
transparent to the OS.
SCSI RAID arrays that provide simultaneous
access to all logical units on the host ports are
recommended.
To ensure symmetry of device IDs and LUNs,
many RAID arrays with dual redundant
controllers must be configured in an
active/passive mode. Refer to Section 2.4.4 Configuring Shared Disk Storage for more information.
14 Chapter 2. Hardware Installation and Operating System Configuration
Hardware Quantity Description Required
SCSI cable One per
member
SCSI cables with 68 pins connect each host bus adapter to a storage enclosure port. Cables have either HD68 or VHDCI connectors. Cables vary
Only for parallel SCSI configurations
based on adapter type.
SCSI terminator
As required by hardware configuration
For a RAID storage enclosure that uses "out" ports (such as FlashDisk RAID Disk Array) and is connected to single-initiator SCSI buses, connect terminators to the "out" ports to terminate the buses.
Only for parallel SCSI configurations and only as necessary for termination
Fibre Channel hub or switch
One or two A Fibre Channel hub or switch may be required. Only for some
Fibre Channel configurations
Fibre Channel cable
As required by hardware configuration
A Fibre Channel cable connects a host bus adapter to a storage enclosure port, a Fibre Channel hub, or a Fibre Channel switch. If a hub
Only for Fibre Channel
configurations or switch is used, additional cables are needed to connect the hub or switch to the storage adapter ports.
Table 2-8. Shared Disk Storage Hardware Table
Hardware Quantity Description Required
Network interface
Two for each member
Each Ethernet connection requires a network interface card installed
No
on all cluster members.
Network crossover cable
One for each channel
A network crossover cable connects a network interface on one member to a network interface on other cluster members, creating an Ethernet connection for
Only for a redundant Ethernet connection (use of channel-bonded Ethernet connection is preferred)
communicating heartbeat.
Table 2-9. Point-To-Point Ethernet Connection Hardware Table
Hardware Quantity Description Required
UPS system One or more Uninterruptible power supply (UPS) systems
protect against downtime if a power outage occurs. UPS systems are highly recommended
Strongly
recommended
for availability for cluster operation. Connect the power cables for the shared storage enclosure and both power switches to redundant UPS systems. Note, a UPS system must be able to provide voltage for an adequate period of time, and should be connected to its own power circuit.
Table 2-10. UPS System Hardware Table
Loading...
+ 178 hidden pages