Page 1
Red Hat GFS 5.2.1
Administrator’s Guide
Page 2
Red Hat GFS 5.2.1: Administrator’s Guide
Copyright © 2004 by Red Hat, Inc.
Red Hat, Inc.
1801 Varsity Drive
Raleigh NC 27606-2072 USA
Phone: +1 919 754 3700
Phone: 888 733 4281
Fax: +1 919 754 3701
PO Box 13588
Research Triangle Park NC 27709 USA
Red Hat GFS Administrator’s Guide(EN)-5.2.1-Print-RHI (2004-01-15T12:12-0400)
Copyright © 2004 by Red Hat, Inc. This material may be distributed only subject to the terms and conditions set forth in the
Open Publication License, V1.0 or later (the latest version is presentlyavailableat http://www.opencontent.org/openpub/).
Distribution of substantively modified versions of this document is prohibited without the explicit permission of the copyright
holder.
Distribution of the work or derivative of the work in any standard (paper) book form for commercial purposes is prohibited
unless prior permission is obtained from the copyright holder.
Red Hat, Red Hat Network, the Red Hat "Shadow Man" logo, RPM, Maximum RPM, the RPM logo, Linux Library,
PowerTools, Linux Undercover, RHmember, RHmember More, Rough Cuts, Rawhide and all Red Hat-based trademarks and
logos are trademarks or registered trademarks of Red Hat, Inc. in the United States and other countries.
Linux is a registered trademark of Linus Torvalds.
Motif and UNIX are registeredtrademarks of The Open Group.
XFree86 is a trademark of The XFree86 Project, Inc, and is pending registration.
Intel and Pentium are registered trademarks of Intel Corporation. Itanium and Celeron are trademarks of Intel Corporation.
AMD, Opteron, Athlon, Duron, and K6 are registered trademarks of Advanced Micro Devices, Inc.
Netscape is a registered trademark of Netscape Communications Corporation in the United States and other countries.
Java and Swing are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. or other countries.
Oracle is a registered trademark, and Oracle8i, Oracle9i, and interMedia are trademarks or registered trademarksof Oracle
Corporation.
Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or
other countries.
SSH and Secure Shell are trademarks of SSH Communications Security, Inc.
FireWire is a trademark of Apple Computer Corporation.
IBM, AS/400, OS/400, RS/6000, S/390, and zSeries are registered trademarks of International Business Machines
Corporation. eServer, iSeries, and pSeries are trademarks of InternationalBusiness Machines Corporation.
All other trademarks and copyrights referred to are the property of their respective owners.
The GPG fingerprint of the security@redhat.com key is:
CA 20 86 86 2B D6 9D FC 65 F6 EC C4 21 91 80 CD DB 42 A6 0E
Page 3
Table of Contents
Introduction..........................................................................................................................................i
1. Audience ................................................................................................................................i
2. Document Conventions..........................................................................................................i
3. More to Come ......................................................................................................................iii
3.1. Send in Your Feedback ......................................................................................... iv
4. Sign Up for Support ............................................................................................................. iv
5. Recommended References................................................................................................... iv
1. GFS Overview ................................................................................................................................. 1
1.1. New and Changed Features................................................................................................ 1
1.2. Performance, Scalability, and Economy ............................................................................ 1
1.2.1. Superior Performance and Scalability ................................................................ 2
1.2.2. Performance, Scalability, Moderate Price........................................................... 3
1.2.3. Economy and Performance ................................................................................. 3
1.3. GFS Functions ................................................................................................................... 4
1.3.1. Cluster Volume Management.............................................................................. 4
1.3.2. Lock Management .............................................................................................. 5
1.3.3. Cluster Management, Fencing, and Recovery .................................................... 5
1.3.4. Cluster Configuration Management.................................................................... 5
1.4. GFS Software Subsystems.................................................................................................6
1.5. Before Configuring GFS .................................................................................................... 8
2. System Requirements ...................................................................................................................11
2.1. Platform Requirements .................................................................................................... 11
2.2. TCP/IP Network...............................................................................................................11
2.3. Fibre Channel Storage Network....................................................................................... 11
2.4. Fibre Channel Storage Devices........................................................................................12
2.5. Network Power Switches................................................................................................. 12
2.6. Console Access ................................................................................................................ 12
2.7. I/O Fencing ...................................................................................................................... 12
3. Installing System Software...........................................................................................................13
3.1. Prerequisite Tasks ............................................................................................................ 13
3.1.1. Net::Telnet Perl Module .......................................................................... 13
3.1.2. Clock Synchronization Software ...................................................................... 13
3.1.3. Stunnel Utility ................................................................................................ 14
3.2. Installation Tasks ............................................................................................................. 14
3.2.1. Installing a Linux Kernel .................................................................................. 14
3.2.2. Installing A GFS RPM...................................................................................... 15
3.2.3. Loading the GFS Kernel Modules ....................................................................16
4. Initial Configuration ..................................................................................................................... 19
4.1. Prerequisite Tasks ............................................................................................................ 19
4.2. Initial Configuration Tasks...............................................................................................19
4.2.1. Setting Up Logical Devices .............................................................................. 19
4.2.2. Setting Up and Starting the Cluster Configuration System .............................. 20
4.2.3. Starting Clustering and Locking Systems......................................................... 20
4.2.4. Setting Up and Mounting File Systems ............................................................ 21
Page 4
5. Using the Pool Volume Manager .................................................................................................23
5.1. Overview of GFS Pool Volume Manager ........................................................................ 23
5.2. Synopsis of Pool Management Commands ..................................................................... 24
5.2.1. pool_tool ....................................................................................................... 24
5.2.2. pool_assemble ..............................................................................................25
5.2.3. pool_info ....................................................................................................... 25
5.2.4. pool_mp ........................................................................................................... 26
5.3. Scanning Block Devices ..................................................................................................27
5.3.1. Usage................................................................................................................. 27
5.3.2. Example ............................................................................................................ 27
5.4. Creating a Configuration File for a New Volume ............................................................ 27
5.4.1. Examples........................................................................................................... 29
5.5. Creating a Pool Volume ................................................................................................... 29
5.5.1. Usage................................................................................................................. 29
5.5.2. Example ............................................................................................................ 30
5.5.3. Comments .........................................................................................................30
5.6. Activating/Deactivating a Pool Volume ........................................................................... 30
5.6.1. Usage................................................................................................................. 30
5.6.2. Examples........................................................................................................... 31
5.6.3. Comments .........................................................................................................31
5.7. Displaying Pool Configuration Information .................................................................... 31
5.7.1. Usage................................................................................................................. 31
5.7.2. Example ............................................................................................................ 31
5.8. Growing a Pool Volume ...................................................................................................32
5.8.1. Usage................................................................................................................. 32
5.8.2. Example procedure ........................................................................................... 32
5.9. Erasing a Pool Volume .....................................................................................................33
5.9.1. Usage................................................................................................................. 33
5.9.2. Example ............................................................................................................ 33
5.9.3. Comments .........................................................................................................33
5.10. Renaming a Pool Volume............................................................................................... 33
5.10.1. Usage...............................................................................................................34
5.10.2. Example .......................................................................................................... 34
5.11. Changing a Pool Volume Minor Number ...................................................................... 34
5.11.1. Usage...............................................................................................................34
5.11.2. Example .......................................................................................................... 35
5.11.3. Comments ....................................................................................................... 35
5.12. Displaying Pool Volume Information ............................................................................ 35
5.12.1. Usage...............................................................................................................35
5.12.2. Examples......................................................................................................... 36
5.13. Using Pool Volume Statistics ......................................................................................... 36
5.13.1. Usage...............................................................................................................36
5.13.2. Examples......................................................................................................... 36
5.14. Adjusting Pool Volume Multipathing ............................................................................ 37
5.14.1. Usage...............................................................................................................37
5.14.2. Examples......................................................................................................... 37
6. Creating the Cluster Configuration System Files ...................................................................... 39
6.1. Prerequisite Tasks ............................................................................................................ 39
6.2. CCS File Creation Tasks.................................................................................................. 39
6.3. Dual Power and Multipath FC Fencing Considerations .................................................. 40
6.4. GNBD Multipath Considerations ....................................................................................40
6.5. Adding the license.ccs File........................................................................................ 41
6.6. Creating the cluster.ccs File...................................................................................... 42
6.7. Creating the fence.ccs File .......................................................................................... 43
6.8. Creating the nodes.ccs File .......................................................................................... 52
Page 5
7. Using the Cluster Configuration System.....................................................................................71
7.1. Creating a CCS Archive................................................................................................... 71
7.1.1. Usage................................................................................................................. 71
7.1.2. Example ............................................................................................................ 71
7.1.3. Comments .........................................................................................................72
7.2. Starting CCS in the Cluster .............................................................................................. 72
7.2.1. Usage................................................................................................................. 72
7.2.2. Example ............................................................................................................ 72
7.2.3. Comments .........................................................................................................72
7.3. Using Other CCS Administrative Options....................................................................... 73
7.3.1. Extracting Files from a CCS Archive ............................................................... 73
7.3.2. Listing Files in a CCS Archive .........................................................................73
7.3.3. Comparing CCS Configuration Files to a CCS Archive................................... 74
7.4. Changing CCS Configuration Files ................................................................................. 74
7.4.1. Example Procedure ...........................................................................................74
7.5. Alternative Methods to Using a CCA Device.................................................................. 75
7.5.1. CCA File and Server .........................................................................................75
7.5.2. Local CCA Files ............................................................................................... 77
7.6. Combining CCS Methods ................................................................................................ 78
8. Using Clustering and Locking Systems ......................................................................................79
8.1. Locking System Overview............................................................................................... 79
8.2. LOCK_GULM................................................................................................................. 79
8.2.1. Selection of LOCK_GULM Servers................................................................. 79
8.2.2. Number of LOCK_GULM Servers .................................................................. 79
8.2.3. Starting LOCK_GULM Servers ....................................................................... 80
8.2.4. Fencing and LOCK_GULM .............................................................................80
8.2.5. Shutting Down a LOCK_GULM Server .......................................................... 80
8.3. LOCK_NOLOCK............................................................................................................ 81
9. Managing GFS .............................................................................................................................. 83
9.1. Making a File System ...................................................................................................... 83
9.1.1. Usage................................................................................................................. 83
9.1.2. Examples........................................................................................................... 84
9.1.3. Complete Options .............................................................................................84
9.2. Mounting a File System ................................................................................................... 85
9.2.1. Usage................................................................................................................. 85
9.2.2. Example ............................................................................................................ 86
9.2.3. Complete Usage ................................................................................................ 86
9.3. Unmounting a File System ..............................................................................................87
9.3.1. Usage................................................................................................................. 87
9.4. GFS Quota Management.................................................................................................. 87
9.4.1. Setting Quotas ...................................................................................................87
9.4.2. Displaying Quota Limits and Usage ................................................................. 88
9.4.3. Synchronizing Quotas ....................................................................................... 90
9.4.4. Disabling/Enabling Quota Enforcement ...........................................................91
9.4.5. Disabling/Enabling Quota Accounting ............................................................. 92
9.5. Growing a File System..................................................................................................... 93
9.5.1. Usage................................................................................................................. 93
9.5.2. Comments .........................................................................................................93
9.5.3. Examples........................................................................................................... 93
9.5.4. Complete Usage ................................................................................................ 94
9.6. Adding Journals to a File System .................................................................................... 94
9.6.1. Usage................................................................................................................. 94
9.6.2. Comments .........................................................................................................95
9.6.3. Examples........................................................................................................... 95
9.6.4. Complete Usage ................................................................................................ 95
Page 6
9.7. Direct I/O .........................................................................................................................96
9.7.1. O_DIRECT .........................................................................................................96
9.7.2. GFS File Attribute.............................................................................................97
9.7.3. GFS Directory Attribute ...................................................................................97
9.8. Data Journaling ................................................................................................................98
9.8.1. Usage................................................................................................................. 98
9.8.2. Examples........................................................................................................... 99
9.9. Configuring atime Updates ............................................................................................ 99
9.9.1. Mount with noatime........................................................................................ 99
9.9.2. Tune GFS atime Quantum ............................................................................ 100
9.10. Suspending Activity on a File System .........................................................................101
9.10.1. Usage.............................................................................................................101
9.10.2. Examples....................................................................................................... 101
9.11. Displaying Extended GFS Information and Statistics ................................................. 101
9.11.1. Usage.............................................................................................................102
9.11.2. Examples....................................................................................................... 102
9.12. Repairing a File System............................................................................................... 102
9.12.1. Usage.............................................................................................................102
9.12.2. Example ........................................................................................................ 103
9.13. Context-Dependent Path Names ..................................................................................103
9.13.1. Usage.............................................................................................................103
9.13.2. Example ........................................................................................................ 104
9.14. Shutting Down a GFS Cluster......................................................................................105
9.15. Restarting a GFS Cluster .............................................................................................105
10. Using the Fencing System......................................................................................................... 107
10.1. How the Fencing System Works.................................................................................. 107
10.2. Fencing Methods.......................................................................................................... 107
10.2.1. APC MasterSwitch........................................................................................ 108
10.2.2. WTI Network Power Switch......................................................................... 108
10.2.3. Brocade FC Switch ....................................................................................... 109
10.2.4. Vixel FC Switch ............................................................................................109
10.2.5. HP RILOE Card............................................................................................109
10.2.6. GNBD ...........................................................................................................110
10.2.7. Fence Notify GNBD ..................................................................................... 110
10.2.8. Manual .......................................................................................................... 110
11. Using GNBD ..............................................................................................................................113
11.1. Considerations for Using GNBD Multipath ................................................................ 113
11.1.1. Linux Page Caching ...................................................................................... 113
11.1.2. Lock Server Startup ......................................................................................113
11.1.3. CCS File Location ........................................................................................ 113
11.1.4. Fencing GNBD Server Nodes.......................................................................114
11.2. GNBD Driver and Command Usage ...........................................................................114
11.2.1. Exporting a GNBD from a Server ................................................................ 115
11.2.2. Importing a GNBD on a Client ..................................................................... 116
12. Software License .......................................................................................................................119
12.1. Overview...................................................................................................................... 119
12.2. Installing a License ...................................................................................................... 119
12.3. Upgrading and Replacing a License ............................................................................ 120
12.3.1. Replacing a License File: Not Changing the Lock Manager ........................ 120
12.3.2. Replacing a License File: Changing the Lock Manager ............................... 120
12.4. License FAQ................................................................................................................. 121
12.5. Solving License Problems............................................................................................ 121
Page 7
A. Upgrading GFS ..........................................................................................................................123
A.1. Overview of Differences between GFS 5.1.x and GFS 5.2.x ....................................... 123
A.1.1. Configuration Information .............................................................................123
A.1.2. GFS License................................................................................................... 124
A.1.3. GFS LockProto ..............................................................................................124
A.1.4. GFS LockTable .............................................................................................. 124
A.1.5. GFS Mount Options....................................................................................... 124
A.1.6. Pool ................................................................................................................ 125
A.1.7. Fencing Agents .............................................................................................. 125
A.2. Upgrade Procedure........................................................................................................125
B. Basic GFS Examples ..................................................................................................................131
B.1. LOCK_GULM, RLM Embedded .................................................................................131
B.1.1. Key Characteristics......................................................................................... 131
B.1.2. Prerequisites ...................................................................................................132
B.1.3. Setup Process.................................................................................................. 133
B.2. LOCK_GULM, RLM External..................................................................................... 137
B.2.1. Key Characteristics......................................................................................... 137
B.2.2. Prerequisites ...................................................................................................139
B.2.3. Setup Process.................................................................................................. 140
B.3. LOCK_GULM, SLM Embedded.................................................................................. 145
B.3.1. Key Characteristics......................................................................................... 145
B.3.2. Prerequisites ...................................................................................................146
B.3.3. Setup Process.................................................................................................. 147
B.4. LOCK_GULM, SLM External ..................................................................................... 151
B.4.1. Key Characteristics......................................................................................... 151
B.4.2. Prerequisites ...................................................................................................152
B.4.3. Setup Process.................................................................................................. 153
B.5. LOCK_GULM, SLM External, and GNBD ................................................................. 157
B.5.1. Key Characteristics......................................................................................... 157
B.5.2. Prerequisites ...................................................................................................159
B.5.3. Setup Process.................................................................................................. 160
B.6. LOCK_NOLOCK ......................................................................................................... 165
B.6.1. Key Characteristics......................................................................................... 165
B.6.2. Prerequisites ...................................................................................................166
B.6.3. Setup Process.................................................................................................. 167
Index................................................................................................................................................. 171
Colophon..........................................................................................................................................177
Page 8
Page 9
Introduction
Welcome to the Red Hat GFS Administrator’s Guide. This book provides information about installing,
configuring, and maintaining GFS (Global File System). The document contains procedures for commonly performed tasks, reference information, and examples of complex operations and tested GFS
configurations.
HTML and PDF versions of all the official Red Hat Enterprise Linux manuals and release notes are
available online at http://www.redhat.com/docs/.
1. Audience
This book is intended primarily for Linux system administrators who are familiar with the following
activities:
• Linux system administration procedures, including kernel configuration
• Installation and configuration of shared storage networks, such as Fibre Channel SANs
2. Document Conventions
When you read this manual, certain words are represented in different fonts, typefaces, sizes, and
weights. This highlighting is systematic; different words are represented in the same style to indicate
their inclusion in a specific category. The types of words that are represented this way include the
following:
command
Linux commands (and other operating system commands, when used) are represented this way.
This style should indicate to you that you can type the word or phrase on the command line
and press [Enter] to invoke a command. Sometimes a command contains words that would be
displayed in a different style on their own (such as file names). In these cases, they are considered
to be part of the command, so the entire phrase is displayed as a command. For example:
Use the cat testfile command to view the contents of a file, named testfile, in the current
working directory.
file name
File names, directory names, paths, and RPM package names are represented this way. This style
should indicate that a particular file or directory exists by that name on your system. Examples:
The .bashrc file in your home directory contains bash shell definitions and aliases for your own
use.
The /etc/fstab file contains information about different system devices and file systems.
Install the webalizer RPM if you want to use a Web server log file analysis program.
application
This style indicates that the program is an end-user application (as opposed to system software).
For example:
Use Mozilla to browse the Web.
Page 10
ii Introduction
[key]
A key on the keyboard is shown in this style. For example:
To use [Tab] completion, type in a character and then press the [Tab] key. Your terminal displays
the list of files in the directory that start with that letter.
[key]-[combination]
A combination of keystrokes is represented in this way. For example:
The [Ctrl]-[Alt]-[Backspace] key combination exits your graphical session and return you to the
graphical login screen or the console.
text found on a GUI interface
A title, word, or phrase found on a GUI interface screen or window is shown in this style. Text
shown in this style is being used to identify a particular GUI screen or an element on a GUI
screen (such as text associated with a checkbox or field). Example:
Select the Require Password checkbox if you would like your screensaver to require a password
before stopping.
top level of a menu on a GUI screen or window
A word in this style indicates that the word is the top level of a pulldown menu. If you click on
the word on the GUI screen, the rest of the menu should appear. For example:
Under File on a GNOME terminal, the New Tab option allows you to open multiple shell
prompts in the same window.
If you need to type in a sequence of commands from a GUI menu, they are shown like the
following example:
Go to Main Menu Button (on the Panel) => Programming => Emacs to start the Emacs text
editor.
button on a GUI screen or window
This style indicates that the text can be found on a clickable button on a GUI screen. For example:
Click on the Back button to return to the webpage you last viewed.
computer output
Text in this style indicates text displayed to a shell prompt such as error messages and responses
to commands. For example:
The ls command displays the contents of a directory. For example:
Desktop about.html logs paulwesterberg.png
Mail backupfiles mail reports
The output returned in response to the command (in this case, the contents of the directory) is
shown in this style.
prompt
A prompt, which is a computer’s way of signifying that it is ready for you to input something, is
shown in this style. Examples:
$
#
[stephen@maturin stephen]$
leopard login:
Page 11
Introduction iii
user input
Text that the user has to type, either on the command line, or into a text box on a GUI screen, is
displayed in this style. In the following example, text is displayed in this style:
To boot your system into the text based installation program, you must type in the text command at the boot: prompt.
replaceable
Text used for examples which is meant to be replaced with data provided by the user is displayed
in this style. In the following example,
version-numberis displayed in this style:
The directory for the kernel source is /usr/src/
version-number/, where
version-numberis the version of the kernel installed on this system.
Additionally, we use several different strategies to draw your attention to certain pieces of information. In order of how critical the information is to your system, these items are marked as note, tip,
important, caution, or a warning. For example:
Note
Remember that Linux is case sensitive. In other words, a rose is not a ROSE is not a rOsE.
Tip
The directory /usr/share/doc/ contains additional documentation for packages installed on your
system.
Important
If you modify the DHCP configuration file, the changes will not take effect until you restart the DHCP
daemon.
Caution
Do not perform routine tasks as root — use a regular user account unless you need to use the root
account for system administration tasks.
Warning
Be careful to remove only the necessary partitions. Removing other partitions could result in data
loss or a corrupted system environment.
Page 12
iv Introduction
3. More to Come
The Red Hat GFS Administrator’s Guide is part of Red Hat’s growing commitment to provide useful
and timely support to Red Hat Enterprise Linux users.
3.1. Send in Your Feedback
If you spot a typo in the Red Hat GFS Administrator’s Guide, or if you have thought of a way
to make this manual better, we would love to hear from you! Please submit a report in Bugzilla
(http://www.redhat.com/bugzilla) against the component rh-gfsg.
Be sure to mention the manual’s identifier:
Red Hat GFS Administrator’s Guide(EN)-5.2.1-Print-RHI (2004-01-15T12:12-0400)
If you mention this manual’s identifier, we will know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If you
have found an error, please include the section number and some of the surrounding text so we can
find it easily.
4. Sign Up for Support
If you have a variant of Red Hat GFS 5.2.1, please remember to sign up for the benefits you are
entitled to as a Red Hat customer.
Registration enables access to the Red Hat Services you have purchased, such as technical support
and Red Hat Network. To register your product, go to:
http://www.redhat.com/apps/activate/
Note
You must activate your product before attempting to connect to Red Hat Network. If your product
has not been activated, Red Hat Network rejects registration to channels to which the system is not
entitled.
Good luck, and thank you for choosing Red Hat GFS!
The Red Hat Documentation Team
5. Recommended References
For additional references about related topics, refer to the following table:
Page 13
Introduction v
Topic Reference Comment
Shared Data Clustering and
File Systems
Shared Data Clusters by Dilip
M. Ranade. Wiley, 2002.
Provides detailed technical
information on cluster file
system and cluster volume
manager design.
Storage Area Networks (SANs) Designing Storage Area
Networks: A Practical
Reference for Implementing
Fibre Channel and IP SANs,
Second Edition by Tom Clark.
Addison-Wesley, 2003.
Provides a concise summary of
Fibre Channel and IP SAN
Technology.
Building SANs with Brocade
Fabric Switches by C.
Beauchamp, J. Judd, and B.
Keo. Syngress, 2001.
Best practices for building
Fibre Channel SANs based on
the Brocade family of switches,
including core-edge topology
for large SAN fabrics.
Building Storage Networks,
Second Edition by Marc Farley.
Osborne/McGraw-Hill, 2001.
Provides a comprehensive
overview reference on storage
networking technologies.
Applications and High
Availability
Blueprints for High
Availability: Designing
Resilient Distributed Systems
by E. Marcus and H. Stern.
Wiley, 2000.
Provides a summary of best
practices in high availability.
Table 1. Recommended References Table
Page 14
vi Introduction
Page 15
Chapter 1.
GFS Overview
GFS is a cluster file system that provides data sharing among Linux-based computers. GFS provides
a single, consistent view of the file system name space across all nodes in a cluster. It allows applications to install and run without much knowledge of the underlying storage infrastructure. GFS is
fully compliant with the IEEE POSIX interface, allowing applications to perform file operations as if
they were running on a local file system. Also, GFS provides features that are typically required in
enterprise environments, such as quotas, multiple journals, and multipath support.
GFS provides a versatile method of networking your storage according to the performance, scalability,
and economic needs of your storage environment.
This chapter provides some very basic, abbreviated information as background to help you understand
GFS. It contains the following sections:
• Section 1.1 New and Changed Features
• Section 1.2 Performance, Scalability, and Economy
• Section 1.3 GFS Functions
• Section 1.4 GFS Software Subsystems
• Section 1.5 Before Configuring GFS
1.1. New and Changed Features
New for this release is GNBD (Global Network Block Device) multipath. GNBD multipath configuration of multiple GNBD server nodes (nodes that export GNBDs to GFS nodes) with redundant
paths between the GNBD server nodes and storage devices. The GNBD servers, in turn, present multiple storage paths to GFS nodes via redundant GNBDs. With GNBD multipath, if a GNBD server
node becomes unavailable, another GNBD server node can provide GFS nodes with access to storage
devices.
With GNBD multipath, you need to take into account additional factors — one being that Linux page
caching needs to be disabled on GNBD servers in a GNBD multipath configuration. For information
about CCS files with GNBD multipath, refer to Section 6.4 GNBD Multipath Considerations. For
information about using GNBD with GNBD multipath, refer to Section 11.1 Considerations for Using
GNBD Multipath.
For upgrade instructions, refer to Appendix A Upgrading GFS.
1.2. Performance, Scalability, and Economy
You can deploy GFS in a variety of configurations to suit your needs for performance, scalability, and
economy. For superior performance and scalability, you can deploy GFS in a cluster that is connected
directly to a SAN. For more economical needs, you can deploy GFS in a cluster that is connected to a
LAN with servers that use the GFS VersaPlex architecture. The VersaPlex architecture allows a GFS
cluster to connect to servers that present block-level storage via an Ethernet LAN. The VersaPlex
architecture is implemented with GNBD (Global Network Block Device), a method of presenting
block-level storage over an Ethernet LAN. GNBD is a software layer that can be run on network
nodes connected to direct-attached storage or storage in a SAN. GNBD exports a block interface from
those nodes to a GFS cluster.
Page 16
2 Chapter 1. GFS Overview
You can configure GNBD servers for GNBD multipath. GNBD multipath allows you to configure
multiple GNBD server nodes with redundant paths between the GNBD server nodes and storage devices. The GNBD servers, in turn, present multiple storage paths to GFS nodes via redundant GNBDs.
With GNBD multipath, if a GNBD server node becomes unavailable, another GNBD server node can
provide GFS nodes with access to storage devices.
The following sections provide examples of how GFS can be deployed to suit your needs for performance, scalability, and economy:
• Section 1.2.1 Superior Performance and Scalability
• Section 1.2.2 Performance, Scalability, Moderate Price
• Section 1.2.3 Economy and Performance
Note
The deployment examples in this chapter reflect basic configurations; your needs might require a
combination of configurations shown in the examples. Also, the examples show GNBD (Global Network Block Device) as the method of implementing the VersaPlex architecture.
1.2.1. Superior Performance and Scalability
You can obtain the highest shared-file performance when applications access storage directly. The
GFS SAN configuration in Figure 1-1 provides superior file performance for shared files and file
systems. Linux applications run directly on GFS clustered application nodes. Without file protocols
or storage servers to slow data access, performance is similar to individual Linux servers with directconnect storage; yet, each GFS application node has equal access to all data files. GFS supports over
300 GFS application nodes.
SAN
Fabric
FC or iSCSI
SAN
GFS
Applications
Shared Files
Figure 1-1. GFS with a SAN
Page 17
Chapter 1. GFS Overview 3
1.2.2. Performance, Scalability, Moderate Price
Multiple Linux client applications on a LAN can share the same SAN-based data as shown in Figure
1-2. SAN block storage is presented to network clients as block storage devices by GNBD servers.
From the perspective of a client application, storage is accessed as if it were directly attached to the
server in which the application is running. Stored data is actually on the SAN. Storage devices and data
can be equally shared by network client applications. File locking and sharing functions are handled
by GFS for each network client.
Note
Clients implementing ext2 and ext3 file systems can be configured to access their own dedicated
slice of SAN storage.
GFS with VersaPlex (implemented with GNBD, as shown in Figure 1-2) and a SAN provide fully
automatic application and device failover with failover software and redundant devices.
LAN
Clients
GNBD
servers
SAN
Fabric
GFS
Applications
Shared Files
Figure 1-2. GFS and GNBD with a SAN
Page 18
4 Chapter 1. GFS Overview
1.2.3. Economy and Performance
Figure 1-3 shows how Linux client applications can take advantage of an existing Ethernet topology
to gain shared access to all block storage devices. Client data files and file systems can be shared with
GFS on each client. Application and device failover can be fully automated with mirroring, failover
software, and redundant devices.
LAN
Clients
GNBD
servers
Disk
A
Mirrored disk pairs for data availability
GFS
Applications
1
Disk
C
Disk
B
1
Disk
A
Disk
C
1
Disk
B
Shared Files
Figure 1-3. GFS and GNBD with Direct-Attached Storage
1.3. GFS Functions
GFS is a native file system that interfaces directly with the VFS layer of the Linux kernel file-system
interface. GFS is a cluster file system that employs distributed metadata and multiple journals for
optimal operation in a cluster.
GFS provides the following main functions:
• Cluster volume management
• Lock management
• Cluster management, fencing, and recovery
• Cluster configuration management
Page 19
Chapter 1. GFS Overview 5
1.3.1. Cluster Volume Management
Cluster volume management provides simplified management of volumes and the ability to dynamically extend file system capacity without interrupting file-system access. With cluster volume management, you can aggregate multiple physical volumes into a single, logical device across all nodes in
a cluster.
Cluster volume management provides a logical view of the storage to GFS, which provides flexibility
for the administrator in how the physical storage is managed. Also, cluster volume management provides increased availability because it allows increasing the storage capacity without shutting down
the cluster. Refer to Chapter 5 Using the Pool Volume Manager for more information about cluster
volume management.
1.3.2. Lock Management
A lock management mechanism is a key component of any cluster file system. The GFS OmniLock
architecture provides the following lock managers:
• Single Lock Manager (SLM) — A simple centralized lock manager that can be configured to run
either on a file system node or on a separate dedicated lock manager node.
• Redundant Lock Manager (RLM) — A high-availability lock manager. It allows the configuration
of a master and multiple hot-standby failover lock manager nodes. The failover nodes provide
failover in case the master lock manager node fails.
The lock managers also provide cluster management functions that control node recovery. Refer to
Chapter 8 Using Clustering and Locking Systems for a description of the GFS lock protocols.
1.3.3. Cluster Management, Fencing, and Recovery
Cluster management functions in GFS monitor node status through heartbeat signals to determine
cluster membership. Also, cluster management keeps track of which nodes are using each GFS file
system, and initiates and coordinates the recovery process when nodes fail. This process involves
recovery coordination from the fencing system, the lock manager, and the file system. The cluster
management functions are embedded in each of the lock management modules described earlier in
Lock Management. Refer to Chapter 8 Using Clustering and Locking Systems for more information
on cluster management.
Fencing is the ability to isolate or "fence off" a cluster node when that node loses its heartbeat notification with the rest of the cluster nodes. Fencing ensures that data integrity is maintained during
the recovery of a failed cluster node. GFS supports a variety of automated fencing methods and one
manual method. In addition, GFS provides the ability to configure each cluster node for cascaded
fencing with the automated fencing methods. Refer to Chapter 10 Using the Fencing System for more
information about the GFS fencing capability.
Recovery is the process of controlling reentry of a node into a cluster after the node has been fenced.
Recovery ensures that storage data integrity is maintained in the cluster while the previously fenced
node is reentering the cluster. As stated earlier, recovery involves coordination from fencing, lock
management, and the file system.
1.3.4. Cluster Configuration Management
Cluster configuration management provides a centralized mechanism for the configuration and
maintenance of configuration files throughout the cluster. It provides high-availability access to
configuration-state information for all nodes in the cluster.
Page 20
6 Chapter 1. GFS Overview
For information about cluster configuration management refer to Chapter 6 Creating the Cluster Configuration System Files and Chapter 7 Using the Cluster Configuration System.
1.4. GFS Software Subsystems
GFS consists of the following subsystems:
• Cluster Configuration System (CCS)
• Fence
• Pool
• LOCK_GULM
• LOCK_NOLOCK
Table 1-1 summarizes the GFS Software subsystems and their components.
Software Subsystem Components Description
Cluster Configuration
System (CCS)
ccs_tool Command used to create CCS archives.
ccs_read Diagnostic and testing command that is
used to retrieve information from
configuration files through ccsd.
ccsd CCS daemon that runs on all cluster nodes
and provides configuration file data to
cluster software.
ccs_servd CCS server daemon that distributes CCS
data from a single server to ccsd daemons
when a shared device is not used for storing
CCS data.
Fence fence_node Command used by lock_gulmd when a
fence operation is required. This command
takes the name of a node and fences it
based on the node’s fencing configuration.
fence_apc Fence agent for APC power switch.
fence_wti Fence agent for WTI power switch.
fence_brocade Fence agent for Brocade Fibre Channel
switch.
fence_vixel Fence agent for Vixel Fibre Channel switch.
fence_rib Fence agent for RIB card.
fence_gnbd Fence agent used with GNBD storage.
fence_notify_gnbd Fence agent for notifying GFS nodes about
fenced GNBD server nodes.
fence_manual Fence agent for manual interaction.
Page 21
Chapter 1. GFS Overview 7
Software Subsystem Components Description
fence_ack_manual User interface for fence_manual agent.
Pool pool.o Kernel module implementing the pool
block-device driver.
pool_assemble Command that activates and deactivates
pool volumes.
pool_tool Command that configures pool volumes
from individual storage devices.
pool_info Command that reports information about
system pools.
pool_grow Command that expands a pool volume.
pool_mp Command that manages pool multipathing.
LOCK_GULM lock_gulm.o Kernel module that is installed on GFS
nodes using the LOCK_GULM lock
module.
lock_gulmd Server/daemon that runs on one or more
nodes and communicates with all the nodes
using LOCK_GULM GFS file systems.
gulm_tool Command that configures and debugs the
lock_gulmd server.
LOCL_NOLOCK lock_nolock.o Kernel module installed on a node using
GFS as a local file system.
GFS gfs.o Kernel module that implements the GFS file
system and is loaded on GFS cluster nodes.
lock_harness.o Kernel module that implements the GFS
lock harness into which GFS lock modules
can plug.
gfs_mkfs Command that creates a GFS file system on
a storage device.
gfs_tool Command that configures or tunes a GFS
file system. This command can also gather
a variety of information about the file
system.
gfs_quota Command that manages quotas on a
mounted GFS file system.
gfs_grow Command that grows a mounted GFS file
system.
gfs_jadd Command that adds journals to a mounted
GFS file system.
gfs_fsck Command that repairs an unmounted GFS
file system.
gfs_mount Command that can be optionally used to
mount a GFS file system.
Page 22
8 Chapter 1. GFS Overview
Software Subsystem Components Description
License license_tool Command that is used to check a license
file.
GNBD gnbd.o Kernel module that implements the GNBD
blade-device driver on clients.
gnbd_serv.o Kernel module that implements the GNBD
server. It allows a node to export local
storage over the network.
gnbd_export Command to create, export and manage
GNBDs on a GNBD server.
gnbd_import Command to import and manage GNBDs
on a GNBD client.
Upgrade gfs_conf Command that retrieves from a cidev
configuration information from earlier
versions of GFS.
Table 1-1. GFS Software Subsystem Components
1.5. Before Configuring GFS
Before you install and configure GFS, note the following key characteristics of your GFS configuration:
Cluster name
Determine a cluster name for your GFS cluster. The cluster name is required in the form of a parameter variable, ClusterName, later in this book.The cluster name can be 1 to 16 characters long.
For example, this book uses a cluster name alpha in some example configuration procedures.
Number of file systems
Determine how many GFS file systems to create initially. (More file systems can be added later.)
File system name
Determine a unique name for each file system. Each file system name is required in the form of
a parameter variable, FSName, later in this book. For example, this book uses file system names
gfs1 and gfs2 in some example configuration procedures.
Number of nodes
Determine how many nodes will mount the file systems. The hostnames and IP addresses of all
nodes will be used.
LOCK_GULM server daemons
Determine the number of LOCK_GULM server daemons to use and on which GFS nodes the
server daemons will run. Multiple LOCK_GULM server daemons (only available with RLM)
provide redundancy. RLM requires a minimum of three nodes, but no more than five nodes. Information about LOCK_GULM server daemons is required for a CCS (Cluster Configuration
System) file, cluster.ccs. Refer to Section 6.6 Creating the cluster.ccs File for information about the cluster.ccs file.
Page 23
Chapter 1. GFS Overview 9
GNBD server nodes
If you are using GNBD, determine how many nodes are needed. The hostname and IP address
of all GNBD server nodes are used.
Fencing method
Determine the fencing method for each GFS node. If you are using GNBD multipath, determine
the fencing method for each GNBD server node (node that exports GNBDs to GFS nodes).
Information about fencing methods is required later in this book for the CCS files, fence.ccs
and nodes.ccs.(Refer to Section 6.7 Creating the fence.ccs File and Section 6.8 Creating the
nodes.ccs File for more information.) To help determine the type of fencing methods available
with GFS, refer to Chapter 10 Using the Fencing System. When using RLM, you must use a
fencing method that shuts down and reboots the node being fenced.
Storage devices and partitions
Determine the storage devices and partitions to be used for creating pool volumes in the file systems. Make sure to account for space on one or more partitions for storing cluster configuration
information as follows: 2 KB per GFS node or 2 MB total, whichever is larger.
Page 24
10 Chapter 1. GFS Overview
Page 25
Chapter 2.
System Requirements
This chapter describes the system requirements for Red Hat GFS 5.2.1 and consists of the following
sections:
• Section 2.1 Platform Requirements
• Section 2.2 TCP/IP Network
• Section 2.3 Fibre Channel Storage Network
• Section 2.4 Fibre Channel Storage Devices
• Section 2.5 Network Power Switches
• Section 2.6 Console Access
• Section 2.7 I/O Fencing
2.1. Platform Requirements
Table 2-1 shows the platform requirements for GFS.
Operating System Hardware Architecture RAM
RHEL3 AS, ES, WS ia64, x86-64, x86
SMP supported
256 MB, minimum
RHEL2.1 AS x86, ia64
SMP supported
256 MB, minimum
SLES8 x86
SMP supported
256 MB, minimum
SuSE9 x86-64
SMP supported
256 MB, minimum
Table 2-1. Platform Requirements
2.2. TCP/IP Network
All GFS nodes must be connected to a TCP/IP network. Network communication is critical to the
operation of the GFS cluster, specifically to the clustering and locking subsystems. For optimal performance and security, it is recommended that a private, dedicated, switched network be used for GFS.
GFS subsystems do not use dual-network interfaces for failover purposes.
2.3. Fibre Channel Storage Network
Table 2-2 shows requirements for GFS nodes that are to be connected to a Fibre Channel SAN.
Page 26
12 Chapter 2. System Requirements
Requirement Description
HBA (Host Bus Adapter) One HBA minimum per GFS node
Connection method Fibre Channel switch
Note: If an FC switch is used for I/O fencing nodes, you
may want to consider using Brocade and Vixel FC switches,
for which GFS fencing agents exist.
Note: When a small number of nodes is used, it may be
possible to connect the nodes directly to ports on the
storage device.
Note: FC drivers may not work reliably with FC hubs.
Table 2-2. Fibre Channel Network Requirements
2.4. Fibre Channel Storage Devices
Table 2-3 shows requirements for Fibre Channel devices that are to be connected to a GFS cluster.
Requirement Description
Device Type FC RAID array or JBOD
Note: Make sure that the devices can operate reliably when
heavily accessed simultaneously from multiple initiators.
Note: Make sure that your GFS configuration does not
exceed the number of nodes an array or JBOD supports.
Size 2 TB maximum, for total of all storage connected to a GFS
cluster. Linux 2.4 kernels do not support devices larger than
2 TB; therefore, the total size of storage available to GFS
cannot exceed 2 TB.
Table 2-3. Fibre Channel Storage Device Requirements
2.5. Network Power Switches
GFS provides fencing agents for APC and WTI network power switches.
2.6. Console Access
Make sure that you have console access to each GFS node. Console access to each node ensures that
you can monitor the cluster and troubleshoot kernel problems.
2.7. I/O Fencing
You need to configure each node in your GFS cluster for at least one form of I/O fencing. For more
information about fencing options, refer to Section 10.2 Fencing Methods.
Page 27
Chapter 3.
Installing System Software
Installing system software consists of installing a Linux kernel and the corresponding GFS software
into each node of your GFS cluster. This chapter explains how to install system software and includes
the following sections:
• Section 3.1 Prerequisite Tasks
• Section 3.2 Installation Tasks
• Section 3.2.1 Installing a Linux Kernel
• Section 3.2.2 Installing A GFS RPM
• Section 3.2.3 Loading the GFS Kernel Modules
3.1. Prerequisite Tasks
Before installing GFS software, make sure that you have noted the key characteristics of your GFS
configuration (refer to Section 1.5 Before Configuring GFS). Make sure that you have installed the
following software:
• Net::Telnet Perl module
• Clock synchronization software
• Stunnel utility (optional)
3.1.1. Net::Telnet Perl Module
The Net::Telnet Perl module is used by several fencing agents and should be installed on all
GFS nodes. The Net::Telnet Perl module should be installed before installing GFS; otherwise,
GFS will not install.
You can download the Net::Telnet Perl module and installation instructions for the module from
the Comprehensive Perl Archive Network (CPAN) website at http://www.cpan.org.
3.1.2. Clock Synchronization Software
Make sure that each GFS node is running clock synchronization software. The system clocks in GFS
nodes need to be within a few minutes of each other for the following reasons:
• Avoiding unnecessary inode time-stamp updates — If the node clocks are not synchronized, the
inode time stamps will be updated unnecessarily, severely impacting cluster performance. Refer to
Section 9.9 Configuring atime Updates for additional information.
• Maintaining license validity — If system times are incorrect, the license may be considered invalid.
Page 28
14 Chapter 3. Installing System Software
Note
One example of time synchronization software is the Network Time Protocol (NTP) software. You can
find more information about NTP at http://www.ntp.org.
3.1.3. Stunnel Utility
The Stunnel utility needs to be installed only on nodes that use the HP RILOE PCI card for I/O fencing. (For more information about fencing with the HP RILOE card, refer to HP RILOE Card on page
147.) Verify that the utility is installed on each of those nodes by looking for /usr/sbin/stunnel.
If you need the Stunnel utility, download it from the Stunnel website at http://www.stunnel.org.
3.2. Installation Tasks
To install system software, perform the following steps:
1. Install a Linux kernel.
Note
This step is required only if you are using Red Hat Enterprise Linux Version 3, Update 1 or earlier,
or if you are using Red Hat Linux Version 2.1.
2. Install the GFS RPM.
3. Load the GFS kernel modules.
3.2.1. Installing a Linux Kernel
Note
This step is required only if you are using Red Hat Enterprise Linux Version 3, Update 1 or earlier, or
if you are using Red Hat Linux Version 2.1.
Installing a Linux kernel consists of downloading and installing a binary kernel built with GFS patches
applied and, optionally, compiling third-party drivers not included with the kernel (for example, HBA
drivers) against that kernel. The GFS kernels have been built by applying GFS patches against vendor
kernels, fully enabling all GFS features.
To install a Linux kernel, follow these steps:
Page 29
Chapter 3. Installing System Software 15
Step Action Comment
1. Acquire the appropriate binary
GFS kernel. Copy or download the
appropriate RPM file for each
GFS node.
For example:
kernel-gfs-smp-2.4.21-9.0.1.EL.i686.rpm
Make sure that the file is appropriate for the
hardware and libraries of the computer on which the
kernel will be installed.
2. At each node, install the new
kernel by issuing the following
rpm command and variable:
rpm -ivh GFSKernelRPM
GFSKernelRPM = GFS kernel RPM file.
3. At each node, issue an rpm
command to check the kernel
level.
For example:
rpm -q kernel-gfs
Verify that the correct kernel has been installed.
4. At each node, make sure that the
boot loader is configured to load
the new kernel.
For example, check the GRUB menu.lst file.
5. Reboot each node into the new
kernel.
6. At each node, verify that the
node is running the the
appropriate kernel as follows:
uname -r
If you need to compile third-party
drivers against the kernel, proceed to the next step. If not, proceed to Section 3.2.2 Installing A
GFS RPM.
Displays the current release level of the kernel
running.
7. Acquire the tar file from RHN
containing the GFS kernel
patches:
GFS_kpatches-5.2.1.tgz
The GFS_kpatches-5.2.1.tgz file contains GFS
kernel patches and patch instructions for supported
kernels.
8. Extract a patch-instructions file
from
GFS_kpatches-5.2.1.tgz
For example:
rh_kernel_patch.txt
Follow the patch instructions and check which kernel
versions are supported.
Table 3-1. Installing a Linux Kernel
Page 30
16 Chapter 3. Installing System Software
3.2.2. Installing A GFS RPM
Installing a GFS RPM consists of acquiring the appropriate GFS software and installing it.
Note
Only one instance of a GFS RPM can be installed on a node. If a GFS RPM is present on a node,
that RPM must be removed before installing another GFS RPM. To remove a GFS RPM, you can use
the rpm command with its erase option (-e).
Tip
GNBD modules are included with the GFS RPM.
To install a GFS RPM, follow these steps:
Step Action Comment
1. Acquire the appropriate GFS software and
copy or download it to each GFS node.
Make sure to select the GFS software that
matches the kernel selected in Section 3.2.1
Installing a Linux Kernel.
2. At each node, issue the following rpm
command:
rpm -ivh GFSRPM
Installs the GFS software. GFSRPM = GFS
RPM file.
3. At each node, issue the following rpm
command to check the GFS version:
rpm -qa | grep GFS
Verify that the GFS software has been
installed. This lists the GFS software
installed in the previous step.
Table 3-2. Installing a GFS RPM
3.2.3. Loading the GFS Kernel Modules
Once the GFS software has been installed on the GFS nodes, the following GFS kernel modules need
to be loaded into the running kernel before GFS can be set up and used:
• ccs.o
• pool.o
• lock_harness.o
• locl_gulm.o
• gfs.o
Page 31
Chapter 3. Installing System Software 17
Note
The GFS kernel modules need to be loaded every time a GFS node is started. It is recommended
that you use a startup script to automate loading the GFS kernel modules.
Note
The procedures in this section are for a GFS configuration that uses LOCK_GULM. If you are using
LOCK_NOLOCK, refer to Appendix B Basic GFS Examples for information about which GFS kernel
modules you should load.
Note
When loading the GFS kernel modules you may see warning messages indicating missing or nonGPL licenses. The messages are not related to the GFS license and can be ignored.
To load the GFS kernel modules, follow the steps in Table 3-3. The steps in Table 3-3 use modprobe
commands to load the GFS kernel modules; all dependent modules will be loaded automatically.
Alternatively, you can load the GFS kernel modules by following the steps in Table 3-4 . The steps in
Table 3-4 use insmod commands to load the GFS kernel modules.
Step Command Description
1. depmod -a Run this only once after RPMs are installed.
2. modprobe pool Loads pool.o and dependent files.
3. modprobe lock_gulm Loads lock_gulm.o and dependent files.
4. modprobe gfs Loads gfs.o and dependent files.
5. lsmod Verifies that all GFS kernel modules are loaded. This
shows a listing of currently loaded modules. It
should display all the modules loaded in the previous
steps and other system kernel modules.
Table 3-3. Loading GFS Kernel Modules
Step Command Description
1. insmod ccs Loads the ccs.o module.
2. insmod pool Loads the pool.o module.
3. insmod lock_harness Loads the lock_harness.o module.
Page 32
18 Chapter 3. Installing System Software
Step Command Description
4. insmod lock_gulm Loads the lock_gulm.o module.
5. insmod gfs Loads the gfs.o module.
6. lsmod Verifies that all GFS kernel modules are loaded. This
shows a listing of currently loaded modules. It
should display all the modules loaded in the previous
steps and other system kernel modules.
Table 3-4. (Alternate Method) Loading Kernel Modules
Page 33
Chapter 4.
Initial Configuration
This chapter describes procedures for initial configuration of GFS and contains the following sections:
• Section 4.1 Prerequisite Tasks
• Section 4.2 Initial Configuration Tasks
• Section 4.2.1 Setting Up Logical Devices
• Section 4.2.2 Setting Up and Starting the Cluster Configuration System
• Section 4.2.3 Starting Clustering and Locking Systems
• Section 4.2.4 Setting Up and Mounting File Systems
4.1. Prerequisite Tasks
Before setting up the GFS software, make sure that you have noted the key characteristics of your GFS
configuration (refer to Section 1.5 Before Configuring GFS) and that you have installed the kernel and
the GFS software into each GFS node (refer to Chapter 3 Installing System Software).
4.2. Initial Configuration Tasks
Initial configuration consists of the following tasks:
1. Setting up logical devices (pools).
2. Setting up and starting the Cluster Configuration System (CCS).
3. Starting clustering and locking systems.
4. Setting up and mounting file systems.
Note
GFS kernel modules must be loaded prior to performing initial configuration tasks. Refer to Section
3.2.3 Loading the GFS Kernel Modules.
Note
For examples of GFS configurations, refer to Appendix B Basic GFS Examples.
The following sections describe the initial configuration tasks.
Page 34
20 Chapter 4. Initial Configuration
4.2.1. Setting Up Logical Devices
To set up logical devices (pools) follow these steps:
1. Create file system pools.
a. Create pool configuration files. Refer to Section 5.4 Creating a Configuration File for a
New Volume.
b. Create a pool for each file system. Refer to Section 5.5 Creating a Pool Volume.
Command usage:
pool_tool -c ConfigFile
2. Create a CCS pool.
a. Create pool configuration file. Refer to Section 5.4 Creating a Configuration File for a
New Volume.
b. Create a pool to be the Cluster Configuration Archive (CCA) device. Refer to Section 5.5
Creating a Pool Volume.
Command usage:
pool_tool -c ConfigFile
3. At each node, activate pools. Refer to Section 5.6 Activating/Deactivating a Pool Volume.
Command usage:
pool_assemble
4.2.2. Setting Up and Starting the Cluster Configuration System
To set up and start the Cluster Configuration System, follow these steps:
1. Create CCS configuration files and place them into a temporary directory. Refer to Chapter 6
Creating the Cluster Configuration System Files.
2. Create a CCS archive on the CCA device. (The CCA device is the pool created in Step 2 of
Section 4.2.1 Setting Up Logical Devices.) Put the CCS files (created in Step 1) into the CCS
archive. Refer to Section 7.1 Creating a CCS Archive.
Command usage:
ccs_tool create Directory CCADevice.
3. At each node, start the CCS daemon, specifying the CCA device at the command line. Refer to
Section 7.2 Starting CCS in the Cluster.
Command usage:
ccsd -d CCADevice
Page 35
Chapter 4. Initial Configuration 21
4.2.3. Starting Clustering and Locking Systems
To start clustering and locking systems, follow these steps:
1. Check the cluster.ccs file to identify/verify which nodes are designated as lock server nodes.
2. Start the LOCK_GULM servers. At each lock-server node, start lock_gulmd. Refer to Section
8.2.3 Starting LOCK_GULM Servers.
Command usage:
lock_gulmd
4.2.4. Setting Up and Mounting File Systems
To set up and mount file systems, follow these steps:
1. Create GFS file systems on pools created in Step 1 of Section 4.2.2 Setting Up and Starting the
Cluster Configuration System. Choose a unique name for each file system. Refer to Section 9.1
Making a File System.
Command usage:
gfs_mkfs -p lock_gulm -t ClusterName:FSName -j NumberJournals
BlockDevice
2. At each node, mount the GFS file systems. Refer to Section 9.2 Mounting a File System.
Command usage:
mount -t gfs BlockDevice MountPoint
Page 36
22 Chapter 4. Initial Configuration
Page 37
Chapter 5.
Using the Pool Volume Manager
This chapter describes the GFS volume manager — named Pool — and its commands. The chapter
consists of the following sections:
• Section 5.1 Overview of GFS Pool Volume Manager
• Section 5.2 Synopsis of Pool Management Commands
• Section 5.4 Creating a Configuration File for a New Volume
• Section 5.3 Scanning Block Devices
• Section 5.5 Creating a Pool Volume
• Section 5.6 Activating/Deactivating a Pool Volume
• Section 5.7 Displaying Pool Configuration Information
• Section 5.8 Growing a Pool Volume
• Section 5.9 Erasing a Pool Volume
• Section 5.10 Renaming a Pool Volume
• Section 5.11 Changing a Pool Volume Minor Number
• Section 5.12 Displaying Pool Volume Information
• Section 5.13 Using Pool Volume Statistics
• Section 5.14 Adjusting Pool Volume Multipathing
5.1. Overview of GFS Pool Volume Manager
Pool is a GFS software subsystem that presents physical storage devices (such as disks or RAID arrays) as logical volumes to GFS cluster nodes. Pool can aggregate storage devices either by concatenating the underlying storage or by striping the storage using RAID 0. Pool is a cluster-wide volume
manager, presenting logical volumes to each GFS node as if the storage were attached directly to each
node. Because Pool is a cluster-wide volume manager, changes made to a volume by one GFS node
are visible to all other GFS nodes in a cluster.
Pool is a dynamically loadable kernel module, pool.o. When pool.o is loaded, it gets registered
as a Linux kernel block-device driver. Before pool devices can be used, this driver module must be
loaded into the kernel. (Once the driver module is loaded, the pool_assemble command can be run
to activate pools.)
Pool includes a set of user commands that can be executed to configure and manage specific pool
devices. Those commands are summarized in the next section.
More advanced, special-purpose features of the Pool volume manager are described later in this chapter.
Page 38
24 Chapter 5. Using the Pool Volume Manager
5.2. Synopsis of Pool Management Commands
Four commands are available to manage pools:
• pool_tool
• pool_assemble
• pool_info
• pool-mp
The following sections briefly describe the commands and provide references to other sections in this
chapter, where more detailed information about the commands and their use is described.
5.2.1. pool_tool
The pool_tool command provides a variety of functions for manipulating and controlling pools
(refer to Table 5-1 and Table 5-2). Some pool_tool functions — such as creating a pool and growing
a pool — require that a file be specified on the command line to provide inputs for the command.
Other pool_tool functions — such as erasing a pool, renaming a pool, changing a minor number,
and printing a pool configuration — act on existing pools and require one or more pool names to be
specified on the command line.
Flag Function Section/Page Reference
-c Create a pool Section 5.5 Creating a Pool Volume
-e Erase a pool Section 5.9 Erasing a Pool Volume
-s Scan devices Section 5.3 Scanning Block Devices
-g Grow a pool Section 5.8 Growing a Pool Volume
Note: The pool_tool -g command supersedes
the pool_grow command as of GFS 5.2.
Although the pool_grow command is still
available, it is not supported in GFS 5.2 and
later.
-r Rename a pool Section 5.10 Renaming a Pool Volume
Note: In releases before GFS 5.2, the -r flag
had a different usage.
-p Print a pool configuration Section 5.7 Displaying Pool Configuration
Information
-m Change a pool minor
number
Section 5.11 Changing a Pool Volume Minor
Number
Table 5-1. pool_tool Command Functions
Flag Option
-D Enable debugging output.
-h Help. Show usage information.
-O Override prompts.
-q Be quiet. Do not display output from the command.
Page 39
Chapter 5. Using the Pool Volume Manager 25
Flag Option
-V Display command version information, then exit.
-v Verbose operation.
Table 5-2. pool_tool Command Options
5.2.2. pool_assemble
The pool_assemble command activates and deactivates pools on a system (refer to Table 5-3 and
Table 5-4). One or more pool names can be specified on the command line, indicating the pools to be
activated or deactivated. If no pools are specified on the command line, all pools will be acted upon.
Flag Function Section/Page Reference
-a Activate pool(s) Section 5.6 Activating/Deactivating a Pool
Volume
-r Deactivate pool(s) Section 5.6 Activating/Deactivating a Pool
Volume
Table 5-3. pool_assemble Command Functions
Flag Option
-D Enable debugging output.
-h Help. Show usage information.
-q Be quiet. Do not display output from the command.
-V Display command version information, then exit.
-v Verbose operation.
Table 5-4. pool_assemble Command Options
5.2.3. pool_info
The pool_info command scans disks directly and displays information about activated pools in a
system; that is, pools that have been assembled with the pool_assemble command (refer to Table
5-5 and Table 5-6). Information about pools that are present but not assembled is excluded from the
information displayed with a pool_info command. One or more pool names can be specified on the
command line to select information about specific pools. If no pool name is specified, information on
all pools is returned.
Flag Function Section/Page Reference
-c Clear statistics Section 5.13 Using Pool Volume Statistics
-i Display information Section 5.12 Displaying Pool Volume
Information
Page 40
26 Chapter 5. Using the Pool Volume Manager
Flag Function Section/Page Reference
-s Display statistics Section 5.13 Using Pool Volume Statistics
-p Display an active
configuration
Section 5.7 Displaying Pool Configuration
Information
Table 5-5. pool_info Command Functions
Flag Option
-D Enable debugging output.
-H Show capacity in human readable form.
-h Help. Show usage information.
-q Be quiet. Do not display output from the command.
-t Set time interval for continual statistics updates.
-V Display command version information, then exit.
-v Verbose operation.
Table 5-6. pool_info Command Options
5.2.4. pool_mp
The pool_mp command is for managing multipathing on running pools (refer to Table 5-7 and Table
5-8). One or more pool names can be specified on the command line to adjust multipathing on specific
pools. If no pools are specified on the command line, all pools will be acted upon.
Flag Function Section/Page Reference
-m Tune multipathing Section 5.14 Adjusting Pool Volume
Multipathing
-r Restore failed paths Section 5.14 Adjusting Pool Volume
Multipathing
Table 5-7. pool_mp Command Functions
Flag Option
-D Enable debugging output.
-h Help. Show usage information.
-q Be quiet. Do not display output from the command.
-V Display command version information, then exit.
-v Verbose operation.
Table 5-8. pool_mp Command Options
Page 41
Chapter 5. Using the Pool Volume Manager 27
5.3. Scanning Block Devices
Scanning block devices provides information about the availability and characteristics of the devices.
That information is important for creating a pool configuration file. You can scan block devices by
issuing the pool_tool command with the -s option. Issuing the pool_tool command with the -s
option scans all visible block devices and reports whether they have an Ext2 or Ext3 file system, LVM
version 1 labels, a partition table, a pool label, or an unknown label on them.
Note
The pool_tool -s command does not detect ondisk labels other than those mentioned in the preceding paragraph.
5.3.1. Usage
pool_tool -s
5.3.2. Example
In this example, the response to the command displays information about one GFS file system, other
file systems that have no labels, and a local file system.
# pool_tool -s
Device Pool Label
====== ==========
/dev/pool/stripe-128K <- GFS filesystem ->
/dev/sda <- partition information ->
/dev/sda1 stripe-128K
/dev/sda2 stripe-128K
/dev/sda3 stripe-128K
/dev/sda4 <- partition information ->
/dev/sda5 <- unknown ->
/dev/sda6 <- unknown ->
/dev/sda7 <- unknown ->
/dev/sda8 <- unknown ->
/dev/sdb <- partition information ->
/dev/sdb1 <- unknown ->
/dev/sdb2 <- unknown ->
/dev/sdb3 <- unknown ->
. .
. .
. .
/dev/sdd4 <- partition information ->
/dev/sdd5 <- unknown ->
/dev/sdd6 <- unknown ->
/dev/sdd7 <- unknown ->
/dev/sdd8 <- unknown ->
/dev/hda <- partition information ->
/dev/hda1 <- EXT2/3 filesystem ->
/dev/hda2 <- swap device ->
/dev/hda3 <- EXT2/3 filesystem ->
Page 42
28 Chapter 5. Using the Pool Volume Manager
5.4. Creating a Configuration File for a New Volume
A pool configuration file is used as input to the pool_tool command when creating or growing a pool
volume. The configuration file defines the name and layout of a single pool volume. Refer to Figure
5-1 for the pool configuration file format. Refer to Table 5-9 for descriptions of the configuration file
keywords and variables.
An arbitrary name can be used for a pool configuration file; however, for consistency, it is recommended that you name the file using the pool name followed by the .cfg extension (poolname.cfg).
For example, the pool configuration file for a pool named pool0 would be defined in configuration
file pool0.cfg.
Before creating a configuration file, you can check to see what devices are available by using the
pool_tool command with the -s option.
poolname name
minor number
subpools number
subpool id stripe devices [type]
pooldevice subpool id device
Figure 5-1. File Structure: Pool Configuration File
File Line and
Keyword
Variable Description
poolname name The name of the pool device that appears in
the /dev/pool/ directory.
minor number Assigns a device minor number (0 to 64) to
a pool. If number is specified as 0 (or if the
minor line is omitted), the minor number of
the pool is assigned dynamically. The
default value is 0.
subpools number Represents the total number of subpools in
the pool. The number value should be set
to a value of 1 unless special data or journal
subpools are used.
subpool id stripe devices [type] The details of each subpool:
id is the subpool identifier. Number the
subpools in order beginning with 0.
stripe is the stripe size in sectors (512
bytes per sector) for each device. A value
of 0 specifies no striping (concatenation).
devices specifies the number of devices
in the subpool.
type is optional and specifies the label
type to attach to the subpool. Values of
gfs_data or gfs_journal are
acceptable. The default value is gfs_data.
Page 43
Chapter 5. Using the Pool Volume Manager 29
File Line and
Keyword
Variable Description
pooldevice subpool id device Adds a storage device to a subpool:
subpool specifies the subpool identifier
to which the device is to be added.
id is the device identifier. Number the
devices in order beginning with 0.
device specifies the device node to be
used (for example, /dev/sda1).
Table 5-9. Pool Configuration File Keyword and Variable Descriptions
5.4.1. Examples
This example creates a 4-disk pool named pool0 that has a stripe size of 64K and an assigned minor
number of 1:
poolname pool0
minor 1 subpools 1
subpool 0 128 4 gfs_data
pooldevice 0 0 /dev/sdb1
pooldevice 0 1 /dev/sdc1
pooldevice 0 2 /dev/sdd1
pooldevice 0 3 /dev/sde1
This example creates a 4-disk pool named pool1 that has a dynamic minor number composed of a
striped subpool and a concatenated subpool:
poolname pool1
minor 0
subpools 2
# striped subpool
subpool 0 128 2 gfs_data
# concatenated subpool
subpool 1 0 2 gfs_data
pooldevice 0 0 /dev/sdb1
pooldevice 0 1 /dev/sdc1
pooldevice 1 0 /dev/sdd1
pooldevice 1 1 /dev/sde1
5.5. Creating a Pool Volume
Once you have created or edited a configuration file (refer to Section 5.4 Creating a Configuration
File for a New Volume), you can create a pool volume running the pool_tool command from any
node in the cluster. Because the pool_tool command writes labels to the beginning of the devices
or partitions, the new pool volume’s devices or partitions must be accessible to the system.
Note
You can activate a pool on any node by running the pool_assemble command. Refer to Section 5.6
Activating/Deactivating a Pool Volume.
Page 44
30 Chapter 5. Using the Pool Volume Manager
5.5.1. Usage
pool_tool -c ConfigFile
ConfigFile
Specifies the file that defines the pool.
5.5.2. Example
In this example, the pool0.cfg file describes the new pool, pool0, created by the command.
pool_tool -c pool0.cfg
5.5.3. Comments
Multiple pools can be created with one pool_tool command by listing multiple pool configuration
files on the command line.
If no flag is specified in the pool_tool command, the function defaults to creating a pool (-c), with
the configuration file specified after the command.
5.6. Activating/Deactivating a Pool Volume
The pool_assemble command activates or deactivates pools on a node.
Note
The pool_assemble command must be run on every node that accesses shared pools; also, it must
be run each time a node reboots.
The pool_assemble command only activates pools from devices visible to the node (those listed in
/proc/partitions/). Disk labels created by the pool_tool command are scanned to determine
which pools exist and, as a result, should be activated. The pool_assemble command also creates
device nodes in the /dev/pool/ directory for devices it has activated.
5.6.1. Usage
Activating a Pool Volume
pool_assemble -a [PoolName]
PoolName
Specifies the pool to activate. More than one name can be listed. If no pool names are specified,
all pools visible to the system are activated.
Deactivating a Pool Volume
pool_assemble -r [PoolName]
Page 45
Chapter 5. Using the Pool Volume Manager 31
PoolName
Specifies the pool to deactivate. More than one name can be listed. If no pool names are specified,
all pools visible to the system are deactivated.
5.6.2. Examples
This example activates all pools on a node:
pool_assemble -a
This example deactivates all pools on a node:
pool_assemble -r
This example activates pool0 on a node:
pool_assemble -a pool0
This example deactivates pool0 on a node:
pool_assemble -r pool0
5.6.3. Comments
The pool_assemble command must be run on every GFS node.
The pool_assemble command should be put in the node’s system-startup scripts so that pools are
activated each time the node boots.
5.7. Displaying Pool Configuration Information
Using the pool_tool command with the -p (print) option displays pool configuration information in
configuration file format. The pool information is displayed in the format equivalent to the configuration file that was used to create the pool. The disk labels that were written when the pool was created
are read to recreate the configuration file.
5.7.1. Usage
pool_tool -p [PoolName]
PoolName
Specifies the pool name(s) for which to display information. If no pool names are specified, all
active pools are displayed.
Page 46
32 Chapter 5. Using the Pool Volume Manager
5.7.2. Example
In this example, the pool_tool -p command displays the configuration for pool0:
# pool_tool -p pool0
poolname pool0
#minor
dynamically assigned
subpools 1
subpool 0 0 1 gfs_data
pooldevice 0 0 /dev/sda1
5.8. Growing a Pool Volume
An existing pool can be expanded while it is activated or deactivated. You can grow a pool by creating
a new pool configuration file (based on an existing pool configuration file), then adding one or more
subpools containing the new devices to be added to the volume.
Refer to Section 5.7 Displaying Pool Configuration Information for information on creating a config-
uration file for an existing pool volume.
5.8.1. Usage
pool_tool -g [ConfigFile]
ConfigFile
Specifies the file describing the extended pool.
Note
The pool_tool -g command supersedes the pool_grow command as of GFS 5.2. Although the
pool_grow command is still available, it is not supported in GFS 5.2 and later.
5.8.2. Example procedure
The following example procedure expands a pool volume.
1. Create a new configuration file from configuration information for the pool volume that you
want to expand (in this example, pool0):
# pool_tool -p /dev/pool/pool0 > pool0-new.cfg
# cat pool0-new.cfg
poolname pool0
subpools 1
subpool 0 128 4 gfs_data
pooldevice 0 0 /dev/sdb1
pooldevice 0 1 /dev/sdc1
pooldevice 0 2 /dev/sdd1
pooldevice 0 3 /dev/sde1
Page 47
Chapter 5. Using the Pool Volume Manager 33
2. Edit the new file, pool0-new.cfg, by adding one or more subpools that contain the devices or
partitions, as indicated in this example:
poolname pool0
subpools 2 <--- Change
subpool 0 128 4 gfs_data
subpool 1 0 1 gfs_data <--- Add
pooldevice 0 0 /dev/sdb1
pooldevice 0 1 /dev/sdc1
pooldevice 0 2 /dev/sdd1
pooldevice 0 3 /dev/sde1
pooldevice 1 0 /dev/sdf1 <--- Add
3. After saving the file, verify that the file has been changed:
# cat pool0-new.cfg
poolname pool0
subpools 2 <--- Changed
subpool 0 128 4 gfs_data
subpool 1 0 1 gfs_data <--- Added
pooldevice 0 0 /dev/sdb1
pooldevice 0 1 /dev/sdc1
pooldevice 0 2 /dev/sdd1
pooldevice 0 3 /dev/sde1
pooldevice 1 0 /dev/sdf1 <--- Added
4. Run the pool_tool command with the grow (-g) option specifying the configuration file:
pool_tool -g pool0-new.cfg
5.9. Erasing a Pool Volume
A deactivated pool can be erased by using the -e option of the pool_tool command. Using
pool_tool -e erases the disk labels written when the pool was created.
5.9.1. Usage
pool_tool -e [PoolName]
PoolName
Specifies the pool to erase. If no pool names are specified, all pools are erased.
5.9.2. Example
This example erases all disk labels for pool0:
pool_tool -e pool0
5.9.3. Comments
The -O (override) flag bypasses the confirmation step.
Page 48
34 Chapter 5. Using the Pool Volume Manager
5.10. Renaming a Pool Volume
The pool_tool command can be used to change the name of a pool.
5.10.1. Usage
pool_tool -r NewPoolName CurrentPoolName
NewPoolName
Specifies the new name of the pool.
CurrentPoolName
Specifies the pool name to be changed.
Note
In releases before GFS 5.2, the -r flag had a different usage.
Note
You must deactivate a pool before renaming it. You can deactivate a pool with the pool_assemble
-r PoolName command.
5.10.2. Example
This example changes the name for pool mypool to pool0:
pool_tool -r pool0 mypool
5.11. Changing a Pool Volume Minor Number
The pool_tool command can be used to change the minor number of a pool.
5.11.1. Usage
pool_tool -m Number PoolName
Number
Specifies the new minor number to be used.
Page 49
Chapter 5. Using the Pool Volume Manager 35
PoolName
Specifies the name of the pool to be changed. The minor number must have a value between 0
and 64. Specifying a minor number of 0 dynamically selects an actual minor number between 65
and 127 at activation time.
Note
You must deactivate a pool before changing its pool volume minor number. You can deactivate a pool
with the pool_assemble -r PoolName command.
5.11.2. Example
This example changes the minor number for pool0 to 6.
pool_tool -m 6 pool0
5.11.3. Comments
Before changing a pool volume minor number, deactivate the pool. For this command to take
effect throughout the cluster, you must reload the pools on each node in the cluster by issuing
a pool_assemble -r PoolName command followed by a pool_assemble -a PoolName
command.
5.12. Displaying Pool Volume Information
The pool_info command can be used to display information about pools.
Using the pool_info command with the -i option displays the following basic information about
the named pool(s): the pool name, the minor number, the device node alias, the capacity, whether or
not the pool is being used, and the multipathing type.
Using the pool_info command with the -v (verbose) option displays complete information about
the named pools, which adds subpool and device details to the output display.
5.12.1. Usage
Basic Display
pool_info -i [PoolName]
PoolName
Specifies the pool name(s) for which to display information. If no pool names are specified, all
active pools are displayed.
Page 50
36 Chapter 5. Using the Pool Volume Manager
Complete Display
pool_info -v [PoolName]
PoolName
Specifies the pool name(s) for which to display information. If no pool names are specified, all
active pools are displayed.
5.12.2. Examples
This example displays basic information about all activated pools:
pool_info -i
This example displays complete information about all activated pools:
pool_info -v
This example displays complete information about pool0:
pool_info -v pool0
5.13. Using Pool Volume Statistics
The pool_info command can be used to display pool read-write information and to clear statistics
from pools.
Using the pool_info command with the -s option displays the number of reads and writes for the
named pool(s) since the last time the pool was activated or statistics were cleared.
Using the pool_info command with the -c option clears statistics from the named pools.
5.13.1. Usage
Display the Number of Reads and Writes
pool_info -s [PoolName]
PoolName
Specifies the pool name for which to display information. If no pool names are specified, all
active pools are displayed.
Clear Statistics
pool_info -c [PoolName]
PoolName
Specifies the pool name(s) from which statistics are cleared. If no pool names are specified,
statistics are cleared from all active pools.
Page 51
Chapter 5. Using the Pool Volume Manager 37
5.13.2. Examples
This example displays statistics for all activated pools:
pool_info -s
This example displays statistics for pool0:
pool_info -s pool0
This example clears statistics for pool0:
pool_info -c pool0
5.14. Adjusting Pool Volume Multipathing
The pool_mp command adjusts multipathing for running pools. Using the pool_mp command with
the -m option, you can change the type of multipathing. Using the pool_mp command with the -r
option, you can reintegrate failed paths.
Note
Pool multipathing must be enabled in the license file to use this feature.
5.14.1. Usage
Change the Type of Multipathing
pool_mp -m {none | failover | n} [PoolName]
{none | failover | n}
Specifies the type of multipathing to be used: either none, failover, or the number of kilobytes, n, used as a round-robin stripe value.
PoolName
Specifies the pool on which to adjust multipathing. If no pool names are specified, this action is
attempted on all active pools.
Reintegrate Failed Paths
pool_mp -r [PoolName]
PoolName
Specifies the pool on which to attempt restoration of any failed paths. If no pool names are
specified, this action is attempted on all active pools.
Page 52
38 Chapter 5. Using the Pool Volume Manager
5.14.2. Examples
This example adjusts the multipathing for all pools to none.
pool_mp -m none
This example adjusts the multipathing for pool0 to failover.
pool_mp -m failover pool0
This example adjusts the multipathing for pool0 to round-robin with a stripe size of 512 KB.
pool_mp -m 512 pool0
This example restores failed paths for all active pools.
pool_mp -r
Page 53
Chapter 6.
Creating the Cluster Configuration System Files
The GFS Cluster Configuration System (CCS) requires the following files:
• license.ccs — The license file accompanies the GFS software and is required to use GFS.
• cluster.ccs — The cluster file contains the name of the cluster and the names of the nodes where
LOCK_GULM servers are run.
• fence.ccs — The fence file describes each device used for fencing.
• nodes.ccs — The nodes file contains an entry for each GFS node and LOCK_GULM server node.
This file specifies the IP address and fencing parameters of each node.
This chapter describes how to create the CCS files and contains the following sections:
• Section 6.1 Prerequisite Tasks
• Section 6.2 CCS File Creation Tasks
• Section 6.3 Dual Power and Multipath FC Fencing Considerations
• Section 6.4 GNBD Multipath Considerations
• Section 6.5 Adding the license.ccs File
• Section 6.6 Creating the cluster.ccs File
• Section 6.7 Creating the fence.ccs File
• Section 6.8 Creating the nodes.ccs File
6.1. Prerequisite Tasks
Before creating the CCS files, make sure that you perform the following prerequisite tasks:
• Choose a cluster name (user variable, ClusterName).
• Create a temporary directory (for example, /root/alpha/) in which to place the new CCS files
that are created. Note the temporary directory path; it is used later as a Directory parameter
when the CCS files are written to a CCA (Cluster Configuration Archive) device. For more information, refer to Section 7.1 Creating a CCS Archive.
• Identify each node that runs the LOCK_GULM server daemons. Those nodes must have entries in
the nodes.ccs file. Refer to Section 8.2 LOCK_GULM.
• Determine if any GFS node has dual power supplies or multiple paths to FC storage.
• Determine if you are using GNBD multipath.
• Determine the type of fencing for each node.
For more information about prerequisite tasks, refer to Section 1.5 Before Configuring GFS
Page 54
40 Chapter 6. Creating the Cluster Configuration System Files
6.2. CCS File Creation Tasks
To create the CCS files perform the following steps:
1. Add the license.ccs file.
2. Create the cluster.ccs file.
3. Create the fence.ccs file.
4. Create the nodes.ccs file.
Note
The contents of CCS files are case sensitive.
6.3. Dual Power and Multipath FC Fencing Considerations
To ensure that fencing completely removes a node that has dual power supplies or multiple paths to
FC storage, both power supplies and all paths to FC storage for that node must be fenced.
To fence dual power supplies and multiple paths to FC storage, you need to consider the following
actions when creating the fence.ccs and nodes.ccs files:
• fence.ccs — For each power supply and each path to FC storage define a fencing device
(fence.ccs:fence_devices/DeviceName).
• nodes.ccs — For each node with dual power supplies, include in the fencing method section
(nodes.ccs:nodes/NodeName/fence/Methodname) a fencing device for each power supply .
For each node having multiple paths to FC storage, include in the fencing method section a fencing
device for each path to FC storage.
GFS supports dual power-supply fencing with the APC MasterSwitch only; it supports multipath FC
fencing with Brocade and Vixel switches. For more information about creating the fence.ccs and
nodes.ccs files, refer to Section 6.7 Creating the fence.ccs File and Section 6.8 Creating the
nodes.ccs File. For more information about fencing, refer to Chapter 10 Using the Fencing System.
6.4. GNBD Multipath Considerations
GNBD multipath allows you to configure multiple GNBD server nodes (nodes that export GNBDs to
GFS nodes) with redundant paths between the GNBD server nodes and storage devices. The GNBD
servers, in turn, present multiple storage paths to GFS nodes via redundant GNBDs. With GNBD
multipath, if a GNBD server node becomes unavailable, another GNBD server node can provide GFS
nodes with access to storage devices.
If you use GNBD multipath, make sure that each GNBD server node is configured with a fencing
method that performs both of the following actions:
• Physically removes the node from the network
• Notifies the GFS nodes that the GNBD server node has been fenced
Page 55
Chapter 6. Creating the Cluster Configuration System Files 41
To fence GNBD server nodes, consider the following actions when creating fence.ccs and
nodes.ccs files:
• fence.ccs — Define fencing devices as follows:
• For fencing GFS nodes using GNBD multipath, a GNBD fencing device must include an
option = multipath parameter (in fence.ccs:fence_devices/DeviceName of the
fencing device).
• Define a fencing device for fencing GNBD node servers.
• Define a notification fencing device (using the fence_notify_gnbd fencing agent) for notify-
ing GFS nodes that a GNBD server node has been fenced.
Warning
Do not specify the GNBD fencing agent (fence_gnbd) as a fencing device for the GNBD server
nodes.
• nodes.ccs — Each GNBD node needs to be defined with a fencing method that includes the
notification fencing device (the fencing device using the fence_notify_gnbd fencing agent) to
indicate to the GFS nodes that a GNBD device is not available if it is fenced.
For more information about creating the fence.ccs and nodes.ccs files, refer to Section 6.7 Creat-
ing the fence.ccs File and Section 6.8 Creating the nodes.ccs File. In addition to considerations
for the fence.ccs and nodes.ccs files, other considerations need to be taken into account for
GNBD multipath. For that information and more about using GNBD multipath, refer to Chapter 11
Using GNBD . For more information about fencing, refer to Chapter 10 Using the Fencing System.
6.5. Adding the license.ccs File
Adding the license.ccs file consists of obtaining the license file, renaming the file, and saving it.
To add the license file, follow these steps:
1. Obtain the GFS license file, gfs_license.txt. Example 6-1 shows a license file.
2. Rename the file to license.ccs and save it into the temporary directory.
Caution
Do not edit the license.ccs file; doing so invalidates it.
Page 56
42 Chapter 6. Creating the Cluster Configuration System Files
license {
name = "example"
license_version = 1
license_number = "0"
license_type = "Enterprise"
node_count = 4
gfs {
directio = "TRUE"
cdpn = "TRUE"
}
locking {
type = "Redundant"
}
fencing {
multiple_agents = "TRUE"
redundant_paths = "TRUE"
cascading_methods = "TRUE"
}
pool {
multipath = "TRUE"
}
start_date = "Tue Apr 22 10:37:56 2003 CDT"
stop_date = "Wed Apr 21 10:37:56 2004 CDT"
start_sec = "1051025876"
stop_sec = "1082561876"
hash = "FFFFFFFF-FFFFFFFF"
}
Example 6-1. License File: license.ccs
6.6. Creating the cluster.ccs File
Creating the cluster.ccs file consists of specifying the following parameters:
• Cluster name
• Each node that runs LOCK_GULM server
• Optional parameters
To create the cluster.ccs file, follow these steps:
1. Create a new file named cluster.ccs using the file structure shown in Figure 6-1. Refer to
Table 6-1 for syntax description.
2. Specify ClusterName (for example, alpha). Refer to Example 6-2.
3. Specify each node (NodeName) that runs LOCK_GULM server (for example, n01, n02, and
n03). Refer to Example 6-2.
4. (Optional) For the heartbeat rate (heartbeat_rate =), specify Seconds. Refer to Example
6-2.
5. (Optional) For the allowed consecutively missed heartbeats (allowed_misses =), specify
Number. Refer to Example 6-2.
6. Save the cluster.ccs file.
Page 57
Chapter 6. Creating the Cluster Configuration System Files 43
Note
Two commonly used optional cluster.ccs parameters are included in this procedure. For a description of other optional parameters, refer to the lock_gulmd(5) man page.
cluster {
name = "ClusterName"
lock_gulm {
servers = ["NodeName",..., "NodeName"]
heartbeat_rate = Seconds <-- Optional
allowed_misses = Number <-- Optional
}
}
Figure 6-1. File Structure: cluster.ccs
Parameter Description
ClusterName The name of the cluster, from 1 to 16 characters long.
NodeName The name of each node that runs the LOCK_GULM server. Each node
name must appear under nodes.ccs:nodes.
Seconds (Optional) For heartbeat_rate, the rate at which the heartbeats are checked by
the server in seconds. Two-thirds of this time is the rate at which the
heartbeats are sent. The default values is 15.
Number (Optional) For allowed_misses, How many consecutive heartbeats can be missed
before a node is marked as expired. The default value is 2.
Table 6-1. File Syntax Description: Variables for cluster.ccs
cluster {
name = "alpha"
lock_gulm {
servers = ["n01", "n02", "n03"]
heartbeat_rate = 21
allowed_misses = 3
}
}
Example 6-2. cluster.ccs
6.7. Creating the fence.ccs File
You can configure each node in a GFS cluster for a variety of fencing devices. To configure fencing
for a node, you need to perform the following tasks:
• Create the fence.ccs file — Define the fencing devices available in the cluster (described in this
section).
• Create the nodes.ccs file — Define which fencing method (or methods) each node should use
(refer to Section 6.8 Creating the nodes.ccs File).
Page 58
44 Chapter 6. Creating the Cluster Configuration System Files
Creating the fence.ccs file consists of defining each fencing device you are going to use. You can
define the following types of fencing devices in the fence.ccs file:
• APC MasterSwitch
• WTI NPS (Network Power Switch)
• Brocade FC (Fibre Channel) switch
• Vixel FC switch
• GNBD
• Fence Notify GNBD
• HP RILOE card
• Manual
The fence.ccs file is used in conjunction with the nodes.ccs file to configure fencing in a cluster.
The nodes.ccs file specifies fencing devices that are defined in the fence.ccs file. The fence.ccs
file can define any combination of fencing devices.
If a node has dual power supplies, you must define a fencing device for each power supply. Similarly,
if a node has multiple paths to FC storage, you must define a fencing device for each path to FC
storage.
For more information about fencing, refer to Chapter 10 Using the Fencing System.
To create the fence.ccs file, follow these steps:
1. Create a file named fence.ccs. Use a file format according to the fencing method as follows.
Refer to Table 6-2 for syntax description.
• APC MasterSwitch — Refer to Figure 6-2.
• WTI NPS (Network Power Switch) — Refer to Figure 6-3.
• Brocade FC (Fibre Channel) switch — Refer to Figure 6-4.
• Vixel FC switch — Refer to Figure 6-5.
• GNBD — For GNBD without GNBD multipath, refer to Figure 6-6. For GNBD with GNBD
multipath, refer to Figure 6-7.
• Fence Notify GNBD — Refer to Figure 6-8.
• HP RILOE card — Refer to Figure 6-9.
• Manual — Refer to Figure 6-10.
2. Type parameters in the file according to the fencing device (or devices) needed:
a. For each APC MasterSwitch fencing device, specify the following parameters:
DeviceName, the fencing agent (agent =) as fence_apc, IPAddress,
LoginName, and LoginPassword. Refer to Example 6-3 for a fence.ccs file that
specifies an APC MasterSwitch fencing device.
b. For each WTI NPS fencing device, specify the following parameters: DeviceName, the
fencing agent (agent =) as fence_wti, IPAddress, and LoginPassword . Refer
to Example 6-4 for a fence.ccs file that specifies a WTI NPS fencing device.
c. For each Brocade FC-switch fencing device, specify the following parameters:
DeviceName, the fencing agent (agent =) as fence_brocade, IPAddress,
LoginName, and LoginPassword. Refer to Example 6-5 for a fence.ccs file that
specifies a Brocade FC-switch fencing device.
Page 59
Chapter 6. Creating the Cluster Configuration System Files 45
d. For each Vixel FC-switch fencing device, specify the following parameters:
DeviceName, the fencing agent (agent =) as fence_vixel, IPAddress, and
LoginPassword. Refer to Example 6-6 for a fence.ccs file that specifies a Vixel
FC-switch fencing device.
e. For each GNBD fencing device, specify the following parameters: DeviceName, the
fencing agent (agent =) as fence_gnbd, and ServerName.
For GNBD multipath, include an option = "multipath" line after the ServerName
line. In addition, for GNBD multipath, you can add two optional lines: retrys =
"Number" and wait_time = "Seconds".
For descriptions of those parameters refer to Table 6-2. Refer to Example 6-7 for a
fence.ccs file that specifies a GNBD fencing device for a configuration that does not
employ GNBD multipath. Refer to Example 6-8 for a fence.ccs file that specifies a
GNBD fencing device for a configuration that does employ GNBD multipath.
Note
For GFS clusters using GNBD multipath, do not configure the GNBD server nodes to be
fenced with the GNBD fencing agent, fence_gnbd.
f. For Fence Notify GNBD (used with GNBD multipath), specify a device for notifying
GFS clients that a GNBD server node has been fenced. Specify the following parameters: DeviceName, the fencing agent (agent =) as fence_notify_gnbd, and each
GNBD client (a GFS node that is using GNBD), client =. Refer to Example 6-9 for a
fence.ccs file that specifies a Fence Notify GNBD fencing device.
g. For each HP-RILOE-card fencing device, specify the following parameters:
DeviceName, the fencing agent (agent =) as fence_rib, HostName, LoginName,
and LoginPassword. Refer to Example 6-10 for a fence.ccs file that specifies an
HP-RILOE-card fencing device.
h. For each manual fencing device, specify DeviceName and the fencing agent (agent =)
as fence_manual. Refer to Example 6-11 for a fence.ccs file that specifies a manual
fencing device.
3. Save the file.
fence_devices{
DeviceName {
agent = "fence_apc"
ipaddr = "IPAddress"
login = "LoginName"
passwd = "LoginPassword"
}
DeviceName {
.
.
.
}
}
Figure 6-2. File Structure: fence_devices, fence_apc
Page 60
46 Chapter 6. Creating the Cluster Configuration System Files
fence_devices{
DeviceName {
agent = "fence_wti"
ipaddr = " IPAddress"
passwd = " LoginPassword"
}
DeviceName {
.
.
.
}
}
Figure 6-3. File Structure: fence_devices, fence_wti
fence_devices{
DeviceName {
agent = "fence_brocade"
ipaddr = "IPAddress"
login = "LoginName"
passwd = "LoginPassword"
}
DeviceName {
.
.
.
}
}
Figure 6-4. File Structure: fence_devices, fence_brocade
fence_devices{
DeviceName {
agent = "fence_vixel"
ipaddr = "IPAddress"
passwd = "LoginPassword"
}
DeviceName {
.
.
.
}
}
Figure 6-5. File Structure: fence_devices, fence_vixel
Page 61
Chapter 6. Creating the Cluster Configuration System Files 47
fence_devices{
DeviceName {
agent = "fence_gnbd"
server = "ServerName"
.
.
.
server = "ServerName"
}
DeviceName {
.
.
.
}
}
Figure 6-6. File Structure: fence_devices, fence_gnbd without GNBD Multipath
fence_devices{
DeviceName {
agent = "fence_gnbd"
server = "ServerName"
.
.
.
server = "ServerName"
option = "multipath"
retrys = "Number"
wait_time = "Seconds"
}
DeviceName {
.
.
.
}
}
Figure 6-7. File Structure: fence_devices, fence_gnbd with GNBD Multipath
fence_devices{
DeviceName {
agent = "fence_notify_gnbd"
client = "IPAddress"
.
.
.
client = "IPAddress"
}
DeviceName {
.
.
.
}
}
Figure 6-8. File Structure: fence_devices, fence_notify_gnbd
Page 62
48 Chapter 6. Creating the Cluster Configuration System Files
fence_devices{
DeviceName {
agent = "fence_rib"
hostname = "HostName"
login = "LoginName"
passwd = "LoginPassword"
}
DeviceName {
.
.
.
}
}
Figure 6-9. File Structure: fence_devices, fence_rib
fence_devices{
DeviceName {
agent = "fence_manual"
}
DeviceName {
.
.
.
}
}
Figure 6-10. File Structure: fence_devices, fence_manual
Parameter Description
DeviceName The name of a fencing device. The DeviceName parameter
specifies the name of a fencing device and makes that fencing
device available for use by the fence section of a nodes.ccs file
(nodes.ccs:NodeName/fence/MethodName). The fence
section of a nodes.ccs file also contains DeviceName
parameters each mapping to a DeviceName in the fence.ccs
file.
HostName The host name of a RILOE card on the network to which stunnel
connections can be made.
IPAddress For fencing with power and Fibre Channel switches: The IP
address of a switch to which Telnet connections can be
established.
For GNBD multipath fencing, used with the
fence_notify_gnbd fencing agent (refer to Figure 6-8): IP
address (or hostname) of the GNBD client (a GFS node that has
imported a GNBD from the GNBD server). IPAddress is for
the client = entry and indicates which GNBD client to notify
if the GNBD server has failed and has been fenced. The
fence.ccs file can have multiple client = entries — one for
each GNBD client.
LoginName The login name for a power switch, an FC switch, or a RILOE
card.
Page 63
Chapter 6. Creating the Cluster Configuration System Files 49
Parameter Description
LoginPassword The password for logging in to a power switch, an FC switch, or
a RILOE card.
multipath Selects GNBD multipath style fencing.
CAUTION: When multipath style fencing is used, if the
kgnbd_portd process of a GNBD server node cannot be
contacted, it is fenced as well, using its specified fencing method.
That means that when a GNBD client (GFS node) is fenced, any
node listed as its GNBD server that does not have the
gnbd_serv module loaded (which starts kgnbd_portd) is also
fenced.
Number The number of times to retry connecting to the GNBD server
after a failed attempt, before the server is fenced. The parameter
entry is for the retrys = entry and is only valid when used with
multipath style fencing. (Refer to the multipath entry in this
table.) The default value of Number is 3.
Seconds The length of time, in seconds, to wait between retries. This
parameter entry is for the wait_time = entry and is only valid
when used with multipath style fencing. (Refer to the multipath
entry in this table.) The default value of Seconds is 2.
ServerName The host name of a GNBD server. Each GNBD server is
represented with a "server =" line in the fence.ccs file. For
example, if you have three GNBD servers, then the fence.ccs
file needs three "server =" lines one for each GNBD server.
Table 6-2. File Syntax Description: Variables for fence.ccs
fence_devices {
apc1 {
agent = "fence_apc"
ipaddr = "10.0.3.3"
login = "apc"
passwd = "apc"
}
apc2 {
agent = "fence_apc"
ipaddr = "10.0.3.4"
login = "apc"
passwd = "apc"
}
}
Example 6-3. APC MasterSwitch Fencing Devices Named apc1 and apc2
Page 64
50 Chapter 6. Creating the Cluster Configuration System Files
fence_devices {
wti1 {
agent = "fence_wti"
ipaddr = "10.0.3.3"
passwd = "password"
}
wti2 {
agent = "fence_wti"
ipaddr = "10.0.3.4"
passwd = "password"
}
}
Example 6-4. WTI NPS Fencing Devices Named wti1 and wti2
fence_devices {
silkworm1 {
agent = "fence_brocade"
ipaddr = "10.0.3.3"
login = "admin"
passwd = "password"
}
silkworm2 {
agent = "fence_brocade"
ipaddr = "10.0.3.4"
login = "admin"
passwd = "password"
}
}
Example 6-5. Brocade FC-Switch Fencing Devices Named silkworm1 and silkworm2
fence_devices {
vixel1 {
agent = "fence_vixel"
ipaddr = "10.0.3.3"
passwd = "password"
}
vixel2 {
agent = "fence_brocade"
ipaddr = "10.0.3.4"
passwd = "password"
}
}
Example 6-6. Vixel FC-Switch Fencing Device Named vixel1 and vixel2
Page 65
Chapter 6. Creating the Cluster Configuration System Files 51
fence_devices {
gnbd {
agent = "fence_gnbd"
server = "nodea"
server = "nodeb"
}
}
This example shows a fencing device named gnbd with two servers: nodea and nodeb.
Example 6-7. GNBD Fencing Device Named gnbd, without GNBD Multipath
fence_devices {
gnbdmp {
agent = "fence_gnbd"
server = "nodea"
server = "nodeb"
option = "multipath" <-- Additional entry
retrys = "5" <-- Number of retries set to 5
wait_time = "3" <-- Wait time between retries set to 3
}
}
This example shows a fencing device named gnbdmp with two servers: nodea and nodeb. Because
GNBD Multipath is employed, an additional configuration entry under gnbdmp is needed: option =
"multipath". Also, for GNBD multipath, the example sets the number of retries to 5 with retrys
= 5, and sets the wait time between retries to 3 with wait_time = 3.
Example 6-8. GNBD Fencing Device Named gnbdmp, with GNBD Multipath
fence_devices {
notify_gnbd {
agent = "fence_notify_gnbd"
client = "10.0.1.1"
client = "10.0.1.2"
client = "10.0.1.3"
}
}
Example 6-9. Fence Notify GNBD Fencing Device Named notify_gnbd
Page 66
52 Chapter 6. Creating the Cluster Configuration System Files
fence_devices {
riloe1 {
agent = "fence_rib"
ipaddr = "10.0.4.1"
login = "admin"
passwd = "password"
}
riloe2 {
agent = "fence_rib"
ipaddr = "10.0.4.2"
login = "admin"
passwd = "password"
}
}
Example 6-10. Two HP-RILOE-Card Fencing Device Named riloe1 and riloe2
In this example, two RILOE fencing devices are defined for two nodes.
fence_devices {
admin {
agent = "fence_manual"
}
}
Example 6-11. Manual Fencing Device Named admin
6.8. Creating the nodes.ccs File
The nodes.ccs file specifies the nodes that run in a GFS cluster and their fencing methods. The nodes
specified include those that run GFS and those that run LOCK_GULM servers. The nodes.ccs file
is used in conjunction with the fence.ccs file to configure fencing in a cluster; the nodes.ccs file
specifies fencing devices that are defined in the fence.ccs file.
Creating the nodes.ccs file consists of specifying the identity and fencing method (or methods) of
each node in a GFS cluster. Specifying the identity consists of assigning a name and an IP address
to the node. Specifying a fencing method consists of assigning a name to the fencing method and
specifying its fencing-device parameters; that is, specifying how a node is fenced.
The way in which a fencing method is specified depends on if a node has either dual power supplies or
multiple paths to FC storage. If a node has dual power supplies, then the fencing method for the node
must specify at least two fencing devices — one fencing device for each power supply. Similarly, if a
node has multiple paths to FC storage, then the fencing method for the node must specify one fencing
device for each path to FC storage. For example, if a node has two paths to FC storage, the fencing
method should specify two fencing devices — one for each path to FC storage. If a node has neither
dual power supplies nor multiple paths to FC storage, then the fencing method for the node should
specify only one fencing device.
You can configure a node with one fencing method or multiple fencing methods. When you configure
a node for one fencing method, that is the only fencing method available for fencing that node. When
you configure a node for multiple fencing methods, the fencing methods are cascaded from one fencing method to another according to the order of the fencing methods specified in the nodes.ccs file.
If a node fails, it is fenced using the first fencing method specified in the nodes.ccs file for that node.
If the first fencing method is not successful, the next fencing method specified for that node is used.
If none of the fencing methods is successful, then fencing starts again with the first fencing method
specified, and continues looping through the fencing methods in the order specified in nodes.ccs
until the node has been fenced.
Page 67
Chapter 6. Creating the Cluster Configuration System Files 53
Refer to Chapter 10 Using the Fencing System for basic fencing details, descriptions of how fencing
is used, and descriptions of available fencing methods.
To create the nodes.ccs file, follow these steps:
1. Create a file named nodes.ccs.
a. If you are configuring a node for one fencing method (not cascaded), specify only one
fencing method per node in the nodes.ccs file. If the node is a GNBD server node
(for use with GNBD multipath), an additional fencing device that uses fencing agent
fence_notify_gnbd needs to be included as the last fencing device for the fencing
method. Use a file format according to the fencing method as follows. In addition, refer
to Figure 6-19 for the file format to be used if the node is a GNBD server node. Refer to
Table 6-3 for syntax description.
• APC MasterSwitch — For a node with a single power supply, refer to Figure 6-11. For
a node with dual power supplies, refer to Figure 6-12.
• WTI NPS — Refer to Figure 6-13.
• Brocade or Vixel FC switch — Refer to Figure 6-14.
• GNBD — Refer to Figure 6-15.
• HP RILOE — Refer to Figure 6-16.
• Manual — Refer to Figure 6-17.
b. If you are configuring a node for cascaded fencing, use the file format in Figure 6-18.
Refer to Table 6-3 for syntax description.
Note
Figure 6-18 and Figure 6-19 do not show device-specific parameters for fencing methods. To determine device-specific parameters, see the appropriate figures listed in Step
1, part a.
2. For each node, specify NodeName, IFName, and the IP address of the node name,
IPAddress.
Note
Make sure that you specify Nodename as the Linux hostname and that the primary IP address
of the node is associated with the hostname. Specifying NodeName other than the Linux hostname (for example the interface name) can cause unpredictable results — especially if the
node is connected to multiple networks. To determine the hostname of a node, use the uname
-n command on the node. To verify the IP address associated with the hostname, issue a ping
command to the hostname.
3. For each node, specify the fencing parameters according to the fencing method you are using,
as follows:
a. If using APC MasterSwitch fencing, specify MethodName, DeviceName,
PortNumber, and SwitchNumber. If you are configuring for dual power supplies,
specify the following parameters for the second fencing device: DeviceName,
PortNumber, and SwitchNumber. Refer to Example 6-12 for a nodes.ccs file that
Page 68
54 Chapter 6. Creating the Cluster Configuration System Files
specifies APC MasterSwitch fencing for a single power supply. Refer to Example 6-13
for a nodes.ccs file that specifies APC MasterSwitch fencing for dual power supplies.
b. If using WTI NPS fencing, specify MethodName, DeviceName, and PortNumber.
Refer to Example 6-14 for a nodes.ccs file that specifies WTI NPS fencing.
c. If using Brocade or Vixel FC-switch fencing, specify MethodName, DeviceName,
and PortNumber. If you are configuring for multiple paths to FC storage, specify the
following parameters for each additional fencing device required: DeviceName and
PortNumber. Refer to Example 6-15 for a nodes.ccs file that specifies Brocade FC-
switch fencing. Refer to Example 6-16 for a nodes.ccs file that specifies Vixel FCswitch fencing.
d. If using GNBD fencing, specify MethodName, DeviceName, and IPAddress. Refer
to Example 6-17 for a nodes.ccs file that specifies GNBD fencing.
e. If using HP RILOE fencing, specify MethodName, DeviceName, and PortNumber.
Refer to Example 6-18 for a nodes.ccs file that specifies HP RILOE fencing.
f. If using manual fencing, specify MethodName, DeviceName, and IPAddress. Refer
to Example 6-19 for a nodes.ccs file that specifies manual fencing.
g. If using cascaded fencing, specify parameters according to the type of fencing methods
and in the order that the fencing methods are to cascade. Refer to Example 6-20 for a
nodes.ccs file that specifies cascaded fencing.
h. If using GNBD multipath, fence the GNBD server nodes using any of the fencing methods
stated in previous steps in the procedure except for GNBD fencing (Step 3, part d). Also, in
the fencing method, include a fencing device that uses the fence_notify_gnbd fencing
agent. Refer to Example 6-21 for a nodes.ccs file that specifies fencing for a GNBD
server node.
4. Save the nodes.ccs file.
Page 69
Chapter 6. Creating the Cluster Configuration System Files 55
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFNAME= "IPAddress"
}
fence {
MethodName {
DeviceName {
port = PortNumber
switch = SwitchNumber
}
}
}
}
NodeName {
.
.
.
}
}
File format for node
identification (same
format for all nodes)
File format for APC
MasterSwitch
fencing method, for
node with single
power supply only
Figure 6-11. File Format: nodes.ccs, APC Single Fencing Method, Single Power Supply
Page 70
56 Chapter 6. Creating the Cluster Configuration System Files
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFNAME = "IPAddress "
}
fence {
MethodName {
DeviceName {
port = PortNumber
switch = SwitchNumber
option = "off"
}
DeviceName {
port = PortNumber
switch = SwitchNumber
option = "off"
}
DeviceName {
port = PortNumber
switch = SwitchNumber
option = "on"
}
DeviceName {
port = PortNumber
switch = SwitchNumber
option = "on"
}
}
}
}
NodeName {
.
.
.
}
}
File format for node
identification (same
format for all nodes)
File format for APC
MasterSwitch
fencing method, for
node with dual
power supplies
Fencing a node with
dual power supplies
requires that both
power supplies be
powered off before
rebooting the node.
To accomplish that,
the nodes.ccs file
requires the use of
the option =
parameter: first, set
to "off" for for the
fencing device of
each power supply,
then set to "on" for
the fencing device
of each power
supply.
Fencing device
for pwr supply 1
Fencing device
for pwr supply 1
Fencing device
for pwr supply 2
Fencing device
for pwr supply 2
Power down
pwr supply 1
Power down
pwr supply 2
Power up
pwr supply 1
Power up
pwr supply 2
Figure 6-12. File Format: nodes.ccs, APC Single Fencing Method, Dual Power Supply
Page 71
Chapter 6. Creating the Cluster Configuration System Files 57
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFNAME= "IPAddress"
}
fence {
MethodName {
DeviceName {
port = PortNumber
}
}
}
}
NodeName {
.
.
.
}
}
File format for node
identification (same
format for all nodes)
File format for
WTI NPS fencing
method
Figure 6-13. File Format: nodes.ccs, WTI NPS, Single Fencing Method
Page 72
58 Chapter 6. Creating the Cluster Configuration System Files
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFNAME= "IPAddress"
}
fence {
MethodName {
DeviceName {
port = PortNumber
}
DeviceName {
port = PortNumber
}
}
}
}
NodeName {
.
.
.
}
}
File format for node
identification (same
format for all nodes)
File format for
Brocade or Vixel
FC-Switch fencing
method
Additional fencing
device: one required
for each additional
FC path
Figure 6-14. File Format: nodes.ccs, Brocade or Vixel FC Switch, Single Fencing Method
Page 73
Chapter 6. Creating the Cluster Configuration System Files 59
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFNAME= "IPAddress"
}
fence {
MethodName {
DeviceName {
ipaddr = "IPAddress"
}
}
}
}
NodeName {
.
.
.
}
}
File format for node
identification (same
format for all nodes)
File format for
GNBD fencing
method
Figure 6-15. File Format: nodes.ccs, GNBD, Single Fencing Method
Page 74
60 Chapter 6. Creating the Cluster Configuration System Files
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFNAME= "IPAddress"
}
fence {
MethodName {
DeviceName {
localport = PortNumber
}
}
}
}
NodeName {
.
.
.
}
}
File format for node
identification (same
format for all nodes)
File format for
HP RILOE fencing
method
Figure 6-16. File Format: nodes.ccs, HP RILOE, Single Fencing Method
Page 75
Chapter 6. Creating the Cluster Configuration System Files 61
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFNAME= "IPAddress"
}
fence {
MethodName {
DeviceName {
ipaddr = "IPAddress"
}
}
}
}
NodeName {
.
.
.
}
}
File format for node
identification (same
format for all nodes)
File format for
manual fencing
method
Figure 6-17. File Format: nodes.ccs, Manual Fencing Method
Page 76
62 Chapter 6. Creating the Cluster Configuration System Files
Node that will use
cascaded fencing
methods
Fencing method 1
Fencing method 2
Fencing method 3
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFName = "IPAddress"
}
fence {
MethodName {
DeviceName {
#Device-specific parameter(s)
}
}
MethodName {
DeviceName {
#Device-specific parameter(s)
}
}
MethodName {
DeviceName {
#Device-specific parameter(s)
}
}
}
}
NodeName {
.
.
.
}
}
Cascades to next
if fencing fails
Cascades to next
if fencing fails
Returns to first
fencing method
if fencing fails
Figure 6-18. File Format: nodes.ccs, Cascaded Fencing
Page 77
Chapter 6. Creating the Cluster Configuration System Files 63
File format for
node identification
(same format for
all nodes)
File format for
fencing GNBD
server node
nodes {
NodeName {
.
.
.
}
NodeName {
ip_interfaces {
IFName = "IPAddress"
}
fence {
MethodName {
DeviceName {
#Device-specific param.
}
DeviceName {
ipaddr = "IPAddress"
}
}
}
}
NodeName {
.
.
.
}
}
Can use any
fencing agent
except fence_gnbd
Must use
fence_notify-gnbd
Figure 6-19. File Format: nodes.ccs, GNBD Server Node Fencing
Parameter Description
DeviceName The name of a fencing device to use with a node. Use a valid
fencing device name specified by a DeviceName parameter in
the fence.ccs file
(fence.ccs:fence_devices/DeviceName).
IFName The interface name of the IP address specified. For example:
eth0
IPAddress For the ip_interfaces section: The IP address of the node on
the interface specified.
For the fence section:
If GNBD fencing — The IP address of this node, the node to be
fenced.
If fencing a GNBD server node — Used with the
fence_notify_gnbd fencing agent. The IP address of this
node, the node to be fenced.
If manual fencing — IP address of this node, the node that needs
to be reset or disconnected from storage.
LoginPassword This is the password of the node to be fenced.
Page 78
64 Chapter 6. Creating the Cluster Configuration System Files
Parameter Description
MethodName A name describing the fencing method performed by the listed
devices. For example, a MethodName of power could be used
to describe a fencing method using an APC MasterSwitch. Or, a
MethodName of Cascade1 could be used to describe a
cascaded fencing method.
NodeName The Linux hostname of the node.
Note: Make sure that you use the Linux hostname and that the
primary IP address of the node is associated with the hostname.
Specifying a NodeName other than the Linux hostname (for
example the interface name) can cause unpredictable results —
especially if the node is connected to multiple networks. To
determine the hostname of a node, you can use the uname -n
command at the node. To verify the IP address associated with
the hostname, you can issue a ping command to the hostname.
PortNumber For power and FC switches: The port number on the switch to
which this node is connected.
For HP RILOE: This is an optional value that defines a local port
to be used. The default value is 8888.
SwitchNumber For use with APC MasterSwitch: When chaining more than one
switch, this parameter specifies the switch number of the port.
This entry is not required when only one switch is present. (The
default value is 1 if not specified.)
UserId The user ID of the node to be fenced.
Table 6-3. File Syntax Description: Variables for nodes.ccs
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
power {
apc1 {
port = 6
switch = 2
}
}
}
}
n02 {
.
.
.
}
}
Example 6-12. Node Defined for APC Fencing, Single Power Supply
Page 79
Chapter 6. Creating the Cluster Configuration System Files 65
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
power {
apc1 { <----------- Fencing device for power supply 1
port = 6
switch = 1
option = "off" <-- Power down power supply 1
}
apc2 { <----------- Fencing device for power supply 2
port = 7
switch = 2
option = "off" <-- Power down power supply 2
}
apc1 { <----------- Fencing device for power supply 1
port = 6
switch = 1
option = "on" <--- Power up power supply 1
}
apc2 { <----------- Fencing device for power supply 2
port = 7
switch = 2
option = "on" <--- Power up power supply 2
}
}
}
}
n02 {
.
.
.
}
}
Example 6-13. Node Defined for APC Fencing, Dual Power Supplies
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
power {
wti1 {
port = 1
}
}
}
}
n02 {
.
.
.
}
}
Example 6-14. Node Defined for WTI NPS Fencing
Page 80
66 Chapter 6. Creating the Cluster Configuration System Files
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
san {
silkworm1 {
port = 3
}
silkworm2 { <--- Additional fencing device, for additional
port = 4 path to FC storage
}
}
}
}
n02 {
.
.
.
}
}
Example 6-15. Node Defined for Brocade FC-Switch Fencing
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
san {
vixel1 {
port = 3
}
vixel2 { <---- Additional fencing device, for additional
port = 4 path to FC storage
}
}
}
}
n02 {
.
.
.
}
}
Example 6-16. Node Defined for Vixel FC-Switch Fencing
Page 81
Chapter 6. Creating the Cluster Configuration System Files 67
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
server {
gnbd {
ipaddr = "10.0.1.1"
}
}
}
}
n02 {
.
.
.
}
}
Example 6-17. Node Defined for GNBD Fencing
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
riloe {
riloe1 {
localport = 2345
}
}
}
}
n02 {
.
.
.
}
}
Example 6-18. Node Defined for HP RILOE Fencing
Page 82
68 Chapter 6. Creating the Cluster Configuration System Files
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
human {
admin {
ipaddr = "10.0.0.1"
}
}
}
}
n02 {
.
.
.
}
}
Example 6-19. Nodes Defined for Manual Fencing
nodes {
n01 {
ip_interfaces {
eth0 = "10.0.1.21"
}
fence {
san { <-- Fencing with Brocade FC switch
brocade1 {
port = 1
}
}
power { <-- Fencing with APC MasterSwitch
apc {
port = 1
switch = 1
}
}
}
}
n02 {
.
.
.
}
}
This example shows a node that can be fenced using a Brocade FC switch or an APC MasterSwitch.
If the node must be fenced, the fencing system first attempts to disable the node’s FC port. If that
operation fails, the fencing system attempts to reboot the node using the power switch.
Example 6-20. Nodes Defined for Cascaded Fencing
Page 83
Chapter 6. Creating the Cluster Configuration System Files 69
nodes {
n01 {
ip_interfaces {
hsi0 = "10.0.0.1"
}
fence {
power { <------------- APC MasterSwitch fencing device
apc1 {
port = 6
switch = 2
}
notify_gnbd { <------- Fence Notify GNBD fencing device,
ipaddr = "10.0.0.1" using fence_notify_gnbd fencing agent
}
}
}
n02 {
.
.
.
}
}
Example 6-21. GNBD Server Node Defined for APC Fencing, Single Power Supply and Fence
Notify GNBD Fencing Device
Page 84
70 Chapter 6. Creating the Cluster Configuration System Files
Page 85
Chapter 7.
Using the Cluster Configuration System
This chapter describes how to use the cluster configuration system (CCS) and consists of the following
sections:
• Section 7.1 Creating a CCS Archive
• Section 7.2 Starting CCS in the Cluster
• Section 7.3 Using Other CCS Administrative Options
• Section 7.4 Changing CCS Configuration Files
• Section 7.5 Alternative Methods to Using a CCA Device
• Section 7.6 Combining CCS Methods
Note
If your GFS cluster is configured for GNBD multipath, there are some considerations you must take
into account for the location of CCS files. Refer to Section 11.1 Considerations for Using GNBD
Multipath.
7.1. Creating a CCS Archive
A CCS archive is a collection of CCS configuration files that can be accessed by the cluster. The
ccs_tool command is used to create a CCS archive from a directory containing .ccs configuration
files. This command writes the archive to a shared pool called the CCA device.
A small pool volume may be used as the CCA device. You can determine the size of the CCA device
pool volume as follows: 2 KB per GFS node or 2 MB total, whichever is larger. (Refer to Section 5.5
Creating a Pool Volume and Section 5.6 Activating/Deactivating a Pool Volume for details on creating
and activating a pool volume for the CCA device.)
7.1.1. Usage
ccs_tool create Directory CCADevice
Directory
The relative path to the directory containing the CCS files for the cluster.
CCADevice
Specifies the name of the CCA device.
Page 86
72 Chapter 7. Using the Cluster Configuration System
7.1.2. Example
In this example, the name of the cluster is alpha, and the name of the pool is
/dev/pool/alpha_cca. The CCS configuration files in directory /root/alpha/ are used to
create a CCS archive on the CCA device /dev/pool/alpha_cca.
ccs_tool create /root/alpha/ /dev/pool/alpha_cca
7.1.3. Comments
• The -O (override) option can be specified after the command name (ccs_tool -O create) to
forcibly write over the current CCA device contents without a prompt.
Warning
Make sure that you specify the right device if you use the override option. Otherwise, data may be
lost.
• Depending on the size of the device, it may take several seconds to create a CCA device for the first
time due to initialization of the device.
• The ccs_tool command uses the Linux raw-device interface to update and read a CCA device
directly, bypassing operating system caches. Caching effects could otherwise create inconsistent
views of the CCA device between cluster nodes.
7.2. Starting CCS in the Cluster
Once a CCS archive has been created on a CCA device (refer to Section 7.1 Creating a CCS Archive
for details, if necessary), the CCS daemon (ccsd) should be started on all cluster nodes. All cluster
nodes must be able to see the CCA device before the daemon is started.
The CCS daemon provides an interface to configuration data that is independent of the specific location where the data is stored.
7.2.1. Usage
ccsd -d CCADevice
CCADevice
Specifies the name of the CCA device.
7.2.2. Example
In this example, the CCS daemon is started on a cluster node. This command should be run on all
cluster nodes:
ccsd -d /dev/pool/alpha_cca
Page 87
Chapter 7. Using the Cluster Configuration System 73
7.2.3. Comments
The CCS daemon (ccsd) uses the Linux raw-device interface to update and read a CCA device directly, bypassing operating system caches. Caching effects could otherwise create inconsistent views
of the CCA device between cluster nodes.
7.3. Using Other CCS Administrative Options
The following sections detail other administrative functions that can be performed by the ccs_tool
command.
7.3.1. Extracting Files from a CCS Archive
When extracting original CCS configuration files from a CCS archive, the ccs_tool extract command creates a new directory specified on the command line and recreates the CCS files in the directory. The CCS archive remains unaffected by this command.
7.3.1.1. Usage
ccs_tool extract CCADevice Directory
CCADevice
Specifies the name of the CCA device.
Directory
The relative path to the directory containing the CCS files for the cluster.
7.3.1.2. Example
In this example, the CCS files contained on the CCA device, /dev/pool/alpha_cca, are extracted
and recreated in directory /tmp/alpha-bak/.
ccs_tool extract /dev/pool/alpha_cca /tmp/alpha-bak/
7.3.2. Listing Files in a CCS Archive
The CCS configuration files contained within a CCS archive can be listed by using the ccs_tool list
command.
7.3.2.1. Usage
ccs_tool list CCADevice
CCADevice
Specifies the name of the CCA device.
Page 88
74 Chapter 7. Using the Cluster Configuration System
7.3.2.2. Example
This example causes the CCS files contained on the CCA device, /dev/pool/alpha_cca, to be
listed.
ccs_tool list /dev/pool/alpha_cca
7.3.3. Comparing CCS Configuration Files to a CCS Archive
The ccs_tool diff command can be used to compare a directory of CCS configuration files with
the configuration files in a CCS archive. The output from the ccs_tool diff command is displayed
for each corresponding file in the specified directory and the CCS archive.
7.3.3.1. Usage
ccs_tool diff CCADevice [Directory ]
CCADevice
Specifies the name of the CCA device.
Directory
The relative path to the directory containing the CCS files for the cluster.
7.3.3.2. Example
In this example, the CCS configuration files in directory /root/alpha/ are compared with the configuration files in CCA device /dev/pool/alpha_cca.
ccs_tool diff /dev/pool/alpha_cca /root/alpha/
7.4. Changing CCS Configuration Files
Based on the LOCK_GULM locking protocol, the following list defines what can or cannot be
changed in a CCS archive while a cluster is running. There are no restrictions to making changes
to configuration files when the cluster is offline.
• New nodes can be defined in the nodes.ccs file.
• Unused node definitions can be removed from the nodes.ccs file.
• New fencing devices can be defined in the fence.ccs file.
• The locking servers array (servers =) in cluster.ccs:cluster/lock_gulm cannot be
changed.
• The fencing parameters for an existing node definition in nodes.ccs can be changed.
• The IP address of an existing node definition in the nodes.ccs file can only be changed if the
node does not have any GFS file systems mounted and is not running a LOCK_GULM server.
Page 89
Chapter 7. Using the Cluster Configuration System 75
7.4.1. Example Procedure
This example procedure shows how to change configuration files in a CCS archive.
1. Extract configuration files from the CCA device into temporary directory /root/alpha-new/.
ccs_tool extract /dev/pool/alpha_cca /root/alpha-new/
2. Make changes to the configuration files in /root/alpha-new/.
3. Create a new CCS archive on the CCA device by using the -O (override) flag to forcibly overwrite the existing CCS archive.
ccs_tool -O create /root/alpha-new/ /dev/pool/alpha_cca
7.5. Alternative Methods to Using a CCA Device
If it is not possible to reserve shared storage for use as a CCA device, you can use two alternative
methods:
• Section 7.5.1 CCA File and Server
• Section 7.5.2 Local CCA Files
Neither of these methods requires shared storage to store CCS data.
7.5.1. CCA File and Server
The first alternative to a CCA device is to use a single network server to serve CCS configuration
files to all nodes in the cluster. If used, this CCS server is a single point of failure in a cluster. If a
single (non-redundant) LOCK_GULM server daemon is being used, it would be reasonable to run a
CCS server on the same node as the LOCK_GULM server. The CCS server does not have failover
capabilities.
The CCS server is called ccs_servd, it can be run on any node in or out of the cluster. When
CCS daemons (ccsd) are started on cluster nodes, the IP address of the node running ccs_servd is
specified instead of the name of the CCA device. The name of the cluster is also passed to ccsd.
The CCS server does not read CCS files directly; rather, it reads a CCA file that is a local file containing a CCS archive.
Steps related to CCS in the setup procedure must be modified to use a CCS server in place of a CCA
device.
Note
ccs_servd provides information to any computer that can connect to it. Therefore, ccs_servd should
not be used at sites where untrusted nodes can contact the CCS server.
7.5.1.1. Creating a CCA File
Like a CCA device, a CCA file is created by the ccs_tool command from a directory of CCS
configuration files. Instead of specifying a CCA device as the last parameter when creating an archive,
a local file name is specified. The ccs_tool command creates the named file, which is the CCA file.
That file should be named ClusterName with a .cca extension. (ClusterName is the usersupplied variable that specifies the name of the cluster.) The CCA file must be located on the node
that runs ccs_servd.
Page 90
76 Chapter 7. Using the Cluster Configuration System
7.5.1.1.1. Usage
ccs_tool create Directory CCAFile
Directory
The relative path to the directory containing the CCS files for the cluster.
CCAFile
Specifies the CCA file to create.
7.5.1.1.2. Example
In this example, the name of the cluster is alpha and the name of the CCA file is alpha.cca. The
CCA file is saved in the /etc/sistina/ccs-build/ directory, which is the default location where
ccs_servd looks for CCA files.
ccs_tool create /root/alpha/ /etc/sistina/ccs-build/alpha.cca
7.5.1.2. Starting the CCS Server
There are two parts to starting CCS in the cluster when using a CCS server. The first is starting
ccs_servd and the second is starting ccsd on all the cluster nodes. When starting ccs_servd,
no command line options are required unless the CCA file is saved in a location other than
/etc/sistina/ccs-build/.
7.5.1.2.1. Usage
ccs_servd
or
ccs_servd -p Path
Path
Specifies an alternative location of CCA files.
7.5.1.2.2. Examples
This example shows starting the CCS server normally; that is, using the default location for CCA files.
ccs_servd
This example shows starting the CCS server using a user-defined location for CCA files. In this case,
CCA files are saved in /root/cca/.
ccs_servd -p /root/cca/
Page 91
Chapter 7. Using the Cluster Configuration System 77
7.5.1.3. Starting the CCS Daemon
When using a CCS server, ccsd must connect to it over the network, and requires two parameters on
the ccsd command line: the IP address (and optional port number) of the node running ccs_servd,
and the name of the cluster.
7.5.1.3.1. Usage
ccsd -s IPAddress[:PortNumber] -c ClusterName
IPAddress
The IP address of the node running the CCS server.
:PortNumber
(Optional) The non-default port number. A colon and port number can optionally follow the
IPAddress to specify a non-default port number: IPAddress:PortNumber.
ClusterName
Specifies the name of the cluster. The CCS server uses this to pick the correct CCA file that is
named for the cluster.
7.5.1.3.2. Example
This example starts ccsd on a node for cluster alpha when using a CCS server with the IP address
shown.
ccsd -s 10.0.5.1 -c alpha
7.5.2. Local CCA Files
An alternative to both a CCA device and a CCS server is to replicate CCA files on all cluster nodes.
Note
Care must be taken to keep all the copies identical.
A CCA file is created using the same steps as for a CCS server. The CCA file is manually copied to
all cluster nodes.
7.5.2.1. Starting the CCS Daemon
When the CCS daemon is started on each node, it must be given the location of the local copy of the
CCA file.
Page 92
78 Chapter 7. Using the Cluster Configuration System
7.5.2.2. Usage
ccsd -f File
File
Specifies the local copy of the CCA file.
7.5.2.3. Example
This example starts ccsd on a node using a local copy of a CCA file.
ccsd -f /etc/sistina/ccs-build/alpha.cca
7.6. Combining CCS Methods
The shared block-device methodology described at the beginning of this chapter is the recommended
method for storing CCS data (Section 7.1 Creating a CCS Archive and Section 7.2 Starting CCS in
the Cluster). The advantages are that there is no server point-of-failure and that updates to the CCS
archive happen atomically.
However, not every cluster has every node connected to the shared storage. For example, a cluster
may be built with external lock servers that do not have access to the shared storage. In that case, the
client/server methodology (Section 7.5.1 CCA File and Server) could be employed, but that approach
introduces a server point-of-failure. Also, local file archives could be used on each node (Section 7.5.2
Local CCA Files), but that approach makes updating the CCS archives difficult.
The best approach for storing CCS data may be a combination of the shared-device method and the
local-files method. For example, the cluster nodes attached to shared storage could use the shareddevice method, and the other nodes in the cluster could use the local-files approach. Combining those
two methods eliminates the possible point-of-failure and reduces the effort required to update a CCS
archive.
Note
When you update a CCS archive, update the shared-device archive first, then update the local
archives. Be sure to keep the archives synchronized.
Page 93
Chapter 8.
Using Clustering and Locking Systems
This chapter describes how to use the clustering and locking systems available with GFS, and consists
of the following sections:
• Section 8.1 Locking System Overview
• Section 8.2 LOCK_GULM
• Section 8.3 LOCK_NOLOCK
8.1. Locking System Overview
The GFS OmniLock interchangeable locking/clustering system is made possible by the
lock_harness.o kernel module. The GFS kernel module gfs.o connects to one end of the
harness, and lock modules connect to the other end. When a GFS file system is created, the lock
protocol (or lock module) that it uses is specified. The kernel module for the specified lock protocol
must be loaded subsequently to mount the file system. The following lock protocols are available
with GFS:
• LOCK_GULM — Implements both OmniLock RLM and SLM and is the recommended choice
• LOCK_NOLOCK — Provides no locking and allows GFS to be used as a local file system
8.2. LOCK_GULM
OmniLock RLM and SLM are both implemented by the LOCK_GULM system.
LOCK_GULM is based on a central server daemon that manages lock and cluster state for all
GFS/LOCK_GULM file systems in the cluster. In the case of RLM, multiple servers can be run
redundantly on multiple nodes. If the master server fails, another "hot-standby" server takes over.
The LOCK_GULM server daemon is called lock_gulmd. The kernel module for GFS nodes using
LOCK_GULM is called lock_gulm.o. The lock protocol (LockProto) as specified when creating a
GFS/LOCK_GULM file system is called lock_gulm (lower case, with no .o extension).
8.2.1. Selection of LOCK_GULM Servers
The nodes selected to run the lock_gulmd server are specified in the cluster.ccs configuration file
(cluster.ccs:cluster/lock_gulm/servers). Refer to Section 6.6 Creating the cluster.ccs
File.
For optimal performance, lock_gulmd servers should be run on dedicated nodes; however, they can
also be run on nodes using GFS. All nodes, including those only running lock_gulmd, must be listed
in the nodes.ccs configuration file (nodes.ccs:nodes).
8.2.2. Number of LOCK_GULM Servers
You can use just one lock_gulmd server; however, if it fails, the entire cluster that depends on it
must be reset. For that reason, you can run multiple instances of the lock_gulmd server daemon
on multiple nodes for redundancy. The redundant servers allow the cluster to continue running if the
master lock_gulmd server fails.
Page 94
80 Chapter 8. Using Clustering and Locking Systems
Over half of the lock_gulmd servers on the nodes listed in the cluster.ccs file
(cluster.ccs:cluster/lock_gulm/servers) must be operating to process locking requests
from GFS nodes. That quorum requirement is necessary to prevent split groups of servers from
forming independent clusters — which would lead to file system corruption.
For example, if there are three lock_gulmd servers listed in the cluster.ccs configuration file,
two of those three lock_gulmd servers (a quorum) must be running for the cluster to operate.
A lock_gulmd server can rejoin existing servers if it fails and is restarted.
When running redundant lock_gulmd servers, the minimum number of nodes required is three; the
maximum number of nodes is five.
8.2.3. Starting LOCK_GULM Servers
If no lock_gulmd servers are running in the cluster, take caution before restarting them — you
must verify that no GFS nodes are hung from a previous instance of the cluster. If there are hung
GFS nodes, reset them before starting lock_gulmd servers. Resetting the hung GFS nodes before
starting lock_gulmd servers prevents file system corruption. Also, be sure that all nodes running
lock_gulmd can communicate over the network; that is, there is no network partition.
The lock_gulmd server is started with no command line options.
8.2.4. Fencing and LOCK_GULM
Cluster state is managed in the lock_gulmd server. When GFS nodes or server nodes fail, the
lock_gulmd server initiates a fence operation for each failed node and waits for the fence to complete
before proceeding with recovery.
The master lock_gulmd server fences failed nodes by calling the fence_node command with the
name of the failed node. That command looks up fencing configuration in CCS to carry out the fence
operation.
When using RLM, you need to use a fencing method that shuts down and reboots a node. With RLM
you cannot use any method that does not reboot the node.
8.2.5. Shutting Down a LOCK_GULM Server
Before shutting down a node running a LOCK_GULM server, lock_gulmd should be terminated
using the gulm_tool command. If lock_gulmd is not properly stopped, the LOCK_GULM server
may be fenced by the remaining LOCK_GULM servers.
Caution
Shutting down one of multiple redundant LOCK_GULM servers may result in suspension of cluster
operation if the remaining number of servers is half or less of the total number of servers listed in the
cluster.ccs file (cluster.ccs:lock_gulm/servers).
Page 95
Chapter 8. Using Clustering and Locking Systems 81
8.2.5.1. Usage
gulm_tool shutdown IPAddress
IPAddress
Specifies the IP address or hostname of the node running the instance of lock_gulmd to be
terminated.
8.3. LOCK_NOLOCK
The LOCK_NOLOCK system allows GFS to be used as a local file system on a single node.
The kernel module for a GFS/LOCK_NOLOCK node is lock_nolock.o. The lock protocol as specified when creating a GFS/LOCK_NOLOCK file system is called lock_nolock (lower case, with no
.o extension).
Caution
Do not allow multiple nodes to mount the same file system while LOCK_NOLOCK is used. Doing so
causes one or more nodes to panic their kernels, and may cause file system corruption.
Page 96
82 Chapter 8. Using Clustering and Locking Systems
Page 97
Chapter 9.
Managing GFS
This chapter describes the tasks and commands for managing GFS and consists of the following
sections:
• Section 9.1 Making a File System
• Section 9.2 Mounting a File System
• Section 9.3 Unmounting a File System
• Section 9.4 GFS Quota Management
• Section 9.5 Growing a File System
• Section 9.6 Adding Journals to a File System
• Section 9.7 Direct I/O
• Section 9.8 Data Journaling
• Section 9.9 Configuring atime Updates
• Section 9.10 Suspending Activity on a File System
• Section 9.11 Displaying Extended GFS Information and Statistics
• Section 9.12 Repairing a File System
• Section 9.13 Context-Dependent Path Names
• Section 9.14 Shutting Down a GFS Cluster
• Section 9.15 Restarting a GFS Cluster
9.1. Making a File System
Making a GFS file system is one of the final tasks in the process of configuring and setting up a
GFS cluster. (Refer to Chapter 4 Initial Configuration for more information.) Once a cluster is set
up and running, additional GFS file systems can be made and mounted without additional clusterconfiguration steps.
A file system is created on a block device, which is usually an activated Pool volume. (Refer to Chapter
5 Using the Pool Volume Manager for further details.) The following information is required to run
the gfs_mkfs command:
• Lock protocol/module name (for example, lock_gulm)
• Cluster name (from cluster.ccs)
• Number of nodes that may be mounting the file system
Page 98
84 Chapter 9. Managing GFS
9.1.1. Usage
gfs_mkfs -p LockProtoName -t LockTableName -j Number BlockDevice
Warning
Make sure that you are very familiar with using the LockProtoName and LockTableName parameters. Improper use of the LockProtoName and LockTableName parameters may cause file system
or lock space corruption.
LockProtoName
Specifies the name of the locking protocol (typically lock_gulm) to use.
LockTableName
This parameter has two parts separated by a colon (no spaces) as follows:
ClusterName:FSName
• ClusterName, the cluster name, is set in the cluster.ccs file
(cluster.ccs:cluster/name).
• FSName, the file system name, can be 1 to 16 characters long, and the name must be unique
among all file systems in the cluster.
Number
Specifies the number of journals to be created by the gfs_mkfs command. One journal is required for each node that mounts the file system. (More journals can be specified to allow for
easier, future expansion.)
BlockDevice
Usually specifies a pool volume, but any block device can be specified.
9.1.2. Examples
In this example, lock_gulm is the locking protocol that the file system uses. The cluster name is
alpha, and the file system name is gfs1. The file system contains eight journals and is created on the
pool0 block device.
gfs_mkfs -p lock_gulm -t alpha:gfs1 -j 8 /dev/pool/pool0
In this example, a second lock_gulm file system is made, which can be used in cluster alpha. The
file system name is gfs2. The file system contains eight journals and is created on the pool1 block
device.
gfs_mkfs -p lock_gulm -t alpha:gfs2 -j 8 /dev/pool/pool1
9.1.3. Complete Options
Table 9-1 describes the gfs_mkfs command options (flags and parameters).
Page 99
Chapter 9. Managing GFS 85
Flag Parameter Description
-b BlockSize Sets the file system block size to BlockSize.
Default block size is 4096 bytes.
-D Enables debugging output.
-h Help. Displays available options, then exits.
-J MegaBytes Specifies the size of the journal in megabytes. Default
journal size is 128 megabytes. The minimum size is
32 megabytes.
-j Number Specifies the number of journals to be created by the
gfs_mkfs command. One journal is required for
each node that mounts the file system.
Note: More journals than are needed can be specified
at creation time to allow for future expansion.
-P Tells the gfs_mkfs command that the underlying
device is a pool. The gfs_mkfs command then asks
the pool about its layout. The -p flag overrides the -j
and -J flags.
-p LockProtoName Specifies the name of the locking protocol to use.
Recognized cluster-locking protocols include:
LOCK_GULM — The standard GFS locking
module.
LOCK_NOLOCK — May be used when GFS is
acting as a local file system (one node only).
-O Prevents the gfs_mkfs command from asking for
confirmation before writing the file system.
-q Quiet. Do not display anything.
-r MegaBytes Specifies the size of the resource groups in megabytes.
Default resource group size is 256 megabytes.
-s Blocks Specifies the journal-segment size in file system
blocks.
-t LockTableName This parameter has two parts separated by a colon
(no spaces) as follows: ClusterName:FSName.
ClusterName is set in the cluster.ccs file
(cluster.ccs:cluster/name).
FSName, the file system name, can be 1 to 16
characters in length, and the name must be unique
among all file systems in the cluster.
-V Displays command version information, then exits.
Table 9-1. Command Options: gfs_mkfs
9.2. Mounting a File System
Before you can mount a GFS file system, the file system must exist (refer to Section 9.1 Making
a File System), the pool volume where the file system exists must be activated, and the supporting
clustering and locking systems must be started (refer to Chapter 4 Initial Configuration). After those
requirements have been met, you can mount the GFS file system as you would any Linux file system.
Page 100
86 Chapter 9. Managing GFS
9.2.1. Usage
mount -t gfs BlockDevice MountPoint
BlockDevice
Specifies the block device where the GFS file system resides.
MountPoint
Specifies the directory where the GFS file system should be mounted.
9.2.2. Example
In this example, the GFS file system on the pool0 block device is mounted on the /gfs1/ directory.
mount -t gfs /dev/pool/pool0 /gfs1
9.2.3. Complete Usage
mount -t gfs BlockDevice MountPoint -o option
The -o option consists of GFS-specific options (refer to Table 9-2) or acceptable standard Linux
mount -o options, or a combination of both. Multiple option parameters are separated by a comma
and no spaces.
Note
The mount command is a Linux system command. In addition to using GFS-specific options described in this section, you can use other, standard, mount command options (for example, -r). For
information about other Linux mount command options, see the Linux mount man page.
Table 9-2 describes the available GFS-specific options that can be passed to GFS at mount time.
Option Description
hostdata=nodename LOCK_GULM file systems use this information to set
the local node name, overriding the usual selection of
node name from uname -n.
lockproto=LockModuleName Allows the user to specify which locking protocol to
use with the file system. If LockModuleName is not
specified, the locking protocol name is read from the
file system superblock.
locktable=LockTableName Allows the user to specify which locking table to use
with the file system.
upgrade Upgrade the on-disk format of the file system so that
it can be used by newer versions of GFS.
Loading...