Red Hat ENTERPRISE LINUX 5 - GLOBAL FILE SYSTEM, Global File System User Manual
Red Hat Enterprise Linux 5
Global File System
Red Hat Global File System
Global File System
Red Hat Enterprise Linux 5 Global File System
Red Hat Global File System
Edition 4
Copyright © 2010 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons
Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available
at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this
document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,
Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity
Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
All other trademarks are the property of their respective owners.
1801 Varsity Drive
Raleigh, NC 27606-2072 USA
Phone: +1 919 754 3700
Phone: 888 733 4281
Fax: +1 919 754 3701
PO Box 13588 Research Triangle Park, NC 27709 USA
This book provides information about configuring, and maintaining Red Hat GFS (Red Hat Global File
System) for Red Hat Enterprise Linux 5.
iii
Introduction v
1. Audience ........................................................................................................................ v
2. Related Documentation ................................................................................................... v
3. Document Conventions ................................................................................................... vi
3.1. Typographic Conventions ..................................................................................... vi
3.2. Pull-quote Conventions ........................................................................................ vii
3.3. Notes and Warnings ........................................................................................... viii
4. Feedback ..................................................................................................................... viii
1. GFS Overview 1
1.1. New and Changed Features ......................................................................................... 2
1.2. Performance, Scalability, and Economy ......................................................................... 2
1.2.1. Superior Performance and Scalability ................................................................. 3
1.2.2. Economy and Performance ................................................................................ 3
1.3. GFS Software Components .......................................................................................... 4
1.4. Before Setting Up GFS ................................................................................................ 5
2. Getting Started 7
2.1. Prerequisite Tasks ........................................................................................................ 7
2.2. Initial Setup Tasks ........................................................................................................ 7
3. Managing GFS 9
3.1. Creating a File System ................................................................................................. 9
3.2. Mounting a File System .............................................................................................. 13
3.3. Unmounting a File System .......................................................................................... 15
3.4. Special Considerations when Mounting GFS File Systems ............................................ 15
3.5. Displaying GFS Tunable Parameters ........................................................................... 16
3.6. GFS Quota Management ............................................................................................ 17
3.6.1. Setting Quotas ................................................................................................ 17
3.6.2. Displaying Quota Limits and Usage .................................................................. 18
3.6.3. Synchronizing Quotas ...................................................................................... 20
3.6.4. Disabling/Enabling Quota Enforcement ............................................................. 21
3.6.5. Disabling/Enabling Quota Accounting ................................................................ 22
3.7. Growing a File System ............................................................................................... 23
3.8. Adding Journals to a File System ................................................................................ 25
3.9. Direct I/O ................................................................................................................... 27
3.9.1. O_DIRECT ....................................................................................................... 27
3.9.2. GFS File Attribute ............................................................................................ 28
3.9.3. GFS Directory Attribute .................................................................................... 28
3.10. Data Journaling ........................................................................................................ 29
3.11. Configuring atime Updates ...................................................................................... 30
3.11.1. Mount with noatime ..................................................................................... 31
3.11.2. Tune GFS atime Quantum ........................................................................... 31
3.12. Suspending Activity on a File System ........................................................................ 32
3.13. Displaying Extended GFS Information and Statistics ................................................... 33
3.13.1. Displaying GFS Space Usage ........................................................................ 33
3.13.2. Displaying GFS Counters ............................................................................... 34
3.13.3. Displaying Extended Status ............................................................................ 36
3.14. Repairing a File System ............................................................................................ 38
3.15. Context-Dependent Path Names ............................................................................... 40
3.16. The GFS Withdraw Function ..................................................................................... 41
A. Revision History 43
Global File System
iv
Index 45
v
Introduction
The Global File System Configuration and Administration document provides information about
configuring and maintaining Red Hat GFS (Red Hat Global File System). A GFS file system can be
implemented in a standalone system or as part of a cluster configuration. For information about Red
Hat Cluster Suite refer to Red Hat Cluster Suite Overview and Configuring and Managing a Red Hat
Cluster.
HTML and PDF versions of all the official Red Hat Enterprise Linux manuals and release notes are
available online at http://www.redhat.com/docs/.
1. Audience
This book is intended primarily for Linux system administrators who are familiar with the following
activities:
• Linux system administration procedures, including kernel configuration
• Installation and configuration of shared storage networks, such as Fibre Channel SANs
2. Related Documentation
For more information about using Red Hat Enterprise Linux, refer to the following resources:
• Red Hat Enterprise Linux Installation Guide — Provides information regarding installation of Red
Hat Enterprise Linux 5.
• Red Hat Enterprise Linux Deployment Guide — Provides information regarding the deployment,
configuration and administration of Red Hat Enterprise Linux 5.
For more information about Red Hat Cluster Suite for Red Hat Enterprise Linux 5, refer to the following
resources:
• Red Hat Cluster Suite Overview — Provides a high level overview of the Red Hat Cluster Suite.
• Configuring and Managing a Red Hat Cluster — Provides information about installing, configuring
and managing Red Hat Cluster components.
• Logical Volume Manager Administration — Provides a description of the Logical Volume Manager
(LVM), including information on running LVM in a clustered environment.
• Global File System 2: Configuration and Administration — Provides information about installing,
configuring, and maintaining Red Hat GFS2 (Red Hat Global File System 2).
• Using Device-Mapper Multipath — Provides information about using the Device-Mapper Multipath
feature of Red Hat Enterprise Linux 5.
• Using GNBD with Global File System — Provides an overview on using Global Network Block
Device (GNBD) with Red Hat GFS.
• Linux Virtual Server Administration — Provides information on configuring high-performance
systems and services with the Linux Virtual Server (LVS).
Introduction
vi
• Red Hat Cluster Suite Release Notes — Provides information about the current release of Red Hat
Cluster Suite.
Red Hat Cluster Suite documentation and other Red Hat documents are available in HTML,
PDF, and RPM versions on the Red Hat Enterprise Linux Documentation CD and online at http://
www.redhat.com/docs/.
3. Document Conventions
This manual uses several conventions to highlight certain words and phrases and draw attention to
specific pieces of information.
In PDF and paper editions, this manual uses typefaces drawn from the Liberation Fonts1 set. The
Liberation Fonts set is also used in HTML editions if the set is installed on your system. If not,
alternative but equivalent typefaces are displayed. Note: Red Hat Enterprise Linux 5 and later includes
the Liberation Fonts set by default.
3.1. Typographic Conventions
Four typographic conventions are used to call attention to specific words and phrases. These
conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold
Used to highlight system input, including shell commands, file names and paths. Also used to highlight
keycaps and key combinations. For example:
To see the contents of the file my_next_bestselling_novel in your current
working directory, enter the cat my_next_bestselling_novel command at the
shell prompt and press Enter to execute the command.
The above includes a file name, a shell command and a keycap, all presented in mono-spaced bold
and all distinguishable thanks to context.
Key combinations can be distinguished from keycaps by the hyphen connecting each part of a key
combination. For example:
Press Enter to execute the command.
Press Ctrl+Alt+F1 to switch to the first virtual terminal. Press Ctrl+Alt+F7 to
return to your X-Windows session.
The first paragraph highlights the particular keycap to press. The second highlights two key
combinations (each a set of three keycaps with each set pressed simultaneously).
If source code is discussed, class names, methods, functions, variable names and returned values
mentioned within a paragraph will be presented as above, in mono-spaced bold. For example:
File-related classes include filesystem for file systems, file for files, and dir for
directories. Each class has its own associated set of permissions.
Proportional Bold
1
https://fedorahosted.org/liberation-fonts/
Pull-quote Conventions
vii
This denotes words or phrases encountered on a system, including application names; dialog box text;
labeled buttons; check-box and radio button labels; menu titles and sub-menu titles. For example:
Choose System → Preferences → Mouse from the main menu bar to launch Mouse
Preferences. In the Buttons tab, click the Left-handed mouse check box and click
Close to switch the primary mouse button from the left to the right (making the mouse
suitable for use in the left hand).
To insert a special character into a gedit file, choose Applications → Accessories
→ Character Map from the main menu bar. Next, choose Search → Find… from
the Character Map menu bar, type the name of the character in the Search field
and click Next. The character you sought will be highlighted in the Character Table.
Double-click this highlighted character to place it in the Text to copy field and then
click the Copy button. Now switch back to your document and choose Edit → Paste
from the gedit menu bar.
The above text includes application names; system-wide menu names and items; application-specific
menu names; and buttons and text found within a GUI interface, all presented in proportional bold and
all distinguishable by context.
Mono-spaced Bold Italic or Proportional Bold Italic
Whether mono-spaced bold or proportional bold, the addition of italics indicates replaceable or
variable text. Italics denotes text you do not input literally or displayed text that changes depending on
circumstance. For example:
To connect to a remote machine using ssh, type ssh username@domain.name at
a shell prompt. If the remote machine is example.com and your username on that
machine is john, type ssh john@example.com.
The mount -o remount file-system command remounts the named file
system. For example, to remount the /home file system, the command is mount -o
remount /home.
To see the version of a currently installed package, use the rpm -q package
command. It will return a result as follows: package-version-release.
Note the words in bold italics above — username, domain.name, file-system, package, version and
release. Each word is a placeholder, either for text you enter when issuing a command or for text
displayed by the system.
Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and
important term. For example:
Publican is a DocBook publishing system.
3.2. Pull-quote Conventions
Terminal output and source code listings are set off visually from the surrounding text.
Output sent to a terminal is set in mono-spaced roman and presented thus:
books Desktop documentation drafts mss photos stuff svn
Introduction
viii
books_tests Desktop1 downloads images notes scripts svgs
Source-code listings are also set in mono-spaced roman but add syntax highlighting as follows:
package org.jboss.book.jca.ex1;
import javax.naming.InitialContext;
public class ExClient
{
public static void main(String args[])
throws Exception
{
InitialContext iniCtx = new InitialContext();
Object ref = iniCtx.lookup("EchoBean");
EchoHome home = (EchoHome) ref;
Echo echo = home.create();
System.out.println("Created Echo");
System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));
}
}
3.3. Notes and Warnings
Finally, we use three visual styles to draw attention to information that might otherwise be overlooked.
Note
Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note
should have no negative consequences, but you might miss out on a trick that makes your
life easier.
Important
Important boxes detail things that are easily missed: configuration changes that only
apply to the current session, or services that need restarting before an update will apply.
Ignoring a box labeled 'Important' won't cause data loss but may cause irritation and
frustration.
Warning
Warnings should not be ignored. Ignoring warnings will most likely cause data loss.
4. Feedback
If you spot a typo, or if you have thought of a way to make this manual better, we would love to
hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against the
component Documentation-cluster.
Be sure to mention the manual's identifier:
Feedback
ix
Bugzilla component: Documentation-cluster
Book identifier: Global_File_System(EN)-5 (2010-03-11T15:10)
By mentioning this manual's identifier, we know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If you have
found an error, please include the section number and some of the surrounding text so we can find it
easily.
x
Chapter 1.
1
GFS Overview
The Red Hat GFS file system is a native file system that interfaces directly with the Linux kernel file
system interface (VFS layer). When implemented as a cluster file system, GFS employs distributed
metadata and multiple journals. Red Hat supports the use of GFS file systems only as implemented in
Red Hat Cluster Suite.
Note
Although a GFS file system can be implemented in a standalone system or as part of a
cluster configuration, for the RHEL 5.5 release and later Red Hat does not support the use
of GFS as a single-node file system. Red Hat does support a number of high-performance
single node file systems which are optimized for single node and thus have generally
lower overhead than a cluster filesystem. Red Hat recommends using these file systems
in preference to GFS in cases where only a single node needs to mount the file system.
Red Hat will continue to support single-node GFS file systems for existing customers.
Note
Red Hat does not support using GFS for cluster file system deployments greater than 16
nodes.
GFS is based on a 64-bit architecture, which can theoretically accommodate an 8 EB file system.
However, the current supported maximum size of a GFS file system is 25 TB. If your system requires
GFS file systems larger than 25 TB, contact your Red Hat service representative.
When determining the size of your file system, you should consider your recovery needs. Running the
gfs_fsck command on a very large file system can take a long time and consume a large amount
of memory. Additionally, in the event of a disk or disk-subsytem failure, recovery time is limited by
the speed of your backup media. For information on the amount of memory the gfs_fsck command
requires, see Section 3.14, “Repairing a File System”.
When configured in a Red Hat Cluster Suite, Red Hat GFS nodes can be configured and managed
with Red Hat Cluster Suite configuration and management tools. Red Hat GFS then provides data
sharing among GFS nodes in a Red Hat cluster, with a single, consistent view of the file system name
space across the GFS nodes. This allows processes on different nodes to share GFS files in the
same way that processes on the same node can share files on a local file system, with no discernible
difference. For information about Red Hat Cluster Suite refer to Configuring and Managing a Red Hat
Cluster.
While a GFS file system may be used outside of LVM, Red Hat supports only GFS file systems that
are created on a CLVM logical volume. CLVM is a cluster-wide implementation of LVM, enabled by the
CLVM daemon clvmd, which manages LVM logical volumes in a Red Hat Cluster Suite cluster. The
daemon makes it possible to use LVM2 to manage logical volumes across a cluster, allowing all nodes
in the cluster to share the logical volumes. For information on the LVM volume manager, see Logical
Volume Manager Administration
Chapter 1. GFS Overview
2
Note
When you configure a GFS file system as a cluster file system, you must ensure that
all nodes in the cluster have access to the shared file system. Asymmetric cluster
configurations in which some nodes have access to the file system and others do not are
not supported.
This chapter provides some basic, abbreviated information as background to help you understand
GFS. It contains the following sections:
• Section 1.1, “New and Changed Features”
• Section 1.2, “Performance, Scalability, and Economy”
• Section 1.3, “GFS Software Components”
• Section 1.4, “Before Setting Up GFS”
1.1. New and Changed Features
This section lists new and changed features included with the initial release of Red Hat Enterprise
Linux 5.
• GULM (Grand Unified Lock Manager) is not supported in Red Hat Enterprise Linux 5. If your GFS
file systems use the GULM lock manager, you must convert the file systems to use the DLM lock
manager. This is a two-part process.
• While running Red Hat Enterprise Linux 4, convert your GFS file systems to use the DLM lock
manager.
• Upgrade your operating system to Red Hat Enterprise Linux 5, converting the lock manager to
DLM when you do.
For information on upgrading to Red Hat Enterprise Linux 5 and converting GFS file systems to use
the DLM lock manager, see Configuring and Managing a Red Hat Cluster.
• Documentation for Red Hat Cluster Suite for Red Hat Enterprise Linux 5 has been expanded and
reorganized. For information on the available documents, see Section 2, “Related Documentation”.
1.2. Performance, Scalability, and Economy
You can deploy GFS in a variety of configurations to suit your needs for performance, scalability, and
economy. For superior performance and scalability, you can deploy GFS in a cluster that is connected
directly to a SAN. For more economical needs, you can deploy GFS in a cluster that is connected to a
LAN with servers that use GNBD (Global Network Block Device).
The following sections provide examples of how GFS can be deployed to suit your needs for
performance, scalability, and economy:
• Section 1.2.1, “Superior Performance and Scalability”
• Section 1.2.2, “Economy and Performance”
Superior Performance and Scalability
3
Note
The deployment examples in this chapter reflect basic configurations; your needs might
require a combination of configurations shown in the examples.
1.2.1. Superior Performance and Scalability
You can obtain the highest shared-file performance when applications access storage directly. The
GFS SAN configuration in Figure 1.1, “GFS with a SAN” provides superior file performance for shared
files and file systems. Linux applications run directly on GFS nodes. Without file protocols or storage
servers to slow data access, performance is similar to individual Linux servers with directly connected
storage; yet, each GFS application node has equal access to all data files. GFS supports up to 125
GFS nodes.
Figure 1.1. GFS with a SAN
1.2.2. Economy and Performance
Multiple Linux client applications on a LAN can share the same SAN-based data as shown in
Figure 1.2, “GFS and GNBD with a SAN”. SAN block storage is presented to network clients as block
storage devices by GNBD servers. From the perspective of a client application, storage is accessed as
if it were directly attached to the server in which the application is running. Stored data is actually on
the SAN. Storage devices and data can be equally shared by network client applications. File locking
and sharing functions are handled by GFS for each network client.
Note
Clients implementing ext2 and ext3 file systems can be configured to access their own
dedicated slice of SAN storage.
Chapter 1. GFS Overview
4
Figure 1.2. GFS and GNBD with a SAN
Figure 1.3, “GFS and GNBD with Directly Connected Storage” shows how Linux client applications
can take advantage of an existing Ethernet topology to gain shared access to all block storage
devices. Client data files and file systems can be shared with GFS on each client. Application failover
can be fully automated with Red Hat Cluster Suite.
Figure 1.3. GFS and GNBD with Directly Connected Storage
1.3. GFS Software Components
Table 1.1, “GFS Software Subsystem Components” summarizes the GFS software components.
Before Setting Up GFS
5
Software Component Description
gfs.ko Kernel module that implements the GFS file system and is
loaded on GFS cluster nodes.
lock_dlm.ko A lock module that implements DLM locking for GFS. It plugs
into the lock harness, lock_harness.ko and communicates
with the DLM lock manager in Red Hat Cluster Suite.
lock_nolock.ko A lock module for use when GFS is used as a local file system
only. It plugs into the lock harness, lock_harness.ko and
provides local locking.
Table 1.1. GFS Software Subsystem Components
1.4. Before Setting Up GFS
Before you install and set up GFS, note the following key characteristics of your GFS file systems:
GFS nodes
Determine which nodes in the Red Hat Cluster Suite will mount the GFS file systems.
Number of file systems
Determine how many GFS file systems to create initially. (More file systems can be added later.)
File system name
Determine a unique name for each file system. Each file system name is required in the form of
a parameter variable. For example, this book uses file system names mydata1 and mydata2 in
some example procedures.
File system size
GFS is based on a 64-bit architecture, which can theoretically accommodate an 8 EB file system.
However, the current supported maximum size of a GFS file system is 25 TB. If your system
requires GFS file systems larger than 25 TB, contact your Red Hat service representative.
When determining the size of your file system, you should consider your recovery needs. Running
the gfs_fsck command on a very large file system can take a long time and consume a large
amount of memory. Additionally, in the event of a disk or disk-subsytem failure, recovery time
is limited by the speed of your backup media. For information on the amount of memory the
gfs_fsck command requires, see Section 3.14, “Repairing a File System”.
Journals
Determine the number of journals for your GFS file systems. One journal is required for each node
that mounts a GFS file system. Make sure to account for additional journals needed for future
expansion, as you cannot add journals dynamically to a GFS file system.
GNBD server nodes
If you are using GNBD, determine how many GNBD server nodes are needed. Note the hostname
and IP address of each GNBD server node for setting up GNBD clients later. For information on
using GNBD with GFS, see the Using GNBD with Global File System document.
Storage devices and partitions
Determine the storage devices and partitions to be used for creating logical volumes (via CLVM) in
the file systems.
6
Chapter 2.
7
Getting Started
This chapter describes procedures for initial setup of GFS and contains the following sections:
• Section 2.1, “Prerequisite Tasks”
• Section 2.2, “Initial Setup Tasks”
2.1. Prerequisite Tasks
Before setting up Red Hat GFS, make sure that you have noted the key characteristics of the GFS
nodes (refer to Section 1.4, “Before Setting Up GFS”). Also, make sure that the clocks on the GFS
nodes are synchronized. It is recommended that you use the Network Time Protocol (NTP) software
provided with your Red Hat Enterprise Linux distribution.
Note
The system clocks in GFS nodes must be within a few minutes of each other to prevent
unnecessary inode time-stamp updating. Unnecessary inode time-stamp updating
severely impacts cluster performance.
2.2. Initial Setup Tasks
Initial GFS setup consists of the following tasks:
1. Setting up logical volumes
2. Making a GFS files system
3. Mounting file systems
Follow these steps to set up GFS initially.
1. Using LVM, create a logical volume for each Red Hat GFS file system.
Note
You can use init.d scripts included with Red Hat Cluster Suite to automate
activating and deactivating logical volumes. For more information about init.d
scripts, refer to Configuring and Managing a Red Hat Cluster.
2. Create GFS file systems on logical volumes created in Step 1. Choose a unique name for each file
system. For more information about creating a GFS file system, refer to Section 3.1, “Creating a
File System”.
You can use either of the following formats to create a clustered GFS file system:
gfs_mkfs -p lock_dlm -t ClusterName:FSName -j NumberJournals BlockDevice
Loading...