Xsan
Deployment and
Tuning Guide
Guidelines and examples for configuring Xsan storage area network volumes
K Apple Computer, Inc.
© 2005 Apple Computer, Inc. All rights reserved.
Under the copyright laws, this manual may not be copied, in whole or in part, without the written consent of Apple.
The Apple logo is a trademark of Apple Computer, Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Option-Shift-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws.
Every effort has been made to ensure that the information in this manual is accurate. Apple is not responsible for printing or clerical errors.
Apple
1 Infinite Loop
Cupertino, CA 95014-2084 408-996-1010 www.apple.com
Apple, the Apple logo, Final Cut Pro, Mac, Macintosh, the Mac logo, Mac OS, and Xserve are trademarks of Apple Computer, Inc., registered in the U.S. and other countries.
Finder and Xsan are trademarks of Apple Computer, Inc.
StorNext and ADIC are registered trademarks of Advanced Digital Information Corporation.
Other company and product names mentioned herein are trademarks of their respective companies. Mention of third-party products is for informational purposes only and constitutes neither an endorsement nor a recommendation. Apple assumes no responsibility with regard to the performance or use of these products.
The product described in this manual incorporates copyright protection technology that is protected by method claims of certain U.S. patents and other intellectual property rights owned by Macrovision Corporation and other rights owners. Use of this copyright protection technology must be authorized by Macrovision Corporation and is intended for home and other limited viewing uses only unless otherwise authorized by Macrovision Corporation. Reverse engineering or disassembly is prohibited.
Apparatus Claims of U.S. Patent Nos. 4,631,603, 4,577,216, 4,819,098 and 4,907,093 licensed for limited viewing uses only.
Simultaneously published in the United States and Canada.
019-0242/09-02-05
Contents
Preface |
5 |
About This Guide |
|
5 |
Using This Guide |
|
6 |
For More Information |
|
6 |
Notation Conventions |
Chapter 1 |
7 |
Setup and Tuning Guidelines |
|
7 |
Setting Up the Ethernet TCP/IP Network |
|
8 |
Set Up a Private Metadata Network |
|
8 |
Use Switches Instead of Hubs |
|
8 |
Setting Up the Fibre Channel Network |
|
8 |
Verify Base Fibre Channel Performance |
|
8 |
If Your Fibre Channel Fabric Is Running Slower Than Expected |
|
9 |
Configuring Xserve RAID Systems |
|
9 |
Install the Latest Firmware |
|
9 |
Connecting Xserve RAID Systems to an Ethernet Network |
|
10 |
Choosing RAID Levels for LUNs |
|
10 |
7-Drive LUN vs 6-Drive LUN With Hot Spare |
|
10 |
Create One LUN per Xserve RAID Controller |
|
11 |
Adjusting Xserve RAID Fibre Channel Settings |
|
11 |
Adjusting Xserve RAID Performance Settings |
|
12 |
Estimating Base Xserve RAID Throughput |
|
13 |
Configuring the Xsan File System |
|
13 |
Organizing LUNs, Storage Pools, and Volumes |
|
13 |
Volume and Storage Pool Settings |
|
15 |
About SAN Write Latency |
Chapter 2 |
17 |
Using the Xsan Tuner Application |
|
17 |
Where to Get Xsan Tuner |
|
17 |
Installing Xsan Tuner |
|
18 |
Starting Xsan Tuner |
|
18 |
About the Tests |
|
18 |
The UNIX Read and Write Tests |
|
18 |
The Final Cut Pro Read and Write Tests |
3
|
19 |
About the Test Files |
|
20 |
Performing a Test |
|
23 |
Interpreting Test Results |
|
23 |
Understanding UNIX Test Results |
|
24 |
Understanding Final Cut Pro Test Results |
Chapter 3 |
27 |
Deployment Examples |
|
27 |
Which Example Should You Follow? |
|
28 |
Example 1: Network Attached Storage |
|
29 |
Objectives of This Configuration |
|
29 |
Deployment Decisions |
|
32 |
Example 2: Storage for Computational Clusters |
|
32 |
Requirements for This Example |
|
32 |
Small Computational Cluster Setup |
|
34 |
A Larger Computational Cluster |
|
35 |
Volume Configuration |
|
36 |
Example 3: Storage for Video or Film Production Group |
|
37 |
Deployment Decisions |
4 |
Contents |
|
|
Preface
This guide shows you how to choose the best Xsan setup and configuration options for your users and applications.
The performance and availability of an Xsan volume depend on a variety of parameters, including how you organize available storage, how you configure the SAN’s Fibre Channel and Ethernet networks, and how you set basic file system parameters such as block size and stripe breadth. This guide goes beyond the basic instructions in the Xsan Administrator’s Guide to help you choose options that result in the best performance from your Xsan volumes. The guide includes:
ÂGuidelines for configuration choices
ÂInstructions for using the Xsan Tuner application to measure SAN and volume performance
ÂSample configurations for network attached storage, high-performance computing, and video production
ÂFor general tips and guidelines, see Chapter 1.
ÂFor help using the Xsan Tuner application, see Chapter 2.
ÂFor sample deployments, see Chapter 3.
5
The Xsan Administrator’s Guide contains basic instructions for setting up Xsan volumes along with information about managing Xsan volumes, including problem-solving tips and command-line alternatives for common tasks.
You can find the guide:
ÂOn the Xsan Installer disc
ÂIn the folder /Library/Documentation/Xsan on any computer where Xsan is installed
ÂAt www.apple.com/server/documentation
You can also check the Xsan web pages at www.apple.com/xsan.
The following conventions are used in this book wherever shell commands or other command-line items are described.
Notation |
Indicates |
monospaced font |
A command or other terminal text |
|
|
$ |
A shell prompt |
|
|
[text_in_brackets] |
An optional parameter |
|
|
(one|other) |
Alternative parameters (type one or the other) |
|
|
underlined |
A parameter you must replace with a value |
|
|
[...] |
A parameter that may be repeated |
|
|
<anglebrackets> |
A displayed value that depends on your SAN configuration |
|
|
6 |
Preface About This Guide |
|
|
Setup and Tuning Guidelines |
1 |
|
|
|
|
This chapter offers guidelines for making configuration choices that can affect SAN performance.
How quickly SAN clients can transfer data to and from Xsan volumes depends on a variety of factors, including:
ÂThe configuration of and load on the SAN’s Ethernet network
ÂThe layout and performance of the SAN’s Fibre Channel network
ÂSettings for the Xserve RAID systems that provide LUNs
ÂThe organization of Xsan volumes and storage pools
ÂXsan file system settings
Ethernet connections are used in several ways in an Xsan storage area network:
ÂXsan clients and controllers use Ethernet to exchange volume metadata.
ÂXsan clients can use Ethernet for access to networks outside the SAN (campus or corporate intranet or the Internet).
ÂXsan controllers can use Ethernet connections for remote management.
ÂXserve RAID systems can use Ethernet connections for system management.
ÂFibre Channel switches can use Ethernet connections for switch management.
You have two basic options:
ÂUse one Ethernet network for all traffic. This is the less expensive option, but is also less secure and might not provide the best possible performance.
ÂUse two separate networks; one for metadata and another for all other IP traffic. This configuration is slightly more expensive (requiring two Ethernet adapters for each computer) but offers greater security and better performance because routine network traffic doesn’t interfere with SAN volume metadata traffic.
7
Non-SAN-related Ethernet traffic can interfere with the exchange of metadata among Xsan controllers and clients. For example, using the same connection for both Xsan metadata exchange and Internet access can slow file system performance. Similarly, using the same Ethernet network to connect client computers to directory services and SAN metadata can affect SAN performance.
If SAN performance is critical for your users or applications, keep all extraneous traffic off the network that clients and controllers use to exchange metadata. For best SAN performance, set up a private Ethernet TCP/IP network for the exclusive use of Xsan clients and controllers. For other types of network traffic, including Internet access, Xserve RAID and Fibre Channel switch management, remote SAN management, or directory services, connect each client or controller to a second, private SAN Ethernet network using a second network adapter.
Ethernet switches generally offer better performance than hubs. Use switches, not hubs, in the SAN Ethernet network.
Xsan uses Fibre Channel connections to:
ÂTransfer user data directly between clients and data storage pools
ÂTransfer metadata between controllers and metadata storage pools
Because the devices connected to a Fibre Channel network automatically adjust their speed to match the slowest device on the fabric, it is important to check that all connections in the fabric are operating at 2 GB/s.
To check Fibre Channel connection performance:
mUse the management software provided with your Fibre Channel switches to test the performance of your Fibre Channel fabric.
The following paragraphs list things you can check if your Fibre Channel fabric is not running at the expected 2 GB/s.
Check Cables
One faulty cable in a fabric can slow the entire network. Check all cables to make sure they are capable of full transmission speed. Use your switch management software to isolate the faulty cable by checking the performance of specific connections.
8 |
Chapter 1 Setup and Tuning Guidelines |
|
|
Use Qualified Transceivers in Matching Pairs
Check with the manufacturers of the devices you are connecting to your fabric to be sure that the transceivers (GBICs) you are using are qualified for use with their devices.
Also, use identical transceivers (same manufacturer and model number) on both ends of each cable. Mismatched optical transceivers (even if they are both separately qualified for use with your devices) can cause Fibre Channel communication errors and degrade SAN performance.
Check Fibre Channel Switch Port Configuration
The Request for State Change Notifications (RSCN) that is generated when a client on the SAN restarts can cause dropped frames in video streams to other clients.
To avoid interrupting SAN traffic to other clients if one client restarts, check your
Fibre Channel switch documentation to see if you can configure the switch to suppress RSCNs on initiator ports. (On Qlogic switches, for example, this feature is called I/O StreamGuard.)
Connect Devices to Specific Blades
If your Fibre Channel switch is based on a blade architecture, you might be able to improve performance by:
ÂConnecting pairs of devices that routinely exchange large volumes of data to the same blade in the switch
ÂDistributing loads across multiple blades instead of concentrating all of the load on one or two blades
Follow these guidelines when you set up your Xserve RAID systems for use as Xsan LUNs.
To be sure you get the best performance and reliability from your Xserve RAID systems, be sure to install the latest available firmware.
To check for firmware updates:
m Visit www.apple.com/support/xserve/raid/
For best performance, don’t connect Xserve RAID controller Ethernet management ports to the SAN’s metadata network. Connect the ports to a separate Ethernet network.
Chapter 1 Setup and Tuning Guidelines |
9 |
|
|
Use RAID 1 for metadata LUNs and RAID 5 for data LUNs.
Use RAID 1 for Metadata LUNs
RAID 1 (mirroring) can give slightly better performance than the default RAID 5 scheme for the small, two-drive metadata LUNs that Xsan uses to store volume information.
A single drive is almost always adequate for storing the primary volume metadata (10 GB of metadata space is enough for approximately 10 million files). The second, mirror drive protects you against metadata loss.
Use RAID 5 for Data LUNs
Xserve RAID systems are optimized for excellent performance and data redundancy using a RAID 5 scheme. (RAID 5 stripes data across the available drives and also distributes parity data across the drives.) Xserve RAID systems ship already configured as RAID 5 LUNs. RAID 0 (striping with no parity) might give slightly better write performance but provides no data recovery protection, so RAID 5 is always a better choice for LUNs used to store user data.
For best performance, use full 7-drive RAID 5 LUNs. Even if a drive fails, the degraded 6-drive array will continue to provide excellent performance. Keep in mind, however, that the array is unprotected against the loss (however unlikely) of a second drive until someone replaces the original faulty drive.
If you can’t afford to have the LUN operating in a degraded state until someone replaces the faulty drive, you can configure your Xserve RAID systems as 6-drive RAID 5 arrays and use the seventh drive as a hot spare. Data on the faulty drive is reconstructed automatically without human intervention.
For high performance data sets, create only one LUN on each Xserve RAID controller (one array on each side of the system). Xserve RAID systems ship with one RAID 5 array on each controller.
Working With LUNs Larger Than 2 Terabytes
The capacity of an Xserve RAID array can exceed 2 terabytes (TB) if the system contains large drive modules. However, Xsan can’t use a LUN that is larger than 2 TB. If you set up your Xserve RAID systems as one array per controller, as suggested above, you can’t take advantage of the array capacity beyond 2 TB. To use as much available space as possible, you can move drive modules to other controllers or slice a large array into two smaller (less than 2 TB) LUNs. Slicing an array might, however, slow SAN performance.
Note: For the best possible SAN performance, don’t slice an array to create multiple
LUNs on a single controller.
10 |
Chapter 1 Setup and Tuning Guidelines |
|
|
There are several Xserve RAID settings that can affect the Fibre Channel performance of the device and the SAN as a whole.
Fibre Channel Speed
Be sure the Fibre Channel connection is set to operate at 2 GB/s.
Fibre Channel Topology
To add an Xserve RAID system to a Fibre Channel fabric, set the topology to Automatic.
Disable Hard Loop ID
Don’t enable hard loop IDs for Xserve RAID systems in a Fibre Channel fabric.
To adjust Xserve RAID Fibre Channel settings:
mOpen RAID Admin, choose a system, click Settings, and enter the management password for the system. Then click Fibre Channel.
Xserve RAID performance settings, affecting parameters such as drive caching, controller caching, and read prefetching, can have a significant effect on Xsan volume performance. Follow these guidelines.
Enable Drive Cache
In addition to the caching performed by the Xserve RAID controller, each drive in an array can perform its own caching at the drive level to improve performance.
Important: If you enable drive cache for an Xserve RAID set, be sure that the system is connected to a UPS. Otherwise, you could lose cached data if the power fails.
To enable drive cache for an Xserve RAID array:
mOpen the RAID Admin application, select the RAID system, and click Settings. Then click Performance and enable Drive Cache for the array.
Enable Controller Write Cache
Without RAID controller write caching, a request to write data to the associated LUN is not considered finished until the data has been completely written to the physical disks that make up the array. Only then can the next write request be processed. (This is sometimes called “write-through caching.”)
When the RAID controller write cache is enabled, a request to write data is considered finished as soon as the data is in the cache. This is sometimes called “write-back caching.”Write requests are processed more quickly because the file system only needs to write to the fast cache memory and doesn’t need to wait for the slower disk drives.
Always be sure to enable write caching on controllers that support metadata storage pools.
Chapter 1 Setup and Tuning Guidelines |
11 |
|
|
Although some large write requests might benefit from caching, often they do not. By placing a volume’s metadata storage pool on a controller separate from the data storage pools, you can configure the metadata controller to use caching and the data controller to run without caching.
When the file system is relying on caching in this way, you must guarantee that data in the cache isn’t lost before it is actually written to disk. Data that has been written to disk is safe if the power fails, but data in a cache is not. So, to be sure that a power failure can’t cause the loss of cached data, protect your Xserve RAID systems with controller backup batteries or an uninterruptable power supply (UPS).
Important: If you enable Controller Write Cache on an Xserve RAID system, be sure that the system includes controller backup batteries and, preferably, is connected to a UPS.
To enable Xserve RAID write cache:
mOpen the RAID Admin application, select the RAID system, and click Settings. Then click Performance and enable Write Cache for each controller.
Set Read Prefetch to 8 Stripes
Read prefetch is a technique that improves file system read performance in cases where data is being read sequentially, as in the case of audio or video streaming, for example. When read prefetch is enabled, the controller assumes that a read request for a particular block of data will be followed by requests for subsequent, adjacent data blocks. To prepare for these requests, the controller reads not only the requested data, but also the following data, and stores it in cache memory. Then, if the data is actually requested, it is retrieved from the fast cache instead of from the slow disk drives.
Read prefetch is always enabled on Xserve RAID systems, though you can adjust the amount of data that is read. If you’re using other RAID systems, check the documentation to find out how to enable read prefetch.
To adjust the Xserve RAID read prefetch size:
mOpen the RAID Admin application, select the RAID system, and click Settings. Then click Performance and select a Read Prefetch size for each controller.
The default of 8 stripes is best for most applications.
To estimate how many Xserve RAID systems you need to support specific throughput requirements, you can assume that one Xserve RAID with 14 drives set up as two RAID 5 arrays can handle a minimum of 160 MB of data per second (80 MB/s per RAID controller). This value is applicable to video streaming applications; other applications might achieve higher data rates.
Overall performance is also affected by SAN latency; see “About SAN Write Latency” on page 15.
12 |
Chapter 1 Setup and Tuning Guidelines |
|
|