Apple XSAN 1.X User Manual

Xsan Deployment and Tuning Guide

Guidelines and examples for configuring Xsan storage area network volumes

Apple Computer, Inc.

Under the copyright laws, this manual may not be copied, in whole or in part, without the written consent of Apple.

The Apple logo is a trademark of Apple Computer, Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Option-Shift-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws.

Every effort has been made to ensure that the information in this manual is accurate. Apple is not responsible for printing or clerical errors.

Apple 1 Infinite Loop Cupertino, CA 95014-2084 408-996-1010 www.apple.com

Apple, the Apple logo, Final Cut Pro, Mac, Macintosh, the Mac logo, Mac OS, and Xserve are trademarks of Apple Computer, Inc., registered in the U.S. and other countries.

Finder and Xsan are trademarks of Apple Computer, Inc.

StorNext and ADIC are registered trademarks of Advanced Digital Information Corporation.

Other company and product names mentioned herein are trademarks of their respective companies. Mention of third-party products is for informational purposes only and constitutes neither an endorsement nor a recommendation. Apple assumes no responsibility with regard to the performance or use of these products.

The product described in this manual incorporates copyright protection technology that is protected by method claims of certain U.S. patents and other intellectual property rights owned by Macrovision Corporation and other rights owners. Use of this copyright protection technology must be authorized by Macrovision Corporation and is intended for home and other limited viewing uses only unless otherwise authorized by Macrovision Corporation. Reverse engineering or disassembly is prohibited.

Apparatus Claims of U.S. Patent Nos. 4,631,603, 4,577,216, 4,819,098 and 4,907,093 licensed for limited viewing uses only.

Simultaneously published in the United States and Canada.

019-0242/09-02-05

Contents

Preface 5 About This Guide

Using This Guide

For More Information

Notation Conventions

Chapter 1 7 Setup and Tuning Guidelines

Setting Up the Ethernet TCP/IP Network

Set Up a Private Metadata Network

Use Switches Instead of Hubs

Setting Up the Fibre Channel Network

Verify Base Fibre Channel Performance

If Your Fibre Channel Fabric Is Running Slower Than Expected

Configuring Xserve RAID Systems

Install the Latest Firmware

Connecting Xserve RAID Systems to an Ethernet Network

10 10 10

11 12 13 13 13 15

Choosing RAID Levels for LUNs 7-Drive LUN vs 6-Drive LUN With Hot Spare Create One LUN per Xserve RAID Controller Adjusting Xserve RAID Fibre Channel Settings Adjusting Xserve RAID Performance Settings Estimating Base Xserve RAID Throughput

Configuring the Xsan File System

Organizing LUNs, Storage Pools, and Volumes Volume and Storage Pool Settings About SAN Write Latency

Chapter 2 17 Using the Xsan Tuner Application

17 17 18 18 18 18

Where to Get Xsan Tuner Installing Xsan Tuner Starting Xsan Tuner

About the Tests

The UNIX Read and Write Tests The Final Cut Pro Read and Write Tests

23 23

About the Test Files Performing a Test Interpreting Test Results

Understanding UNIX Test Results

Understanding Final Cut Pro Test Results

Chapter 3 27 Deployment Examples

Which Example Should You Follow?

Example 1: Network Attached Storage

29 29

32 32 32

35 36 37

Objectives of This Configuration

Deployment Decisions Example 2: Storage for Computational Clusters

Requirements for This Example

Small Computational Cluster Setup

A Larger Computational Cluster

Volume Configuration Example 3: Storage for Video or Film Production Group

Deployment Decisions

Contents

About This Guide

This guide shows you how to choose the best Xsan setup and configuration options for your users and applications.

The performance and availability of an Xsan volume depend on a variety of parameters, including how you organize available storage, how you configure the SAN’s Fibre Channel and Ethernet networks, and how you set basic file system parameters such as block size and stripe breadth. This guide goes beyond the basic instructions in the

Administrator’s Guide

from your Xsan volumes. The guide includes:

Guidelines for configuration choices

Instructions for using the Xsan Tuner application to measure SAN and volume performance

Sample configurations for network attached storage, high-performance computing, and video production

to help you choose options that result in the best performance

Xsan

Preface

Using This Guide

For general tips and guidelines, see Chapter 1.

For help using the Xsan Tuner application, see Chapter 2.

For sample deployments, see Chapter 3.

For More Information

The

Xsan Administrator’s Guide

contains basic instructions for setting up Xsan volumes along with information about managing Xsan volumes, including problem-solving tips and command-line alternatives for common tasks.

You can find the guide:

On the Xsan Installer disc

In the folder /Library/Documentation/Xsan on any computer where Xsan is installed

At www.apple.com/server/documentation

You can also check the Xsan web pages at www.apple.com/xsan.

Notation Conventions

The following conventions are used in this book wherever shell commands or other command-line items are described.

Notation Indicates

monospaced font

[text_in_brackets]

(one|other)

underlined

[...]

A command or other terminal text

A shell prompt

An optional parameter

Alternative parameters (type one or the other)

A parameter you must replace with a value

A parameter that may be repeated

A displayed value that depends on your SAN configuration

6 Preface

About This Guide

Setup and Tuning Guidelines

This chapter offers guidelines for making configuration choices that can affect SAN performance.

How quickly SAN clients can transfer data to and from Xsan volumes depends on a variety of factors, including:

The configuration of and load on the SAN’s Ethernet network

The layout and performance of the SAN’s Fibre Channel network

Settings for the Xserve RAID systems that provide LUNs

The organization of Xsan volumes and storage pools

Xsan file system settings

Setting Up the Ethernet TCP/IP Network

Ethernet connections are used in several ways in an Xsan storage area network:

Xsan clients and controllers use Ethernet to exchange volume metadata.

Xsan clients can use Ethernet for access to networks outside the SAN (campus or corporate intranet or the Internet).

Xsan controllers can use Ethernet connections for remote management.

Xserve RAID systems can use Ethernet connections for system management.

Fibre Channel switches can use Ethernet connections for switch management.

You have two basic options:

Use one Ethernet network for all traffic. This is the less expensive option, but is also less secure and might not provide the best possible performance.

Use two separate networks; one for metadata and another for all other IP traffic. This configuration is slightly more expensive (requiring two Ethernet adapters for each computer) but offers greater security and better performance because routine network traffic doesn’t interfere with SAN volume metadata traffic.

Set Up a Private Metadata Network

Non-SAN-related Ethernet traffic can interfere with the exchange of metadata among Xsan controllers and clients. For example, using the same connection for both Xsan metadata exchange and Internet access can slow file system performance. Similarly, using the same Ethernet network to connect client computers to directory services and SAN metadata can affect SAN performance.

If SAN performance is critical for your users or applications, keep all extraneous traffic off the network that clients and controllers use to exchange metadata. For best SAN performance, set up a private Ethernet TCP/IP network for the exclusive use of Xsan clients and controllers. For other types of network traffic, including Internet access, Xserve RAID and Fibre Channel switch management, remote SAN management, or directory services, connect each client or controller to a second, private SAN Ethernet network using a second network adapter.

Use Switches Instead of Hubs

Ethernet switches generally offer better performance than hubs. Use switches, not hubs, in the SAN Ethernet network.

Setting Up the Fibre Channel Network

Xsan uses Fibre Channel connections to:

Transfer user data directly between clients and data storage pools

Transfer metadata between controllers and metadata storage pools

Verify Base Fibre Channel Performance

Because the devices connected to a Fibre Channel network automatically adjust their speed to match the slowest device on the fabric, it is important to check that all connections in the fabric are operating at 2 GB/s.

To check Fibre Channel connection performance:

Use the management software provided with your Fibre Channel switches to test the performance of your Fibre Channel fabric.

If Your Fibre Channel Fabric Is Running Slower Than Expected

The following paragraphs list things you can check if your Fibre Channel fabric is not running at the expected 2 GB/s.

Check Cables

One faulty cable in a fabric can slow the entire network. Check all cables to make sure they are capable of full transmission speed. Use your switch management software to isolate the faulty cable by checking the performance of specific connections.

8 Chapter 1

Setup and Tuning Guidelines

Use Qualified Transceivers in Matching Pairs

Check with the manufacturers of the devices you are connecting to your fabric to be sure that the transceivers (GBICs) you are using are qualified for use with their devices.

Also, use identical transceivers (same manufacturer and model number) on both ends of each cable. Mismatched optical transceivers (even if they are both separately qualified for use with your devices) can cause Fibre Channel communication errors and degrade SAN performance.

Check Fibre Channel Switch Port Configuration

The Request for State Change Notifications (RSCN) that is generated when a client on the SAN restarts can cause dropped frames in video streams to other clients.

To avoid interrupting SAN traffic to other clients if one client restarts, check your Fibre Channel switch documentation to see if you can configure the switch to suppress RSCNs on initiator ports. (On Qlogic switches, for example, this feature is called I/O StreamGuard.)

Connect Devices to Specific Blades

If your Fibre Channel switch is based on a blade architecture, you might be able to improve performance by:

Â Connecting pairs of devices that routinely exchange large volumes of data to the

same blade in the switch

Â Distributing loads across multiple blades instead of concentrating all of the load on

one or two blades

Configuring Xserve RAID Systems

Follow these guidelines when you set up your Xserve RAID systems for use as Xsan LUNs.

Install the Latest Firmware

To be sure you get the best performance and reliability from your Xserve RAID systems, be sure to install the latest available firmware.

To check for firmware updates:

m Visit www.apple.com/support/xserve/raid/

Connecting Xserve RAID Systems to an Ethernet Network

For best performance, don’t connect Xserve RAID controller Ethernet management ports to the SAN’s metadata network. Connect the ports to a separate Ethernet network.

Chapter 1 Setup and Tuning Guidelines 9

Choosing RAID Levels for LUNs

Use RAID 1 for metadata LUNs and RAID 5 for data LUNs.

Use RAID 1 for Metadata LUNs

RAID 1 (mirroring) can give slightly better performance than the default RAID 5 scheme for the small, two-drive metadata LUNs that Xsan uses to store volume information. A single drive is almost always adequate for storing the primary volume metadata (10 GB of metadata space is enough for approximately 10 million files). The second, mirror drive protects you against metadata loss.

Use RAID 5 for Data LUNs

Xserve RAID systems are optimized for excellent performance and data redundancy using a RAID 5 scheme. (RAID 5 stripes data across the available drives and also distributes parity data across the drives.) Xserve RAID systems ship already configured as RAID 5 LUNs. RAID 0 (striping with no parity) might give slightly better write performance but provides no data recovery protection, so RAID 5 is always a better choice for LUNs used to store user data.

7-Drive LUN vs 6-Drive LUN With Hot Spare

For best performance, use full 7-drive RAID 5 LUNs. Even if a drive fails, the degraded 6-drive array will continue to provide excellent performance. Keep in mind, however, that the array is unprotected against the loss (however unlikely) of a second drive until someone replaces the original faulty drive.

If you can’t afford to have the LUN operating in a degraded state until someone replaces the faulty drive, you can configure your Xserve RAID systems as 6-drive RAID 5 arrays and use the seventh drive as a hot spare. Data on the faulty drive is reconstructed automatically without human intervention.

Create One LUN per Xserve RAID Controller

For high performance data sets, create only one LUN on each Xserve RAID controller (one array on each side of the system). Xserve RAID systems ship with one RAID 5 array on each controller.

Working With LUNs Larger Than 2 Terabytes

The capacity of an Xserve RAID array can exceed 2 terabytes (TB) if the system contains large drive modules. However, Xsan can’t use a LUN that is larger than 2 TB. If you set up your Xserve RAID systems as one array per controller, as suggested above, you can’t take advantage of the array capacity beyond 2 TB. To use as much available space as possible, you can move drive modules to other controllers or slice a large array into two smaller (less than 2 TB) LUNs. Slicing an array might, however, slow SAN performance.

Note: For the best possible SAN performance, don’t slice an array to create multiple LUNs on a single controller.

10 Chapter 1 Setup and Tuning Guidelines

Adjusting Xserve RAID Fibre Channel Settings

There are several Xserve RAID settings that can affect the Fibre Channel performance of the device and the SAN as a whole.

Fibre Channel Speed

Be sure the Fibre Channel connection is set to operate at 2 GB/s.

Fibre Channel Topology

To add an Xserve RAID system to a Fibre Channel fabric, set the topology to Automatic.

Disable Hard Loop ID

Don’t enable hard loop IDs for Xserve RAID systems in a Fibre Channel fabric.

To adjust Xserve RAID Fibre Channel settings:

m Open RAID Admin, choose a system, click Settings, and enter the management

password for the system. Then click Fibre Channel.

Adjusting Xserve RAID Performance Settings

Xserve RAID performance settings, affecting parameters such as drive caching, controller caching, and read prefetching, can have a significant effect on Xsan volume performance. Follow these guidelines.

Enable Drive Cache

In addition to the caching performed by the Xserve RAID controller, each drive in an array can perform its own caching at the drive level to improve performance.

Important: If you enable drive cache for an Xserve RAID set, be sure that the system is

connected to a UPS. Otherwise, you could lose cached data if the power fails.

To enable drive cache for an Xserve RAID array:

m Open the RAID Admin application, select the RAID system, and click Settings. Then click

Performance and enable Drive Cache for the array.

Enable Controller Write Cache

Without RAID controller write caching, a request to write data to the associated LUN is not considered finished until the data has been completely written to the physical disks that make up the array. Only then can the next write request be processed. (This is sometimes called “write-through caching.”)

When the RAID controller write cache is enabled, a request to write data is considered finished as soon as the data is in the cache. This is sometimes called “write-back caching.” Write requests are processed more quickly because the file system only needs to write to the fast cache memory and doesn’t need to wait for the slower disk drives.

Always be sure to enable write caching on controllers that support metadata storage pools.

Chapter 1 Setup and Tuning Guidelines 11

Although some large write requests might benefit from caching, often they do not. By placing a volume’s metadata storage pool on a controller separate from the data storage pools, you can configure the metadata controller to use caching and the data controller to run without caching.

When the file system is relying on caching in this way, you must guarantee that data in the cache isn’t lost before it is actually written to disk. Data that has been written to disk is safe if the power fails, but data in a cache is not. So, to be sure that a power failure can’t cause the loss of cached data, protect your Xserve RAID systems with controller backup batteries or an uninterruptable power supply (UPS).

Important: If you enable Controller Write Cache on an Xserve RAID system, be sure that

the system includes controller backup batteries and, preferably, is connected to a UPS.

To enable Xserve RAID write cache:

m Open the RAID Admin application, select the RAID system, and click Settings. Then click

Performance and enable Write Cache for each controller.

Set Read Prefetch to 8 Stripes

Read prefetch is a technique that improves file system read performance in cases where data is being read sequentially, as in the case of audio or video streaming, for example. When read prefetch is enabled, the controller assumes that a read request for a particular block of data will be followed by requests for subsequent, adjacent data blocks. To prepare for these requests, the controller reads not only the requested data, but also the following data, and stores it in cache memory. Then, if the data is actually requested, it is retrieved from the fast cache instead of from the slow disk drives.

Read prefetch is always enabled on Xserve RAID systems, though you can adjust the amount of data that is read. If you’re using other RAID systems, check the documentation to find out how to enable read prefetch.

To adjust the Xserve RAID read prefetch size:

m Open the RAID Admin application, select the RAID system, and click Settings. Then click

Performance and select a Read Prefetch size for each controller.

The default of 8 stripes is best for most applications.

Estimating Base Xserve RAID Throughput

To estimate how many Xserve RAID systems you need to support specific throughput requirements, you can assume that one Xserve RAID with 14 drives set up as two RAID 5 arrays can handle a minimum of 160 MB of data per second (80 MB/s per RAID controller). This value is applicable to video streaming applications; other applications might achieve higher data rates.

Overall performance is also affected by SAN latency; see “About SAN Write Latency” on page 15.

12 Chapter 1 Setup and Tuning Guidelines

+ 27 hidden pages