Xsan
Administrator’s Guide
Information for setting up and managing Xsan volumes in a storage area network using Xsan Admin or the command line
For Xsan 1.1
K Apple Computer, Inc.
© 2005 Apple Computer, Inc. All rights reserved.
Under the copyright laws, this manual may not be copied, in whole or in part, without the written consent of Apple.
The Apple logo is a trademark of Apple Computer, Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Option-Shift-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws.
Every effort has been made to ensure that the information in this manual is accurate. Apple is not responsible for printing or clerical errors.
Apple
1 Infinite Loop
Cupertino, CA 95014-2084 408-996-1010 www.apple.com
Apple, the Apple logo, Final Cut Pro, Mac, Macintosh, the Mac logo, Mac OS, Power Mac, and Xserve are trademarks of Apple Computer, Inc., registered in the U.S. and other countries.
Finder and Xsan are trademarks of Apple Computer, Inc.
StorNext and ADIC are registered trademarks of Advanced Digital Information Corporation.
Other company and product names mentioned herein are trademarks of their respective companies. Mention of third-party products is for informational purposes only and constitutes neither an endorsement nor a recommendation. Apple assumes no responsibility with regard to the performance or use of these products.
The product described in this manual incorporates copyright protection technology that is protected by method claims of certain U.S. patents and other intellectual property rights owned by Macrovision Corporation and other rights owners. Use of this copyright protection technology must be authorized by Macrovision Corporation and is intended for home and other limited viewing uses only unless otherwise authorized by Macrovision Corporation. Reverse engineering or disassembly is prohibited.
Apparatus Claims of U.S. Patent Nos. 4,631,603, 4,577,216, 4,819,098 and 4,907,093 licensed for limited viewing uses only.
Simultaneously published in the United States and Canada.
034-3114-A/05-30-05
Preface |
9 |
About This Guide |
|
10 |
What’s New in Xsan 1.1 |
|
11 |
Version Compatibility |
|
11 |
Upgrading From an Earlier Version of Xsan |
|
12 |
Notation Conventions |
Chapter 1 |
13 |
Overview of Xsan |
|
13 |
What Is Xsan? |
|
14 |
Xsan Storage Area Networks |
|
15 |
Shared SAN Volumes |
|
15 |
Controllers and Clients |
|
15 |
SAN Connections |
|
16 |
How Xsan Storage Is Organized |
|
16 |
LUNs (RAID Arrays) |
|
17 |
Storage Pools |
|
17 |
Volumes |
|
18 |
Folders With Affinities |
|
19 |
How Xsan Utilizes Available Storage |
|
19 |
Metadata and Journal Data |
|
19 |
Striping at a Higher Level |
|
19 |
Security |
|
20 |
Expanding Storage |
|
20 |
Xsan Capacities |
Chapter 2 |
21 |
Setting Up a Storage Area Network |
|
21 |
Hardware and Software Requirements |
|
21 |
Supported Computers |
|
22 |
Supported Storage Devices |
|
22 |
Fibre Channel Fabric |
|
23 |
Ethernet TCP/IP Network |
|
23 |
Directory Services |
|
24 |
Outgoing Mail Service |
|
25 |
Planning Your SAN |
3
26 Planning Considerations and Guidelines
31Connecting Computers and Storage Devices
31Preparing LUNs (RAID Arrays and Slices)
32Using the Xsan Admin Application
32Installing Xsan Admin Separately
32Connecting Through a Firewall
32Xsan Admin Preferences
32Getting Help
32Using the Command Line
33SAN and Volume Setup Summary
34Setting Up an Xsan Volume on a Storage Area Network
48Renaming a SAN
48Deleting a SAN
49Setting Up Additional SANs
Chapter 3 |
51 |
Managing SAN Storage |
|
52 |
Adding Storage |
|
52 |
Restoring the Previous Volume Configuration After Changes |
|
52 |
About Rearranging Fibre Channel Connections |
|
53 |
Adding LUNs to a Storage Pool |
|
55 |
Adding a Storage Pool to a Volume |
|
56 |
Adding a Volume to a SAN |
|
57 |
Assigning a Storage Pool Affinity to a Folder |
|
58 |
Assigning an Affinity to a Folder Within a Folder |
|
59 |
Removing an Affinity |
|
60 |
Changing Storage Pool Settings |
|
61 |
Renaming a Storage Pool |
|
61 |
Choosing the Types of Files Stored on a Storage Pool |
|
62 |
Setting Storage Pool Stripe Breadth |
|
63 |
Setting Storage Pool Access Permissions |
|
63 |
Setting the Selection Method for Multiple Connections |
|
64 |
Changing Volume Settings |
|
65 |
Renaming a Volume |
|
66 |
Setting the Block Allocation Size |
|
67 |
Setting the Volume Allocation Strategy |
|
67 |
Checking Volume Fragmentation |
|
68 |
Defragmenting a Volume |
|
68 |
Checking the Integrity of a Volume |
|
69 |
Repairing a Volume |
Chapter 4 |
71 |
Managing Clients and Users |
|
72 |
Adding a Client |
|
73 |
Adding a Client to a StorNext SAN |
4 |
Contents |
|
|
73Mounting a Volume on a Client
74Controlling Client and User Access
74Controlling Access to Folders on Volumes
74Unmounting a Volume on a Client
74Restricting a Client to Read-Only Access
75Removing a Client From a SAN
75Removing Xsan Software From a Computer
76Setting User and Group Quotas
77About Xsan Quotas
78Checking User Quota Use
79Helping Clients Check Their Own Quotas
79 Creating Local Home Directories for Network Accounts
Chapter 5 |
81 |
Managing Metadata Controllers |
|
82 |
Adding a Controller |
|
83 |
Setting Controller Failover Priority |
|
83 |
Switching to a Standby Controller |
|
84 |
Finding Out Which Controller Is Hosting a Volume |
|
85 |
Listing the Volumes Hosted by a Controller |
|
86 |
Changing a Controller’s IP Address |
|
87 |
Upgrading Controller Software |
|
87 |
Monitoring Controller Status |
Chapter 6 |
89 |
Monitoring SAN Status |
|
90 |
Locking Xsan Admin Views for Secure Monitoring |
|
90 |
Checking Overall SAN Status |
|
91 |
Checking Overall Volume Status |
|
92 |
Checking Free Space on a Volume |
|
92 |
Checking Free Space on a Storage Pool |
|
92 |
Checking Quota Use |
|
93 |
Viewing a Controller’s CPU and Network Utilization |
|
93 |
Viewing File System CPU and Memory Utilization |
|
94 |
Setting Up Status Notifications |
|
94 |
Checking the Status of File System Processes |
|
95 |
Viewing Xsan Logs |
|
95 |
Checking Volume Clients |
|
96 |
Checking for Fibre Channel Connection Failures |
|
96 |
Checking the State of Xserve RAID Systems |
Chapter 7 |
97 |
Solving SAN Problems |
|
97 |
You’re Unable to Install the Xsan Software |
|
97 |
Some Computers Aren’t Listed in Xsan Admin |
|
97 |
You’re Unable to Connect to a Computer Using Xsan Admin |
Contents |
5 |
|
|
97You’re Unable to Mount a Volume on a Client
98Xserve RAID LUNs Aren’t Accessible Over Fibre Channel
98You’re Unable to Add a Storage Pool
99Some LUNs Aren’t Listed in Xsan Admin
99 Some LUNs Are Listed Twice in Xsan Admin
100Problems Using Command-Line Tools
100A Client User Sees Error Code –1425
100A LUN Doesn’t Have as Much Space as Expected
100Files and Folders Created by Mac OS 9 Computers Show the Wrong Creation Date
101You’re Unable to Rename an Xsan Volume in the Finder
101You’re Unable to Restart a Volume After Adding LUNs or Storage Pools
101Fibre Channel Performance Is Poorer Than Expected
101You’re Unable to Add LUNs to a Storage Pool
102A Client is Unable to Use a Volume After a Fibre Channel Interruption
102SAN Performance Declines Periodically and Predictably
Appendix A |
103 |
Combining Xsan and StorNext Clients and Controllers |
|
103 |
Compatible Software Versions |
|
104 |
Licensing |
|
104 |
Terminology |
|
105 |
Adding Macintosh Clients to a StorNext SAN |
|
106 |
Using Xsan Controllers With StorNext Clients |
Appendix B |
107 |
Using the Command Line |
|
107 |
Using the Shell Commands |
|
107 |
Sending Commands to Remote Computers |
|
107 |
Viewing the Man Pages |
|
108 |
Notation Conventions |
|
108 |
The Commands |
|
109 |
Viewing or Changing Volume and Storage Pool Settings (cvadmin) |
|
112 |
Copying Files or Folders (cvcp) |
|
113 |
Checking or Repairing a Volume (cvfsck) |
|
114 |
Labeling, Listing, and Unlabeling LUNs (cvlabel) |
|
114 |
Creating a Folder With an Affinity (cvmkdir) |
|
115 |
Creating and Pre-Allocating a File (cvmkfile) |
|
116 |
Initializing a Volume (cvmkfs) |
|
116 |
Applying Volume Configuration Changes (cvupdatefs) |
|
116 |
Starting a Volume Controller (fsm) |
|
117 |
Starting a Port Mapper Process (fsmpm) |
|
117 |
Defragmenting a File, Directory, or Volume (snfsdefrag) |
|
119 |
Mounting an Xsan Volume |
|
119 |
Unmounting an Xsan Volume |
|
119 |
Viewing Logs |
6 |
Contents |
|
|
120 The Configuration Files
120 The Volume Configuration File
123 Additional Examples
123 The Volume Auto-Start List
125 The Controller List
Glossary 127
Index 131
Contents |
7 |
|
|
Preface
Use this guide to learn how to set up and manage Xsan volumes on a storage area network.
This guide shows how to use Xsan to combine Xserve RAID arrays and slices into large, easy-to-expand volumes of storage that clients use like local disks but are actually shared over a high-speed Fibre Channel fabric.
Chapter 1 provides an overview of Xsan and how you can use it to organize RAID arrays into shared volumes of storage.
Chapter 2 includes hardware and software requirements, SAN planning guidelines, and basic steps for setting up an Xsan volume.
Chapter 3 contains instructions for expanding storage, creating folders with affinities, changing volume and storage pool settings, and checking, defragmenting, and repairing SAN volumes.
Chapter 4 shows how to add client computers to a SAN, mount volumes on clients, control client and user access to SAN files, and control user space using quotas.
Chapter 5 contains information on managing volume metadata controllers.
Chapter 6 includes instructions for monitoring and automatically reporting the condition of a SAN.
Chapter 7 lists solutions to common problems you might encounter.
Appendix A contains information to help you combine Xsan controllers or clients with ADIC StorNext controllers or clients in the same SAN.
Appendix B describes command-line utilities and configuration files you can use to manage an Xsan SAN using Terminal.
9
When you run the Xsan installer it installs either Xsan 1.1 for Mac OS X v10.3 or Xsan 1.1 for Mac OS X v10.4 to match the operating system of the target computer. Xsan 1.1 offers these new features and capabilities:
ÂYou can use the new Lock View command in Xsan Admin to monitor a SAN from an unsecured computer without the risk of a passerby mounting, unmounting, starting, or stopping volumes or otherwise changing the SAN configuration.
ÂYou can have Xsan Admin send an email notification to warn you if the Xsan software serial number on a controller or client is about to expire.
ÂYou can purchase and use site-licensed serial numbers.
ÂYou can take advantage of both block-level and byte-level locking.
ÂYou can use the Xsan User Quotas application to check user quotas from a client without using Xsan Admin.
Â(Mac OS X version 10.4 only) You can create files and volumes as large as 1024 terabytes (TB).
10 |
Preface About This Guide |
|
|
The following table shows which versions of Xsan and StorNext controllers and clients can be used in the same SAN.
Controller |
Client |
Compatible? |
Xsan 1.0.x or 1.1 (Mac OS X v10.3) |
Xsan 1.0.x or 1.1 (Mac OS X v10.3) |
Yes |
|
|
|
|
Xsan 1.1 (Mac OS X v10.4)) |
No |
|
|
|
|
StorNext 2.4 |
Yes |
|
|
|
|
StorNext 2.5 |
No |
|
|
|
Xsan 1.1 (Mac OS X v10.4) |
Xsan 1.0.x or 1.1 (Mac OS X v10.3) |
Yes |
|
|
|
|
Xsan 1.1 (Mac OS X v10.4) |
Yes |
|
|
|
|
StorNext 2.4 |
No |
|
|
|
|
StorNext 2.5 |
Yes |
|
|
|
StorNext 2.4 |
Xsan 1.0.x or 1.1 (Mac OS X v10.3) |
Yes |
|
|
|
|
Xsan 1.1 (Mac OS X v10.4) |
No |
|
|
|
|
StorNext 2.4 |
Yes |
|
|
|
|
StorNext 2.5 |
No |
|
|
|
StorNext 2.5 |
Xsan 1.0.x or 1.1 (Mac OS X v10.3) |
Yes |
|
|
|
|
Xsan 1.1 (Mac OS X v10.4) |
Yes |
|
|
|
|
StorNext 2.4 |
Yes |
|
|
|
|
StorNext 2.5 |
Yes |
|
|
|
Important: Note that if any client on your SAN is running Xsan 1.1 on Mac OS X v10.4, your controllers must all be running the same (Xsan 1.1 on Mac OS X v10.4). You can’t use a controller running Mac OS X v10.3 if any client is running Xsan 1.1 on Mac OS X v10.4.
For more information on migrating your SAN storage to Xsan 1.1 from an earlier version of Xsan, including tips for upgrading with the least impact on existing storage, see the Xsan Migration Guide at www.apple.com/server/documentation.
Preface About This Guide |
11 |
|
|
The following conventions are used in this book wherever shell commands or other command-line items are described.
Notation |
Indicates |
monospaced font |
A command or other terminal text |
|
|
$ |
A shell prompt |
|
|
[text_in_brackets] |
An optional parameter |
|
|
(one|other) |
Alternative parameters (type one or the other) |
|
|
underlined |
A parameter you must replace with a value |
|
|
[...] |
A parameter that may be repeated |
|
|
<anglebrackets> |
A displayed value that depends on your SAN configuration |
|
|
12 |
Preface About This Guide |
|
|
Overview of Xsan |
1 |
|
|
|
|
This chapter gives you an overview of Xsan and storage area networks.
Read this chapter for an overview of Xsan and how you can use it to set up a storage area network (SAN) to provide fast, shared storage.
Xsan is a storage area network file system and a management application (Xsan Admin) that you can use to provide users or applications on client computers with shared highspeed access to expandable storage.
Fibre |
Storage |
Channel |
pools |
RAID
arrays (LUNs)
13
A storage area network is a way of connecting computers to storage devices that gives users very fast access to files and gives administrators the ability to expand storage capacity as needed without interrupting users.
An Xsan SAN consists of:
ÂVolumes of shared storage, stored on Xserve RAID systems, available to clients as mounted volumes that they can use like local disks
ÂAt least one computer acting as a metadata controller that coordinates access to the shared volumes
ÂClient computers that access storage in accordance with established permissions and quotas
ÂUnderlying Fibre Channel and Ethernet networks
The following illustration shows the physical components of a typical Xsan SAN.
Metadata |
Standby |
controller |
controller |
Clients
Fibre
Channel switch
Xserve RAID storage
Ethernet - TCP/IP
(Private)
Ethernet - TCP/IP |
Intranet/ |
(Public) |
Internet |
14 |
Chapter 1 Overview of Xsan |
|
|
Users and applications see shared SAN storage as local volumes. Xsan volumes are logical disks made up of groups of RAID arrays. The elements you combine to create an Xsan volume are described under “How Xsan Storage Is Organized” on page 16.
When you add a computer to an Xsan SAN, you specify whether it will play the role of client, controller, or both.
Controllers
When you set up an Xsan SAN, you assign at least one computer to act as the controller. The controller manages the SAN volume metadata, maintains a file system journal, and controls concurrent access to files. Metadata includes such information as where files are actually stored and what portions of available storage are allocated to new files.
For high availability, you can add more than one controller to a SAN, as shown in the illustration on page 14. If the primary controller fails, the standby controller takes over. Controllers can also act as clients, so you can use a standby controller as a working client while the primary controller is operational.
Clients
The computers that users or applications use to access SAN volumes are called clients. Clients communicate with controllers over the Ethernet network but use Fibre Channel to send and retrieve file data to and from the RAID systems that provide storage for the volumes.
Xsan uses independent networks to connect storage devices, metadata controllers, and client computers: a Fibre Channel network and one or two Ethernet networks.
User Data Over Fibre Channel
User data is transferred over high-speed Fibre Channel connections. Controllers also use a Fibre Channel connection to move metadata to and from the volume.
Metadata Over Ethernet
To eliminate unnecessary traffic on the Fibre Channel connections, controllers and clients use an Ethernet network to exchange file system metadata. (When a controller reads or writes metadata on a volume, it uses Fibre Channel.) The Xsan Admin application also uses the Ethernet connection to let you manage the SAN.
To prevent Internet or intranet traffic from interfering with metadata communications, you can set up separate Ethernet networks as shown in the illustration.
Chapter 1 Overview of Xsan |
15 |
|
|
Fibre Channel Multipathing
Xsan can take advantage of multiple Fibre Channel connections between clients and storage. Xsan can alternate between connections for each read and write, or assign each LUN in a volume to one of the connections when the volume is mounted.
Users use an Xsan volume the same way they use a local disk. What they don’t see is that the SAN volume actually consists of numerous physical disks combined on several levels using RAID techniques.
The following illustration shows an example of how disk space provided by the individual drive modules in Xserve RAID systems is combined into a volume that users see as a large local disk.
Shared SAN
volume
Faster
Affinity |
Safer |
Affinity |
|
|
|
|
|
|
|
Storage pool |
|
|
|
|
|
|
|
|
|
|
|
|
|
Storage pool |
|
|
|
|
|
||||||||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Striping) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Striping) |
|
|
|
|
|
|
||||||||||
|
|
LUN |
|
|
|
LUN |
|
|
|
|
LUN |
|
|
|
LUN |
|
|
|
|
LUN |
|
|
|
LUN |
|
|
|
|
LUN |
|
|
|
LUN |
|
||||||
|
RAID 0 |
|
|
RAID 0 |
|
|
|
RAID 0 |
|
|
RAID 0 |
|
|
|
RAID 5 |
|
|
RAID 5 |
|
|
|
RAID 5 |
|
|
RAID 5 |
|
||||||||||||||
|
|
array |
|
|
array |
|
|
|
|
array |
|
|
array |
|
|
|
|
array |
|
|
array |
|
|
|
|
array |
|
|
array |
|
||||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The following paragraphs describe these storage elements and how you organize them to create shared Xsan volumes.
The smallest storage element you work with in Xsan is a logical storage device called a LUN (a SCSI logical unit number). In most storage area networks a LUN represents a group of drives such as a RAID array or a JBOD (just a bunch of disks) device. In Xsan, LUNs are Xserve RAID arrays or slices.
16 |
Chapter 1 Overview of Xsan |
|
|
You create a LUN when you use RAID Admin to create an Xserve RAID array. The controller hardware and software in the Xserve RAID system combine individual drive modules into an array based on the RAID scheme you choose. Each array appears on the network as a separate LUN. If you slice an array, each slice appears as a LUN.
One of your first tasks when you set up a SAN volume is to prepare LUNs. If the two RAID 5 arrays on a new Xserve RAID are not right for your application, you can use RAID Admin to create arrays based on other RAID schemes. For help choosing schemes for your LUNs, see “Choosing RAID Schemes for LUNs” on page 28.
The illustration on page 16 shows four Xserve RAID systems hosting two arrays each. Half of the arrays use a RAID 0 scheme (striping only) for speed while the others use RAID 5 (distributed parity) to ensure against data loss. Xsan sees the arrays as LUNs that can be combined to create a volume.
After your Xserve RAID LUNs are set up, you label and initialize them for use with the Xsan file system using Xsan Admin.
LUNs are combined to form storage pools. A storage pool in a small volume might consist of a single RAID array, but storage pools in many volumes consist of multiple arrays.
Xsan distributes file data in parallel across the LUNs in a storage pool using a RAID 0 (striping) scheme. So, you can improve a client’s access speed by distributing available storage over several LUNs in a storage pool.
You can set up storage pools that have different performance or recoverability characteristics and assign folders to them using affinities. Users can then select where to store files based on their need for speed or safety. See “Folders With Affinities” on page 18.
As an example, the illustration on page 16 shows eight LUNs combined into two storage pools, one pool consisting of RAID 0 (fast but not recoverable) arrays and the other made up of RAID 5 (not as fast, but recoverable) arrays. Xsan stripes data across the four LUNs in each storage pool.
You use Xsan Admin to add available LUNs to specific storage pools.
Storage pools are combined to create the volumes that users see. From the user’s perspective, the SAN volume looks and behaves just like a large local disk, except that:
ÂThe size of the volume can grow as you add underlying arrays or storage pools
ÂOther users on the SAN can access files on the volume at the same time
Chapter 1 Overview of Xsan |
17 |
|
|
In the example illustrated on page 16, two storage pools are combined to create a single shared volume.
You create volumes and mount them on client computers using the Xsan Admin application.
The following screen image shows how LUNs, storage pools, and volumes appear as you organize them in the Xsan Admin application. This example shows a SAN named “Editing SAN” with a single shared volume named “SanVol.” Storage for the volume is provided by two storage pools,“Meta” and “Data,” the first based on a single LUN and the second on two. Each of the LUNs is a 3-disk RAID 5 array on an Xserve RAID using 115 GB drive modules.
SAN
Volume
Storage pool
LUN
To control which storage pool is used to store specific files (for example, to provide different levels of service for different users or applications), you can associate a folder on an Xsan volume with one of the storage pools that make up the volume.
If, for example, you set up storage pools with different balances of performance and data redundancy, users can choose between faster and safer storage by putting files in the appropriate folder.
18 |
Chapter 1 Overview of Xsan |
|
|
In the illustration on page 16, a predefined folder has an affinity for the faster storage pool that is based on RAID 0 arrays. Any file that a user copies into this folder is automatically stored on the faster arrays. A second folder is associated with the more secure RAID 5 storage.
Xsan stores both user files and file system data on SAN volumes, and stripes data across the LUNs in a volume for better performance.
Xsan records information about the files in an Xsan volume using metadata files and file system journals. File system metadata includes information such as which specific parts of which disks are used to store a particular file and whether the file is being accessed. The journal data includes a record of file system transactions that can help ensure the integrity of files in the event of a failure.
These files are managed by the Xsan metadata controller, but are stored on SAN volumes, not on the controller itself. By default, metadata and journal data are stored on the first storage pool you add to a volume. You can use Xsan Admin to choose where these files are stored when you add storage pools to a new volume.
When you write a file to a RAID array using RAID 0 (striping), the file is broken into segments that are spread across the individual disk drives in the array. This improves performance by writing pieces of the file in parallel (instead of one piece at a time) to the individual disks in the array. Xsan applies this same technique at a second, higher level in the storage hierarchy. Within each storage pool in a volume, Xsan stripes file data across the individual LUNs that make up the storage pool. Once again, performance is improved because data is written in parallel.
You can tune SAN performance by adjusting the amount of data written to each LUN in a storage pool (the “stripe breadth”) to suit a critical application.
As SAN administrator, you can control access to shared volumes in several ways.
First, users cannot browse or mount SAN volumes. Only a SAN administrator can specify which volumes are mounted on which client computers. One way you can control access to data is to mount a volume only on appropriate client computers.
To prevent users from modifying data on a volume, you can mount the volume with read-only access.
Chapter 1 Overview of Xsan |
19 |
|
|
You can also control user access to folders on a volume by specifying owner, group, and general access permissions as you would in the Finder.
You can also set up zones in the underlying Fibre Channel network to segregate users and volumes.
There are two ways you can add free space to an Xsan volume:
ÂAdd Xserve RAID systems (new LUNs) to existing storage pools
ÂAdd entire new storage pools to volumes
Both methods require you to unmount and remount the volume on clients.
You can also add new volumes to a SAN at any time.
For information on expanding Xsan storage, see “Adding Storage” on page 52.
The following table lists limits and capacities for Xsan volumes.
Parameter |
Maximum |
Number of computers in a SAN (controllers and clients) |
64 |
|
|
Number of storage pools in a volume |
512 |
|
|
Number of LUNs in a storage pool |
32 |
|
|
Number of LUNs in a volume |
512 |
|
|
Number of files in a volume |
4,294,967,296 |
|
|
LUN size |
2 TB |
|
|
Volume size |
16 TB (Mac OS X v10.3) |
|
1024 TB (Mac OS X v10.4) |
|
|
File size |
16 TB (Mac OS X v10.3) |
|
1024 TB (Mac OS X v10.4) |
|
|
Volume name length |
70 characters |
|
|
File or folder name length |
251 characters |
|
|
SAN name length |
255 characters |
|
|
Storage pool name length |
255 characters |
|
|
LUN name (label or disk name) |
242 characters |
|
|
20 |
Chapter 1 Overview of Xsan |
|
|
Setting Up a Storage Area |
2 |
Network |
This chapter lists requirements, suggests planning tips, and gives instructions for setting up an Xsan SAN.
This chapter contains:
ÂXsan hardware and software requirements (page 21)
ÂPlanning guidelines (page 25)
ÂSAN setup instructions (page 33)
Your SAN environment needs to satisfy requirements in these areas:
ÂSupported computers
ÂSupported storage devices
ÂFibre Channel fabric, adapters, and switches
ÂEthernet network
ÂDirectory services (optional)
ÂOutgoing mail service (optional)
Xsan controller and client computers must meet these minimum requirements:
Systems
ÂXserve
ÂXserve G5
ÂPower Mac G4 dual 800 MHz or faster
ÂPower Mac G5
Memory
ÂClients should have a minimum of 256 MB of RAM.
ÂControllers should have a minimum of 512 MB of RAM. For optimum performance, add an additional 512 MB of RAM for each SAN volume hosted by the controller.
21
Supported Operating Systems
ÂMac OS X or Mac OS X Server v.10.3.9 (Xsan 1.1 for Mac OS X v10.3)
ÂMac OS X or Mac OS X Server v10.4 or later (Xsan 1.1 for Mac OS X v10.4)
Windows, AIX, IRIX, Linux, and Solaris clients must be running ADIC’s StorNext File
System version 2.4 or 2.5. For complete compatibility information, see “Version
Compatibility” on page 11.
Although you can use any standard LUN storage device, this guide assumes you are using Xserve RAID systems for your storage devices.
Important: Be sure to install the latest firmware update on your Xserve RAID systems before you use them with Xsan.
For demanding applications such as video editing, use Xserve RAID systems that have:
ÂA full set of 14 Apple Drive Modules
Â512 MB of cache in each controller (1 GB total)
Unlike file system metadata, which controllers and clients exchange over Ethernet, actual file content in an Xsan SAN is transferred over Fibre Channel connections (as is metadata that controllers access on a volume). To set up the connections, you need:
ÂApple Fibre Channel PCI or PCI-X cards for each client and controller computer
ÂOne or more supported Fibre Channel switches
ÂFibre Channel cables connecting computers and storage devices to the switches to form a Fibre Channel fabric
Fibre Channel PCI Cards
Install Apple Fibre Channel PCI or PCI-X cards in all Macintosh computers that will connect to the SAN.
Fibre Channel Switches
The following Fibre Channel switches have been tested with Xsan, Xserve RAID systems, and the Apple Fibre Channel PCI and PCI-X cards:
ÂBrocade Silkworm 3200, 3800, 3900, and 12000 series
ÂQLogic SANbox 2–8, SANbox 2–16, SANbox 2–64, and SANbox 5200
ÂEmulex SAN Switch 355, 375, and 9200
ÂCisco MDS 9000 family
For the latest additions to this list of qualified switches, see the Xsan webpages at
www.apple.com/xsan.
22 |
Chapter 2 Setting Up a Storage Area Network |
|
|
Fabric Configuration
You must connect the computers, storage devices, and switches in your Fibre Channel network to form a Fibre Channel “fabric.” In a fabric, Fibre Channel cables connect node ports (F or N_Port). See the documentation that came with your Fibre Channel switches for more information.
Note: If you are using a Vixel 355 switch, you must connect Xserve RAID systems to an FL (arbitrated loop) port on the switch.
You cannot use Fibre Channel Arbitrated Loop (FL ports) with an Xsan SAN, with the exception noted above for Vixel 355 switches.
Computers in the SAN must also be connected to an Ethernet network. Xsan uses this network instead of the Fibre Channel network to transfer file system metadata, reserving the Fibre Channel connections for actual file contents.
If the computers in your SAN need to communicate with directory servers, a corporate or campus intranet, or the Internet, you should connect each SAN client and controller to two separate Ethernet networks: one private subnet for the SAN and a separate connection for directory, intranet, or Internet traffic. This is important if you plan to use the SAN for high-performance applications such as video editing.
IP Addresses and Domain Names
For best results, assign fixed, non-routed IP addresses to all clients, controllers, and storage devices connected to the SAN Ethernet network. You can use the following ranges of IP addresses in your private (non-routed) subnet:
Private address range |
Associated subnet mask |
Comments |
10.0.0.0 – 10.255.255.255 |
255.0.0.0 |
10/8 |
|
|
|
172.16.0.0 – 172.31.255.255 |
255.240.0.0 |
172.16/12 |
|
|
|
192.168.0.0 – 192.168.255.255 |
255.255.0.0 |
192.168/16 |
|
|
|
If you plan to use user and group privileges to control access to files and folders on the SAN, you can simplify management by setting up or joining a user and group directory. Although not required, a central directory lets you manage users and groups on one computer instead of having to visit all SAN clients and controllers.
If you already have a directory, you can use the Directory Access application on each controller and client to access the directory for user and group information.
If you don’t use a central directory service, you need to set up users and groups on each SAN computer.
Chapter 2 Setting Up a Storage Area Network |
23 |
|
|
Important: If you create users and groups on each SAN computer, be sure that 1) each user or group has a numeric user ID (UID) or group ID (GID) that is unique throughout the SAN and 2) each user or group defined on more than one computer has the same UID or GID on each computer.
If you don’t have access to a directory, you can use the directory services in Mac OS X Server to set up an LDAP directory of SAN users and groups.
Note: If you do create your user accounts in a centralized directory, be aware that some applications running on SAN client computers, such as Final Cut Pro, for example, work better when users have local home directories. For help setting up local home directories for users with network accounts, see “Creating Local Home Directories for Network Accounts” on page 79.
Xsan can send SAN status notifications via email on your local subnet or corporate network without using a separate mail server. However, to send notifications outside your local network, you need an SMTP server to act as a mail gateway. If you don’t have access to an outgoing mail server, you can use the mail service in Mac OS X Server to set one up.
24 |
Chapter 2 Setting Up a Storage Area Network |
|
|
It’s easy to add storage to an existing Xsan SAN, but reorganizing a SAN after you set it up is not so simple. So, it’s important to plan the layout and organization of your SAN and its storage before you set it up.
An Xsan SAN is composed of:
ÂStorage devices (usually Xserve RAID systems)
ÂLUNs (logical unit numbers, usually RAID arrays)
ÂStorage pools (groups of LUNs)
ÂVolumes (groups of storage pools visible to users)
ÂClients (computers that use volumes)
ÂControllers (computers that manage volume metadata)
ÂUnderlying Fibre Channel and Ethernet networks
Before you use Xsan Admin to set up a SAN, decide how you want to organize these components. Take the time to create a drawing or a table that organizes available hardware into RAID arrays, storage pools, volumes, client computers, and controllers in a way that meets both your users’ needs and your needs as SAN administrator.
First, consider these questions:
ÂHow much storage do you need?
ÂHow do you want to present available storage to users?
ÂWhat storage organization makes the most sense for user workflow?
ÂWhat levels of performance do your users require?
ÂHow important is constant availability?
ÂWhat are your requirements for security?
Your answers to the above questions will help you decide the following:
ÂWhat RAID schemes should you use for your RAID arrays?
ÂHow many SAN volumes do you need?
ÂHow should individual volumes be organized?
ÂWhich LUNs go in each storage pool?
ÂWhich storage pools make up each volume?
ÂWhich clients, users, and groups should have access to each volume?
ÂWhich computers will act as controllers?
ÂDo you need standby controllers?
ÂDo you want to use controllers as clients also?
ÂWhere do you want to store file system metadata and journal data?
ÂWhat allocation strategy should you use?
Chapter 2 Setting Up a Storage Area Network |
25 |
|
|
Review the considerations and guidelines on the following pages for help translating your answers into a suitable SAN design.
The following paragraphs might help you make some of your SAN design decisions.
How Much Storage?
Because it’s easy to add storage to an Xsan SAN, you only need to decide on an adequate starting point. You can then add storage for user data as needed.
You can’t expand a storage pool that is used to store volume metadata and journal data. For information on estimating your metadata and journal data storage requirements, see “Estimating Metadata and Journal Data Storage Needs” on page 31.
Note that the number of Xserve RAID systems you use affects not only available space but also SAN performance. See “Performance Considerations” below.
How Should Users See Available Storage?
If you want the users working on a particular project to see a volume dedicated to their work, create a separate volume for each project. If it’s acceptable for a user to see a folder for his or her work on a volume with others’ folders, you can create a single volume and organize it into project folders.
Workflow Considerations
How much file sharing is required by your users’ workflow? If, for example, different users or groups work on the same files, either simultaneously or in sequence, it makes sense to store those files on a single volume to avoid having to maintain or hand off copies. Xsan uses file locking to manage shared access to a single copy of the files.
Performance Considerations
If your SAN supports an application (such as high resolution video capture and playback) that requires the fastest possible sustained data transfers, design your SAN with these performance considerations in mind:
ÂSet up the LUNs (RAID arrays) using a RAID scheme that offers high performance. See “Choosing RAID Schemes for LUNs” on page 28.
ÂGroup your fastest LUNs in storage pools reserved for the application. Reserve slower devices for a volume dedicated to less demanding or supporting applications.
ÂTo increase parallelism, spread LUNs across different Xserve RAID controllers. For example, instead of creating a single 4-disk LUN on one side of an Xserve RAID, create two 2-disk LUNs, one on each side, and add these LUNs to a storage pool.
Xsan then stripes data across the two LUNs and benefits from simultaneous transfers through two controllers.
ÂTo increase parallelism in a relatively small storage pool (the size of one or a few drive modules), create a slice of similar size across all the drives on a controller instead of creating the storage pool from just one or two drive modules.
26 |
Chapter 2 Setting Up a Storage Area Network |
|
|
ÂSpread file transfers across as many drives and RAID controllers as possible. Try creating slices across the drives in RAID systems, then combine these slices into a storage pool.
ÂTo increase throughput, connect both ports on client Fibre Channel cards to the fabric and set the multipathing method for the storage pool to Rotate.
ÂStore user files, file system metadata, and journal data on separate storage pools, and create these storage pools using LUNs from different RAID controllers.
ÂUse a router to isolate the Ethernet network used by the SAN from a company intranet or the Internet, or better, use a second Ethernet network (including a second Ethernet card in each SAN computer) for the SAN.
ÂIf your SAN uses directory services, mail services, or other services on a separate server, use a second, separate Ethernet network to connect SAN computers to that server.
ÂAs a rule of thumb, consider that a single Xserve RAID controller, after file system overhead, can transfer roughly 80 MB of user data per second (160 MB per Xserve RAID system). If your SAN must support an application running on multiple clients that requires specific throughput on each client, you can use this number to estimate the number of Xserve RAID systems necessary to support the aggregate transfer rate.
Availability Considerations
If high availability is important for your data, set up at least one standby controller in addition to your primary controller. Also, consider setting up dual Fibre Channel connections between each client, controller, and storage device using redundant Fibre Channel switches.
Important: Losing a metadata controller without a standby can result in the loss of all data on a volume. A standby controller is recommended.
Also, if you have a standby controller, you can upgrade the Xsan software without interrupting the SAN. For more information, see “Upgrading Controller Software” on page 87.
Security Considerations
If your SAN will support projects that need to be completely secure and isolated from each other, you can create separate volumes for each project to eliminate any possibility of the wrong client or user accessing files stored on a volume.
As SAN administrator, you control which client computers can use a volume. Clients can’t browse for or mount SAN volumes on their own. You use Xsan Admin to specify which clients a volume is mounted on.
You can also assign user and group permissions to folders you create on a volume or use standard file access permissions to control access to other items.
Chapter 2 Setting Up a Storage Area Network |
27 |
|
|
Choosing RAID Schemes for LUNs
Much of the reliability and recoverability of data in a SAN is not provided by Xsan itself but by the RAID arrays you combine to create your storage pools and volumes. Before you set up a SAN, you use RAID Admin to prepare LUNs based on specific RAID schemes.
Important: If a LUN belonging to an Xsan volume fails and can’t be recovered, all data on the volume is lost. It is strongly recommended that you use only redundant LUNs (LUNs based on RAID schemes other than RAID 0) to create your Xsan volumes.
LUNs configured as RAID 0 arrays (striping only) or LUNs based on single drives are difficult or impossible to recover if they fail. Unprotected LUNs such as these should only be used for volumes that contain scratch files or other data that you can afford to lose.
Xserve RAID systems ship already configured as recoverable, redundant RAID 5 arrays.
Xserve RAID supports all popular RAID levels. Each RAID scheme offers a different balance of performance, data protection, and storage efficiency, as summarized in the following table.
RAID level |
Storage efficiency |
Read performance |
Write performance |
Data redundancy |
RAID 0 |
Highest |
Very High |
Highest |
No |
|
|
|
|
|
RAID 1 |
Low |
High |
Medium |
Yes |
|
|
|
|
|
RAID 3 |
High to very high |
Medium |
Medium |
Yes |
|
|
|
|
|
RAID 5 |
High to very high |
High |
High |
Yes |
|
|
|
|
|
RAID 0+1 |
Low |
High |
High |
Yes |
|
|
|
|
|
RAID 10, 30, and 50 schemes assume the use of AppleRAID software striping and aren’t appropriate for use with Xsan, which performs its own striping. For more help choosing RAID schemes for your arrays, see the Xserve RAID User’s Guide or the Xserve RAID Technology Overview (at www.apple.com).
Deciding on the Number of Volumes
A volume is the largest unit of shared storage in the SAN. If your users need shared access to files, you should store those files on the same volume. This makes it unnecessary for them to pass copies of the files among themselves.
On the other hand, if security is critical, one way to control client access is to create separate volumes and mount only the authorized volume on each client.
For a more typical balance of security and shared access, a flexible compromise is to create a single volume and use folder access privileges to control access.
Note: The maximum size of a volume is 16 TB in Mac OS X v10.3 and 1024 TB in
Mac OS X v10.4. If you need more storage, you need to create more than one volume.
28 |
Chapter 2 Setting Up a Storage Area Network |
|
|
Deciding How to Organize a Volume
You can help users organize data on a volume or restrict users to specific areas of the volume by creating predefined folders. You can control access to these folders by assigning access permissions using Xsan Admin.
You can assign folders to specific storage pools using affinities. You can, for example, create a folder for data that requires fast access and assign that folder to your fastest storage pool.
Assigning LUNs to Storage Pools
You should set up a storage pool using LUNs that have similar capacity and performance characteristics.
To provide high performance, Xsan uses the RAID 0 scheme to stripe data across the LUNs in a storage pool. This requires that the LUNs in the pool be the same size. If you set up a storage pool using LUNs of different sizes, Xsan uses available space on each LUN equal to the capacity of the smallest LUN. If the LUNs vary in size, this can result in wasted capacity. For example, if you assign 240 GB and 360 GB RAID arrays to a storage pool, 120 GB of the larger array will not be used. By combining LUNs with similar capacities, you avoid wasting available storage.
If you want to set up a storage pool for use by a high performance application, assign similarly high speed LUNs. Assign slower LUNs to a storage pool where you keep data that doesn’t have critical performance requirements.
Creating storage pools from LUNs that are hosted on different drive modules and different RAID controllers increases performance by increasing the parallelism of data transfers. For example, a storage pool consisting of two LUNs, each a single drive module on the left side of an Xserve RAID, will not be as fast as a similarly sized storage pool made up of two LUNs that are single slices across all seven drives, one slice on each controller. In the first case, all transfers go through a single RAID controller to just two drives; in the second case the same transfer is spread across two RAID controllers and fourteen drives.
Assigning Storage Pools to Volumes
After you decide how to combine available LUNs into storage pools, assign the storage pools to the volumes you want to create.
For best performance, create separate storage pools for file system metadata and journal data.
Note: No storage pool or volume can be larger than 16 TB (Mac OS X v10.3) or 1024 TB (Mac OS X v10.4).
Chapter 2 Setting Up a Storage Area Network |
29 |
|
|
Deciding Which Clients to Mount a Volume On
If you create multiple volumes, decide which volumes should be mounted on which clients.
Choosing Controllers
You must choose at least one computer to be the SAN controller, the computer that is responsible for managing file system metadata.
Note: File system metadata and journal data are stored on selected SAN volumes, not on the controller itself. For more information, see “Choosing Where to Store Metadata and Journal Data” on page 30.
If you have a small number of clients or if performance is not critical you can use a single computer as both controller and client. You can even set up a SAN consisting of a single storage device and a single computer that acts as both controller and client (to provide network attached storage, for example).
If high availability is important, you should use at least two controllers, one as the primary controller and one as a standby. You can specify additional controllers as needed, and set their failover priorities to determine the order in which they are tried if the primary controller stops responding.
Choosing Standby Controllers
To be sure that SAN volumes are always available, set up at least one standby controller that can take over if your primary metadata controller fails. A standby controller also makes it possible for you to upgrade software on the controllers without interrupting user access to SAN volumes.
Combining Clients and Controllers
The same computer can function as both a metadata controller and a client. It’s possible, for example, to set up a SAN consisting of a single Xserve RAID and one computer that acts as both controller and client. Any computer you specify as a controller can also act as a client.
If, for example, you don’t have a computer to dedicate as a standby controller, you can assign a computer that is normally used as a client to take over controller duties if the primary controller fails.
To keep clients and controllers separate, you can set up client-only computers for your users.
Choosing Where to Store Metadata and Journal Data
The metadata and journal data that describe a volume are not stored on the volume’s metadata controller but on the volume itself. By default, they are stored on the first storage pool in the volume. If the volume consists of more than one storage pool, you can choose which storage pool is used to store metadata and journal data.
30 |
Chapter 2 Setting Up a Storage Area Network |
|
|