Cambex Bus Adapter User Manual
Dynamic Path Failover Driver for AIX
FibreQuik Host Bus Adapter
Installation and User Guide
Cambex Corporation
115 Flanders Road
Westborough, MA 01581
Customer support
support@cambex.com
Document: 081-468-025
Date: 12/07/06
Rev.: H
Table of Contents
DYNAMIC PATH FAILOVER FOR AIX................................................ 3
Introduction............................................................................................ 3
SAN Management................................................................................... 4
Storage Management ............................................................................. 4
Path Management.................................................................................. 4
SCSI and FCP Drivers........................................................................... 5
Load Balancing...................................................................................... 5
HOW PREFERRED PATHS WORK........................................................ 6
Multiple LUNs........................................................................................ 6
More than two paths............................................................................... 6
Multiple Hosts........................................................................................ 7
ADAPTER SPECIFICATIONS.................................................................. 9
Trademarks ............................................................................................ 9
SOFTWARE AND HARDWARE INSTALLATION............................. 10
Driver Installation (command line)...................................................... 10
Driver Installation (smit) ..................................................................... 12
Hardware Installation.......................................................................... 13
AutoRecover......................................................................................... 14
AIX Configuration................................................................................ 15
Persistent Bindings............................................................................... 16
SMIT COMMANDS.................................................................................. 17
Commands............................................................................................ 17
Path Management Commands ............................................................. 18
Manage Licensing................................................................................ 19
How to set path preferences................................................................. 20
TROUBLESHOOTING ............................................................................ 22
ERROR MESSAGES................................................................................. 26
UPGRADING THE DRIVER................................................................... 27
APPENDIX A............................................................................................. 28
Identifying Fibre Channel Components............................................... 28
PC2000 LED Meanings ....................................................................... 31
PC4000 LED Meanings ....................................................................... 32
1
DECLARATION OF CONFORMITY.................................................... 33
Warning................................................................................................ 33
SAFETY NOTICES................................................................................... 34
FCC Warning....................................................................................... 34
Safety Notice ........................................................................................ 34
DYNAMIC PATH FAILOVER LICENSE AGREEMENT .................. 35
2
Dynamic Path Failover for AIX
Introduction
The Cambex FibreQuik™ Fibre Channel host bus adapters
provide attachment of Fibre Channel devices to PCI or Micro
Channel based servers. They support the SCSI-FCP protocol.
When used with the Cambex DPF (Dynamic Path Failover)
software, two adapters in a system provide greater throughput
and total protection from failure of a fibre channel component.
In addition, the Cambex AutoRecover daemon, cbxdpfar, will
automatically detect when a failed path has been repaired and
make it available as a standby path. For arrays that support
preferred paths, AutoRecover will automatically resume
operation on the repaired path, otherwise, an operator may
manually change paths to do load balancing.
Storage (SAN) and communications (LAN) networks have
resulted in a physical and logical separation between the storage
device (typically a SAN-connected RAID array), the storage
user (typically an Enterprise Server, which we will refer to
below as the server) and the administration station (typically a
remote workstation).
As storage systems have evolved from JBOD (just bunch of
disks) to complex shared RAID systems, the management
software has also grown in complexity. The following sections
describe the different levels of management software.
3
SAN Management
SAN Management is the top layer. It is responsible for
assigning the required access for each server to each storage
device. The unit of storage is the LUN (Logical Unit Number)
which acts the same as a single disk directly attached to the
server. In reality, a LUN is made up of portions of a number of
disks under the direction of a RAID controller in the storage
device. SAN Management typically requires a process running
on each managed server communicating with the manager
running on the administering station.
Storage Management
The Storage Management layer, or RAID manager, is the layer
that communicates with the RAID controllers on the storage
device. This is the layer used to create, modify or delete LUNs.
The Storage Management layer may provide SAN Management
functions, but only for the specific storage device being
managed. Storage management typically runs on a server
attached to the storage device (in-band) but may also run over a
network (out-of-band).
Path Management
Path Management includes a variety of software for managing
the various fibre channel components such as HBAs (Host Bus
Adapters), hubs and switches. It is used to display path status
and allow control over multiple paths between servers and
storage. To avoid having to access a large number of unrelated
software interfaces, integration of the various Path Management
components is usually done using a management tool
4
communicating via a network connection to each managed
device.
The Cambex Dynamic Path Failover driver is a Path
Management component that allows multiple paths to a storage
device and provides automatic failover between available paths.
Cambex Dynamic Path Failover software provides both a
command line interface and access through smit, the standard
integrated management tool for AIX.
SCSI and FCP Drivers
AIX makes two types of device drivers available. SCSI device
drivers support SCSI-2 disk and tape devices. FCP device
drivers support newer SCSI-3 disk and tape devices, which
provides a larger address space for target ID and LUN. When
using the FCP device driver, the adapter will be configured with
a name like fscsi2.
Load Balancing
Selection of which path to use happens in one of three ways.
Static load balancing allows the operator to select which path is
used to access a LUN. This is the default method. Preferred
Paths allow certain RAID arrays to automatically specify a path
choice. See the next section on Preferred Paths for more
information. Finally, turning on Load Balancing will send I/O
on all available paths using round robin scheduling.
5
How Preferred Paths work
The basic idea of multipathing is fairly simple. Access a disk on
one of two paths. If the path fails, move I/O activity to the other
path. We build on that basis by adding more factors into the
configuration and seeing what effect they have.
A path is defined as an end-to-end connection through a SAN,
ignoring any intermediate routing points. A complex SAN may
have multiple connections from an HBA to a storage device, but
this is invisible to the software. The combination of one HBA to
one target is one path, regardless of how it is routed.
Multiple LUNs
The effect of multiple LUNs on a RAID controller is that we
can choose to distribute them so as to balance the I/O across
multiple paths. The choice can be made in one of two ways. If
one RAID controller has preference over the other, we call this
"controller preference". If the selection is made in the host
software we call this "target preference". A third type of
preference is to lock a LUN to a particular HBA, called "HBA
preference"
.
More than two paths
With four paths (for example) we have to sort the paths into an
order in which they are chosen when the primary path fails. This
order should depend on getting the best performance when a
path segment fails. For example, if our performance is limited
by the IOPS that can be handled by each RAID controller, then
we want to balance the LUNs between the RAID controllers,
6
and set up preferences so that the remaining paths to the same
controller are preferred over paths to the alternate controller.
Example 1) Suppose we have two HBAs, a switch, and two
RAID controllers. A total of four cables, and four paths to each
LUN (2 HBAs x 2 RAID controllers). We use the RAID
controller preference to select which controller each LUN is
accessed on. The target preference lets me balance those
accesses across the two HBAs.
Example 2) We have two switches, 2 HBAs and 2 RAID
controllers, a fully redundant system. We add an inter-switch
link for in-band management, but we do not want to use this as
a data path if there are any other paths available. We set the ISL
paths' target preference lower than the direct paths, so they will
be used last.
Example 3) A 2G RAID array is connected to a 2G switch,
which does speed matching to two 1G HBAs in the host. To
avoid saturating either of the 1G paths, We use HBA preference
when balancing the LUNs across the paths.
Multiple Hosts
When we have more than two hosts accessing the same storage,
we find that the limit is usually the number of controller ports.
The HBAs will run below the maximum bandwidth, but the
controllers need to be balanced. Use the controller preferences
and target preferences to balance the loads.
Normally, each host accesses different LUNs through either
controller, so all paths are independent. One special case is
where a RAID controller has LUN groups. All LUNs in a LUN
group are accessed on the same controller. If the different LUNs
7
are accessed by different hosts, we could have the problem of
"thrashing". One host tries to move the LUN group to preferred
controller A while the other host tries to move the group to
preferred controller B. Anti-thrashing code prevents this
problem in the multipathing software.
8
Adapter Specifications
PC2000LC-HPSP
Bus PCI PCI-X 2.0
Data rate (each dir.) 100/200 100/200/400 MB/s
Cable length - copper 30
- multimode optical 500 70 m
- single mode optical 10
Voltage 3.3 or 5 3.3 V
Power 7.2 8.0 W
Shipping weight 10 10 oz
PC2000LC-A
Trademarks
Dynamic Path Failover and FibreQuik are trademarks of
Cambex Corporation.
PC4000 (Single Port)
PC4002 (Dual Port)
9
Software and Hardware Installation
Driver Installation (command line)
The examples assume that the driver CD is mounted at /mnt and
that you are installing on AIX 5.1. The name and revision level
of the uninstalled driver image may be examined by issuing the
command (as root):
installp -ld <device>
where <device> is the path to the drivers, such as /mnt/AIX4/
for the CD or the name of a driver directory on the hard disk.
Example output would be:
Fileset Name Level I/U Q Content
==============================================
PC1000.driver.obj 2.5.10.25 I b usr
# PC1000 Fibre Channel Adapter Driver (AIX 5L)
cbxdpf.driver.obj 5.1.0.23 I b usr
# Cambex Dynamic Path Failover Driver
Before installing a new version of the driver, delete the previous
version of the driver. Note that any file systems must be
unmounted, volume groups exported and SCSI devices must be
removed before removing the driver. Be sure to have a
complete backup of all data before updating the driver. The
command to remove the driver is:
installp -u <name>.driver.obj
where names are cbxdpf and PC1000 or PC4000 as shown
above.
10
Loading...