Symantec HP-UX User Manual
Veritas Storage Foundation
Cluster File System
Installation Guide
HP-UX
5.0
™
N18486G
Veritas Storage Foundation Cluster File System
Installation Guide
Copyright © 2006 Symantec Corporation. All rights reserved.
SFCFS 5.0
Symantec, the Symantec logo, Veritas, and Veritas Storage Foundation Cluster File
System are trademarks or registered trademarks of Symantec Corporation or its affiliates
in the U.S. and other countries. Other names may be trademarks of their respective
owners.
The product described in this document is distributed under licenses restricting its use,
copying, distribution, and decompilation/reverse engineering. No part of this document
may be reproduced in any form by any means without prior written authorization of
Symantec Corporation and its licensors, if any.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED
CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED
WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR
NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH
DISCLAIMERS ARE HELD TO BE LEGALLY INVALID, SYMANTEC CORPORATION SHALL
NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION
WITH THE FURNISHING PERFORMANCE, OR USE OF THIS DOCUMENTATION. THE
INFORMATION CONTAINED IN THIS DOCUMENTATION IS SUBJECT TO CHANGE
WITHOUT NOTICE.
The Licensed Software and Documentation are deemed to be “commercial computer
software” and “commercial computer software documentation” as defined in FAR
Sections 12.212 and DFARS Section 227.7202.
Symantec Corporation
20330 Stevens Creek Blvd.
Cupertino, CA 95014
www.symantec.com
Third-party legal notices
Third-party software may be recommended, distributed, embedded, or bundled
with this Veritas product. Such third-party software is licensed separately by its
copyright holder. All third-party copyrights associated with this product are
listed in the accompanying release notes.
HP-UX is a registered trademark of Hewlett-Packard Development Company, L.P.
Licensing and registration
Veritas Storage Foundation Cluster File System is a licensed product. See the
Veritas Storage Foundation Cluster File System Installation Guide for license
installation instructions.
Technical support
For technical assistance, visit http://support.veritas.com and select phone or
email support. Use the Knowledge Base search feature to access resources such
as TechNotes, product alerts, software downloads, hardware compatibility lists,
and our customer email notification service.
Contents
Chapter 1 Installing and configuring the product
Hardware overview ..............................................................................................10
Shared storage ..............................................................................................11
Fibre channel switch ...................................................................................11
Cluster platforms .........................................................................................11
Software components ..........................................................................................12
Required HP-UX patches ....................................................................................15
Preinstallation ......................................................................................................17
Release Notes ................................................................................................17
Product licensing .........................................................................................17
Setting PATH and MANPATH environment variables ..........................18
Secure communication ................................................................................18
Veritas Enterprise Administrator .............................................................19
Prerequisites .........................................................................................................19
Installing the product .........................................................................................20
Configuring the Components .............................................................................23
Using the log files ................................................................................................25
Installation log file .......................................................................................25
Response file .................................................................................................25
Summary file ................................................................................................25
Verifying the configuration files .......................................................................26
Low Latency Transport configuration files .............................................26
Checking Low Latency Transport operation ...........................................27
Group Membership and Atomic Broadcast configuration files ............29
Checking Group Membership and Atomic Broadcast operation ..........29
Checking cluster operation ........................................................................30
Verifying agent configuration ...........................................................................32
Synchronizing time on Cluster File Systems ...................................................32
Configuring VCS ...................................................................................................32
main.cf file ....................................................................................................33
SFCFS HA Only .............................................................................................34
VCS application failover services ......................................................................34
6
Chapter 2 Upgrading the product
Preparing to upgrade the product ..................................................................... 36
Planning the upgrade .................................................................................. 36
Upgrade paths ..............................................................................................36
Upgrade Overview ............................................................................................... 38
Phased upgrade ............................................................................................38
Full upgrade .................................................................................................. 38
Upgrading from 3.5 to 5.0 .................................................................................. 39
Phased upgrade ............................................................................................39
Full upgrade .................................................................................................. 42
Upgrading from 4.1 to 5.0 .................................................................................. 45
Phased upgrade ............................................................................................45
Full upgrade .................................................................................................. 48
Upgrading the disk layout versions .................................................................. 50
Chapter 3 Adding and removing a node
Adding a node to a cluster .................................................................................. 54
Configuring SFCFS and CVM agents on the new node .................................. 56
Removing a node from a cluster ........................................................................ 57
Chapter 4 Uninstalling the product
Appendix A Troubleshooting and recovery
Installation issues ................................................................................................63
Incorrect permissions for root on remote system .................................. 63
Resource temporarily unavailable ............................................................ 64
Inaccessible system ..................................................................................... 64
Storage Foundation Cluster File System problems ........................................ 64
Unmount failures ........................................................................................ 65
Mount failures .............................................................................................. 65
Command failures ........................................................................................ 66
Performance issues ..................................................................................... 66
High availability issues ...............................................................................67
Chapter
Installing and configuring the product
This chapter describes how to install the Veritas Storage Foundation Cluster
File System (SFCFS). SFCFS requires several Veritas software packages to
configure a cluster and to provide messaging services. These packages include
the Veritas Cluster Server (VCS) to monitor systems and application services,
Veritas Low Latency Transport (LLT) and Veritas Group Membership and
Atomic Broadcast (GAB) for messaging and cluster membership, the Veritas
Volume Manager (VxVM) to create the shared volumes necessary for cluster file
systems, and the Veritas File System (VxFS) package.
Topics covered in this chapter include:
1
■ Hardware overview
■ Software components
■ Preinstallation
■ Prerequisites
■ Installing the product
■ Using the log files
■ Verifying the configuration files
■ Verifying agent configuration
■ Synchronizing time on Cluster File Systems
■ Configuring VCS
■ VCS application failover services
10 Installing and configuring the product
Hardware overview
Hardware overview
VxFS cluster functionality runs optimally on a Fibre Channel fabric. Fibre
Channel technology provides the fastest, most reliable, and highest bandwidth
connectivity currently available. By employing Fibre Channel technology, SFCFS
can be used in conjunction with the latest Veritas Storage Area Network (SAN)
applications to provide a complete data storage and retrieval solution.
The figure below shows the configuration of a cluster file system on a Fibre
Channel fabric with a disk array.
Figure 1-1 Four Node SFCFS Cluster Built on Fibre Channel Fabric
Public
Private
Network
Public
Fiber Optic Private LAN
Fibre Channel Switch
Fiber Optic Connection
Disk Array
Shared storage
Shared storage can be one or more shared disks or a disk array connected either
directly to the nodes of the cluster or through a Fibre Channel Switch. Nodes can
also have non-shared or local devices on a local I/O channel. It is advisable to
have /, /usr, /var and other system partitions on local devices.
Fibre channel switch
Each node in the cluster must have a Fibre Channel I/O channel to access shared
storage devices. The primary component of the Fibre Channel fabric is the Fibre
Channel switch.
Cluster platforms
There are several hardware platforms that can function as nodes in a cluster file
system cluster.
See the Storage Foundation Cluster File System Release Notes.
Install the HP-UX 11i 64-bit operating system with the September 2004 HP-UX
11i Version 2.0 or later on each node and install a Fibre Channel host bus
adapter to allow connection to the Fibre Channel switch.
11Installing and configuring the product
Hardware overview
Note: For a cluster to work correctly, all nodes must have the same time. If you
are not running the Network Time Protocol (NTP) daemon, make sure the time
on all the systems comprising your cluster is synchronized.
12 Installing and configuring the product
Software components
Software components
Storage Foundation for Cluster File System is the name of the Veritas Cluster
File System product and its supporting software packages. Storage Foundation
Cluster File System HA (SFCFS HA) provides support for application failover
functionality of Veritas Cluster Server (VCS) in addition to other SFCFS features.
Packages installed with SFCFS only
The software packages listed below are required for implementing cluster file
system functionality. They are available on the software disc in the depot
directory and are installed on each node in the cluster using the
installation script.
The following table shows the package name and contents for each package:
Package Contents
VRTSperl Veritas Perl 5.8.8 Redistribution
VRTSvlic Veri tas Licensing
VRTSicsco Symantec Common Infrastructure
installer
VRTSpbx Symantec Private Branch Exchange
VRTSsmf Symantec Service Management Framework
VRTSat Symantec Product Authentication Service
VRTSobc33 Veritas Enterprise Administrator Core Service
VRTSob Veritas Enterprise Administrator Service
VRTSobgui Veritas Enterprise Administrator
VRTSccg Veritas Enterprise Administrator Central Control Grid
VRTSmh Veritas Centralized Management for Storage Foundation -
Managed Host
VRTSaa Veritas Enterprise Administrator Action Agent
VRTSspt Veritas Software Support Tools
SYMClma Symantec License Inventory Agent
VRTSllt Veritas Low Latency Transport
VRTSgab Veritas Group Membership and Atomic Broadcast
VRTSvxfen Veri tas I/O Fen cing
Software components
Package Contents
VRTSvcs Veritas Cluster Server
VRTSacclib Veritas ACC Library
VRTSvcsag Veritas Cluster Server Bundled Agents
VRTSvcsmg Veritas Cluster Server Message Catalogs
VRTSjre Veritas Java Runtime Environment Redistribution
VRTSjre15 Veritas Java Runtime Environment Redistribution
VRTScutil Veritas Cluster Utilities
VRTSweb Veritas Java Web Server
VRTScscw Veritas Cluster Server Configuration Wizards
VRTSvxvm Veritas Volume Manager Binaries
VRTSdsa Veritas Datacenter Storage Agent
VRTSfspro Veritas File System Management Services Provider
VRTSvmpro Veritas Volume Manager Management Services Provider
13Installing and configuring the product
VRTSdcli Veritas Distributed Command Line Interface
VRTSalloc Veritas Volume Manager Intelligent Storage Provisioning
VRTSvdid Veritas Device Identification API
VRTSvsvc Veritas Volume Server and Client Provider
VRTSddlpr Veritas Device Discovery Layer Services Provider
VRTSvrpro Veritas Volume Replicator Client Extension and Provider for
Veritas Enterprise Administrator
VRTSvcsvr Veritas Cluster Server Agents for VVR
VRTSvrw Veri tas Volume Repli cator Web Console
VRTSvxfs Veritas File System
VRTSfsman Veritas File System Manual Pages
VRTSfssdk Veritas File System Software Developer Kit
VRTSglm Veri tas Gro u p Lock M ana ger
VRTScavf Veritas Cluster Server Agents for Cluster File System
14 Installing and configuring the product
Software components
Optional packages for SFCFS and SFCFS HA
Packages Contents
VRTScfsdc Veritas Cluster File System Documentation
VRTScmccc Veritas Cluster Management Console Cluster Connector
VRTScmcs Veritas Cluster Management Console (Single Cluster Mode)
VRTScscm Veritas Cluster Server Cluster Manager
VRTScssim Veritas Cluster Server Simulator
VRTSfsdoc Veritas File System Documentation
VRTSvcsdc Veritas Cluster Server Documentation
VRTSvcsmn Veritas Cluster Server Man Pages
VRTSvmdoc Veritas Volume Manager Documentation
VRTSvrdoc Veritas Volume Replicator Documentation
VRTSfsmnd Veritas File System SDK Manuals
Note: SFCFS 5.0 operates only on HP-UX 11i 64-bit operating system with the
September 2004 HP-UX 11i Version 2.0 or later. All cluster nodes must be
running this OS version.
For cluster file system functionality to work reliably on HP-UX, you must have
the required HP-UX patches installed.
See “Required HP-UX patches” on page 15.
Required HP-UX patches
HP-UX required patches include the following:
HP-UX Patch ID Description
PHCO_32385 Enables fscat(1M).
PHCO_32387 Enables getext(1M).
PHCO_32388 Enables setext(1M).
PHCO_32389 Enables vxdump(1M).
PHCO_32390 Enables vxrestore(1M).
PHCO_32391 Enables vxfsstat(1M).
PHCO_32392 Enables vxtunefs(1M).
PHCO_32393 Enables vxupgrade(1M).
PHCO_32488 Enables LIBC for VxFS 4.1 and later file systems.
PHCO_32523 Enhancement to quota(1) for supporting large uids.
15Installing and configuring the product
Required HP-UX patches
PHCO_32524 Enhancement to edquota for supporting large uids.
PHCO_32551 Enhancement to quotaon/quotaoff for supporting large uids.
PHCO_32552 Enhancement to repquota for supporting large uids.
PHCO_32596 Enables df(1M).
PHCO_32608 Enables bdf(1M).
PHCO_32609 Enables fstyp(1M).
PHCO_32610 Enables mount(1M).
PHCO_32611 Fix fs_wrapper to accept vxfs from subtype.
PHCO_33238 swapon(1M) cumulative patch.
PHCO_34036 LVM com m a n ds pa tch .
PHCO_34208 SAM cumilative patch.
PHCO_34191 Cumulative libc patch.
PHSS_32674 Obam patch (backend for the SAM patch).
PHKL_31500 Sept04 Base Patch
PHKL_32272 Changes to fix intermittent failures in getacl/setacl.
16 Installing and configuring the product
Required HP-UX patches
HP-UX Patch ID Description
PHKL_32430 Changes to separate vxfs symbols from libdebug.a, so that
PHKL_32431 Changes to disallow mounting of a file system on a vnode having
PHKL_33312 LVM Cum u l ativ e Pat ch.
PHKL_34010 Cumulative VM Patch.
In addition to the above patches the EnableVXFS bundle needs to be installed
before installing the SFCFS 5.0. This bundle is a HP bundle and contains
enhancements to various commands to understand the new disk layout Version
6 and later. The EnableVXFS bundle contains the following patches:
Patch ID Description
FSLibEnh Enhancement to LIBC libraries to understand VxFS disk layout
DiskQuota-Enh Enhancements to various quota related commands to support
symbols of VxFS 4.1and later are easily available in q4/p4.
VNOMOUNT set. Enhancements for supporting quotas on large
uids.
Vers ion 6 and later.
large uids.
FSCmdsEnh Enhancements to the mount command to support VxFS 5.0.
Note: Install all the latest required HP-UX patches before you install SFCFS. You
can use the
swlist command to determine whether the correct update and
patches are installed. The installation procedure terminates if the correct
patches are not found. Make sure that EnableVXFS bundle has revision
B.11.23.04 or later after installing the latest patches.
Patches that supersede the ones in this list. To verify that you have the latest
patches, go to the Veritas support website to view the following TechNote:
http://support.veritas.com/docs/281875
Note: Most of the above patches are available in the Feature11i bundle. The
Feature11i bundle and the EnableVxFS bundle are available from HP software
download site http://h20293.www2.hp.com/. Search for Ve r i tas 5 .0 on this site
and follow the instructions.
Preinstallation
Release Notes
Read the Release Notes for all products included with this product. Portable
Document Format (.pdf) versions of the Release Notes are included on the
software disc in the
storage_foundation_cluster_file_system/release_notes directory
and on the documentation disc that came with your software.
Because product Release Notes are not installed by any packages, it is
recommended that you copy them from the disc to the /opt/VRTS/docs
directory on your system so that they are available for future reference.
Product licensing
Product keys are available on the License Key certificate. The certificate also
includes the number of product licenses purchased. A single key enables product
installation on the number and the type of systems for which you purchased the
license. A key may enable the operation of more products than specified on the
certificate, but you are legally limited to the number of product licenses
purchased.
The VRTSvlic package executes Veritas product licensing. The following
commands and their manual pages are available on the system after VRTSvlic
is installed:
vxlicinst installs a license key for a Veritas product
vxlicrep displays currently installed licenses
vxlictest retrieves features and their descriptions encoded in a license
key
If you encounter problems while licensing your product, visit the Veritas
licensing support website at
http://www.veritas.com/buy/vLicense/vLicenseHome.jhtml.
Veritas products are installed under the /opt directory on the specified host
systems. Verify that the directory /opt exists and has write permissions for
root before starting the installation procedure. Do not make /opt a VxFS file
system.
17Installing and configuring the product
Preinstallation
18 Installing and configuring the product
Preinstallation
Also, you can get the patches from Hewlett-Packard’s Patch Database offered
under the Maintenance and Support section of the HP Services & Support - IT
Resource Center. HP’s Patch Database provides fast, accurate searches for the
latest recommended and superseded patches available for Veritas File System or
Veritas Volume Manager.
Setting PATH and MANPATH environment variables
The software and online manual pages for the packages comprising SFCFS are
installed in several different directories. However, there are symbolic links to all
commands in the /opt/VRTS/bin directory, and symbolic links to all manual
pages in /opt/VRTS/man. To make all SFCFS commands and manual pages
accessible when you do the installation, add /opt/VRTS/bin to your PATH and
/opt/VRTS/man to your MANPATH environment variables. Command line
examples in this guide assume these environment variables are set.
To prevent conflicts with VxFS manual pages previously installed with
JFS/OnLineJFS 3.5, the VxFS 5.0 manual pages are installed in the
/opt/VRTS/vxfs5.0/man directory. The /opt/VRTS/vxfs5.0/man
directory is automatically added to /etc/MANPATH when the VxFS 5.0 package
is installed. Make sure that the /opt/VRTS/man directory or the
/opt/VRTS/vxfs5.0/man directory goes before /usr/share/man in you
MANPATH environment variable so that the latest version of the VxFS manual
pages display.
Secure communication
Establishing a secure communication between nodes is required to install and
configure a cluster. The
greater level of security than the
See the Veritas Storage Foundation and High Availability Solutions Getting
Started Guide.
ssh command provides a remote communication and a
rsh command.
Veritas Enterprise Administrator
The Veritas Enterprise Administrator (VEA) client can be installed and run on
any machine that supports the Java Runtime Environment.
VEA is required to access the graphical user interface (GUI) for Veritas Storage
Foundation. You can use the GUI to administer disks, volumes, file systems, and
database functionality on local or remote machines.
One of the following packages needs to be installed and running on the client:
■ Veritas Enterprise Administrator (VRTSobgui)
This is the client package for HP-UX.
■ Veritas Enterprise Administrator for Windows (windows/VRTSobgui.msi)
This is the client package for Windows.
Check the Veritas Storage Foundation Release Notes for any patch information
before you install VEA.
Requirements
The following are system recommendations for the GUI:
OS Requirements
19Installing and configuring the product
Prerequisites
HP-UX 512MB of memory
Windows XP, NT, ME, 2000, or 98300MHz Pentium with at least 256MB of memory
Alternatively, remove the file /sbin/init.d/vmsa-server.
Prerequisites
Each cluster node must be connected to the public network and each must have
a unique host name by which it can be addressed on the public network. The
local node from which you install does not have to be part of the cluster.
Provide the following information when installing the SFCFS:
■ The cluster name, beginning with a letter (a-z, A-Z).
■ A unique ID from 0-65535 for the cluster. Within the public subnet, a new
■ The Storage Foundation Cluster File System is also supported without I/O
cluster using a duplicate cluster ID can cause existing clusters to fail.
fencing enabled. However, without I/O fencing enabled, split brain scenarios
can result in data corruption.
20 Installing and configuring the product
Installing the product
■ The host names of the cluster nodes.
■ The device names of the network interface cards (NICs) used for the private
networks among nodes.
■ Establishing communication between nodes is required to install Veritas
software from a remote system, or to install and configure a cluster. The
node from which the installation utility is run must have permissions to run
rsh (remote shell) or ssh (secure shell) utilities as root on all cluster nodes
or remote systems.
See Veritas Storage Foundation and High Availability Solutions Getting
Started Guide.
■ Symantec recommends configuring the cluster with I/O fencing enabled. I/O
fencing requires shared devices to support SCSI-3 Persistent Reservations
(PR). Enabling I/O fencing prevents data corruption caused by a split brain
scenario.
The Storage Foundation Cluster File System is supported without I/O
fencing enabled. However, without I/O fencing enabled, split brain
scenarios can result in data corruption.
Installing the product
The product installer is the recommended method to license and install the
product. The installer also enables you to configure the product, verify
preinstallation requirements, and view the product’s description.
At most points during an installation, you can type b (“back”) to return to a
previous section of the installation procedure. The back feature of the
installation scripts is context-sensitive, so it returns to the beginning of a
grouped section of questions. If an installation procedure hangs, use Control–c
to stop and exit the program. There is a short delay before the script exits.
The following sample procedure is based on the installation of a Veritas Storage
Foundation Cluster File System HA cluster with two nodes: “system01” and
“system02.” If you are installing on standalone systems only, some steps are
unnecessary, and these are indicated. Default responses are enclosed by
parentheses. Press Return to accept defaults.
Note: If you have obtained a Veritas product from an electronic download site,
the single product download files do not contain the
script, so you must use the product installation script to install the product. For
example, if you download Veritas Cluster File System, use the
script instead of the installer script.
installer installation
installsfcfs
Installing the product
To install the product
1 Log in as superuser.
2 Insert the appropriate media disc into your system’s DVD-ROM drive
connected to your system.
3 Determine the block device file for the DVD drive:
# ioscan -fnC disk
Make a note of the device file as it applies to your system.
4 Create a directory in which to mount the software disc and mount the disc
using the appropriate drive name. For example:
# mkdir -p /dvdrom
# /usr/sbin/mount -F cdfs /dev/dsk/c3t2d0 /dvdrom
5 Change directory to /dvdrom:
# cd /dvdrom
6 Run the installer command to install SFCFS. The installer script uses
ssh to communicate with remote nodes as default:
# ./installer
If you want to use rsh you must specify on the command line:
# ./installer -rsh
7 From the Installation menu, choose the I option for Install and select 6 the
Veritas Storage Foundation Cluster File System. You are prompted to
continue with the installation.
8 Enter y to continue with the installation.
Do you wish to continue with the installation? [y,n,q] (y)
9 Enter one or more system names to install SFCFS.
Enter the system names separted by spaces on which to install
SFCFS: system01 system02
10 Enter the license key for system01.
Enter a SFCFS license key for system01?
11 Enter y to accept another license key or enter n to proceed.
Do you want to enter another license key for system02?
[y,n,q] (n)
12 Enter 1, 2 or 3 to be installed on all systems.
Select the packages to be installed on all systems?
[1-3,q,?] (2)
13 Press Return to continue.
Press [Return] to continue:
14 Reboot all the nodes on which SFCFS is installed and proceed to
“Configuring the Components.”
21Installing and configuring the product
22 Installing and configuring the product
Configuring the Components
Configuring the Components
This sections describes the configuration of SFCFS components.
To configure the components
1 Log in as superuser.
2 Run the
installer command to install the SFCFS. For example:
# cd /cdrom
# ./installer
3 From the Installation menu, choose the C option for Configuration and
select 6 the Veritas Storage Foundation Cluster File System. You are
prompted to enter one or more system names.
Enter the system names separted by spaces on which to
configure SFCFS: system01 system02
4 Enter one or more system names to configure SFCFS. You are prompted to
enter Return to continue.
Press [Return] to continue:
5 Press Return to continue. You are prompted to configure I/O fencing in
enabled mode.
Will you be configuring I/O Fencing in enabled mode? [y,n,q]
(y)
6 Enter y or n to configure I/O fencing in enable mode. You are prompted to
configure VCS for SFCFS.
Enter the unique cluster name: [?]
Enter the unique Cluster ID number between 0-65535: [b,?]
Enter the NIC for the first private heartbeat link on
system01: [b,?]
Would you like to configure a second private heartbeat link?
[y,n,q,b,?] (y)
Enter the NIC for the second private heartbeat link on
hpslia05: [b,?] (lan0)
Are you sure you want to use lan0 for the second private
heartbeak link? [y,n,q,b,?] (n)
Do you want to configure an additional low priority
heartbeat link? [y,n,q,b,?] (n)
Are you using the same NICs for private heartbeat links on
all systems? [y,n,q,b,?] (y)
.
.
.
Is this information correct? [y,n,q] (y)
Loading...