Hp COMPAQ PROLIANT 1850R Modifying Physical Cluster Resources in a Compaq Parallel Database Cluster

White Paper

December 1998
ECG001/1298

Modifying Physical Cluster

Prepared by High Availability
Products Group

Compaq Computer Corporation

Contents

Managing Changes to Drive

Ordering .....................................3

Symptoms................................3

When does this situation

occur?......................................3

Preventing the Issue .................4

Simplifying Detection of the

Issue........................................ 4

What Oracle Component is

Affected?..................................4

Removing a Shared Storage

Array...........................................8

Adding a Shared Storage

Array...........................................9

Adding or Removing Drives
from a Shared Storage Array ...11

Replacing a Failed Drive.........11

Adding a Drive to Increase

Storage Capacity.................... 12

Resources in a Compaq Parallel
Database Cluster

Abstract: At some time during the life of a Compaq Parallel
Database Cluster it is likely that some modifications to the physical
resources of the cluster will be required. This paper describes how to
perform some of the more likely modifications, such as:

• Managing Changes to Drive Ordering

• Removing a Shared Storage Array

• Adding a Shared Storage Array

• Adding or Removing Shared Storage Drives

Additional scenarios are described in Chapter 5 of the Compaq
Parallel Database Cluster Administrator Guide.

Modifying Physical Cluster Resources in a Compaq Parallel Database Cluster 2

Notice

The information in this publication is subject to change without notice and is provided “AS IS” WITHOUT
WARRANTY OF ANY KIND. THE ENTIRE RISK ARISING OUT OF THE USE OF THIS
INFORMATION REMAINS WITH RECIPIENT. IN NO EVENT SHALL COMPAQ BE LIABLE FOR
ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL, PUNITIVE OR OTHER DAMAGES
WHATSOEVER (INCLUDING WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS
PROFITS, BUSINESS INTERRUPTION OR LOSS OF BUSINESS INFORMATION), EVEN IF
COMPAQ HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

The limited warranties for Compaq products are exclusively set forth in the documentation accompanying
such products. Nothing herein should be construed as constituting a further or additional warranty.

This publication does not constitute an endorsement of the product or products that were tested. The
configuration or configurations tested or described may or may not be the only available solution. This test
is not a determination or product quality or correctness, nor does it ensure compliance with any federal
state or local requirements.

Product names mentioned herein may be trademarks and/or registered trademarks of their respective
companies.

Compaq, Contura, Deskpro, Fastart, Compaq Insight Manager, LTE, PageMarq, Systempro, Systempro/LT,
ProLiant, TwinTray, ROMPaq, LicensePaq, QVision, SLT, ProLinea, SmartStart, NetFlex, DirectPlus,
QuickFind, RemotePaq, BackPaq, TechPaq, SpeedPaq, QuickBack, PaqFax, Presario, SilentCool,
CompaqCare (design), Aero, SmartStation, MiniStation, and PaqRap, registered United States Patent and
Trademark Office.

Netelligent, Armada, Cruiser, Concerto, QuickChoice, ProSignia, Systempro/XL, Net1, LTE Elite,
Vocalyst, PageMate, SoftPaq, FirstPaq, SolutionPaq, EasyPoint, EZ Help, MaxLight, MultiLock,
QuickBlank, QuickLock, UltraView, Innovate logo, Wonder Tools logo in black/white and color, and
Compaq PC Card Solution logo are trademarks and/or service marks of Compaq Computer Corporation.

Microsoft, Windows, Windows NT, Windows NT Server and Workstation, Microsoft SQL Server for
Windows NT are trademarks and/or registered trademarks of Microsoft Corporation.

Oracle is a registered trademark and Oracle8 is a trademark of Oracle Corporation
NetWare and Novell are registered trademarks and intraNetWare, NDS, and Novell Directory Services are

trademarks of Novell, Inc.
Pentium is a registered trademark of Intel Corporation.
Copyright ©1998 Compaq Computer Corporation. All rights reserved. Printed in the U.S.A.
Modifying Physical Cluster Resources in a Compaq Parallel Database Cluster

White Paper prepared by High Availability Products Group
First Edition (December 1998)

Document Number ECG001/1298

ECG001/1298

Modifying Physical Cluster Resources in a Compaq Parallel Database Cluster 3

Managing Changes to Drive Ordering

The Compaq Parallel Database Cluster encounters difficulties when the order in which drives are
brought online changes. For the drives associated with the fibre channel arrays, NT assigns disk
numbers based on the order in which the arrays are powered on. Oracle uses these disk number
assignments via the links created with the SETLINKS utility. The order in which drives are
brought online directly affects the disk numbers assigned to the disks, which in turn affects
whether Oracle8 Server can find its database files.

Symptoms

The primary symptom of this issue is that the database will not start up correctly. Some other
symptoms that might be seen are:

• The user may see Oracle errors when trying to mount the database.

• The database will mount correctly, however, the data will not be correct.

• The PGMS service will not start.

• When using Windows NT disk administrator to see the disk configuration:

• one or more of the disks associated with the shared storage will not be seen or will be

marked off-line; or

• the disks will be shown in the wrong order

When does this situation occur?

The situation occurs when the order in which the database’s drives are brought online is different
than the order in which the drives were brought online when the database cluster was originally
set up. Following are some specific examples:

• When more than one shared storage array exists, if the arrays are powered on in a different

order than they were when the cluster was initially configured, the order of the shared drives
will change for all cluster nodes.

• When a drive is added, internally, to a cluster node, the order of the shared drives will change

for that node.

• When an internal drive is removed from a cluster node, the order of the shared drives will

change for that node.

• When a drive is added to a shared storage array, if the drive is added anywhere other than the

last drive in the last array, the order of the drives will change for all cluster nodes.

• When an existing drive is removed from a shared storage array, if the drive is removed from

anywhere other than the last drive in the last array, the order of the drives will change for all
cluster nodes.

ECG001/1298

• When an array is added to the cluster’s shared storage, if the array is added anywhere other

than the last in the sequence of arrays, the order of the drives will change for all cluster
nodes.

Modifying Physical Cluster Resources in a Compaq Parallel Database Cluster 4

• When an existing array is removed from the cluster’s shared storage, if the array was not the

last one brought online, the order of the drives will change for all cluster nodes.

Preventing the Issue

It is highly recommended that a power-on sequence for the shared storage arrays be determined
during the initial configuration of the cluster. Thereafter, this power-on sequence must be
followed exactly to ensure that each node sees the disk in a proper order in the Windows NT Disk
Administrator utility. When adding shared storage (a physical drive or an array), make it the last
drive (or array) to be brought online. In the case of adding an array, make sure it is the last array
to be powered-on; and add it to the power-on sequence as such.

On each node use disk administrator to verify that the disks are brought on-line in the expected
order before starting the Oracle instance. Also, verify that the PGMS service is set to start
manually.

Simplifying Detection of the Issue

A method that simplifies detection of the issue is to create logical partitions that are not all the
same size. With different-sized logical partitions, disk administrator can be used to see the size of
the partitions, which can be quickly and easily compared against the sizes written in the suggested
Oracle worksheet. If the drive order in the worksheet does not compare with the drive order
shown in disk administrator, the drives were brought online out of sequence.

For example, the shared disks associated with the database could have a small unused partition as
the first partition on the disk. Assuming there are three disks associated with the database, a
sample configuration might be:

• Harddisk1 Partition1 is sized to 11 MB

• Harddisk2 Partition1 is sized to 12 MB

• Harddisk3 Partition1 is sized to 13 MB

When the cluster is powered on, if disk administrator shows harddisk1, partition1 as 12MB or
13MB, the drives were brought on-line out of sequence. The cluster should be shut down and
restarted correctly.

What Oracle Component is Affected?

How does the drive ordering issue manifest itself in Oracle8 Server? The focal point of this issue
resides in the Oracle symbolic link table files; frequently identified as ORALINKx.TBL. These
files create symbolic links, which are used by Oracle8 Server to map specific Oracle database
files to specific hard drive partitions. A link is required for each database file. A two node cluster
requires nine database files and one cluster file, and therefore requires ten links. The first
symbolic link table file is used by a two-node cluster. Each additional node requires two more
database files, two more links, and one more symbolic link table file .

Oracle suggests using a worksheet to define the symbolic links. For more information about the
worksheet, refer to the Oracle Parallel Server Getting Started Release 8.0.5 for Windows NT
manual. An example of such a worksheet follows. The example assumes the cluster consists of
two nodes – Node1 has two internal disk drives and Node2 has one internal disk drive.
Additionally, it assumes the shared storage consists of two fibre channel arrays - the first array
has three RAID logical drives and the second array has two RAID logical drives. Therefore,

ECG001/1298