Reproduction in any manner whatsoever without the written permission of Dell Inc. is strictly forbidden.
Trademarks used in this text: Dell, the DELL logo, and PowerEdge are trademarks of Dell Inc.; EMC, PowerPath, and Navisphere are registered
trademarks of EMC Corporation; Intel and Xeon are registered trademarks of Intel Corporation; Red Hat is a registered trademark of Red Hat, Inc.
Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products.
Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own.
Configuring Shared Storage for Oracle Clusterware and the
Database Using ASM
Installing Oracle RAC 10g
Before You Begin
Installing Oracle Clusterware
Installing the Oracle Database 10g Software
RAC Post Deployment Fixes and Patches
Configuring the Listener
Creating the Seed Database Using OCFS2
Creating the Seed Database Using ASM
Configuring the Public Network
Configuring Database Storage
Configuring Database Storage Using the Oracle ASM Library Driver
Installing Oracle Database 10g
Installing the Oracle Database 10g 10.2.0.2 Patchset
Configuring the Listener
Creating the Seed Database
Adding a New Node to the Network Layer
Configuring Shared Storage on the New Node
Adding a New Node to the Oracle Clusterware Layer
Adding a New Node to the Database Layer
Reconfiguring the Listener
Adding a New Node to the Database Instance Layer
Removing a Node From the Cluster
Reinstalling the Software
Additional Information
Supported Software Versions
Determining the Private Network Interface
This document provides information about installing, configuring, reinstalling, and using
Oracle Database 10g Enterprise Edition with the Oracle Real Application Clusters (RAC) software on
your Dell|Oracle supported configuration. Use this document in conjunction with the Dell Deployment,
Red Hat Enterprise Linux, and Oracle RAC 10g software CDs to install your software.
NOTE: If you install your operating system using only the operating system CDs, the steps in this document may not
be applicable.
This document covers the following topics:
•Software and hardware requirements
®
•Installing and configuring Red Hat
•Verifying cluster hardware and software configurations
•Configuring storage and networking for Oracle RAC
•Installing Oracle RAC
•Configuring and installing Oracle Database 10
•Adding and removing nodes
•Reinstalling the software
•Additional information
•Troubleshooting
•Getting help
•Obtaining and using open source files
For more information on Dell supported configurations for Oracle, see the Dell|Oracle Tested and
Validated Configurations website at www.dell.com/10g.
Enterprise Linux
g
(single node)
Oracle RAC 10g Deployment Service
If you purchased the Oracle RAC 10g Deployment Service, your Dell Professional Services representative
will assist you with the following:
•Verifying cluster hardware and software configurations
•Configuring storage and networking
•Installing Oracle RAC 10
g
Release 2
Deployment Guide5
Software and Hardware Requirements
Before you install the Oracle RAC software on your system:
•Download the Red Hat CD images from the Red Hat website at
•Locate your Oracle CD kit.
•Download the
Dell Deployment CD
images that are appropriate for the solution being installed from
the Dell|Oracle Tested and Validated Configurations website at
downloaded CD images to CDs.
Table 1-1 lists basic software requirements for Dell supported configurations for Oracle. Table 1-2
through Table 1-3 list the hardware requirements. For more information on the minimum software
versions for drivers and applications, see "Supported Software Versions."
Table 1-1. Software Requirements
Software ComponentConfiguration
Red Hat Enterprise Linux AS EM64T (Version 4)Update 3
Oracle Database 10gVersio n 10 .2
• Enterprise Edition, including the RAC option for clusters
• Enterprise Edition for single-node configuration
EMC® PowerPath
NOTE: Depending on the number of users, the applications you use, your batch processes, and other factors, you
may need a system that exceeds the minimum hardware requirements in order to achieve desired performance.
®
Version 4.5.1
rhn.redhat.com
.
www.dell.com/10g
. Burn all these
NOTE: The hardware configuration of all the nodes must be identical.
See the Dell|Oracle Tested and Validated Configurations website at
www.dell.com/10g for information on supported configurations
Eight ports
License Agreements
NOTE: Your Dell configuration includes a 30-day trial license of Oracle software. If you do not have a license for
this product, contact your Dell sales representative.
Important Documentation
For more information on specific hardware components, see the documentation included with
your system.
For Oracle product information, see the How to Get Started guide in the Oracle CD kit.
Before You Begin
Before you install the Red Hat Enterprise Linux operating system, download the Red Hat Enterprise
Linux Quarterly Update ISO images from the Red Hat Network website at rhn.redhat.com and burn
these images to CDs.
To download the ISO images, perform the following steps:
1
Navigate to the Red Hat Network website at
2
Click
Channels
3
In the left menu, click
.
Easy ISOs
.
rhn.redhat.com
.
Deployment Guide7
4
In the
Easy ISOs
The ISO images for all Red Hat products appear.
5
In the
Channel Name
Linux software.
6
Download the ISOs for your Red Hat Enterprise Linux software as listed in your Solution Deliverable
List (SDL) from the Dell|Oracle Tested and Validated Configurations website at
7
Burn the ISO images to CDs.
page left menu, click
menu, click the appropriate ISO image for your Red Hat Enterprise
All
.
www.dell.com/10g
Installing and Configuring Red Hat Enterprise Linux
NOTICE: To ensure that the operating system is installed correctly, disconnect all external storage devices from
the system before you install the operating system.
This section describes the installation of the Red Hat Enterprise Linux AS operating system and the
configuration of the operating system for Oracle Database deployment.
Installing Red Hat Enterprise Linux Using the Deployment CDs
1
Disconnect all external storage devices from the system.
2
Locate your
3
Insert the
The system boots to the
4
When the deployment menu appears, type 1 to select
Linux 4 U3 (x86_64)
5
When another menu asking deployment image source appears, type 1 to select
Deployment CD.
Dell Deployment CD
Dell Deployment CD 1
Dell Deployment CD
.
and the
into the CD drive and reboot the system.
Red Hat Enterprise Linux AS EM64T
.
Oracle 10g R2 EE on Red Hat Enterprise
CDs.
Copy solution by
.
NOTE: This procedure may take several minutes to complete.
6
When prompted, insert
A deployment partition is created and the contents of the CDs are copied to it. When the copy
operation is completed, the system automatically ejects the last CD and boots to the
deployment partition.
When the installation is completed, the system automatically reboots and the Red Hat Setup
Agent appears.
7
In the
Red Hat Setup Agent Welcome
Do not create any operating system users at this time.
8
When prompted, specify a
8Deployment Guide
Dell Deployment CD 2
window, click
root password
.
and each Red Hat installation CD into the CD drive.
Next
to configure your operating system settings.
9
10
When the
When the
Network Setup
Security Level
window appears, click
window appears, disable the firewall. You may enable the firewall after
completing the Oracle deployment.
11
Log in as
root
.
Configuring Red Hat Enterprise Linux
1
Log in as
2
Insert the
mount /dev/cdrom
/media/cdrom/install.sh
root
.
Dell Deployment CD 2
into the CD drive and type the following commands:
Next
. You will configure network settings later.
The contents of the CD are copied to the
procedure is completed, type
3
Ty p e
cd /dell-oracle-deployment/scripts/standard
umount /dev/cdrom
containing the scripts installed from the
NOTE: Scripts discover and validate installed component versions and, when required, update components
to supported levels.
4
Ty p e
./005-oraclesetup.py
5
Ty p e
source /root/.bash_profile
6
Ty p e
./010-hwCheck.py
to configure the Red Hat Enterprise Linux for Oracle installation.
to verify that the CPU, RAM, and disk sizes meet the minimum
/usr/lib/dell/dell-deploy-cd
directory. When the copy
and remove the CD from the CD drive.
to navigate to the directory
Dell Deployment CD
.
to start the environment variables.
Oracle Database installation requirements.
If the script reports that a parameter failed, update your hardware configuration and run the script
again (see Table 1-2 and Table 1-3 for updating your hardware configuration).
7
Connect the external storage device.
8
Reload the HBA driver(s) using
rmmod
and
modprobe
commands. For instance, for Emulex HBAs,
reload the lpfc driver by issuing
rmmod lpfc
modprobe lpfc
For QLA HBAs, identify the drivers that are loaded (
lsmod | grep qla
), and reload these drivers.
Updating Your System Packages Using Red Hat Network
Red Hat periodically releases software updates to fix bugs, address security issues, and add new features.
You can download these updates through the Red Hat Network (RHN) service. See the Dell|Oracle
Tested and Validated Configurations website at www.dell.com/10g for the latest supported
configurations before you use RHN to update your system software to the latest revisions.
NOTE: If you are deploying Oracle Database on a single node, skip the following sections and see "Configuring and
Deploying Oracle Database 10g (Single Node)."
Deployment Guide9
Verifying Cluster Hardware and Software Configurations
Dell|EMC Fibre Channel
storage systems
client systems
PowerEdge systems
(Oracle Database)
Gb Ethernet switches (private network)
Dell|EMC Fibre Channel switches
(SAN)
LAN/WAN
CAT 5e/6 (public NIC)
CAT 5e/6 (copper gigabit NIC)
fiber optic cables
additional fiber optic cables
Before you begin cluster setup, verify the hardware installation, communication interconnections, and
node software configuration for the entire cluster. The following sections provide setup information for
hardware and software Fibre Channel cluster configurations.
Fibre Channel Cluster Setup
Your Dell Professional Services representative completed the setup of your Fibre Channel cluster. Verify
the hardware connections and the hardware and software configurations as described in this section.
Figure 1-1 and Figure 1-3 show an overview of the connections required for the cluster, and Table 1-4
summarizes the cluster connections.
Figure 1-1. Hardware Connections for a Fibre Channel Cluster
Each PowerEdge system node One Category 5 enhanced (CAT 5e) or CAT 6 cable from public NIC to local area
network (LAN)
One CAT 5e or CAT 6 cable from private Gigabit NIC to Gigabit Ethernet switch
One CAT 5e or CAT 6 cable from a redundant private Gigabit NIC to a redundant
Gigabit Ethernet switch
One fiber optic cable from optical HBA 0 to Fibre Channel switch 0
One fiber optic cable from HBA 1 to Fibre Channel switch 1
Each Dell|EMC Fibre
Channel storage system
Each Dell|EMC Fibre
Channel switch
Each Gigabit Ethernet switch One CAT 5e or CAT 6 connection to the private Gigabit NIC on each PowerEdge
Two CAT 5e or CAT 6 cables connected to the LAN
One to four fiber optic cable connections to each Fibre Channel switch; for
example, for a four-port configuration:
•One
fiber optic cable
•One
fiber optic cable
•One
fiber optic cable
•One
fiber optic cable
One to four fiber optic cable connections to the Dell|EMC Fibre Channel storage
system
One fiber optic cable connection to each PowerEdge system’s HBA
system
One CAT 5e or CAT 6 connection to the remaining Gigabit Ethernet switch
from SPA port 0 to Fibre Channel switch 0
from SPA port 1 to Fibre Channel switch 1
from SPB port 0 to Fibre Channel switch 1
from SPB port 1 to Fibre Channel switch 0
Verify that the following tasks are completed for your cluster:
•All hardware is installed in the rack.
•All hardware interconnections are set up as shown in Figure 1-1 and
Figure 1-3, and
listed in Table 1-4.
•All logical unit numbers (LUNs), redundant array of independent disk (RAID) groups, and storage
groups are created on the Dell|EMC Fibre Channel storage system.
•Storage groups are assigned to the nodes in the cluster.
Before continuing with the following sections, visually inspect all hardware and interconnections for
correct installation.
Deployment Guide11
Fibre Channel Hardware and Software Configurations
•Each node must include the minimum hardware peripheral components as described in Table 1-2.
•Each node must have the following software installed:
–Red Hat Enterprise Linux software (see Table 1-1)
–Fibre Channel HBA driver
•The Fibre Channel storage system must be configured with the following:
–A minimum of three LUNs created and assigned to the cluster storage group (see Table 1-5)
–A minimum LUN size of 5 GB
Table 1-5. LUNs for the cluster storage group
LUNMinimum SizeNumber of PartitionsUsed For
First LUN512 MBthree of 128 MB eachVoting disk, Oracle Cluster
Registry (OCR), and
storage processor (SP) file
Second LUNLarger than the size of your databaseoneDatabase
Third LUNMinimum twice the size of your
second LUN
oneFlash Recovery Area
Cabling Your Storage System
You can configure your Oracle cluster storage system in a direct-attached configuration or a four-port
SAN-attached configuration, depending on your needs. See the following procedures for both
configurations.
12Deployment Guide
Figure 1-2. Cabling in a Direct-Attached Fibre Channel Cluster
HBA ports (2)
node 1
node 2
HBA ports (2)
0
1
0
1
CX700 storage system
SP-B
SP-A
0
1
2
3
SP ports
3
2
1
0
Direct-Attached Configuration
To configure your nodes in a direct-attached configuration (see Figure 1-2), perform the following steps:
Connect one optical cable from HBA0 on node 1 to port 0 of SP-A.
1
2
Connect one optical cable from HBA1 on node 1 to port 0 of SP-B.
3
Connect one optical cable from HBA0 on node 2 to port 1 of SP-A.
4
Connect one optical cable from HBA1 on node 2 to port 1 of SP-B.
Deployment Guide13
Figure 1-3. Cabling in a SAN-Attached Fibre Channel Cluster
HBA ports (2)
node 1
node 2
HBA ports (2)
SP-B
SP-A
0
1
2
3
SP ports
3
2
1
0
sw0
sw1
01
0
1
CX700 storage system
SAN-Attached Configuration
To configure your nodes in a four-port SAN-attached configuration (see Figure 1-3), perform the
following steps:
1
Connect one optical cable from SP-A port 0 to Fibre Channel switch 0.
2
Connect one optical cable from SP-A port 1 to Fibre Channel switch 1.
3
Connect one optical cable from SP-A port 2 to Fibre Channel switch 0.
4
Connect one optical cable from SP-A port 3 to Fibre Channel switch 1.
5
Connect one optical cable from SP-B port 0 to Fibre Channel switch 1.
6
Connect one optical cable from SP-B port 1 to Fibre Channel switch 0.
14Deployment Guide
7
Connect one optical cable from SP-B port 2 to Fibre Channel switch 1.
8
Connect one optical cable from SP-B port 3 to Fibre Channel switch 0.
9
Connect one optical cable from HBA0 on node 1 to Fibre Channel switch 0.
10
Connect one optical cable from HBA1 on node 1 to Fibre Channel switch 1.
11
Connect one optical cable from HBA0 on node 2 to Fibre Channel switch 0.
12
Connect one optical cable from HBA1 on node 2 to Fibre Channel switch 1.
Configuring Storage and Networking for Oracle RAC 10g
This section provides information and procedures for setting up a Fibre Channel cluster running a
seed database:
•Configuring the public and private networks
•Securing your system
•Verifying the storage configuration
•Configuring shared storage for Cluster Ready Services (CRS) and Oracle Database
Oracle RAC 10g is a complex database configuration that requires an ordered list of procedures.
To configure networks and storage in a minimal amount of time, perform the following procedures in order.
Configuring the Public and Private Networks
This section presents steps to configure the public and private cluster networks.
NOTE: Each node requires a unique public and private internet protocol (IP) address and an additional public
IP address to serve as the virtual IP address for the client connections and connection failover. The virtual
IP address must belong to the same subnet as the public IP. All public IP addresses, including the virtual IP address,
should be registered with Domain Naming Service and routable.
Depending on the number of NIC ports available, configure the interfaces as shown in Table 1-6.
Table 1-6. NIC Port Assignments
NIC PortThree Ports AvailableFour Ports available
1Public IP and virtual IPPublic IP
2Private IP (bonded)Private IP (bonded)
3Private IP (bonded)Private IP (bonded)
4NAVirtual IP
Deployment Guide15
Configuring the Public Network
NOTE: Ensure that your public IP address is a valid, routable IP address.
If you have not already configured the public network, do so by performing the following steps on
each node:
1
Log in as
2
Edit the network device file
root
.
/etc/sysconfig/network-scripts/ifcfg-eth#
, where # is the number of the
network device, and configure the file as follows:
For example, the line for node 1 would be as follows:
HOSTNAME=node1.domain.com
4
Ty p e :
service network restart
5
Ty p e
ifconfig
6
To check your network configuration, ping each public IP address from a client on the LAN outside
to verify that the IP addresses are set correctly.
the cluster.
7
Connect to each node to verify that the public network is functioning and type
to verify that the secure shell (
Configuring the Private Network Using Bonding
ssh)
command is working.
ssh <public IP>
Before you deploy the cluster, configure the private cluster network to allow the nodes to communicate
with each other. This involves configuring network bonding and assigning a private IP address and
hostname to each node in the cluster.
To set up network bonding for Broadcom or Intel NICs and configure the private network, perform the
following steps on each node:
1
Log in as
2
Add the following line to the
root
.
/etc/modprobe.conf
file:
alias bond0 bonding
16Deployment Guide
3
For high availability, edit the
/etc/modprobe.conf
file and set the option for link monitoring.
The default value for miimon is 0, which disables link monitoring. Change the value to
100 milliseconds initially, and adjust it as needed to improve performance as shown in the following
example. Type:
options bonding miimon=100 mode=1
In the
4
/etc/sysconfig/network-scripts/
directory, create or edit the
ifcfg-bond0
configuration file.
For example, using sample network parameters, the file would appear as follows:
to verify that the private interface is functioning.
and ignore any warnings.
The private IP address for the node should be assigned to the private interface bond0.
7
When the private IP addresses are set up on every node, ping each IP address from one node to ensure
that the private network is functioning.
Deployment Guide17
8
Connect to each node and verify that the private network and
ssh
are functioning correctly by typing:
ssh <private IP>
9
On
each node,
modify the
/etc/hosts
file by adding the following lines:
127.0.0.1 localhost.localdomain localhost
<private IP node1> <private hostname node1>
<private IP node2> <private hostname node2>
<public IP node1> <public hostname node1>
<public IP node2> <public hostname node2>
<virtual IP node1> <virtual hostname node1>
<virtual IP node2> <virtual hostname node2>
NOTE: The examples in this and the following step are for a two-node configuration; add lines for each
additional node.
10
On
each node
, create or modify the
/etc/hosts.equiv
file by listing all of your public IP addresses or host
names. For example, if you have one public hostname, one virtual IP address, and one virtual hostname
for each node, add the following lines:
<virtual IP or hostname node1> oracle
<virtual IP or hostname node2> oracle
11
Log in as
oracle
by typing:
rsh <public hostname nodex>
where x is the node number.
18Deployment Guide
, connect to each node to verify that the remote shell (
rsh
) command is working
Verifying the Storage Configuration
While configuring the clusters, create partitions on your Fibre Channel storage system. In order to create
the partitions, all the nodes must be able to detect the external storage devices. To verify that each node
can detect each storage LUN or logical disk, perform the following steps:
1
For Dell|EMC Fibre Channel storage system, verify that the EMC Navisphere® agent and the correct
version of PowerPath (see Table 1-7) are installed on each node, and that each node is assigned to the
correct storage group in your EMC Navisphere software. See the documentation that came with your
Dell|EMC Fibre Channel storage system for instructions.
NOTE: The Dell Professional Services representative who installed your cluster performed this step. If you
reinstall the software on a node, you must perform this step.
2
Visually verify that the storage devices and the nodes are connected correctly to the Fibre Channel
switch (see Figure 1-1 and Table 1-4).
3
Verify that you are logged in as
4
On
each node
, type:
more /proc/partitions
The node detects and displays the LUNs or logical disks, as well as the partitions created on those
external devices.
NOTE: The listed devices vary depending on how your storage system is configured.
A list of the LUNs or logical disks that are detected by the node is displayed, as well as the partitions
that are created on those external devices. PowerPath pseudo devices appear in the list, such as
/dev/emcpowera, /dev/emcpowerb
root
, and
.
/dev/emcpowerc
.
5
In the
/proc/partitions
file, ensure that:
•All PowerPath pseudo devices appear in the file with similar device names across all nodes.
For example,
/dev/emcpowera
, /
dev/emcpowerb
, and
/dev/emcpowerc
.
•The Fibre Channel LUNs appear as SCSI devices, and each node is configured with the same
number of LUNs.
For example, if the node is configured with a SCSI drive or RAID container attached to a Fibre
Channel storage device with three logical disks,
internal drive, and
emcpowera, emcpowerb
, and
sda
identifies the node’s RAID container or
emcpowerc
identifies the LUNs (or PowerPath
pseudo devices).
If the external storage devices do not appear in the /proc/partitions file, reboot the node.
Deployment Guide19
Disable SELinux
To run the Oracle database, you must disable SELinux.
To temporarily disable SELinux, perform the following steps:
Log in as
1
2
At the command prompt, type:
setenforce 0
To permanently disable SELinux, perform the following steps on all the nodes:
1
Open your
2
Locate the kernel command line and append the following option:
Configuring Shared Storage Using the ASM Library Driver
1
Log in as
2
Open a terminal window and perform the following steps on all nodes:
a
b
root
.
Ty p e
service oracleasm configure
Type the following inputs for all the nodes:
udevstart
Default user to own the driver interface [ ]:
Default group to own the driver interface []:
Start Oracle ASM library driver on boot (y/n) [n]:
Fix permissions of Oracle ASM disks on boot (y/n) [y]:
24Deployment Guide
on all the nodes and repeat step 4.
oracle
dba
y
y
3
On the
4
Repeat step 3 for any additional ASM disks that need to be created.
5
Verify that the ASM disks are created and marked for ASM usage.
In the terminal window, type the following and press <Enter>:
service oracleasm listdisks
The disks that you created in step 3 appear.
For example:
ASM1
ASM2
6
Ensure that the remaining nodes are able to access the ASM disks that you created in step 3.
On each remaining node, open a terminal, type the following, and press <Enter>:
service oracleasm scandisks
first node
service oracleasm createdisk ASM1 /dev/emcpowerb1
service oracleasm createdisk ASM2 /dev/emcpowerc1
, in the terminal window, type the following and press <Enter>:
Installing Oracle RAC 10g
This section describes the steps required to install Oracle RAC 10g, which involves installing CRS and
installing the Oracle Database 10g software. Dell recommends that you create a seed database to verify
that the cluster works correctly before you deploy it in a production environment.
Before You Begin
To prevent failures during the installation procedure, configure all the nodes with identical system
clock settings.
Synchronize your node system clock with a Network Time Protocol (NTP) server. If you cannot access an
NTP server, perform one of the following procedures:
•Ensure that the system clock on the Oracle Database software installation node is set to a later time
than the remaining nodes.
•Configure one of your nodes as an NTP server to synchronize the remaining nodes in the cluster.
If your system is configured correctly, the following message appears:
Pre-check for database installation was successful.
2
As user
root
, type:
xhost +
3
As user
4
Log in as
root
oracle
, mount the
<CD_mountpoint>/runInstaller
The Oracle Universal Installer starts.
5
In the
Welc om e
28Deployment Guide
node2
are the public host names.
not
configured correctly, see "Troubleshooting" for more information.
Oracle Database 10g
CD.
, and type:
window, click
Next
.
6
In the
Select Installation Type
7
In the
Specify Home Details
/opt/oracle/product/10.2.0/db_1
NOTE: The Oracle home name in this step must be different from the Oracle home name that you identified
during the CRS installation. You cannot install the Oracle 10g Enterprise Edition with RAC into the same home
name that you used for CRS
8
In the
Specify Hardware Cluster Installation Mode
9
In the
Product-Specific Prerequisite Checks
column for each system check, and then click
NOTE: In some cases, a warning may appear regarding swap size. Ignore the warning and click Yes
to proceed.
10
In the
Select Configuration Option
11
In the
Summary
window, click
window, select
window in the
and click
Next
.
window, select
Install
.
Enterprise Edition
Path
window, ensure that
Next
The Oracle Database software is installed on your cluster.
and click
Next
.
field, verify that the complete Oracle home path is
.
window, click
Select All
Succeeded
and click
Next
appears in the
.
Status
.
Install database Software only
and click
Next
.
Next, the
12
Follow the instructions in the window and click OK.
13
In the
Execute Configuration Scripts
End of Installation
window, click
window appears.
Exit
.
RAC Post Deployment Fixes and Patches
This section provides the required fixes and patch information for deploying Oracle RAC 10g.
Reconfiguring the CSS Miscount for Proper EMC PowerPath Failover
When an HBA, switch, or EMC Storage Processor (SP) failure occurs, the total PowerPath failover time
to an alternate device may exceed 105 seconds. The default CSS disk time-out for Oracle 10g R2 version
10.2.0.1 is 60 seconds. To ensure that the PowerPath failover procedure functions correctly, increase the
CSS time-out to 120 seconds.
For more information, see Oracle Metalink Note 294430.1 on the Oracle Metalink website at
metalink.oracle.com.
To increase the CSS time-out:
1
Shut down the database and CRS on all nodes except on one node.
2
On the running node, log in as user
crsctl set css misscount 120
3
Reboot all nodes for the CSS setting to take effect.
root
and type:
Deployment Guide29
Installing the Oracle Database 10g 10.2.0.2 Patchset
Downloading and Extracting the Installation Software
1
On the
2
Create a folder for the patches and utilities at
3
Open a web browser and navigate to the Oracle Support website at
4
Log in to your Oracle Metalink account.
5
Search for the patch number 4547817 with Linux x86-64 (AMD64/EM64T) as the platform.
6
Download the patch to the
7
To unzip the downloaded zip file, type the following in a terminal window and press <
first node
, log in as
oracle
.
/opt/oracle/patches
/opt/oracle/patches
directory.
.
metalink.oracle.com
unzip p4547817_10202_LINUX-x86-64.zip
Upgrading Oracle Clusterware Installation
1
On the
2
Shut down Oracle Clusterware. To do so, type the following in the terminal window and press
<
Enter
first node
>:
, log in as
root
.
crsctl stop crs
3
On the remaining nodes, open a terminal window and repeat step 1 and step 2.
4
On the
5
In the terminal window, type the following and press <
first node
, log in as
oracle
.
Enter
>:
export ORACLE_HOME=/crs/oracle/product/10.2.0/crs
.
Enter
>:
6
Start the Oracle Universal Installer. To do so, type the following in the terminal window and
Enter
press <
>:
cd /opt/oracle/patches/Disk1/
./runInstaller
The
10
7
8
9
Welc om e
Click
In the
In the
In the
Next
screen appears.
.
Specify Home Details
Specify Hardware Cluster Installation Mode
Summary
The Oracle Universal Installer scans your system, displays all the patches that are required to be
installed, and installs them on your system. When the installation is completed, the
screen appears.
NOTE: This procedure may take several minutes to complete.
30Deployment Guide
screen, click
screen, click
Install
.
Next
.
screen, click
Next
.
End of Installation
11
Read all the instructions that are displayed in the message window, which appears.
NOTE: Do not shut down the Oracle Clusterware daemons, as you already performed this procedure in step 1
and step 2.
12
Open a terminal window.
13
Log in as
14
Type the following and press <
root
.
Enter
>:
$ORA_CRS_HOME/install/root102.sh
15
Repeat step 12 through step 14 on the remaining nodes, one node at a time.
16
On the
17
Click
18
Click
Upgrading the RAC Installation
1
On the
2
Log in as
3
Run the Oracle Universal Installer from the same node that you installed the Oracle Database software.
a
b
c
first node
Exit
Yes
to exit the Oracle Universal Installer.
first node
On the
Log in as
, return to the
.
, open a terminal window.
oracle
.
first node
oracle
, open a terminal window.
.
End of Installation
screen.
Shut down the Oracle Clusterware node applications on all nodes.
Enter
In the terminal window, type the following and press <
Start the Oracle Universal Installer. To do so, type the following in the terminal window, and
press <
Enter
>:
cd /opt/oracle/patches/Disk1/
./runInstaller
The
Welc om e
screen appears.
Deployment Guide31
10
Click
Next
.
11
In the
12
13
Specify Home Details
In the
Specify Hardware Cluster Installation Mode
In the
Summary
screen, click
screen, click
Install
.
Next
.
screen, click
Next
.
The Oracle Universal Installer scans your system, displays all the patches that are required to be
installed, and installs them on your system. When the installation is completed, the
End of Installation
screen appears.
Next, a message window appears, prompting you to run
14
Open a terminal window.
15
Type the following and press <
Enter
>:
root.sh
as user
root
.
/opt/oracle/product/10.2.0/db_1/root.sh
16
Repeat step 14 and step 15 on the remaining nodes, one node at a time.
When the installation is completed, the
NOTE: This procedure may take several minutes to complete.
17
In the
End of Installation
18
Click
Yes
to exit the Oracle Universal Installer.
19
On the
20
Log in as
21
Type the following and press <
first node
oracle
, open a terminal window.
.
screen, click
Enter
End of Installation
Exit
.
>:
screen appears.
srvctl start nodeapps -n <nodename>
Where <
22
On all the remaining nodes, shut down CRS by issuing the following command:
is the database instance identifier assigned to the node.
This example assumes that
26
On
one node
srvctl status database -d dbname
where
If the database instances are running, confirmation appears on the screen.
If the database instances are
srvctl start database -d dbname
where
dbname
dbname
, type:
is the global identifier name that you defined for the database in DBCA.
is the global identifier name that you defined for the database in DBCA.
each node
racdb
is the global database name that you defined in DBCA.
not
running, type:
:
Securing Your System
oracle
To prevent unauthorized users from accessing your system, Dell recommends that you disable rsh after
you install the Oracle software.
To disab l e rsh, type:
chkconfig rsh off
Setting the Password for the User oracle
Dell strongly recommends that you set a password for the user oracle to protect your system. Complete
the following steps to create the oracle password:
1
Log in as
2
Ty p e
NOTE: Additional security setup may be performed according to the site policy, provided the normal database
operation is not disrupted.
root
.
passwd oracle
and follow the instructions on the screen to create the
Deployment Guide37
oracle
password.
Configuring and Deploying Oracle Database 10g (Single Node)
This section provides information about completing the initial setup or completing the reinstallation
procedures as described in "Installing and Configuring Red Hat Enterprise Linux." This section covers
the following topics:
•Configuring the Public Network
•Configuring Database Storage
•Installing the Oracle Database
•Configuring the Listener
•Creating the Seed Database
Configuring the Public Network
Ensure that your public network is functioning and that an IP address and host name are assigned to
your system.
Configuring Database Storage
Configuring Database Storage Using ex3 File System
If you have additional storage device, perform the following steps:
1
Log in as
2
Ty p e :
cd /opt/oracle
root
.
3
Ty p e :
mkdir oradata recovery
Using the
4
sdb1
5
Using the
if your storage device is
6
Verify the new partition by typing:
cat /proc/partitions
If you do not detect the new partition, type:
sfdisk -R /dev/sdb
sfdisk -R /dev/sdc
7
Ty p e :
mke2fs -j /dev/sdb1
mke2fs -j /dev/sdc1
38Deployment Guide
fdisk
if your storage device is
fdisk
utility, create a partition where you want to store your recovery files (for example,
utility, create a partition where you want to store your database files (for example,
sdb
).
sdc1
sdc
).
8
Edit the
/etc/fstab
file for the newly created file system by adding entries such as:
mount /dev/sdb1 /opt/oracle/oradata
mount /dev/sdc1 /opt/oracle/recovery
10
Ty p e :
chown -R oracle.dba oradata recovery
Configuring Database Storage Using Oracle ASM
The following example assumes that you have two storage devices (sdb and sdc) available to create a
disk group for the database files, and a disk group to be used for flash back recovery and archive
log files, respectively.
1
Log in as
2
Create a primary partition for the entire device by typing:
root
.
fdisk /dev/sdb
3
Create a primary partition for the entire device by typing:
fdisk /dev/sdc
Configuring ASM Storage Using the RAW Device Interface
1
Edit the
/etc/sysconfig/rawdevices
file and add the following lines:
/dev/raw/ASM1 /dev/sdb1
/dev/raw/ASM2 /dev/sdc1
2
Restart the RAW Devices Service by typing:
service rawdevices restart
Configuring Database Storage Using the Oracle ASM Library Driver
This section provides procedures for configuring the storage device using ASM.
NOTE: Before you configure the ASM Library Driver, disable SELinux.
To temporarily disable SELinux, perform the following steps:
Log in as
1
2
At the command prompt, type:
setenforce 0
root
.
Deployment Guide39
To permanently disable SELinux, perform the following steps:
1
Open your
2
Locate the kernel command line and append the following option:
Installing the Oracle Database 10g 10.2.0.2 Patchset
Downloading and Extracting the Installation Software
1
Log in as
2
Create a folder for the patches and utilities at
3
Open a web browser and navigate to the Oracle Metalink website at
4
Log in to your Oracle Metalink account.
5
Search for the patch number 4547817 with Linux x86-64 (AMD64/EM64T) as the platform.
6
Download the patch to the
7
To unzip the downloaded zip file, type the following in a terminal window and press <Enter>:
unzip p4547817_10202_LINUX-x86-64.zip
Upgrading the Database Software
1
Open a terminal window.
2
Log in as
3
Ensure that ORACLE_HOME is set to
4
As the user
/etc/init.d/init.cssd stop
5
Start the Oracle Universal Installer. To do so, type the following in the terminal window, and
press <Enter>:
/opt/oracle/patches/Disk1/runInstaller
oracle
.
/opt/oracle/patches
oracle
/opt/oracle/patches
.
directory.
/opt/oracle/product/10.2.0/db_1
root
, stop the
NOTE: This procedure may take a few minutes to complete.
cssd
process. To do so, type the following and press <Enter>:
.
metalink.oracle.com
.
.
The
6
7
8
9
Welc om e
Click
In the
In the
In the
Next
screen appears.
.
Specify Home Details
Specify Hardware Cluster Installation Mode
Summary
The Oracle Universal Installer scans your system, displays all the patches that are required to be
installed, and installs them on your system. When the installation is completed, the
screen appears.
Next, a message window appears, prompting you to run
10
In a terminal window, type the following and press <Enter>:
/opt/oracle/product/10.2.0/db_1/root.sh
42Deployment Guide
screen, click
screen, click
Install
.
Next
.
screen, click
root.sh
Next
as user
.
root
End of Installation
.
11
Press <Enter> to accept the default answers to the questions generated by
NOTE: This procedure may take several minutes to complete.
12
When you complete executing
13
In the
End of Installation
14
Click
Yes
to exit the Oracle Universal Installer.
15
Restart the
cssd
process. To do so, type the following and press <Enter>:
root.sh
, go back to
screen, click
Exit
Execute Configuration Scripts
.
/etc/init.d/init.cssd start
Configuring the Listener
1
Log in as
2
Start the X Window System by typing:
root
.
startx
3
Open a terminal window and type:
xhost +
4
Log in as
5
Ty p e
6
Accept the default settings and click
oracle
netca
.
to start the Oracle Net Configuration Assistant.
Next
on all the screens to complete the listener configuration.
Creating the Seed Database
root.sh
.
window and click OK.
Creating the Seed Database Using ext3 File System
Perform the following steps to create a seed database with the DBCA:
1
Log in as
2
Start the Oracle DBCA by typing:
oracle
.
dbca
3
In the
Welc om e
4
In the
Operations
5
In the
Database Templates
6
In the
Database Identification
Global Database Name
7
In the
Management Options
8
In the
Database Credentials
9
In the
Storage Options
10
In the
Database File Locations
window, click
window, click
window, click
and the
window, complete password selections and entries and click
window, select
Next
.
Create a Database
Custom Database
and click
Next
and click
window, type the name of the database that you are creating in the
SID Prefix
window, click
window, click
fields, and click
Next
.
File System
Next
and click
.
Next
Next
.
Next
.
.
Next
.
.
Deployment Guide43
11
In the
Recovery Configuration
window, click
Browse
, select the flashback recovery area that you created
in "Configuring Database Storage Using ex3 File System" (for example,
change the Flash Recovery Area size as needed, and click
12
In the
13
14
15
16
Database Content
In the
Initialization Parameters
In the
Database Storage
In the
Creation Options
In the
Confirmation
NOTE: The seed database creation may take more than an hour to complete.
window, click
window, click
window, click
window, click
Next
.
Next
Next
.
Create Database
window, click OK to create the database.
When the database creation procedure is completed, the
17
Click
Exit
.
18
Ty p e :
Next
.
and click
Password Management
export ORACLE_SID=dbname
where
dbname
19
To verify that the database is operating, perform the following steps:
a
Ty p e
b
Type the following query at the
is the global identifier name that you defined for the database in DBCA.
sqlplus "/ as sysdba"
to display the
SQL>
prompt:
SQL>
SELECT * FROM v$instance;
c
If the database is not running and you receive an error message, type
prompt to start the database instance on the node.
.
Finish
prompt.
/opt/oracle/recovery
.
window appears.
startup
at the
SQL>
),
Creating the Seed Database Using Oracle ASM
If you configured your storage using Oracle ASM, perform the following steps to create a seed database
with the DBCA:
1
As the user
oracle
, start DBCA by typing:
dbca &
2
In the
Welc om e
3
In the
Operations
4
In the
Database Templates
5
In the
Database Identification
6
In the
Management Options
7
In the
Database Credentials
password entries, and click
8
In the
Storage Options
window, click
window, click
window, click
window, click
Next
window, click
Next
.
Create a Database
Custom Database
window, enter a
window, click
Next
Use the Same Password for All Accounts
.
ASM
and click
and click
Next
.
and click
Global Database Name
.
Next
.
Next
.
such as
oradb
, complete
and click
44Deployment Guide
Next
.
9
In the
Create ASM Instance
10
When a message appears indicating that DBCA is ready to create and start the ASM instance,
click
OK
.
11
In the
ASM Disk Groups
12
In the
Create Disk Group
a
Enter a name for the disk group to be created, such as
window, enter the password for user
window, under
Available Disk Groups
SYS
, click
and click
Create New
Next
.
window, enter the storage information for the database files and click OK.
databaseDG
, select
External Redundancy
and select the disks to include in the disk group.
b
If you are using the RAW device interface, select
c
If you are using the ASM library driver and you cannot access the candidate disks, click
Disk Discovery String
and type
ORCL:*
as the string, and then select
/dev/raw/ASM1
.
ASM1
.
A window appears indicating that disk group creation is in progress.
d
If you are using the ASM library driver and the candidate disks are not listed, click
13
14
Discover String
Under
Available Disk Groups
In the
Disk Group
a
Enter a name for the disk group to be created, such as
Redundancy
b
If you are using the RAW device interface, select
and enter
ORCL:*
, click
as the string.
Create New
.
window, enter the information for the flashback recovery files and clickOK.
flashbackDG
, select
External
, and select the disks to include in the disk group.
/dev/raw/ASM2
.
A window appears indicating that disk group creation is in progress.
c
If you are using the ASM library driver and you cannot access the candidate disks, click
Disk Discovery String
15
In the
ASM Disk Groups
(for example,
16
In the
Database File Locations
click
Next
.
17
In the
Recovery Configuration
step 14 (for example,
18
In the
Database Content
19
In the
Initialization Parameters
20
In the
Database Storage
21
In the
Creation Options
databaseDG
, type
ORCL:*
as the string, and then select
ASM2
.
window, check the disk group that you would like to use for Database Storage
) and click
flashbackDG
window, click
window, click
window, select
Next
.
window, check
window, click
Use Common Location for All Database Files
Browse
, select the flashback group that you created in
), change the Flash Recovery Area size as needed, and click
Next
.
window, select
Next
Typical
and click
.
Create Database
Next
and click
.
Finish
.
.
Change
Change Disk
Change
, and
Next
,
.
Deployment Guide45
22
In the
Confirmation
NOTE: Creating the seed database may take more than an hour.
When the database creation is completed, the
23
Click
Exit
.
24
When database creation is completed, type the following commands to add the ORACLE_SID
environment variable entry in the
NOTE: See the section "Securing Your System" and follow the steps for additional security setup.
oradb
is the global database name that you defined in DBCA.
Adding and Removing Nodes
This section describes the steps to add a node to an existing cluster and the steps to remove a node from
a cluster.
NOTE: The new node must have the same hardware and operating system configuration as the existing node(s).
To add a node to an existing cluster:
•Add the node to the network layer.
•Configure shared storage.
•Add the node to Oracle Clusterware, database, and the database instance layers.
To remove a node from an existing cluster, reverse the process by removing the node from the database
instance, the database, and the Oracle Clusterware layers.
For more information about adding an additional node to an existing cluster, see the Oracle Real Application Clusters 10g Administration document on the Oracle website at www.oracle.com.
Adding a New Node to the Network Layer
To add a new node to the network layer:
1
Install the Red Hat Enterprise Linux operating system on the new node. See "Installing and
Configuring Red Hat Enterprise Linux."
2
Configure the public and private networks on the new node. See "Configuring the Public and
Private Networks."
3
Verify that each node can detect the storage LUNs or logical disks. See "Verifying the Storage
Configuration."
46Deployment Guide
Configuring Shared Storage on the New Node
To extend an existing RAC database to your new nodes, configure storage for the new nodes so that the
storage is the same as on the existing nodes. This section provides the appropriate procedures for ASM.
Configuring Shared Storage Using ASM
If you are using ASM, ensure that the new nodes can access the ASM disks with the same permissions as
the existing nodes.
To configure the ASM disks:
1
Log in as
2
At the command prompt, type:
setenforce 0
To permanently disable SELinux:
Open your
1
2
Locate the kernel command line and append the following option:
window displays the status of the node addition process.
on the local node and
Execute Configuration Scripts
window, click
.
Exit
directory of this existing node, type
root.sh
.
and click
on the new node as user
window, click OK.
Yes
in the
Exit
window.
./addNode.sh
root
,
4
In the
Specify Cluster Nodes for Node Addition
Next
click
.
If all the verification checks pass, the
5
Click
Install
.
The
Cluster Node Addition Progress
6
When prompted, as user
root
Cluster Node Addition Summary
window displays the status of the node addition process.
run
/opt/oracle/product/10.2.0/db_1/root.sh
window, verify that the new node is selected and
window appears.
on the new node and
press <Enter> when asked to enter the full path name of the local bin directory.
7
When
8
root.sh
In the
End of Installation
finishes running in the
window, click
Execute Configuration Scripts
Exit
and click
Yes
when asked to confirm.
window, click OK.
Reconfiguring the Listener
This section describes the steps to reconfigure the listener, which is required for remote client
connection to a database.
NOTE: The steps below assume that you are willing to stop the listener to reconfigure the existing listener.
Otherwise, the steps may be a little different than the steps below.
On one node only, perform the following steps:
1
Log in as
2
Start the X Window System by typing:
startx
3
Open a terminal window and type:
xhost +
root
.
4
As user
oracle
, stop the listener by typing:
lsnrctl stop
5
When this is successful, type
6
Select
Cluster Configuration
7
In the Real Application Clusters,
8
In the
9
In the
10
In the
menu and click
11
In the
Welc om e
Listener Configuration→ Listener
Listener Configuration→ Select Listener
Listener Configuration→ Select Protocols
window, select
Next
.
netca
to start the Net Configuration Assistant.
and click
Listener Configuration
Next
.
Active Nodes
window, select
window, click
and click
Reconfigure
Select All Nodes
window, select
window, select
Next
.
and click
LISTENER
TCP
and click
and click
Next
Next
.
.
from the pull down
Next
.
Deployment Guide49
12
In the
13
14
15
Listener Configuration→ TCP/IP Protocol
Next
and click
In the
Listener Configuration→ More Listeners?
In the
Listener Configuration Done
Click
Finish
.
.
window, click
window, select
window, select No and click
Next
.
Use the standard port number of 1521
Next
.
Adding a New Node to the Database Instance Layer
1
On one of the existing nodes, as user
dbca &
2
In the
Welc om e
3
In the
Operations
4
In the
Instance Management
5
In the
List of Cluster Databases
If your user name is not operating system-authenticated, the DBCA prompts you for a user name and
password for a database user with SYSDBA privileges.
6
Enter the user name
List of Cluster Database Instances
The
RAC database that you selected and the status of each instance.
7
In the
List of Cluster Database Instances
8
In the
Instance Naming and Node Selection
window, select the new node name, and click
9
In the
Instance Storage
10
In the
Summary
A progress bar appears, followed by a message asking if you want to extend ASM to the new node(s).
window, click
window, click
window, click
sys
and the password, and click
window, click
window click OK to add the database instance.
oracle
Next
.
Instance Management
window, select the existing database.
Finish
, start DBCA by typing:
and click
Add Instance
window appears, showing the instances associated with the
window, click
window, enter the instance name at the top of the
Next
.
.
and click
Next
Next
.
.
Next
Next
.
.
11
Click
Yes
.
The following message appears:
Do you want to perform another operation?
12
Click No.
13
On any node, determine that the instance is successfully added by typing:
srvctl status database -d <database name>
NOTE: See the section "Securing Your System" and follow the steps for additional security setup.
50Deployment Guide
Removing a Node From the Cluster
When you perform the procedures in this section, ensure that you select and remove the correct node
from the cluster.
Deleting the Node From the Database Instance Layer
1
Log in as
2
From one of the remaining nodes, type:
dbca &
In the
3
4
In the
5
In the
6
In the
If your user name is not operating system-authenticated, the DBCA prompts you for a user name and
password for a database user with SYSDBA privileges.
7
Enter the user name
The
RAC database that you selected and the status of each instance.
8
Select the instance to delete and click
This instance cannot be the local instance from where you are running DBCA. If you select the local
instance, the DBCA displays an
click
oracle
Welc om e
Operations
Instance Management
List of Cluster Databases
.
window, click
window, click
sys
and the password, and click
Next
.
Instance Management
window, click
window, select a RAC database from which to delete an instance.
List of Cluster Database Instances
Next
Error
dialog. If this occurs, click OK, select another instance, and
Finish
.
and click
Delete an instance
Next
and click
.
Next
.
Next
.
window appears, showing the instances associated with the
.
If services are assigned to this instance, the
DBCA Services Management
window appears. Use this
window to reassign services to other instances in the cluster database.
9
In the
Summary
10
Verify the information about the instance deletion operation and click OK.
window, click OK.
A progress bar appears while DBCA removes the instance and its Oracle Net configuration. When the
operation is completed, a dialog prompts whether you want to perform another operation.
11
Click No to exit.
12
Verify that the node was removed by typing:
srvctl config database -d <database name>
Deployment Guide51
Reconfiguring the Listener
1
Ty p e
netca
.
2
In the
Real Application Clusters→ Configuration
click
Next
.
3
In the
Real Application Clusters→ Active Nodes
Next
click
4
In the
5
In the
6
In the
When a message
click
7
In the
8
In the
9
Click
To Stop and Remove ASM From the Node That is Deleted
.
Welc om e
Listener Configuration→ Listener
window, select
Listener Configuration
window, select
Listener Configuration→ Select Listener
Are you sure you want to delete listener LISTENER?
Yes
.
Listener Configuration→ Listener Deleted
Listener Configuration Done
Finish
.
window, click
window, select
window, select the node that you want to delete and
window, select
window, click
Next
On one of the remaining nodes, perform the following steps:
1
Open a terminal window.
2
Ty p e :
srvctl stop asm -n <node_name>
and click
Delete
.
Cluster Configuration
Next
.
and click
LISTENER
Next
.
Next
.
and click
and
Next
.
appears,
where
<node_name>
3
Ty p e :
is the node you want to remove from the cluster.
srvctl remove asm -n <node_name>
where
<node_name>
Deleting a Node From the Database Layer
1
On the node being deleted, log in as
2
Type the following command, using the public name of the node you are deleting (for example, if you
are removing
node3-pub
is the node you want to remove from the cluster.
):
srvctl stop nodeapps -n node3-pub
Ignore error CRS-0210 which complains about the listener.
3
On the node being deleted, log in as
4
If you wish to remove the Oracle Database software, type the following command:
To determine the node number of any node, type the following command:
/crs/oracle/product/10.2.0/crs/bin/olsnodes -n
On the node that you are deleting, if you wish to remove the Oracle CRS software, type the
3
following command:
rm -rf /crs/oracle/product/10.2.0/crs/*
<public-nodename>
is public name and
root
, type the following command:
<node-number>
is the node number of the node
Reinstalling the Software
NOTICE: Reinstalling the software erases all information on the hard drives.
NOTICE: You must disconnect all external storage devices from the system before you reinstall the software.
NOTICE: Dell recommends that you perform regular backups of your database and individual nodes so that you do
not lose valuable data. Reinstall the node software only if you have no other options.
Installing the software using the Dell DeploymentCD created a redeployment partition on your hard drive
that contains all of the software images that were installed on your system. The redeployment partition
allows for quick redeployment of the Oracle software.
Reinstalling the software by using this method requires that you boot the system to the redeployment
partition. When the system boots to this partition, it automatically reinstalls the Red Hat Linux
operating system.
To reinstall software using this method, perform the following steps:
1
Disconnect the external storage device.
2
Log in as
3
Edit the grub configuration file by typing:
vi /etc/grub.conf
root
on the system on which you want to reinstall the software.
and press <Enter>.
4
In the file, change the default to 3.
5
Save the file and restart your system.
Deployment Guide53
For information about configuring the system for use, see "Configuring Red Hat Enterprise Linux" and
continue through the remaining sections to reconfigure your system.
Additional Information
Supported Software Versions
Table 1-7 lists the supported software at the time of release. For the latest supported hardware and software,
see the Dell|Oracle Tested and Validated Configurations website at www.dell.com/10g and download the
Oracle Database 10g EM64T Version 2.0 Solution Deliverable List for the latest supported versions.
Table 1-7. Supported Software Versions
Software ComponentSupported Versions
Red Hat Enterprise Linux AS EM64T (Version 4) Update 3 kernel 2.6.9-34.ELsmp, 2.6.9-34.ELlargsmp
Oracle Database version10.2.0.2
PowerPath for Linux4.5.1
DKMS2.0.11-1
QLogic HBA QLE2362 (QLA2322)8.01.02-d4
QLogic HBA QLE2460 (QLA2400)8.01.02-d4
QLogic HBA QLE2462 (QLA2400)8.01.02-d4
Emulex HBA LP1000 & LP1150e (lpfc)8.0.16.18
PERC 4e/Si, PERC 4e/Di (megaraid_mbox)2.20.4.6
PERC 5/e, PERC 5/I (megaraid_sas)00.00.02.00
Intel PRO/100 S NIC driver (e100)6.1.16-k3-NAPI
Intel PRO/1000 XT/MT/MT DP NIC driver (e1000)6.1.16-k3-NAPI
Broadcom NetXtreme BCM5704 (tg3)3.43-rh
Broadcom NetXtreme BCM5708 (bnx2)1.4.36b
54Deployment Guide
Determining the Private Network Interface
To determine which interface device name is assigned to each network interface, perform the
following steps:
1
Determine the types of NICs in your system.
See Table 1-8 to identify which integrated NICs are present in your system.
For add-in NICs, you may have Intel PRO/100 family or PRO/1000 family cards or Broadcom
NetXtreme Gigabit cards. You may have to open your system and view the add-in cards to identify your
card.
Table 1-8. Integrated NICs
SystemIntegrated NICsDriver Name
PowerEdge 1950Broadcom NetXtreme II BCM5708bnx2
PowerEdge 2950Broadcom NetXtreme II BCM5708bnx2
PowerEdge 2900Broadcom NetXtreme II BCM5708bnx2
PowerEdge 1850Intel PRO/1000e1000
PowerEdge 2850Intel PRO/1000e1000
PowerEdge 6850Broadcom NetXtreme BCM5704tg3
2
Verify that a Broadcom NetXtreme Gigabit or Intel PRO/1000 family NIC is connected with
a Cat 5e cable to the Gigabit Ethernet switch, which is your private NIC.
3
Determine the driver module your private NIC uses (see Table 1-8 above).
4
View the
more /etc/modprobe.conf
/etc/modprobe.conf
file by typing:
Several lines appear with the format
interface number and
For example, the line
driver-module
alias eth1 tg3
alias ethx driver-module
, where x is the Ethernet
is the module you determined in step 3.
appears if your operating system assigned eth1 to
a Broadcom NetXtreme Gigabit NIC.
5
Determine which Ethernet interfaces (ethx) are assigned to the type of Gigabit NIC that is connected
to the Gigabit switch.
If only one entry exists in
/etc/modprobe.conf
for your driver module type, then you have successfully
identified the private network interface.
6
If you have more than one of the same type of NIC in your system, experiment to determine which
Ethernet interface is assigned to each NIC.
For each Ethernet interface, follow the steps in "Configuring the Private Network Using Bonding" for
the correct driver module until you have identified the correct Ethernet interface.
Deployment Guide55
Troubleshooting
Table 1-9 provides recommended actions for problems that you may encounter while deploying and
using your Red Hat Enterprise Linux and Oracle software.
file and verify that the private IP addresses are being
used for all instances.
NETCANETCA fails,
resulting in
database creation
errors.
NETCANETCA cannot
configure remote
nodes or a RAW
device validation
error occurs while
running DBCA.
The public network,
hostname, or virtual
IP is not listed in the
/etc/hosts.equiv file.
The /etc/hosts.equiv
file either does not
exist or does not
include the assigned
public or virtual IP
addresses.
Before launching netca, ensure that a hostname is
assigned to the public network and that the public and
virtual IP addresses are listed in the /etc/hosts.equiv file.
Verify that the /etc/hosts.equiv file on each node
contains the correct public and virtual IP address. Try to
rsh to other public names and VIP addresses as the
user oracle.
4698419. See the
Oracle Metalink
website at
metalink.oracle.com.
EMC PowerPath
device names are not
uniform across the
nodes.
The Cluster Ready
Services CSS
daemon cannot write
to the quorum disk.
Check and make sure
you have public and
private node names
defined and that you
can ping the node
names.
The OCR file and
Voting disk are
inaccessible.
Apply patch 4698419 available on the Oracle Metalink
website at metalink.oracle.com.
Before you install Oracle Clusterware, restart PowerPath
and ensure that the PowerPath device names are
uniform across the nodes.
• Attempt to start the service again by rebooting the
node or typing
/crs/oracle/product/10.2.0/crs/
• Verify that each node has access to the quorum disk
and the user
• Check the last line in the file
$ORA_CRS_HOME/css/log/ocssd.log
• If you see
flush writes to (votingdisk)
following:
/etc/hosts
–The
IP addresses for all node hostnames, including the
virtual IP addresses.
– You can ping the public and private hostnames.
– The quorum disk is writable.
Attempt to start the service again by rebooting the node
or by running root.sh from
/crs/oracle/product/10.2.0/crs/ after correcting the
networking issues.
Correct the I/O problem and attempt to start the
service again by rebooting the node or by running
root.sh from /crs/oracle/product/10.2.0/crs/.
Verify that the node can ping over the private
interconnect to the remaining nodes in the cluster.
4
Ty p e :
/etc/inet.d/init.crs enable
5
Reboot your system.
6
In some cases, the network has a latency of up to
30 seconds before it can ping the remaining nodes in
the cluster after reboot. If this situation occurs, add
the following line to the beginning of your
/etc/inet.d/init.crs
file and reboot your system:
/bin/sleep 30
DBCAThere is no response
when you click OK
in the DBCA
Java Runtime
Environment timing
issue.
Click again. If there is still no response, restart DBCA.
Summary window.
Software
installation
You receive dd
failure error
messages while
Using copies, rather
than the original
Red Hat CDs.
When burning the CD images (ISOs), use the proper
options such as -dao if using cdrecord command.
installing the
software using Dell Deployment CD 1.
Software
installation
When connecting
to the database as a
user other than
Required permissions
are not set on the
remote node.
On all remote nodes, as user root, type: chmod 6751
$ORACLE_HOME
oracle, you receive
the error messages
ORA01034:
ORACLE not
available and
Linux Error
13: Permission
denied.
• Ensure that the system clock on the Oracle software
installation node is set to a later time than the
remaining nodes.
• Configure one of your nodes as an NTP server to
synchronize the remaining nodes in the cluster.
Software
installation
When you run
root.sh, the utility
fails to format the
OCR disk.
The utility fails to
format the OCR disk.
This issue is
documented in
Download and apply Oracle patch 4679769, found on
the Oracle Metalink website at metalink.oracle.com.
Oracle Metalink
under bug 4679769.
NetworkingThe cluster
verification check
fails.
Your public network
IP address is not
routable.
Assign a valid, routable public IP address.
For example:
192.168.xxx.xxx
Fibre
Channel
storage
system
ASM Library
Driver
You receive I/O
errors and warnings
when you load the
Fibre Channel HBA
driver module.
When you type
service
The HBA driver,
BIOS, or firmware
needs to be updated.
Check the Solution Deliverable List on the Dell|Oracle
Tested and Validated Configurations website at
www.dell.com/10g for the supported versions. Update as
required the driver, BIOS, and firmware for the Fibre
Channel HBAs.
SELinux is enabled.Disable SELinux by following the steps in the section
"Configuring Shared Storage for Oracle Clusterware."
oracleasm
start, the
procedure fails.
Operating
System
When you add a
new peripheral
Kudzu is disabled.Manually run Kudzu after you add the new peripheral
to your system.
device to your
PowerEdge system,
the operating
system does not
recognize the
device.
Deployment Guide61
Getting Help
Dell Support
For detailed information on the use of your system, see the documentation that came with your system
components.
For white papers, Dell supported configurations, and general information, visit the Dell|Oracle Tested
and Validated Configurations website at www.dell.com/10g.
For Dell technical support for your hardware and operating system software and to download the latest
updates for your system, visit the Dell Support website at support.dell.com. Information about
contacting Dell is provided in your system’s Installation and Troubleshooting Guide.
Dell Enterprise Training and Certification is now available; see www.dell.com/training for more
information. This training service may not be offered in all locations.
Oracle Support
For training information on your Oracle software and application Clusterware, see the Oracle website at
www.oracle.com or see your Oracle documentation for information about contacting Oracle.
Technical support, downloads, and other technical information are available at the Oracle MetaLink
website at metalink.oracle.com
.
Obtaining and Using Open Source Files
The software contained on the Dell DeploymentCD is an aggregate of third-party programs as well as
Dell programs. Use of the software is subject to designated license terms. All software that is designated
as "under the terms of the GNU GPL" may be copied, distributed, and/or modified in accordance with
the terms and conditions of the GNU General Public License, Version 2, June 1991. All software that is
designated as "under the terms of the GNU LGPL" (or "Lesser GPL") may be copied, distributed, and/or
modified in accordance with the terms and conditions of the GNU Lesser General Public License,
Version 2.1, February 1999. Under these GNU licenses, you are also entitled to obtain the corresponding
source files by contacting Dell at 1-800-WWW-DELL. Please see SKU 420-4534 when making such
request. You may be charged a nominal fee for the physical act of transferring a copy.
62Deployment Guide
Index
A
adding and removing
nodes, 46
additional configuration
options
adding and removing
nodes, 46
additional information, 54
determining the private
network interface, 55
ASM
configuring database
storage, 39
B
bonding, 16
C
cluster
Fibre Channel hardware
connections, example, 10
cluster setup
Fibre Channel, 10
configuring
database storage
(single node), 38
database storage (single node)
using ASM, 39
database storage (single node)
using ex3, 38
configuring (continued)
Oracle Database 10g
(single node), 38
Oracle RAC 10g, 15
Red Hat Enterprise Linux, 9
shared storage, 20
shared storage for CRS, 23