Third-party Information Provided to You Courtesy of Dell
Dell/EMC CX3 Series
Oracle RAC 10g
Reference Architecture Guide
Abstract
This document provides an overview of the architecture of the EMC solution for Oracle
RAC 10g on Dell/EMC CX3 Series arrays over FPC. This solution was developed by
the EMC Commercial Solutions Practice.
ii Oracle RAC 10g Dell/EMC CX3 Series FCP Reference Architecture Guide A01
EMC believes the information in this publication is accurate as of its publication date.
The information is subject to change without notice.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY
CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE
INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC
CORPORATION AND DELL INC. MAKE NO REPRESENTATIONS OR
WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS
PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication
requires an applicable software license.
Trademark Information
EMC2, EMC, EMC ControlCenter, AlphaStor, ApplicationXtender, Celerra, CentraStar,
CLARalert, CLARiiON, Connectrix, Co-StandbyServer, Dantz, Direct Matrix
Architecture, DiskXtender, Documentum, EmailXtender, EmailXtract, HighRoad,
Legato, Legato NetWorker, Navisphere, OpenScale, PowerPath, RepliStor,
ResourcePak, Retrospect, Smarts, SnapShotServer, SnapView/IP, SRDF, Symmetrix are
trademarks of EMC Corporation. Dell, the DELL logo, and PowerEdge are trademarks
of Dell Inc. All other trademarks used herein are the property of their respective owners.
Oracle RAC 10g Dell/EMC CX3 Series FCP Reference Architecture Guide
A01
THE INFORMATION CONTAINED IN THIS DOCUMENT, INCLUDING ALL
INSTRUCTIONS, CAUTIONS, AND REGULATORY APPROVALS AND
CERTIFICATIONS, IS PROVIDED BY EMC AND HAS NOT BEEN INDEPENDENTLY
VERIFIED OR TESTED BY DELL. DELL CANNOT BE RESPONSIBLE FOR DAMAGE
CAUSED AS A RESULT OF EITHER FOLLOWING OR FAILING TO FOLLOW THESE
INSTRUCTIONS. ALL STATEMENTS OR CLAIMS REGARDING THE PROPERTIES,
CAPABILITIES, SPEEDS OR QUALIFICATIONS OF THE PART REFERENCED IN
THIS DOCUMENT ARE MADE BY EMC AND NOT BY DELL. DELL SPECIFICALLY
DISCLAIMS KNOWLEDGE OF THE ACCURACY, COMPLETENESS OR
SUBSTANTIATION FOR ANY SUCH STATEMENTS. ALL QUESTIONS OR
COMMENTS RELATING TO SUCH STATEMENTS OR CLAIMS SHOULD BE
DIRECTED TO EMC CORPORATION.
Oracle RAC 10g Dell/EMC CX3 Series FCP Reference Architecture Guide ix
Page 9
Preface
Preface
As part of an effort to improve and enhance the performance and capabilities of its
product line, EMC from time to time releases revisions of its hardware and software.
Therefore, some functions described in this guide may not be supported by all revisions
of the software or hardware currently in use. For the most up-to-date information on
product features, refer to your product release notes.
This document provides an overview of the architecture of the EMC solution for Oracle
RAC 10g on Dell/EMC CX3 Series arrays over FPC. This solution was developed by
the EMC Commercial Solutions Practice.
Purpose
Information in this document can be used as the basis for a solution build, white paper,
best practices document, or training.
Information in this document can also be used by other EMC organizations (for
example, the technical services or sales organization) as the basis for producing
documentation for a technical services or sales kit.
Audience
This document is intended for Dell and EMC customers, partners, and personnel.
Scope
This document describes the architecture of an EMC solution built and tested by the
EMC Commercial Solutions Practice, located in Research Triangle Park, N.C.
Implementation instructions and best practices are beyond the scope of this document.
See the related documents, below.
x Oracle RAC 10g Dell/EMC CX3 Series FCP Reference Architecture Guide
Page 10
Preface
Related documents
The following documents, available from the EMC Powerlink website
(
http://powerlink.emc.com/ ) provide additional, relevant information:
♦ Oracle Database 10g/Oracle RAC 10g CLARiiON CX3 Series FCP Best Practices
Mid-size enterprises face the same challenges as their larger counterparts when it comes
to managing database environments. Typical challenges include cost, control, resource
utilization, and scaling.
Oracle RAC 10g supports numerous functions ranging from manufacturing to online
web applications. These deployments may be in smaller data centers that often support
applications ranging from custom database applications to standard packages such as
SAS, Siebel, PeopleSoft, and SAP, or they may be in large, centralized data centers.
Mid-size enterprises need scalable and RAID-protected storage but must be extremely
cost-conscious. They may lack the IT resources to deploy, manage, and maintain
complex environments at the departmental level.
The technology solution
The Oracle RAC 10g Dell/EMC CX3 Series FCP solution enables a mid-size enterprise
to deploy a Dell/EMC storage area network (SAN) architecture with FCP connectivity
for its Oracle RAC 10g database applications.
Solution advantages
Table 1-1 Solution advantages
The solution offers the following advantages:
Benefit Details
Lower total cost of ownership
(TCO)
Manageability
Simplified Real Application
Clusters (RAC) implementation
Higher availability
Increased flexibility
Improved data protection
Lower acquisition, administration,
and maintenance costs
Easier implementation,
provisioning, and LUN
management
Leverages ASM for simple
administration of storage
Implements a clustering
architecture that provides very
high levels of data availability
Easily make databases, or copies
of database, available (via
clones) to other servers using
storage based cloning
Integrate both availability and
backup as well as disaster
protection
Each of the configurations includes the following components:
♦ Client, RAC interconnect, and storage networks consisting of dedicated network
switches and VLANs
♦ Oracle RAC 10g servers connected to the client, RAC interconnect, and storage
networks
All solutions other
than test/dev
Test/dev solution
♦ Clients connected to the Oracle RAC 10g servers via the public network
♦ A Dell/EMC CX3 Series (CX3-20C) storage array connected to the Oracle RAC 10g
servers via the storage network
Figure 2-1 Overall architecture of all Oracle RAC 10g Dell/EMC CX3 Series FCP solutions other than
test/dev
Figure 2-2 Overall architecture of the Oracle RAC 10g Dell/EMC CX3 Series FCP test/dev solution
General characteristics
Each solution configuration has the following general characteristics:
♦ Each configuration consists of a four-node active/active Oracle RAC 10g cluster.
♦ Oracle database files, online redo log files, archived log files and the flashback
recovery area each reside on their own ASM disk group, stored on one or more
storage LUNs. Online redo log files are mirrored across two different ASM disk
groups using Oracle software multiplexing. All ASM disk groups are set to external
redundancy.
♦ The control files are mirrored across the online redo log disk groups.
♦ The voting disk and ocr files are stored on an OCFS2 file system mounted on all
RAC hosts.
♦ Storage LUNs are designed to satisfy the I/O demands of individual database
objects, using RAID 5, RAID 3, RAID 1-0 or RAID 1, as appropriate.
♦ Trunked RAC interconnect network connections use jumbo frames.
♦ All database files are stored on the Dell/EMC CX3 storage array, making database
server replacement relatively simple.
♦ Oracle RAC 10g for x86 or x86-64 runs under Red Hat Enterprise Linux.
♦ FCP provides storage connectivity via a fully-switched environment.
The following are the characteristics of the FCP network employed in testing this
solution.
FCP
♦ Fully redundant FCP switches are used.
♦ Two target ports on each of the Dell/EMC CX3 storage processors are utilized. Each
target port is connected to a separate FCP switch.
♦ Two redundant single-ported HBAs are used on each Oracle RAC 10g server. Each
HBA is connected to a separate FCP switch.
♦ Zoning is employed on the FCP switches to ensure that all RAC hosts have
redundant paths to access all LUNs on the Dell/EMC CX3 series array.
♦ EMC PowerPath software is employed on the RAC hosts to ensure path failover in
the event of HBA failure, switch or switch port failure, or SP or SP port failure
Resiliency testing is employed to ensure seamless, consistent failover of all elements of
the FCP network, including the RAC servers, the FCP switches, and the Dell/EMCarray
itself. All elements of this solution are found to be robust and reliable, being able to
survive the failure of any other element in the solution.
Storage architecture
Setting up storage involves:
♦ Creating RAID groups
♦ Binding LUNs
♦ Allocating hot spares
♦ Creating a storage group
♦ Assigning hosts to the storage group
♦ Assigning LUNs to the storage group
High availability and failover
The Dell/EMC CX3 Series has built-in high-availability features. These HA features
allow the Dell/EMC CX3 Series to survive various failures without a loss of access to
the Oracle RAC 10g database. These features protect against the following:
♦ Storage processor failure
♦ FCP target port failure
Once the RAID groups are created, EMC Navisphere software is used to create the
LUNs which are used the store the database. These LUNs are added to a storage group
which is accessible to the RAC hosts.
In general, Automatic Storage Management (ASM) is used to store all of the database
objects. The exception is the Oracle cluster registry file and the voting disk. Both of
these files must be stored on shared storage that is not on an ASM file system. Oracle
Cluster File System (OCFS2) is used to store these files. The following table contains a
detailed description of all of the database objects and where they are stored.
Online redo logs and
control file (one of the
mirrored copies)
Online redo logs and
control file (the other of
the mirrored copies)
Flashback recovery area
(all backups stored here)
Archived log dump
destination
Cloned online redo logs
and control file (one of
the mirrored copies)
Cloned online redo logs
and control file (the other
of the mirrored copies)
Database server architecture
This section describes the database server architecture.
Oracle Database 10g server FCP network architecture
Each Oracle RAC 10g server has two single-ported HBAs. Each HBA on each server is
connected to a separate, redundant FCP switch. Zoning on each switch is used to ensure
redundant pathways to the Dell/EMC storage. PowerPath software is used on the hosts to
ensure path failover in the event of an HBA or switch port failure.
Oracle Database 10g server IP network architecture
Each Oracle RAC 10g server has three network interfaces. Two interfaces connect the
server to the RAC interconnect network, enabling the heartbeat and other network I/O
required by Oracle cluster ready services.
One interface connects to the client network. Table
describes its use
Table 2-4 Database server network interface configuration
Oracle cluster ready services (CRS)
Oracle cluster ready services (CRS) are enabled on each of the Oracle RAC 10g servers.
The servers operate in active/active mode to provide local protection against a server
failure and to provide load balancing.
High availability and failover
TCP/IP provides the ability to establish redundant paths for sending I/O from a
networked computer to another networked computer. This approach uses the link
aggregation protocol, commonly referred to as “trunking”. Redundant paths facilitate
high availability and load balancing for the networked connection. Two trunked NICs on
each Oracle RAC 10g server are used for the RAC interconnect. Although the network
diagrams shown
above exclude the switches for simplicity, and fully switched
environment is used for the RAC interconnect.
Application architecture
The Oracle RAC 10g binary files, including the Oracle Cluster Ready Services software,
are installed on the database servers’ local disks. Datafiles, online redo log files, archive
log files, tempfiles, and CRS files reside on the Dell/EMC CX3 Series. These file
systems are designed (in terms of the RAID level and number of disks used) to be
appropriate for each type of file. See
The Oracle RAC 10g - Dell/EMC CX3 Series FCP solution uses the following hardware
resources.
Equipment Quality Configuration
Dell™
PowerEdge™
2950 server
Brocade M4400 Two 16 ports per switch
Dell/EMC CX3-20 One Two Storage Processors
Four
Two 3.0-GHz Intel Xeon dual-core
processors
12 GB of RAM
Two 73 GB 10K internal SAS disks
Two onboard GbE Ethernet NICs
One additional dual-port GbE Ethernet
NIC
Two QLogic QLE2460 FCP HBAs
Two target FCP ports per Storage
Processor
Two or three FC shelves
One LCFC shelf
30 or 45 146 GB FC disks (depending
on configuration)
Fifteen 320 GB LCFC disks
Table 3-1 Hardware specifications
Software resources
Table 3-2 Software specifications
The Oracle RAC 10g – Dell/EMC CX3 Series FCP solution uses the software resources
in Table
3-2.
Software title Number of licenses.
Red Hat Enterprise Linux One per database server
Oracle RAC 10g Enterprise Edition One per database server
EMC Navisphere Agent One per database server
EMC PowerPath One per database server
EMC FLARE