Dell Fluid Cache for DAS Troubleshooting
Balamurugan B
Krishna Kamal Kapa
Naveen Iyengar
Dell Oracle Database Solutions
Engineering
April 2013
Improving Oracle OLTP database
performance with Dell Fluid Cache for
DAS
This technical whitepaper describes how the performance of an Oracle
Online Transaction Processing database can be improved by using Dell
Fluid Cache for Direct Attach Storage.
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
This document is for informational purposes only and may contain typographical errors and
technical inaccuracies. The content is provided as is, without express or implied warranties of any
kind.
© 2013 Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions
in typography or photography. Dell, the Dell logo, and PowerEdge are trademarks of Dell Inc. Intel and
Xeon are registered trademarks of Intel Corporation in the U.S. and other countries. Microsoft,
Windows, and Windows Server are either trademarks or registered trademarks of Microsoft Corporation
in the United States and/or other countries. Other trademarks and trade names may be used in this
document to refer to either the entities claiming the marks and names or their products. Dell disclaims
proprietary interest in the marks and names of others.
April 2013| Rev 1.0
ii
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Contents
Executive summary ..................................................................................................... 5
Introduction .............................................................................................................. 5
Dell Fluid Cache for DAS Overview ................................................................................. 6
Solution design and reference architecture ........................................................................ 6
Traditional HDD-based storage solution (Baseline Configuration) ............................................ 7
Fluid Cache for DAS-based storage solution (Fluid Cache) ..................................................... 9
Test methodology ..................................................................................................... 11
Test environment .................................................................................................... 11
Baseline configuration test methodology ....................................................................... 11
Fluid Cache based solution test methodology .................................................................. 13
Performance results and analysis .................................................................................. 14
Conclusion .............................................................................................................. 19
References ............................................................................................................. 20
Appendix A. Server configuration ................................................................................. 21
Appendix B. Backend storage configuration .................................................................... 22
Appendix C. Software configuration ............................................................................. 23
Appendix C.1. Fluid Cache based solution configuration .................................................. 24
Appendix C.2. Setting up ownership and permission for Oracle disks ................................... 26
Appendix C.3. ASM diskgroup configuration .................................................................. 28
Appendix C.4. Parameter settings ............................................................................. 29
Appendix C.4.1. Kernel parameter settings .................................................................. 29
Appendix C.4.2. User security limits settings ................................................................ 30
Appendix C.4.3. Oracle database parameter settings ...................................................... 30
Appendix D. Next Steps ............................................................................................. 31
Appendix D.1. Fluid Cache for DAS Reference Architecture Solution ID ................................ 31
Appendix D.2. Database management software ............................................................. 31
Tables
Table 1. Server configuration details for baseline configuration ........................................... 21
Table 2. Server configuration details for Fluid Cache based solution ..................................... 21
Table 3. Backend storage configuration ........................................................................ 22
Table 4. Virtual Disk configuration for baseline. .............................................................. 22
iii
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Table 5. Virtual Disk configuration for Fluid Cache based solution. ....................................... 23
Table 6. Software Versions ........................................................................................ 23
Table 7. ASM disk group configuration for the baseline ...................................................... 28
Table 8. ASM disk group configuration for the Fluid Cache based solution ............................... 29
Table 9. Kernel parameters settings for Oracle database ................................................... 29
Table 10. Database parameter settings .......................................................................... 30
Figures
Figure 1. Architecture: Baseline configuration ................................................................... 8
Figure 2. Architecture: Fluid Cache based storage solution .................................................. 10
Figure 3. Performance benchmarking architecture ............................................................ 11
Figure 4. Performance graph: TPS behavior spanning entire test duration ............................... 15
Figure 5. Performance graph: ART behavior spanning entire test duration ............................... 15
Figure 6. Performance graph: CPU Utilization - IOWaits ...................................................... 16
Figure 7. Performance graph: Total CPU Utilization .......................................................... 17
Figure 8. Performance graph: Max TPS performance .......................................................... 18
Figure 9. Performance graph: Average response time (ART) at 3100 user load .......................... 18
Figure 10. Performance graph: Max user load with 2secs ART SLA ........................................... 19
iv
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Executive summary
With increasing capacity and more affordable prices, the very fast, low-latency, flash-based Peripheral
Component Interconnect Express (PCIe) Solid State Devices (SSDs) are finding a place in modern
enterprise data centers. As a result, many enterprise hardware companies have begun to integrate and
offer these PCIe SSDs in their commodity servers. One of the ways to leverage these server-side PCIe
SSDs is by using a host-based caching software that can cache the application workloads and thereby
accelerate performance. This method can provide an excellent solution to overcome the performance
bottleneck of an existing traditional spinning disk-based storage solution. Alternatively, this method
can provide an excellent solution for new environments looking for very low-latency application
performance. Dell Fluid Cache for DAS (Fluid Cache) is a new host-based caching solution that
combines the Fluid Cache for DAS software with ultra-high speed Dell PowerEdge
SSDs to create a high-performance cache within the server itself.
This paper provides a reference architecture of the Fluid Cache based storage solution and
demonstrates how the performance of an online transaction processing (OLTP) database can be
accelerated using this solution. The study compares the performance of the Fluid Cache based storage
solution to a traditional hard disk drive (HDD)-based storage solution used as the baseline
configuration.
The key performance improvement findings from this study show that the Fluid Cache based solution
compared to the baseline configuration
TM
Express Flash PCIe
Delivers approximately 60% more transactions per second (TPS)
Delivers approximately 95% improvement in average response time (ART) at 3100 user load
Supports approximately 34% more users while delivering two seconds or less ART
Introduction
In a typical OLTP database environment the data blocks that are read or are written to are small in size
and are distributed at random places on the storage disks. Due to this nature of an OLTP environment,
its performance in a traditional spinning HDD-based storage is greatly dependent on the speed at which
the disk head can seek the data blocks to read or write to. However, the speed or the movement of the
disk head is mechanically restricted; therefore, each storage disk can deliver limited input/output
operations per second (IOPS) performance. So in order to keep up with the increased demand for IOPS
performance, the number of spinning disks must be increased. However, beyond a certain point, adding
additional disks will increase capacity but not performance, since the storage controller has reached its
maximum processing capabilities.
With very fast enterprise-class PCIe SSDs now being integrated and offered in standard commoditybased servers can help to solve this problem. One of the ways to leverage these server-side PCIe SSDs is
to make use of a host-based caching software that can use these PCIe SSDs as a caching device and
thereby accelerate the database performance. However, a lot of the host-based caching software
currently in the market only support caching of the read data. As a result, the database is still
dependent on the backend spinning disks to write either the new or the modified dirty data to, and this
limits the overall performance. Dell Fluid Cache for DAS (Fluid Cache), a new host caching software,
overcomes this limitation by allowing caching of both read and write I/Os on to the local PCIe SSDs.
5
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
This whitepaper studies the performance of an Oracle OLTP database in a Fluid Cache based solution. It
compares the performance of the Fluid Cache based solution against a traditional HDD-based storage
solution used as the baseline configuration.
The following sub-section introduces the Fluid Cache for DAS product, and the following section
describes the design and reference architecture of the baseline and the Fluid Cache based storage
solution evaluated in this whitepaper. The test methodology used in the study is then described.
Comparative performance results and analysis follow. The paper ends with a conclusion on the best use
case environments that can benefit from Fluid Cache based storage solution.
This paper also includes several appendices that provide details of the hardware, software, test tools,
and other configuration and tuning details that were relevant to this study. An appendix also provides
the next steps for customers who are interested in purchasing this reference architecture.
Dell Fluid Cache for DAS Overview
Dell Fluid Cache for DAS (Fluid Cache) is a host caching software that allows you to create a virtual
cache pool on supported Dell PowerEdge systems. For the rest of this paper, Dell Fluid Cache for DAS
will be referred to as “Fluid Cache for DAS” or “Fluid Cache.”
Fluid Cache uses Dell PowerEdge Express Flash PCIe SSDs installed on supported Dell systems to provide
a read and write cache pool. You can install up to four PCIe SSDs in a Dell system. The supported PCIe
SSD capacities are 175 GB and 350 GB. These PCIe SSDs can be combined to create cache pool capacity
ranging from a minimum of 175 GB to a maximum of 1400 GB. The cache pool is used to accelerate
response times with significant improvements in input/output operations per second (IOPS).
Some of the features of the Fluid Cache software are:
Faster cache reads, writes, read-after-writes, and re-reads
Data protection, as writes are replicated across multiple PCIe SSDs
Orderly hot swap and hot plug capability that allows adding or removing a device without
halting or rebooting the system
Support of two modes of write operation: write-back and write-through
The Fluid Cache for DAS cache pool stays persistent through reboots and the cache pool is highlyavailable and resilient to failures. It ensures write cache data protection by utilizing unique journaling
and block replication technology. It actively and intelligently caches data at the block level and keeps
frequently accessed data sets on high-performance Express Flash PCIe SSD storage. This greatly reduces
latency and accelerates response times for Linux Oracle applications.
More information about Fluid Cache for DAS can be found in the user’s guide at
http://www.dell.com/support.
Solution design and reference architecture
The Fluid Cache software used in this whitepaper supports only DAS environments, and hence the scope
of the study in this whitepaper is limited to DAS-based solutions.
6
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
To accurately and systematically evaluate both the solutions, the first step was to design a baseline
configuration based on the traditional HDD-based storage. This baseline configuration was enhanced
with Fluid Cache solution. This approach allows a fair comparison of the performance merits or
demerits of Fluid Cache solution against the baseline configuration.
The following sections describe the design and reference architecture of the DAS-based storage
solutions evaluated in this study:
Traditional HDD-based storage solution (Baseline Configuration)
Fluid Cache for DAS-based storage solution (Fluid Cache)
Traditional HDD-based storage solution (Baseline Configuration)
This section describes the design and architecture of the traditional HDD-based storage solution that
was used as a baseline.
The goal of the baseline configuration was to illustrate the problem stated in the introduction i.e. to
architect a typical hardware configuration that illustrates the limitation of the baseline configuration
where the database performance cannot be improved by adding additional disks since the limits of the
storage controller have been reached.
Figure 1 shows this configuration. A Dell PowerEdge R720 is used as a single node database server. The
Dell PowerEdge Express Flash 350GB SLC PCIe SSDs that are factory installed with the server are not
used in the baseline performance testing. For more details on the R720 server configuration used in this
study, refer to Table 1. The server is attached to four fully populated Dell PowerVaultTM MD1220
storage enclosures using a single Dell PERC H810 external RAID controller. The H810 is connected to the
four daisy-chained PowerVault MD1220 enclosures in unified mode1. In unified mode, the H810
automatically detects the redundant path and balances the I/O load through both paths to the backend
enclosures2.
Note: A single PERC H810 supports a maximum of four daisy-chained PowerVault MD1220s2.
7
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Architecture: Baseline configuration Figure 1.
As shown in Figure 1, eight virtual disks (VDs) were configured for Oracle data files and four VDs were
configured for Oracle Flash Recovery Area (FRA) across the four enclosures. Twelve disks were
configured as global hot spares to ensure that there is at least one hot spare for each of the twelve VDs
used for Oracle database. The VDs used for the Oracle data files were configured with eight disks in
RAID10. The VDs used for FRA were configured as RAID5 with 4+1 disks. These RAID levels were
configured based on Dell’s recommendation and best practices for Oracle OLTP database3. For more
details on the VD configuration, refer to Table 4.
The database server was configured in accordance with Dell’s standard best practices on installing
single node single instance Oracle 11gR2 databases4.
Automatic Shared Memory Management (ASMM) was enabled and the sga_target and
sga_max_size ASMM parameters were both set to 8GB. This size of the System Global Area (SGA)
allows the database to stress the storage disks by generating more physical I/Os than logical I/Os from
the main memory.
Each of the VDs created on the backend storage for the Oracle database was added as an Automatic
Storage Management (ASM) disk to its respective ASM disk group. Figure 1 shows the following two
Oracle ASM disk groups created on the backend storage:
DATA_DG ― contains Redo Log Groups, Undo tablespace, Temp tablespace, Quest tablespace
(for TPC-C schema) and other seed database tablespaces
FRA_DG ― contains Archive logs and Flashback logs
8
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
For more details on the configuration of the ASM disk groups, refer to Table 7. For the rest of the
paper, the ASM disks added to the ‘DATA_DG’ will be referred to as data ASM disks and the ASM disks
added to the ‘FRA_DG’ will be referred to as FRA ASM disks.
For more details on the hardware and software configuration of the baseline, refer to the appendices.
In the rest of the paper, this configuration will be referred to as the “baseline configuration” or
“baseline”.
Fluid Cache for DAS-based storage solution (Fluid Cache)
This section describes how the baseline configuration described in the Traditional HDD-based storage
solution (Baseline Configuration) section was enhanced and configured as the Fluid Cache for DASbased storage solution. In the rest of the paper, this solution will be referred to as “Fluid Cache based
solution.”
In this solution, on top of the hardware and software configuration of the baseline, Fluid Cache
software was installed and enabled in the R720 database server. This solution was tested with Fluid
Cache enabled in write-back mode. This mode requires a minimum of two Express Flash PCIe SSDs to
ensure data protection and high availability that is achieved by unique journaling and block replication
technology.
As shown in Figure 2, a single Fluid Cache for DAS cache pool of ~650GB was created using two of the
350GB SLC Express Flash PCIe SSDs in the R720 database server. Since small random read/write I/O
workload benefits the most with caching, only the data ASM disks were enabled for caching in writeback mode. Caching was not enabled on the FRA ASM disks since the I/O pattern to this is small
sequential and caching is not expected to benefit.
9
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Architecture: Fluid Cache based storage solution Figure 2.
When Fluid Cache is configured with the cache pool (SSDs) and the caching on the disks to be cached
(data ASM disks) are enabled, a new Fluid Cache disk is created for each of the data ASM disks.
Example: If /dev/sdc is the block device configured as an ASM disk, then a new Fluid Cache disk
named /dev/fldc0 gets created that maps to this ASM disk.
The Fluid Cache disk is a standard /dev/xxx device and is transparent to the Oracle database
application. The appropriate ownership and permission to the data ASM disks and to the new Fluid
Cache disks are set. As the metadata of the ASM disk group is kept on the ASM disk itself and not in the
Oracle dictionary, the ASM instance scans the ASM disks based on the parameter asm_diskstring and
reads the header information from the newly-discovered Fluid Cache disks. Thus ASM is able to mount
the DATA_DG ASM disk group like in the baseline configuration.
For details on configuring Fluid Cache, refer to the following sections in the appendix:
Virtual Disk configuration for Fluid Cache based solution.
Appendix C.1.1 Enabling Fluid Cache
Appendix C.2.2. Udev settings for Fluid Cache based solution
ASM disk group configuration for the Fluid Cache based solution
10
Loading...