Balamurugan B
Krishna Kamal Kapa
Naveen Iyengar
Dell Oracle Database Solutions
Engineering
April 2013
Improving Oracle OLTP database
performance with Dell Fluid Cache for
DAS
This technical whitepaper describes how the performance of an Oracle
Online Transaction Processing database can be improved by using Dell
Fluid Cache for Direct Attach Storage.
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
This document is for informational purposes only and may contain typographical errors and
technical inaccuracies. The content is provided as is, without express or implied warranties of any
kind.
© 2013 Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions
in typography or photography. Dell, the Dell logo, and PowerEdge are trademarks of Dell Inc. Intel and
Xeon are registered trademarks of Intel Corporation in the U.S. and other countries. Microsoft,
Windows, and Windows Server are either trademarks or registered trademarks of Microsoft Corporation
in the United States and/or other countries. Other trademarks and trade names may be used in this
document to refer to either the entities claiming the marks and names or their products. Dell disclaims
proprietary interest in the marks and names of others.
April 2013| Rev 1.0
ii
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Contents
Executive summary ..................................................................................................... 5
Introduction .............................................................................................................. 5
Dell Fluid Cache for DAS Overview ................................................................................. 6
Solution design and reference architecture ........................................................................ 6
Traditional HDD-based storage solution (Baseline Configuration) ............................................ 7
Fluid Cache for DAS-based storage solution (Fluid Cache) ..................................................... 9
Test methodology ..................................................................................................... 11
Test environment .................................................................................................... 11
Baseline configuration test methodology ....................................................................... 11
Fluid Cache based solution test methodology .................................................................. 13
Performance results and analysis .................................................................................. 14
Conclusion .............................................................................................................. 19
References ............................................................................................................. 20
Appendix A. Server configuration ................................................................................. 21
Appendix B. Backend storage configuration .................................................................... 22
Appendix C. Software configuration ............................................................................. 23
Appendix C.1. Fluid Cache based solution configuration .................................................. 24
Appendix C.2. Setting up ownership and permission for Oracle disks ................................... 26
Appendix C.3. ASM diskgroup configuration .................................................................. 28
Appendix C.4. Parameter settings ............................................................................. 29
Appendix C.4.1. Kernel parameter settings .................................................................. 29
Appendix C.4.2. User security limits settings ................................................................ 30
Appendix C.4.3. Oracle database parameter settings ...................................................... 30
Appendix D. Next Steps ............................................................................................. 31
Appendix D.1. Fluid Cache for DAS Reference Architecture Solution ID ................................ 31
Appendix D.2. Database management software ............................................................. 31
Tables
Table 1. Server configuration details for baseline configuration ........................................... 21
Table 2. Server configuration details for Fluid Cache based solution ..................................... 21
Table 3. Backend storage configuration ........................................................................ 22
Table 4. Virtual Disk configuration for baseline. .............................................................. 22
iii
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Table 5. Virtual Disk configuration for Fluid Cache based solution. ....................................... 23
Table 6. Software Versions ........................................................................................ 23
Table 7. ASM disk group configuration for the baseline ...................................................... 28
Table 8. ASM disk group configuration for the Fluid Cache based solution ............................... 29
Table 9. Kernel parameters settings for Oracle database ................................................... 29
Table 10. Database parameter settings .......................................................................... 30
Figures
Figure 1. Architecture: Baseline configuration ................................................................... 8
Figure 2. Architecture: Fluid Cache based storage solution .................................................. 10
Figure 3. Performance benchmarking architecture ............................................................ 11
Figure 4. Performance graph: TPS behavior spanning entire test duration ............................... 15
Figure 5. Performance graph: ART behavior spanning entire test duration ............................... 15
Figure 6. Performance graph: CPU Utilization - IOWaits ...................................................... 16
Figure 7. Performance graph: Total CPU Utilization .......................................................... 17
Figure 8. Performance graph: Max TPS performance .......................................................... 18
Figure 9. Performance graph: Average response time (ART) at 3100 user load .......................... 18
Figure 10. Performance graph: Max user load with 2secs ART SLA ........................................... 19
iv
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Executive summary
With increasing capacity and more affordable prices, the very fast, low-latency, flash-based Peripheral
Component Interconnect Express (PCIe) Solid State Devices (SSDs) are finding a place in modern
enterprise data centers. As a result, many enterprise hardware companies have begun to integrate and
offer these PCIe SSDs in their commodity servers. One of the ways to leverage these server-side PCIe
SSDs is by using a host-based caching software that can cache the application workloads and thereby
accelerate performance. This method can provide an excellent solution to overcome the performance
bottleneck of an existing traditional spinning disk-based storage solution. Alternatively, this method
can provide an excellent solution for new environments looking for very low-latency application
performance. Dell Fluid Cache for DAS (Fluid Cache) is a new host-based caching solution that
combines the Fluid Cache for DAS software with ultra-high speed Dell PowerEdge
SSDs to create a high-performance cache within the server itself.
This paper provides a reference architecture of the Fluid Cache based storage solution and
demonstrates how the performance of an online transaction processing (OLTP) database can be
accelerated using this solution. The study compares the performance of the Fluid Cache based storage
solution to a traditional hard disk drive (HDD)-based storage solution used as the baseline
configuration.
The key performance improvement findings from this study show that the Fluid Cache based solution
compared to the baseline configuration
TM
Express Flash PCIe
Delivers approximately 60% more transactions per second (TPS)
Delivers approximately 95% improvement in average response time (ART) at 3100 user load
Supports approximately 34% more users while delivering two seconds or less ART
Introduction
In a typical OLTP database environment the data blocks that are read or are written to are small in size
and are distributed at random places on the storage disks. Due to this nature of an OLTP environment,
its performance in a traditional spinning HDD-based storage is greatly dependent on the speed at which
the disk head can seek the data blocks to read or write to. However, the speed or the movement of the
disk head is mechanically restricted; therefore, each storage disk can deliver limited input/output
operations per second (IOPS) performance. So in order to keep up with the increased demand for IOPS
performance, the number of spinning disks must be increased. However, beyond a certain point, adding
additional disks will increase capacity but not performance, since the storage controller has reached its
maximum processing capabilities.
With very fast enterprise-class PCIe SSDs now being integrated and offered in standard commoditybased servers can help to solve this problem. One of the ways to leverage these server-side PCIe SSDs is
to make use of a host-based caching software that can use these PCIe SSDs as a caching device and
thereby accelerate the database performance. However, a lot of the host-based caching software
currently in the market only support caching of the read data. As a result, the database is still
dependent on the backend spinning disks to write either the new or the modified dirty data to, and this
limits the overall performance. Dell Fluid Cache for DAS (Fluid Cache), a new host caching software,
overcomes this limitation by allowing caching of both read and write I/Os on to the local PCIe SSDs.
5
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
This whitepaper studies the performance of an Oracle OLTP database in a Fluid Cache based solution. It
compares the performance of the Fluid Cache based solution against a traditional HDD-based storage
solution used as the baseline configuration.
The following sub-section introduces the Fluid Cache for DAS product, and the following section
describes the design and reference architecture of the baseline and the Fluid Cache based storage
solution evaluated in this whitepaper. The test methodology used in the study is then described.
Comparative performance results and analysis follow. The paper ends with a conclusion on the best use
case environments that can benefit from Fluid Cache based storage solution.
This paper also includes several appendices that provide details of the hardware, software, test tools,
and other configuration and tuning details that were relevant to this study. An appendix also provides
the next steps for customers who are interested in purchasing this reference architecture.
Dell Fluid Cache for DAS Overview
Dell Fluid Cache for DAS (Fluid Cache) is a host caching software that allows you to create a virtual
cache pool on supported Dell PowerEdge systems. For the rest of this paper, Dell Fluid Cache for DAS
will be referred to as “Fluid Cache for DAS” or “Fluid Cache.”
Fluid Cache uses Dell PowerEdge Express Flash PCIe SSDs installed on supported Dell systems to provide
a read and write cache pool. You can install up to four PCIe SSDs in a Dell system. The supported PCIe
SSD capacities are 175 GB and 350 GB. These PCIe SSDs can be combined to create cache pool capacity
ranging from a minimum of 175 GB to a maximum of 1400 GB. The cache pool is used to accelerate
response times with significant improvements in input/output operations per second (IOPS).
Some of the features of the Fluid Cache software are:
Faster cache reads, writes, read-after-writes, and re-reads
Data protection, as writes are replicated across multiple PCIe SSDs
Orderly hot swap and hot plug capability that allows adding or removing a device without
halting or rebooting the system
Support of two modes of write operation: write-back and write-through
The Fluid Cache for DAS cache pool stays persistent through reboots and the cache pool is highlyavailable and resilient to failures. It ensures write cache data protection by utilizing unique journaling
and block replication technology. It actively and intelligently caches data at the block level and keeps
frequently accessed data sets on high-performance Express Flash PCIe SSD storage. This greatly reduces
latency and accelerates response times for Linux Oracle applications.
More information about Fluid Cache for DAS can be found in the user’s guide at
http://www.dell.com/support.
Solution design and reference architecture
The Fluid Cache software used in this whitepaper supports only DAS environments, and hence the scope
of the study in this whitepaper is limited to DAS-based solutions.
6
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
To accurately and systematically evaluate both the solutions, the first step was to design a baseline
configuration based on the traditional HDD-based storage. This baseline configuration was enhanced
with Fluid Cache solution. This approach allows a fair comparison of the performance merits or
demerits of Fluid Cache solution against the baseline configuration.
The following sections describe the design and reference architecture of the DAS-based storage
solutions evaluated in this study:
Traditional HDD-based storage solution (Baseline Configuration)
Fluid Cache for DAS-based storage solution (Fluid Cache)
Traditional HDD-based storage solution (Baseline Configuration)
This section describes the design and architecture of the traditional HDD-based storage solution that
was used as a baseline.
The goal of the baseline configuration was to illustrate the problem stated in the introduction i.e. to
architect a typical hardware configuration that illustrates the limitation of the baseline configuration
where the database performance cannot be improved by adding additional disks since the limits of the
storage controller have been reached.
Figure 1 shows this configuration. A Dell PowerEdge R720 is used as a single node database server. The
Dell PowerEdge Express Flash 350GB SLC PCIe SSDs that are factory installed with the server are not
used in the baseline performance testing. For more details on the R720 server configuration used in this
study, refer to Table 1. The server is attached to four fully populated Dell PowerVaultTM MD1220
storage enclosures using a single Dell PERC H810 external RAID controller. The H810 is connected to the
four daisy-chained PowerVault MD1220 enclosures in unified mode1. In unified mode, the H810
automatically detects the redundant path and balances the I/O load through both paths to the backend
enclosures2.
Note: A single PERC H810 supports a maximum of four daisy-chained PowerVault MD1220s2.
7
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Architecture: Baseline configuration Figure 1.
As shown in Figure 1, eight virtual disks (VDs) were configured for Oracle data files and four VDs were
configured for Oracle Flash Recovery Area (FRA) across the four enclosures. Twelve disks were
configured as global hot spares to ensure that there is at least one hot spare for each of the twelve VDs
used for Oracle database. The VDs used for the Oracle data files were configured with eight disks in
RAID10. The VDs used for FRA were configured as RAID5 with 4+1 disks. These RAID levels were
configured based on Dell’s recommendation and best practices for Oracle OLTP database3. For more
details on the VD configuration, refer to Table 4.
The database server was configured in accordance with Dell’s standard best practices on installing
single node single instance Oracle 11gR2 databases4.
Automatic Shared Memory Management (ASMM) was enabled and the sga_target and
sga_max_size ASMM parameters were both set to 8GB. This size of the System Global Area (SGA)
allows the database to stress the storage disks by generating more physical I/Os than logical I/Os from
the main memory.
Each of the VDs created on the backend storage for the Oracle database was added as an Automatic
Storage Management (ASM) disk to its respective ASM disk group. Figure 1 shows the following two
Oracle ASM disk groups created on the backend storage:
DATA_DG ― contains Redo Log Groups, Undo tablespace, Temp tablespace, Quest tablespace
(for TPC-C schema) and other seed database tablespaces
FRA_DG ― contains Archive logs and Flashback logs
8
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
For more details on the configuration of the ASM disk groups, refer to Table 7. For the rest of the
paper, the ASM disks added to the ‘DATA_DG’ will be referred to as data ASM disks and the ASM disks
added to the ‘FRA_DG’ will be referred to as FRA ASM disks.
For more details on the hardware and software configuration of the baseline, refer to the appendices.
In the rest of the paper, this configuration will be referred to as the “baseline configuration” or
“baseline”.
Fluid Cache for DAS-based storage solution (Fluid Cache)
This section describes how the baseline configuration described in the Traditional HDD-based storage
solution (Baseline Configuration) section was enhanced and configured as the Fluid Cache for DASbased storage solution. In the rest of the paper, this solution will be referred to as “Fluid Cache based
solution.”
In this solution, on top of the hardware and software configuration of the baseline, Fluid Cache
software was installed and enabled in the R720 database server. This solution was tested with Fluid
Cache enabled in write-back mode. This mode requires a minimum of two Express Flash PCIe SSDs to
ensure data protection and high availability that is achieved by unique journaling and block replication
technology.
As shown in Figure 2, a single Fluid Cache for DAS cache pool of ~650GB was created using two of the
350GB SLC Express Flash PCIe SSDs in the R720 database server. Since small random read/write I/O
workload benefits the most with caching, only the data ASM disks were enabled for caching in writeback mode. Caching was not enabled on the FRA ASM disks since the I/O pattern to this is small
sequential and caching is not expected to benefit.
9
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Architecture: Fluid Cache based storage solution Figure 2.
When Fluid Cache is configured with the cache pool (SSDs) and the caching on the disks to be cached
(data ASM disks) are enabled, a new Fluid Cache disk is created for each of the data ASM disks.
Example: If /dev/sdc is the block device configured as an ASM disk, then a new Fluid Cache disk
named /dev/fldc0 gets created that maps to this ASM disk.
The Fluid Cache disk is a standard /dev/xxx device and is transparent to the Oracle database
application. The appropriate ownership and permission to the data ASM disks and to the new Fluid
Cache disks are set. As the metadata of the ASM disk group is kept on the ASM disk itself and not in the
Oracle dictionary, the ASM instance scans the ASM disks based on the parameter asm_diskstring and
reads the header information from the newly-discovered Fluid Cache disks. Thus ASM is able to mount
the DATA_DG ASM disk group like in the baseline configuration.
For details on configuring Fluid Cache, refer to the following sections in the appendix:
Virtual Disk configuration for Fluid Cache based solution.
Appendix C.1.1 Enabling Fluid Cache
Appendix C.2.2. Udev settings for Fluid Cache based solution
ASM disk group configuration for the Fluid Cache based solution
10
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Test methodology
This section describes the test environment, test methodologies and the tools used to benchmark the
two solutions described in the Solution design and reference architecture section. Most of the steps
described in the Baseline configuration test methodology sub-section are used for Fluid Cache based
solution testing as well. The subsequent sub-sections describe the methods and procedures that are
specific to that particular solution.
Test environment
Dell Quest Benchmark Factory (BMF) software was used as the benchmarking tool for the OLTP
performance testing in this whitepaper. As shown in Figure 3, BMF was setup on a dedicated server and
was connected to the solution under test via a GigE Ethernet link. “Solution under test” represents
each of the storage solutions described in the Solution design and reference architecture section. BMF
was used to create the 450GB TPC-C database schema set and was also used to generate the OLTP
workload against that database schema.
Performance benchmarking architecture Figure 3.
Baseline configuration test methodology
This section describes the test methodology used to evaluate the baseline configuration. The steps
listed in this section forms the basis for Fluid Cache based solution section that follows.
1. Solution hardware setup: The baseline configuration hardware was configured as described in
the Traditional HDD-based storage solution (Baseline Configuration) section.
2. Database Server setup: The operating system (OS), network, storage and database pre-
requisites were configured on the PowerEdge R720 database server according to Dell’s
recommended best practices4.
11
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
2.1 The BIOS System Profile was changed to ‘Performance’. Rest of the BIOS options were
at default settings.
2.2 For details on VD configuration, refer to Table 4.
2.3 The Oracle ASM disk ownership and permission settings were configured in the
/etc/udev rules. For more details, refer to Appendix C.2.1. Udev settings for
baseline configuration.
2.4 The OS kernel parameters were tuned for performance testing. For more details, refer
to Appendix C.4.1. Kernel parameter settings.
2.5 Oracle’s OS Watcher (OSW) was installed to capture the performance metrics (CPU and
memory usage) of the host and the storage array (IOPS).
3. Oracle database setup: Oracle 11gR2 grid and database software for single node single
instance was installed4.
3.1 For details on configuration of ASM disks and ASM disk groups, refer to Table 7.
3.2 Oracle’s Automatic Shared Memory Management (ASMM) feature was enabled.
sga_target and sga_max_size were set to 8GB to stress the storage disks by
generating more physical I/Os than logical I/Os from the main memory
3.3 The database parameters were tuned for the performance test. For more details, refer
to Table 10.
4. Benchmarking tool and TPC-C database schema setup:
4.1 Dell Quest Benchmark Factory (BMF) was used to create a TPC-C test Oracle schema
data set of 450GB (300GB tables + 150GB Indexes) by using a benchmarking scale
factor of 4700 and was loaded into the DATA_DG
5. Database Stats collection: Database stats on the TPC-C schema was manually collected one
time in the beginning using the below SQL Query
SQL> exec dbms_stats.gather_schema_stats('QUEST', estimate_percent =>15);
where QUEST is the 450GB TPC-C schema created from BMF
6. Database schema backup:
6.1 The TPC-C schema was backed-up and exported using Oracle’s Data Pump on to two
local 300GB 15K SAS HDDs on the database server for later use.
6.2 In order to quickly restore the database to its original clean state after each of the
test iterations, Oracles flashback guaranteed restore point was created. For more
details, refer to the wiki article How to quickly restore to a clean database using
Oracle’s restore point.
7. Test load setup: BMF was used to generate the OLTP workload against the 450GB TPC-C
database schema.
7.1 OLTP workload was configured with a user load range of 500-5000 in steps of 100. The
max userload value was determined by doing sample tests to ensure that the highest
user load was able to stress the solution beyond the 2 seconds average response time
7.2 The sampling duration for each iteration was unchanged from the default value of
4mins.
7.3 The think time and the keying time latency setting for each of the TPC-C transaction
mix was set to 1/10th the default value.
8. Pre-test setup:
12
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
8.1 Before each test, it was ensured that the Open Manage ‘dataeng’ service was
stopped to avoid any performance overhead.
8.2 Before each test, it was ensured that Oracle AWR report was not being collected to
avoid any performance overhead.
9. Test start:
9.1 The OSWatcher script was started on the database server to capture the performance
metrics.
9.2 The OLTP workload setup in step 8 was started from the BMF server.
10. Test end: The test was allowed to run until the average response time (ART) as displayed by
BMF crossed the 2 seconds limit.
11. Post-test results capture:
11.1 The User load, TPS and the Average Response Time (ART) results were captured using
BMF.
11.2 CPU utilization, IOWaits, and memory utilization stats were extracted from OSWatcher
results using a custom script.
12. Re-running the test:
12.1 Database was restored to the clean state using the flashback guaranteed restore point
created in step 6.2
12.2 Server was rebooted to ensure the server is in a clean state
12.3 Repeated step 8, step 9, step 10 and step 11 in that order to test the solution again, if
necessary.
Fluid Cache based solution test methodology
This section describes the test methodology that is specific to this solution testing. Hardware,
software, test methods and configuration settings listed in the Baseline configuration test methodology
section apply to this solution testing as well, unless specified below.
1. Restore database: Database was restored to the clean state before installing or enabling Fluid
Cache for DAS caching using the flashback guaranteed restore point created in step 6.2 of the
Baseline configuration test methodology section. This was done to ensure cold cache.
2. Express Flash PCIe SSD hardware and software setup: Only two 350GB Express Flash PCIe
SSDs were used for this solution testing giving a cache pool size of ~650GB. Fluid Cache
software was installed using the Fluid Cache for DAS user’s guide8.
3. Fluid Cache setup: Fluid Cache was tested in write-back mode only. For more details on the
configuration steps, refer to Appendix C.1.1 Enabling Fluid Cache.
4. IO Queue Depth: The Fluid Cache IO queue depth was increased from the default value of 64
to 256 in the /sys/kernel/config/fldc/io_queue_depth parameter file.
5. Testing and results capture: Step 12 from the Baseline configuration test methodology section
was repeated to test this storage solution. This step corresponds to restoring the database to a
clean state, restarting the OLTP benchmark and capturing the results.
6. Re-running the test:
6.1 To flush all the data block references and to start with cold cache again, Fluid Cache
was disabled. For details on the steps followed, refer to Appendix C.1.2 Removing
Fluid Cache.
6.2 Database was restored to the clean state using the flashback guaranteed restore point
created in step 6.2 of the Baseline configuration test methodology section.
6.3 Rebooted the server to ensure the server is in a clean state.
13
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
6.4 Re-setup Fluid Cache as noted in Appendix C.1.1 Enabling Fluid Cache.
6.5 Repeat step 5 to run the test again, if necessary.
7. Disabling Fluid Cache:
7.1 Fluid Cache was disabled. For details on the steps followed, refer to Appendix C.1.2
Removing Fluid Cache.
Performance results and analysis
This section presents the performance results and analysis of the two storage solutions described in the
Solution design and reference architecture section:
Traditional HDD-based storage solution (Baseline Configuration)
Fluid Cache for DAS-based storage solution (Fluid Cache)
For the remainder of this section each of the solutions will be referred to with the short description
mentioned in bold in the brackets above. These short descriptions will also be used as legends in the
graphs to represent each of the solution.
Note: The results presented in this section are intended only to showcase the relative performance of
the two solutions configured as described in this document. The results are not indicative of the
maximum capabilities of any system, database software, storage, or the solution.
For each of the solutions under test, multiple tests were conducted to check for consistency of the
results. Two seconds was used as the acceptable max average response time (ART) for each of the
tests. The tests were either manually terminated or the results were discarded past that point. Also, to
account for the standard deviation between each test run, the average of the results was used to
evaluate the performance of each of the storage solution.
Figure 4 shows the transactions per second (TPS) performance behavior spanning the entire duration of
the test for the two storage solutions evaluated in this paper. The graph plots the TPS behavior until
the average response time (ART) of two seconds was exceeded.
14
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
0
200
400
600
800
1000
1200
1400
1600
1800
500 900 1300 1700 2100 2500 2900 3300 3700 4100 4500
TPS
User Load
OLTP (R720) - Transactions/Sec (TPS)
Baseline Fluid Cache
0
0.5
1
1.5
2
500 900 1300 1700 2100 2500 2900 3300 3700 4100 4500
ART (secs)
User Load
OLTP (R720) - Average Response Time (ART)
Baseline Fluid Cache
Performance graph: TPS behavior spanning entire test duration Figure 4.
As seen in Figure 4, both the storage solutions deliver similar TPS until a certain user load. Beyond that
user load point, the TPS performances for both the solutions start to differ and hit their peak
performance. This behavior occurs due to a sharp rise in average response time (ART) as seen in Figure
5. For example, in Figure 4, the Fluid Cache based solution reaches its peak TPS performance at 3200
user load. And, Figure 5 shows that the ART for the Fluid Cache based solution starts to climb sharply
around the same 3200 user load.
Performance graph: ART behavior spanning entire test duration Figure 5.
15
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
0
20
40
60
80
100
%
Increasing User Load -->
OLTP (R720) - CPU Utilization - IOWaits
Baseline Fluid Cache
The CPU utilization captured during the benchmarking tests using the OSWatcher tool helps to further
investigate the cause of the sharp increase in the ART. As seen in Figure 6, the peak CPU IOWait for the
baseline configuration is around 60%, while the peak CPU IOWait for the Fluid Cache based solution is
around 10%. A high CPU IOWait typically indicates that the CPU is waiting on the backend storage I/O
to finish. This high IOWait behavior is observed in case of the baseline configuration because the
backend spinning disks, with its limited IOPS capability, cannot complete the increasing number of
transactions without the increase in response time. This confirms that the backend storage eventually
becomes the bottleneck in the case of the baseline configuration.
Performance graph: CPU Utilization - IOWaits Figure 6.
The 10% peak CPU IOWait in the case of the Fluid Cache based solution suggests that the storage was
not the cause for the increase in the average response time as the user load increased. The total CPU
utilization graph in Figure 7 helps to further investigate the cause of the increase in the ART in the
case of the Fluid Cache based solution.
16
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
0
20
40
60
80
100
%
Increasing User Load -->
OLTP (R720) - Total CPU Utilization
Baseline Fluid Cache
Performance graph: Total CPU Utilization Figure 7.
Figure 7 shows, that in the case of the Fluid Cache based solution, as the user load keeps increasing,
the total CPU utilization also keeps increasing. In the Fluid Cache based solution, the cache pool
configured on the Dell Express Flash PCIe SSDs is transparent to the database application. As a result,
with caching enabled on the backend store, the database reads, re-reads, and writes are transparently
managed by the Fluid Cache software.
With the Fluid Cache based solution configured in write-back mode, the modified or the dirty data
flushed from the SGA buffer cache in the main-memory by the database is written to the Fluid Cache
for DAS cache pool. Fluid Cache transfers this cached dirty data back to the backend storage at a later
time. Without Fluid Cache, this dirty data would be flushed directly back to the spinning disk. As a
result, from the database point of view, most of the physical read and write transactions are
completed with very low latency even at high user loads in a Fluid Cache based solution. This
attributes to the increased CPU utilization. However, as seen in Figure 7, the CPU eventually hits 100%
utilization that results in response times crossing the two-second threshold. Thus, the CPU eventually
becomes the bottleneck in case of the Fluid Cache based solution.
17
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
983
1577
0
200
400
600
800
1000
1200
1400
1600
1800
Baseline Fluid Cache
TPS
OLTP (R720) - Max TPS
60%
1.712
0.09
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Baseline Fluid Cache
ART (sec)
OLTP (R720) - ART at 3100 User Load
95%
Performance graph: Max TPS performance Figure 8.
Figure 8 compares the max TPS performance of the baseline and the Fluid Cache based solution. As can
be seen from that graph, Fluid Cache based solution delivers 60% more TPS than the baseline
configuration.
Performance graph: Average response time (ART) at 3100 user load Figure 9.
18
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
3200
4300
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Baseline Fluid Cache
Userload
OLTP (R720) - Max Userload with 2secs ART SLA
34%
Figure 9 compares the average response time (ART) performance of both the storage solutions at 3100
user load. As can be seen from the graph, the Fluid Cache based solution delivers 95% lower ART at
3100 user load, which is near the peak performance of the baseline configuration.
Figure 10 compares the maximum user load that the two storage solutions were able to deliver while
keeping the response time at two seconds or less. As seen in Figure 10, the Fluid Cache based solution
delivered 34% more user load compared to the baseline configuration.
Performance graph: Max user load with 2secs ART SLA Figure 10.
The results in this section show that by adding Fluid Cache based solution to a saturated baseline
configuration can vastly improve the OLTP database performance without having to change any existing
hardware or software.
Conclusion
This paper demonstrates how a Dell PowerEdge server configured with Dell Fluid Cache for DAS can be
easily added to a traditional backend spinning HDD-based storage solution to significantly improve
OLTP performance in a DAS environment.
Fluid Cache can be a great solution to boost performance for the following DAS environments and use
cases
Existing traditional spinning HDD-based storage environments looking to boost performance for
OLTP databases. This can be achieved by replacing their existing single node database server
with a Dell’s PowerEdge server that supports Dell PowerEdge Express Flash PCIe SSDs and
configured with Dell Fluid Cache for DAS.
19
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Existing traditional spinning HDD-based storage environments that cannot grow in performance
because storage controller has reached its maximum processing capabilities. This can be
achieved by replacing their existing single node database server with a Dell’s PowerEdge server
that supports Dell PowerEdge Express Flash PCIe SSDs and configured with Dell Fluid Cache for
DAS.
New single node DAS solutions looking for low latency, high TPS for OLTP databases.
Single or multiple large OLTP databases (>700GB) looking for low latency and high TPS
performance and that cannot be consolidated directly onto the server-side PCIe SSDs.
References
1. Dell™ PowerVault™ MD1200 and MD1220 Storage Enclosures Hardware Owner’s Manual
http://support.dell.com/support/edocs/systems/md1220/en/HOM/PDF/HOMA00EN.pdf
2. Dell PowerEdge RAID Controller (PERC) H310, H710, H710P, and H810 User’s Guide
http://support.dell.com/support/edocs/storage/Storlink/PERC_8/en/UG_PERC8_en.pdf
3. Comprehending the Tradeoffs between Deploying Oracle
®
Database on RAID 5 and RAID 10 Storage
Configurations
http://i.dell.com/sites/content/business/solutions/whitepapers/en/Documents/tradeoffs-raid-5raid-10.pdf
4. Single node single instance Oracle
®
11gR2 database installation guide
http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_solutions/3885.4-1-1-10-how-do-i-install-a-standalone-oracle-database.aspx
5. Dell PowerEdge Express Flash PCIe SSD user’s guide
ftp://ftp.dell.com/Manuals/all-products/esuprt_ser_stor_net/esuprt_dell_adapters/dellpoweredge-exp-fsh-pcie-ssd_User%27s%20Guide_en-us.pdf
6. Dell PowerEdge R720
http://www.dell.com/us/enterprise/p/poweredge-r720/pd?~ck=anav
7. Dell Quest Benchmark Factory (BMF)
http://www.quest.com/benchmark-factory/
8. Dell Fluid Cache for DAS
User’s guide can be found at: www.dell.com/support
20
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Server Configuration
Database Server
PowerEdge R720
Bios: v1.5.1
iDRAC7 ESM FW: v1.30.30
LC2: v1.1.0.1108
Backplane, 12G SEP FW: v1.00
Backplane (Flash) Expander ESM FW: v1.03
CPLD: 103
CPU
2 x Intel Xeon E5-2690 @ 2.90GHz 8c 8GT/s L2/L3: 2MB/20MB
Memory
256GB (16x16GB RDIMMs); 1600MHz; 1.5V;
Populated DIMM Slots: A1-A8; B1-B8
PERC H810 Adapter
FW Package: 21.1.0-0007
Driver: 00.00.05.40-rh2 (native)
Populated in PCIe Slot 7
PERC H310 mini for internal drives
FW: 20.11.0-0002
Driver: 00.00.05.40-rh2 (native)
On board Network Adapter
BCM5720 QP rNDC 1GbE
FW: v7.0.47
Driver: v7.2.14
Internal drives
Two 146GB 15K SAS HDDs (Operating System)
Two 300GB 15K SAS HDDs (TPC-C Database Schema Backup)
FW: YS08
Server Configuration
Dell PowerEdge Express Flash PCIe
SSDs
2 x 350GB SLC Express Flash PCIe SSDs
Driver: v1.3.7
FW: B1490208
PCIe extender adapter card
Populated in PCIe slot 5
Rest of the configuration is the same as baseline, as described in Table 1
Appendix A. Server configuration
This section describes the details of the hardware configuration used for each of the storage solutions
described in the Solution design and reference architecture section.
Server configuration details for baseline configuration Table 1.
Server configuration details for Fluid Cache based solution Table 2.
21
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Backend Storage Configuration
Storage
4 x Dell PowerVault MD1220s storage enclosures
Backplane (primary & secondary)
HW rev. A01;
FRU configuration Image rev. A00
EMM
FW v1.01
Power Supplies
HW rev. A00; FW v01.00.14
HDDs
Each Enclosure: 24 x 146GB 15K SAS HDD (Total 96 HDDs)
HDD Firmware
Enclosure0/Enclosure1: Seagate:ST9146852SS; FW:HT65
Enclosure2/Enclosure3: Seagate:ST9146853SS; FW:YS07
Configuration Mode
Unified
Virtual
Disk
(VD)
OS Block Device
RAID Group
(RG) Type
#HDDs/
RG
VD Size
Cache Policy
Segment
Size
DATA1
/dev/sdc1*
10 8 544.50GB
r=NRA+;w=WB
64KB
DATA2
/dev/sdd1*
10 8 544.50GB
r=NRA+;w=WB
64KB
FRA1
/dev/sde1*
5 5 544.50GB
r=ARA^;w=WB
64KB
DATA3
/dev/sdf1*
10 8 544.50GB
r=NRA+;w=WB
64KB
DATA4
/dev/sdg1*
10 8 544.50GB
r=NRA+;w=WB
64KB
FRA2
/dev/sdh1*
5 5 544.50GB
r=ARA^;w=WB
64KB
DATA5
/dev/sdi1*
10 8 544.50GB
r=NRA+;w=WB
64KB
DATA6
/dev/sdj1*
10 8 544.50GB
r=NRA+;w=WB
64KB
FRA3
/dev/sdk1*
5 5 544.50GB
r=ARA^;w=WB
64KB
DATA7
/dev/sdl1*
10 8 544.50GB
r=NRA+;w=WB
64KB
DATA8
/dev/sdm1*
10 8 544.50GB
r=NRA+;w=WB
64KB
Appendix B. Backend storage configuration
This section describes the details of the backend storage configuration used for each of the storage
solutions described in the Solution design and reference architecture section.
Backend storage configuration Table 3.
Virtual Disk configuration for baseline. Table 4.
22
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Virtual
Disk
(VD)
OS Block Device
RAID Group
(RG) Type
#HDDs/
RG
VD Size
Cache Policy
Segment
Size
FRA4
/dev/sdn1*
5 5 544.50GB
r=ARA^;w=WB
64KB
Block Device
Device Size
Cache Policy
Partition Type
/dev/rssda1*
350GB
default
default (fdisk)
/dev/rssdb1*
350GB
default
default (fdisk)
Rest of the VD configuration is the same as baseline, as described in Table 4
Software Versions
Operating System
Oracle Linux 6.2 – 2.6.32-220.el6.x86_64 (Red Hat Compatible
Kernel)
Database
Oracle 11g R2 (standalone database) – 11.2.0.3
Dell Fluid Cache for DAS
v1.0.0
Dell Open Manage
v7.1.2
OSWatcher (stats capture)
OSWatcher Black Box v5.0
Dell Quest Benchmark Factory
(BMF)
v6.8.1
Oracle database Client on BMF
Server
Oracle 11g R2 Client
*
- single disk partition spanning the entire disk; + - No Read Ahead; ^ - Adaptive Read Ahead
Virtual Disk configuration for Fluid Cache based solution. Table 5.
*
- single disk partition spanning the entire disk;
Appendix C. Software configuration
This section provides the software versions, software installation, and software configuration steps for
all the storage solutions described in the Solution design and reference architecture section.
Software Versions Table 6.
23
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Appendix C.1. Fluid Cache based solution configuration
Appendix C.1.1 Enabling Fluid Cache
This section describes the steps to configure Fluid Cache on the Express Flash PCIe SSDs and on the
backend store. This section assumes that the Fluid Cache for DAS v1.0.0 software has already been
installed and the fluid-cache service is up and running.
1. Stop the Oracle High Availability Service (OHAS) stack
1.1. $> su – grid
1.2. $> sqlplus / as sysasm
1.3. SQL> crsctl stop has
1.4. SQL> exit
2. Add the following two Dell Express Flash PCIe SSDs to the Fluid Cache pool
2.1. $> fldc --list --ssd
2.2. $> fldc --add --ssd=/dev/rssda1 –y
2.3. $> fldc --add --ssd=/dev/rssdb1 -y
2.4. $> fldc --status
3. Enable caching on the backend store. The following steps enable caching only on the VDs belonging
to the DATA_DG of the baseline configuration, as listed in Table 7
3.1. $> fldc --enable --disk=/dev/sdc1 --mode=wb [creates
3.2. $> fldc --enable --disk=/dev/sdd1 --mode=wb [creates
3.3. $> fldc --enable --disk=/dev/sdf1 --mode=wb [creates
3.4. $> fldc --enable --disk=/dev/sdg1 --mode=wb [creates
3.5. $> fldc --enable --disk=/dev/sdi1 --mode=wb [creates
3.6. $> fldc --enable --disk=/dev/sdj1 --mode=wb [creates
3.7. $> fldc --enable --disk=/dev/sdl1 --mode=wb [creates
3.8. $> fldc --enable --disk=/dev/sdm1 --mode=wb [creates
4. Check the status of the cache
4.1. $> fldc --status
5. Follow Appendix C.2.2. Udev settings for Fluid Cache based solution to set up the Oracle
ownership and permission in the /etc/udev rules file for the Fluid Cache devices that got created
in step 3.
/dev/fldc0]
/dev/fldc1]
/dev/fldc2]
/dev/fldc3]
/dev/fldc4]
/dev/fldc5]
/dev/fldc6]
/dev/fldc7]
6. Start the Oracle High Availability Service (OHAS) stack
6.1. $> su – grid
6.2. grid@$> sqlplus / as sysasm
6.3. SQL> crsctl start has
24
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
6.4. SQL> exit
6.5. Run the following command as grid user to check if the entire OHAS stack is up and running
6.5.1. grid@$> crsctl stat –res -t
Appendix C.1.2 Removing Fluid Cache
This section describes the steps to disable and remove Fluid Cache.
1. Stop the Oracle High Availability Service (OHAS) stack
1.1. $> su – grid
1.2. grid@$> sqlplus / as sysasm
1.3. SQL> crsctl stop has
1.4. SQL> exit
2. Disable caching on the backend store. Disabling may take some time. Therefore, use the fldc --
status command to check if the block device has been completely removed before proceeding
further to disable the next block device.
2.1. $> fldc --disable --disk=/dev/sdc1
2.2. $> fldc --disable --disk=/dev/sdd1
2.3. $> fldc --disable --disk=/dev/sdf1
2.4. $> fldc --disable --disk=/dev/sdg1
2.5. $> fldc --disable --disk=/dev/sdi1
2.6. $> fldc --disable --disk=/dev/sdj1
2.7. $> fldc --disable --disk=/dev/sdl1
2.8. $> fldc --disable --disk=/dev/sdm1
3. Remove the Dell Express Flash PCIe SSDs from the Fluid Cache pool
3.1. $> fldc --remove --ssd=/dev/rssda1
3.2. $> fldc --remove --ssd=/dev/rssdb1
4. Edit ownership and permission in the /etc/udev/rules.d/20-Dell_Oracle.rules
4.1. Remove the following ownership line for fldc
KERNEL==”fldc*”, OWNER:=”grid”, GROUP:=”asmadmin”, MODE:=”0660”
4.2. Uncomment the block devices belonging to the DATA_DG commented out in step 2 of Appendix
C.2.2. Udev settings for Fluid Cache based solution
5. Restart udev
5.1. $> start_udev
6. Follow Fluid Cache for DAS user’s guide
8
to remove the Fluid Cache software
7. Start the Oracle High Availability Service (OHAS) stack
25
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
7.1. $> su – grid
7.2. grid@$> sqlplus / as sysasm
7.3. SQL> crsctl start has
7.4. SQL> exit
7.5. Run the following command as grid user to check if the entire OHAS stack is up and running
7.5.1. grid@$> crsctl stat –res -t
Appendix C.2. Setting up ownership and permission for Oracle disks
This section describes the udev rules that need to be set for both the baseline and the Fluid
Cache based storage solution.
Appendix C.2.1. Udev settings for baseline configuration
1. It is recommended to set the udev rules for the block devices to be used for ASM disk using
their WWIDs. Identify the WWIDs for all the VDs that are to be configured for Oracle ASM by
running the following command
$> scsi_id --page=0x83 --whitelisted --device=/dev/sdX
where sdX is the name of the block device to be used for Oracle
E.g. output:
sdc 360026b900061855e000008a54ea5356a
2. Edit the /etc/udev/rules.d/20-Dell_Oracle.rules file (create a new file if it doesn’t
already exist) and add the following lines to set the permissions needed for the baseline
configuration. Replace each of the <WWID_for_sdX> below with the correct equivalent WWID
result obtained from step 1 for that sdX block device. The example below adds all the twelve
block devices to be configured for the baseline configuration.
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdc>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdd>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sde>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdf>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdg>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
26
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdh>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdi>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdj>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdk>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdl>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdm>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdn>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
3. Restart the udev rules to apply the rules.
3.1. $> start_udev
Appendix C.2.2. Udev settings for Fluid Cache based solution
After enabling caching on the block devices that belong to the DATA_DG, Fluid Cache creates an
equivalent fldc cache block device as noted in Appendix C.1.1 Enabling Fluid Cache. In order for ASM
to be able to mount the DATA_DG again, we need to disable Oracle permission on those VDs and give
permission to the equivalent fldc block devices. In order to achieve this edit the
/etc/udev/rules.d/20-Dell_Oracle.rules file configured for the baseline configuration as
described in Appendix C.2.1. Udev settings for baseline configuration and do the following steps:
1. Add the following line to set the permission for the Fluid Cache block devices
KERNEL==”fldc*”, OWNER:=”grid”, GROUP:=”asmadmin”, MODE:=”0660”
2. Comment out the lines containing the block devices enabled for caching while leaving the block
devices belonging to the FRA_DG as is.
# KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdc>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
# KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdd>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
27
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
ASM
Diskgroup
Name
Diskgroup Member Disks
(ASM disks)
ASM
Redundancy
Size
Allocation Unit
(AU) Size
Grid and
Database
Compatibility
DATA_DG
/dev/sdc1, /dev/sdd1,
/dev/sdf1, /dev/sdg1,
/dev/sdi1, /dev/sdj1,
/dev/sdl1, /dev/sdm1
external
4356 GB
1MB (default)
11.2.0
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sde>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
# KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdf>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
# KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdg>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdh>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
# KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdi>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
# KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdj>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdk>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
# KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdl>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
# KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdm>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
KERNEL=="sd*", PROGRAM="scsi_id --page=0x83 --whitelisted -device=/dev/%k", RESULT=="<WWID_for_sdn>", OWNER:="grid",
GROUP:="asmadmin", MODE:=”0660”
Appendix C.3. ASM diskgroup configuration
The ASM parameter asm_diskstring is set to “/dev/*d*”. This ensures that ASM instance is able to
scan both /dev/sd* block devices and the Fluid Cache /dev/fldc* devices.
ASM disk group configuration for the baseline Table 7.
28
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
ASM
Diskgroup
Name
Diskgroup Member Disks
(ASM disks)
ASM
Redundancy
Size
Allocation Unit
(AU) Size
Grid and
Database
Compatibility
FRA_DG
/dev/sde1, /dev/sdh1,
/dev/sdk1, /dev/sdn1
external
2178 GB
1MB (default)
11.2.0
ASM
Diskgroup
Diskgroup Member Disks
(ASM disks)
ASM
Redundancy
Size
Allocation Unit
(AU) Size
DATA_DG
/dev/fldc0,
/dev/fldc1,
/dev/fldc2,
/dev/fldc3,
/dev/fldc4,
/dev/fldc5,
/dev/fldc6,
/dev/fldc7,
external
4356 GB
1MB (default)
FRA_DG
/dev/sde,/dev/sdh,
/dev/sdk,/dev/sdn
external
2178 GB
1MB (default)
Parameter
Value
fs.file-max
6815744
net.ipv4.ip_local_port_range
9000 65500
net.core.rmem_default
262144
net.core.rmem_max
4194304
net.core.wmem_default
262144
net.core.wmem_max
1048576
ASM disk group configuration for the Fluid Cache based solution Table 8.
Appendix C.4. Parameter settings
The sections below provide the settings and configurations at the operating system level and at the
Oracle Database level.
Appendix C.4.1. Kernel parameter settings
The following kernel parameters were set in the /etc/sysctl.conf file according to Dell and
Oracle’s recommendation and best practices for all the solution testing.
Kernel parameters settings for Oracle database Table 9.
29
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Parameter
Value
kernel.shmall
1073741824
kernel.shmmax
135400749056
kernel.shmmni
4096
kernel.sem
250 32000 100 142
Parameter
Value
db_writer_processes
8
processes
11000
sessions
16544
open_cursors
11000
transactions
11000
undo_retention
1800
dml_locks
44000
parallel_max_servers
1280
audit_trail
none
Appendix C.4.2. User security limits settings
The following user security limits were set and appended to the /etc/security/limits.conf file
according to Dell and Oracle’s recommendations and best practices for all the solution testing.
grid hard nofile 131072
grid soft nproc 131072
oracle soft nofile 131072
oracle hard nofile 131072
oracle soft nproc 131072
oracle hard nproc 131072
oracle soft core unlimited
oracle hard core unlimited
oracle soft memlock 50000000
oracle hard memlock 50000000
Appendix C.4.3. Oracle database parameter settings
The table below provides the database parameters used for testing all the solutions described in this
paper.
Database parameter settings Table 10.
30
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
Appendix D. Next Steps
Appendix D.1. Fluid Cache for DAS Reference Architecture Solution ID
The Dell Fluid Cache for DAS solution reference architecture evaluated in this paper can be ordered
using the Dell Star solution ID# 2995244
This Solution ID includes the following components listed in the Appendix:
Table 1: Server configuration details for Fluid Cache based solution
Table 2: Backend storage configuration
Dell Fluid Cache for DAS software
System profile in the BIOS set to ‘Performance’
Note: The solution ID does not include the software configuration listed in the Appendix C. Software
configuration section. All the software configurations detailed in this paper must be performed
manually.
Appendix D.2. Database management software
Quest Database Management Software Quest’s award-winning database management software
strengthens Dell’s end-to-end solution capabilities. Quest’s database management solutions for Oracle
databases are designed to meet the challenges faced by IT professionals of all skill levels, such as
working with multiple database servers, complex data environments, and reporting needs.
Dell’s software tools allow greater efficiency by:
combining automation of administrative and management tasks
optimizing performance
boosting productivity for database and data professionals
Dell Quest’s solutions of different database management tools allow you to achieve the highest levels
of database quality, performance, maintainability, and availability.
Toad® for Oracle ensures great productivity in development and administration of Oracle databases.
Toad combines the deepest functionality available with extensive automation and intuitive workflows.
With Toad, database professionals of all skill and experience levels can collaborate and work efficiently
and accurately.
Foglight® for Oracle is a scalable, web-based, 24x7 database monitoring tool. It provides constant
remote database monitoring and correlates performance data from across your technology stack. It
makes your job easier by extending the capabilities of native tools. It guides you through response-time
and wait-event analyses, so you instantly understand existing and developing database performance
conditions. Plus, Foglight’s IntelliProfile™ technology ensures your post-change database environment
31
Improving Oracle OLTP database performance with Dell Fluid Cache for DAS
is thriving to help eliminate deployment risks. Foglight for Oracle empowers you to take action
immediately and maximize your Oracle database resources.
SharePlex® for Oracle is a simple, affordable, impact-free Oracle database replication
solution. SharePlex is a mature, high-performance, high availability technology that offers a low-cost
alternative to other Oracle replication tools. Unlike other solutions, SharePlex provides data compare
and repair, in-flight data integrity, plus monitoring and alerting functionalities – all in one package. It
ensures business continuity while meeting your database operational goals, providing a real-time copy
of production data – without impacting your OLTP system’s performance and availability.
NetVault LiteSpeed® for Oracle reduces Oracle backup and recovery times by 50 percent or more and
decreases backup size by up to 90 percent. Tailored to the unique needs of an Oracle DBA, NetVault
LiteSpeed for Oracle significantly lowers storage costs and delivers a solid return on investment within
minutes of installation. Plus, it secures your critical data with Oracle backup data-compression
technology and up to four levels of encryption. It integrates with RMAN via Oracle’s Media Manager
architecture and seamlessly integrates with Tivoli Storage Manager (TSM) and Symantec NetBackup.
32
Loading...