Dell Fluid Cache For DAS
User's Guide
Notes, Cautions, and Warnings
NOTE: A NOTE indicates important information that helps you make better use of your computer.
CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the
problem.
WARNING: A WARNING indicates a potential for property damage, personal injury, or death.
© 2013 Dell Inc.
Trademarks used in this text: Dell™, the Dell logo, Dell Boomi™, Dell Precision™ , OptiPlex™, Latitude™, PowerEdge™, PowerVault™,
PowerConnect™, OpenManage™, EqualLogic™, Compellent™, KACE™, FlexAddress™, Force10™ and Vostro™ are trademarks of Dell
Inc. Intel®, Pentium®, Xeon®, Core® and Celeron® are registered trademarks of Intel Corporation in the U.S. and other countries. AMD
is a registered trademark and AMD Opteron™, AMD Phenom™ and AMD Sempron™ are trademarks of Advanced Micro Devices, Inc.
Microsoft®, Windows®, Windows Server®, Internet Explorer®, MS-DOS®, Windows Vista® and Active Directory® are either trademarks
or registered trademarks of Microsoft Corporation in the United States and/or other countries. Red Hat® and Red Hat
Enterprise Linux® are registered trademarks of Red Hat, Inc. in the United States and/or other countries. Novell® and SUSE® are
registered trademarks of Novell Inc. in the United States and other countries. Oracle® is a registered trademark of Oracle Corporation
and/or its affiliates. Citrix®, Xen®, XenServer® and XenMotion® are either registered trademarks or trademarks of Citrix Systems, Inc. in
the United States and/or other countries. VMware
trademarks of VMware, Inc. in the United States or other countries.
Corporation.
2013 - 03
®
,
Virtual SMP
®
®
,
vMotion
®
is a registered trademark of International Business Machines
IBM
,
vCenter
®
and
vSphere
®
are registered trademarks or
®
®
Rev. A00
Contents
Notes, Cautions, and Warnings...................................................................................................2
1 Dell Fluid Cache For DAS Overview.........................................................................................7
Best Practices For Using Fluid Cache For DAS........................................................................................................7
Related Documentation............................................................................................................................................7
Contacting Dell.........................................................................................................................................................8
Documentation Feedback.........................................................................................................................................8
2 Getting Started With Dell Fluid Cache For DAS.....................................................................9
Hardware Prerequisites For Installing Fluid Cache For DAS....................................................................................9
Software And Firmware Prerequisites For Installing Fluid Cache............................................................................9
Licensing Information.............................................................................................................................................10
Downloading Fluid Cache For DAS.........................................................................................................................10
Installing Fluid Cache For DAS ..............................................................................................................................11
Uninstalling Fluid Cache For DAS...........................................................................................................................12
3 Configuring And Managing Fluid Cache For DAS................................................................13
Configuring Fluid Cache For DAS Using fldc CLI....................................................................................................13
Managing Fluid Cache For DAS With fldc CLI........................................................................................................15
4 fldc CLI Commands...................................................................................................................17
Some Basic fldc Commands...................................................................................................................................17
Displaying Supported PCIe SSDs...........................................................................................................................17
Displaying DAS In The System...............................................................................................................................17
Displaying Status Of Fluid Cache For DAS Service................................................................................................17
Displaying Status Of PCIe SSDs.............................................................................................................................17
Displaying Status Of Fluid Cache Disk....................................................................................................................18
Displaying Status Of Specific Back End Virtual Disk..............................................................................................18
Displaying Status Of Specific PCIe SSD.................................................................................................................18
Displaying Status Of fldc Service...........................................................................................................................18
Displaying Status Of Fluid Cache For DAS License................................................................................................18
Adding Or Applying Fluid Cache For DAS License.................................................................................................19
Exporting Fluid Cache For DAS Configuration To A .TXT File.................................................................................19
Importing Fluid Cache For DAS Configuration From A .TXT File.............................................................................19
Displaying N Events For Fluid CacheDisplaying All Events For Fluid Cache Based On Time.................................20
Adding A PCIe SSD To The Cache Pool..................................................................................................................20
Removing A Working PCIe SSD From The Cache Pool..........................................................................................20
Enabling Caching On A Back End PERC Virtual Disk..............................................................................................21
Disabling Caching On A Back End PERC Virtual Disk.............................................................................................21
Flushing Dirty Data From A Fluid Cache Disk..........................................................................................................22
Reactivating A PCIe SSD In A Failed State.............................................................................................................22
Reactivating A Failed Back End PERC Virtual Disk.................................................................................................22
Displaying Statistics For Configured Cache And Disks..........................................................................................22
fldcstat Command Options......................................................................................................................................23
fldcstat Command Summary Report.......................................................................................................................23
fldcstat Command Verbose Report.........................................................................................................................24
5 Troubleshooting Fluid Cache For DAS...................................................................................25
Basic Troubleshooting Steps..................................................................................................................................25
Fluid Cache Installation Fails..................................................................................................................................25
Fluid Cache For DAS Fails To Start.........................................................................................................................26
Fluid Cache For DAS Port Conflict Daemon Error...................................................................................................26
Fluid Cache Service For DAS Has Stopped............................................................................................................26
PCIe SSD Cannot Be Added As A Cache Device...................................................................................................26
PCIe SSD Has Failed...............................................................................................................................................27
PCIe SSD Name Changes.......................................................................................................................................28
Fluid Cache Disk Does Not Appear.........................................................................................................................28
Fluid Cache Disk Status Has Failed........................................................................................................................29
Cache Status Is Displayed As Degraded................................................................................................................29
Enabling Caching On Disk Has Failed.....................................................................................................................30
Cannot Add Virtual Disk..........................................................................................................................................30
Disabling Cache Fails..............................................................................................................................................31
Fluid Cache I/O Errors.............................................................................................................................................31
Operating System Fails...........................................................................................................................................31
Fluid Cache For DAS Error Messages And Resolution Steps.................................................................................32
Message ID: 2703| JOURNAL_INSUFFICIENT_MIRRORS_EVENT...................................................................32
Message ID: 2704| JOURNAL_CLUSTER_ID_MISMATCH...............................................................................32
Message ID: 2705| JOURNAL_ACCESS_ERROR..............................................................................................32
Message ID: 2900| CACHEDEV_FAILURE_EVENT............................................................................................32
Message ID: 2901| STORAGE_DEAD_EVENT...................................................................................................32
Message ID: 2909| STORAGE_UNKNOWN_STATE_EVENT............................................................................32
Message ID: 2911| BLOCK_DEVICE_FAILURE..................................................................................................32
Message ID: 2915| LICENSE_INVALID_EVENT................................................................................................32
Message ID: 2916| LICENSE_EVAL_VALID_EVENT.........................................................................................33
Message ID: 2917| LICENSE_EVAL_EXPIRED_90_EVENT...............................................................................33
Message ID: 2918| LICENSE_EVAL_EXPIRED_180_EVENT..............................................................................33
Message ID: 2919| LICENSE_EXPIRED_NO+CONFIG+CHANGE......................................................................33
Message ID: 2922| INSUFFICIENT_MEMORY..................................................................................................33
Message ID: 2923| CACHEDEV_VIEW_INCOMPLETE......................................................................................33
Message ID: 2930| STORAGE_MODE_DEGRADED_TO_WRITE_THROUGH....................................................33
Message ID: 2931| STORAGE_MODE_DEGRADED_TO_PASS_THROUGH......................................................33
6 Disaster Recovery Scenarios For Fluid Cache For DAS.....................................................35
Data Collection Utility For Fluid Cache For DAS.....................................................................................................35
Restoring Fluid Cache For DAS Configuration........................................................................................................35
7 Frequently Asked Questions....................................................................................................37
6
1
Dell Fluid Cache For DAS Overview
Dell Fluid Cache software for direct-attached storage (DAS) is a host caching software that allows you to create a
virtual cache pool for supported Dell PowerEdge systems.
Fluid Cache for DAS uses Dell PowerEdge Express Flash PCIe SSDs installed on supported Dell systems to provide a
read and write cache pool. You can install up to four PCIe SSDs in a Dell system. The supported PCIe SSD capacities are
175 GB and 350 GB. These PCIe SSDs can be combined to create a cache pool capacity ranging from a minimum of 175
GB to a maximum of 1400 GB. The cache pool is used to accelerate response times with significant improvements in I/O
operations per second (IOPS).
Some of the features of the Fluid Cache for DAS software are:
• Faster cache reads, writes, read-after-writes, and re-reads.
• Data protection as writes are replicated across multiple PCIe SSDs.
• Orderly hot swap and hot plug capability that allows adding or removing a device without halting or rebooting
the system.
• Cache pool capacity of up to 1400 GB.
NOTE: All PCIe SSDs used for creating the cache pool must be of the same capacity.
• Support for 2.5 inch PCIe SSD form factor.
Best Practices For Using Fluid Cache For DAS
Some best practices that you can follow when using Fluid Cache for DAS include:
• For best performance, accelerate random writes by enabling write-back caching. Write-back mode is enabled
with a minimum of two PCIe SSDs added to the cache pool.
• It is recommended that you do not implement a file system on the Fluid Cache devices.
• To obtain the maximum benefit from caching, the working data set must not be larger than the cache.
Related Documentation
NOTE: For all storage controllers and PCIe SSD documents, go to dell.com/support/manuals, and select Choose
from a list of all Dell products → Continue → Servers, Storage & Networking → Dell Adapters.
NOTE: For all Dell OpenManage documents, go to dell.com/openmanagemanuals.
NOTE: For all operating system documents, go to dell.com/support/manuals, and select Choose from a list of all Dell
products → Continue → Software, Electronics & Peripherals → Software → Operating System.
NOTE: For all PowerEdge and PowerVault documentation, go to dell.com/support/manuals and enter the system
Service Tag to get your system documentation.
•
Dell PowerEdge RAID Controller (PERC) H310, H710, H710P, and H810 User's Guide
installation, management, and troubleshooting of PERC cards.
•
Dell PowerEdge Express Flash PCIe SSD User’s Guide
troubleshooting of Express Flash SSDs.
— Discusses features, installation, management, and
— Discusses features,
7
• Product-specific Dell PowerVault Owner's Manual — Provides information about system features and
describes how to troubleshoot the system and install or replace system components.
• Product-specific Dell PowerEdge Owner's Manual — Provides information about system features and describes
how to troubleshoot the system and install or replace system components.
Contacting Dell
NOTE: Dell provides several online and telephone-based support and service options. If you do not have an active
Internet connection, you can find contact information on your purchase invoice, packing slip, bill, or Dell product
catalog. Availability varies by country and product, and some services may not be available in your area.
To contact Dell for sales, technical support, or customer-service issues:
1. Go to dell.com/contactdell.
2. Select your country or region from the interactive world map.
When you select a region, the countries for that region are displayed.
3. Select the appropriate language under the country of your choice.
4. Select your business segment.
The main support page for the selected business segment is displayed.
5. Select the appropriate option depending on your requirement.
NOTE: If you have purchased a Dell system, you may be asked for the Service Tag.
Documentation Feedback
If you have feedback for this document, write to documentation_feedback@dell.com. Alternatively, you can click on the
Feedback link in any of the Dell documentation pages, fill up the form, and click
Submit to send your feedback.
8
2
Getting Started With Dell Fluid Cache For DAS
Fluid Cache for DAS monitors your cache usage, protects your writes, programmatically moves data blocks in and out of
the cache as needed, and does it all transparently to the application.
The following steps provide a broad summary to configure and run Fluid Cache for DAS:
1. Add up to four PCIe SSDs to the Dell PowerEdge system.
NOTE: Ensure that all the four PCIe SSDs are of the same capacity.
2. Configure the PCIe SSDs to create a cache pool.
3. Enable caching of back end PERC virtual disks or partitions.
4. Point your applications to the Fluid Cache block device(s) instead of the back end storage.
5. When workload requirements change, dynamically disable or enable caching on disks.
Hardware Prerequisites For Installing Fluid Cache For DAS
• Fluid Cache is supported only on Dell PowerEdge systems that support Dell PowerEdge Express Flash PCIe
SSDs. These PowerEdge systems include PowerEdge R620, R720, R820, and T620.
• Supported PowerEdge RAID Controller (PERC) cards include PERC H810, H710P, and H710.
• A minimum of two PCIe SSDs are required for write-back support.
• For installing Fluid Cache for DAS, you need 100 MB of disk space.
• The supported PCIe SSD capacities are 175 GB and 350 GB.
NOTE: You cannot mix PCIe SSDs of different capacities to run Fluid Cache for DAS. You can add up to four
PCIe SSDs of the same capacity to form the cache pool.
• The minimum memory required to run Fluid Cache for DAS is 16 GB. For optimum operation, it is recommended
that you install 32 GB of RAM.
NOTE: PCIe SSDs in Write Protect mode cannot be used for the cache pool and are marked as invalid.
Software And Firmware Prerequisites For Installing Fluid Cache
The supported operating systems for Fluid Cache for DAS include:
• Red Hat Enterprise Linux 6.2, 2.6.32-220.el6.x86_64.
• Red Hat Enterprise Linux 6.3, 2.6.32-279.el6.x86_64.
• SUSE Linux Enterprise Server 11 SP2, 3.0.13-0.27 (default).
Other software and firmware prerequisites include:
• Dell OpenManage Storage Administrator version 7.1.2 only.
• Latest updated PCIe SSD driver downloaded and installed from dell.com/support.
NOTE: For information on managing PCIe SSDs using Dell OpenManage Server Administrator, see the
PowerEdge Express Flash PCIe SSD User’s Guide
• Latest firmware for PERC H810, H710P, and H710 cards. Supported firmware is A03 21.1.0-0007 or higher.
at dell.com/support/manuals.
Dell
9
Ensure the following opensource component packages are available in your system before you install Fluid Cache. See
the operating system distribution media for the supported version of each package.
Table 1. Operating System Packages
Operating System Package Name Package Version
Red Hat Enterprise Linux 6.2 and 6.3 perl-XML-LibXML 1.70 – 5.el6
perl-XML-NamespaceSupport 1.10 – 3.el6
perl-XML-SAX 0.96 – 7.el6
rrdtool 1.3.8 – 6.el6
sg3_utils 1.28 – 4.el6
libxslt 1.1.26 – 2.el6
libuuid 2.17.2 – 12.el6
SUSE Linux Enterprise Server 11 SP2 perl-XML-LibXML 1.66 – 1.14
perl-XML-NamespaceSupport 1.09 – 1.22
perl-XML-SAX 0.96 – 2.7
rrdtool 1.3.4 – 2.8
sg3_utils 1.31 – 1.11.14
libxslt 1.1.24 – 19.15
libuuid1 2.19.1 – 6.29.3
Licensing Information
Dell Fluid Cache for DAS is only functional when a valid license is installed. A license for Fluid Cache for DAS is only
available through Dell Sales. For customers located in the United States of America, call 1-800-WWW-DELL. For other
countries, login to
The following licenses are available:
• A 90-day evaluation license which offers full functionality for 90 days. You cannot extend the evaluation period
or make changes to Fluid Cache after the 90 days have expired.
• A perpetual license that must be installed on each node.
You can view your Fluid Cache license in OpenManage Server Administrator.
NOTE: After the order is processed, you are provided with the license for Fluid Cache for DAS in an e-mail. To
install a license, at the prompt, run the command: fldc --apply --license (include the path to the
license_file). If a previous license exists, the new license overwrites it.
dell.com and change your country from the bottom of the web page to get the sales number.
Downloading Fluid Cache For DAS
To download Dell Fluid Cache for DAS software:
1. Go to dell.com/support.
2. You may be asked to select your line of business. Select Support for Enterprise IT.
3. Enter your Dell PowerEdge Service Tag and click Submit.
4. Navigate to the Drivers & Downloads section.
10
5. Select your operating system.
6. Click Caching Solutions to expand the menu.
Fluid Cache for DAS software is listed. Click the link to download the software.
Installing Fluid Cache For DAS
You can install Fluid Cache for DAS using the Fluid Cache setup.sh shell script. When you run the installation script, it
uses the options you provide. If the installation script encounters missing dependencies or discovers other issues with
your installation, it stops and an error message is displayed. Follow the prompts and system messages to correct any
issues. Rerun the setup.sh shell script to restart the installation. After the installation completes, configure your
Fluid Cache for DAS environment.
1. Go to dell.com/support.
2. Download and extract the Fluid Cache tar.gz package into a directory of your choice.
See Downloading Fluid Cache For DAS.
3. Change (cd) to the root of that directory:
a) Locate the
example:
To obtain the license, see Licensing Information.
b) From the prompt, run: ./setup.sh
The End User License Agreement (EULA) is displayed.
4. Read the EULA.
At the bottom of the EULA page, the following message is displayed: Do you agree to the above
license terms? ('y' for yes | press 'Enter' to exit):
5. To accept the EULA, type y and press <Enter>.
At the prompt, the following options are displayed:
Type i to install Fluid Cache
Type q to quit
6. Type i to install the product.
7. At the prompt, type a Fluid Cache Identifier of your choice and press <Enter> or accept the default FQDN and press
<Enter>.
The install wizard prompts you to confirm that you have entered the correct information with the following
message:
Do you want to continue with these settings? ('y' for yes | 'r' to
reconfigure | 'q' to quit install):
<name>_license.xml
/tmp/fluidcache_node_license.xml
file and write down the fully qualified path to the license file, for
NOTE: By default, the Fluid Cache Identifier is the hostname of the system on which you are installing Fluid
Cache. The installation script allows you to enter a customized name for the Fluid Cache installation that helps
you identify the cache subsystem. After entry, the name is repeated in various future system messages.
8. To accept the settings for Fluid Cache for DAS, type y and press <Enter>.
9. At the prompt, enter a fully qualified path to a valid license file and press <Enter>.
The license file is the XML file you received as an attachment in your Dell order e-mail and copied to a temporary
directory on the target server.
NOTE: When you enter your license file name, be sure to include the path. For example: /tmp/
fluidcache_node_license.xml
The installation script provides a summary of all the installation variables you have chosen up to this point. You
must confirm that the information is correct before continuing. Then, at the prompt, the script asks whether you
11
want to start the Fluid Cache service with the message: Would you like to start Fluid Cache
now? ('y' for yes | 'n' for no and exit install):
10. To start the Fluid Cache service, type y and press <Enter>.
11. To ensure that the product has started successfully, run the command: /opt/dell/fluidcache/bin/fldc
--status
This command returns a set of information similar to the following:
Fluid Cache Service
Cache size: 0 GiB
Status: ok
SSDs in Cache: 0
Disk being cached: 0
Write Back: unavailable
Cache ID: <any node>.example.com
License: node – limited to 4 nodes
License Status: valid
SSDs in cache No SSDs are configured in the cache
pool
Disks being cached No Disks are being cached
Fluid Cache for DAS is successfully installed. You must now configure Fluid Cache for DAS. See Configuring And
Managing Fluid Cache For DAS. If the installation was not successful, the following error message is displayed:
bash: /opt/dell/fluidcache/bin/fldc: No such file or directory. To resolve the issue,
see the topic “Fluid Cache Installation Fails” in the section
Troubleshooting Fluid Cache For DAS.
Uninstalling Fluid Cache For DAS
Use Dell OpenManage Server Administrator or the fldc tool to disable caching on the back end disks.
1. To disable using fldc, at the prompt, run the command: fldc --disable --disk=<disk_name>
NOTE: Run this command for each configured disk.
2. Remove all cache devices from the cache pool. To remove a PCIe SSD, run the command: fldc --remove --
ssd=<ssd_name>
NOTE: Run this command for each configured disk.
3. Run the command: rpm -qa | grep DAS to get the Fluid Cache for DAS package name.
4. Use the rpm -e command to remove the Fluid Cache for DAS package from the server. For example, run the
command: rpm –e <extract_package_name>
5. Optionally, you can remove the /opt/dell/fluidcache directory to remove all components of Fluid Cache for DAS from
your installation. At the prompt, run the command: rm -rf /opt/dell/fluidcache
NOTE: If you attempt to reinstall Fluid Cache, you may see a warning message that advises that you have had
a previous configuration. You cannot clear that warning and install FluidCache unless you remove the
FluidCache directory. Remove the FluidCache directory after you are sure that all data previously managed by
Fluid Cache is secure.
Fluid Cache for DAS is now uninstalled.
12
3
Configuring And Managing Fluid Cache For DAS
After Fluid Cache for DAS is installed and running, configure and manage your Fluid Cache environment. Fluid Cache for
DAS is configured at the command line using fldc commands or in Dell OpenManage Server Administrator.
NOTE: This document contains instructions for configuring and managing Fluid Cache for DAS using the fldc CLI
only. For information on configuring and managing Fluid Cache for DAS using OpenManage Server Administrator,
see the
Dell OpenManage User's Guide
Configuring Fluid Cache For DAS Using fldc CLI
The fldc command line tool is located in the following directory: /opt/dell/fluidcache/bin
To review the available fldc commands, see fldc CLI Commands.
1. Add the first PCIe SSD cache device to the cache pool. Run the command: fldc --add --ssd=/dev/rssda
The following message is displayed: This operation will destroy all existing data
on /dev/rssda. Are you sure you want to proceed? [y=yes, n=no]
2. To confirm that you are adding the PCIe SSD cache device to the cache pool, type: y
The following message is displayed: SUCCESS: /dev/rssda added to fluid cache pool
3. Add the second PCIe SSD cache device to the cache pool. Run the command: fldc --add --ssd=/dev/
rssdb
The following message is displayed: This operation will destroy all existing data
on /dev/rssdb Are you sure you want to proceed? [y=yes, n=no].
4. To confirm that you are adding the second PCIe SSD cache device to the cache pool, type: y
The following message is displayed: SUCCESS: /dev/rssdb added to fluid cache pool.
5. Repeat step 3 and step 4 to add additional PCIe SSDs to a maximum of four SSDs.
6. Enable caching on one or multiple back end virtual disks in either of the two modes: write-back (WB) or write-
through (WT). Repeat either step a or step b below based on the type of mode you select for your back end virtual
disk(s).
at dell.com/support/manuals.
Write-back mode In addition to reading cached data, WB mode allows the caching of writes. WB mode
requires the data to be written to a minimum of two PCIe SSDs. This mode also allows
applications to manipulate (read and write) large data sets in cache. Both reads and
writes are accelerated.
Write-through
mode
a) For WB mode, run the command: fldc --enable -disk=/dev/sdb --mode=wb
The system displays the following message: SUCCESS: Caching enabled on Disk:/dev/sdb
Cache Mode:write-back Created Virtual Cache Disk:/dev/fldc0.
Use WT mode to force writes to both the cache and virtual back end disk simultaneously.
In this mode, reads are accelerated and writes are not accelerated. Under certain
conditions, system events can change the caching mode automatically to WT. For
example, if a replica block cannot be allocated, Fluid Cache changes modes from WB to
WT to ensure data protection.
13
b) For WT mode, run the command: fldc --enable --disk=/dev/sdb --mode=wt
The system displays the following message: SUCCESS: Caching enabled on Disk:/dev/sdb
Cache Mode:write-through Created Virtual Cache Disk:/dev/fldc0
NOTE: During installation, Fluid Cache for DAS adds a line to /etc/fstab. The /etc/fstab file is a
Linux file used to auto-mount file systems at system startup. It is important that you “comment out” (#) lines in
/etc/fstab file that identify current disks that you plan to cache using Fluid Cache for DAS. For an
the
example of the commenting out process, see the topic “Fluid Cache Disk for DAS Does Not Appear” in
Troubleshooting Fluid Cache For DAS.
7. Display the status of Fluid Cache to confirm it is configured and operational by running the command: fldc --
status
A set of information is displayed similar to the following:
Fluid
Cache
Service:
Cache
size:
Status: ok
SSDs in
Cache:
Disks
being
cached:
Write
Back:
Cache
ID:
License: node - limited to 4 nodes
License
Status:
SSDs in cache:
SSD UID Size-GiB Status
/dev/
rssda
/dev/
rssdb
Disks being cached:
Disk
Status:
/dev/sdb
/dev/
fldc0
652 GiB
2
1
available
<any_node>.example.com
valid
500A075102026041 326 ok
500A075102026374 326 ok
Fluid Cache
Disk UID
6D4AE520A7D6FE0
01796D94409CFB9
35
Size-GiB Mode Status
465 wb ok
.
14
CAUTION: A new Fluid Cache disk /dev/fldcx is created as a result of enabling Fluid Cache. You must use the new
Fluid Cache disk /dev/fldcx for all data access. Any additional data access to the corresponding virtual
disk /dev/sdx or its partitions may lead to data corruption. Fluid Cache for DAS has exclusive control of the back
end device. You cannot make changes to devices controlled by Fluid Cache without first disabling caching.
NOTE: Fluid cache is designed to cache I/O to any disk presented by a PERC controller. These back end virtual
disks are referred to as the PERC virtual disks.
You have now completed the Fluid Cache configuration steps.
Managing Fluid Cache For DAS With fldc CLI
Fluid Cache for DAS provides the fldc CLI to allow you to configure, manage, and script the cache.
Using fldc, you can perform the following actions from the command-line interface:
• List supported devices and disks.
• Add, remove, enable, disable, and reactivate PCIe SSDs and Fluid Cache disks.
• Display the status of PCIe SSD devices or back end virtual disks (persistent storage) in cache.
• Display events by quantity or time of sequence.
• Apply licenses.
• Export or import cache configurations.
Some basic Linux commands to manage Fluid Cache for DAS are provided below. Run these commands at the command
prompt:
• Start the Fluid Cache service: service fluid_cache start
• Stop the Fluid Cache service: # service fluid_cache stop
• Enable auto-starting the Fluid Cache service: # chkconfig fluid_cache on
• Disable auto-starting the Fluid Cache service: # chkconfig fluid_cache off
• Check the Fluid Cache service: # service fluid_cache status
NOTE: For the fldc command syntax, see fldc CLI Commands.
15
16
4
fldc CLI Commands
Use fldc in the following format: fldc [options…] <actions><arguments>. For a complete list of commands,
run the command: # fldc --help
Some Basic fldc Commands
• Displays help text: fldc --help
• Bypasses all warnings and completes the operation: -f/--force
• Shows all information: -v/--verbose
• Shows version number: fldc --version
• Answers all interactive questions automatically with “Yes”: -y/--yes
Displaying Supported PCIe SSDs
Command
Description Displays a list of supported PCIe SSDs in the system. Use this command to review a list of PCIe
fldc --list --ssd
SSDs identified and supported by Fluid Cache. The information provided includes the
availability of the device for use by Fluid Cache (available, unavailable), UID, PCIe SSD size,
and includes a reason for the displayed state.
Displaying DAS In The System
Command
Description Displays a list of direct attached storage (DAS) disks in the system. Use this command to
fldc --list --disk
review the list of disks identified and supported by Fluid Cache. The list includes the following
information about the device: UID, disk size (in GiB), device state, and its availability for
caching, and includes a reason for the state.
Displaying Status Of Fluid Cache For DAS Service
Command
Description Use this command to review the status of the Fluid Cache service. Information displayed
fldc --status
includes the UID, disk size (in GiB), and status of the PCIe SSDs and UIDs, disk size (in GiB),
mode, and status of the cached disks.
Displaying Status Of PCIe SSDs
Command
fldc --status --ssd [-v verbose option]
17
Description Displays the status of all PCIe SSD devices. Use this command to review the status of the PCIe
SSD devices configured in the cache pool. Information displayed includes the PCIe SSD name,
UID, disk size (in GiB), and status.
Displaying Status Of Fluid Cache Disk
Command
Description Displays the current status of the Fluid Cache disk(s) configured for caching. Use this
fldc --status --disk [-v verbose option]
command for all Fluid Cache disks. Information displayed includes the Fluid Cache disk name,
UID, disk size (in GiB), Mode, Share %, Dirty %, and status.
Displaying Status Of Specific Back End Virtual Disk
Command
Description Displays the current status of a particular back end virtual disk in the cache pool. Use this
fldc --status --disk=<disk_name or uid>
command to review the status of a particular Fluid Cache disk. Information displayed includes
the back end virtual disk name, Fluid Cache disk name, UID, disk size (in GiB), Mode, and
status.
Displaying Status Of Specific PCIe SSD
Command
Description Displays the current status of a particular PCIe SSD in the cache pool. Use this command to
fldc --status --ssd=<dev_name or uid>
review the status of a PCIe SSD. Information displayed includes the PCIe SSD name, UID, disk
size (in GiB), and status.
Displaying Status Of fldc Service
Command
Description Use this command to display the overall status of the Fluid Cache service. Information
fldc --status --cache
displayed includes cache size, status, number of PCIe SSDs in cache, number of disks being
cached, mode, disk status (available/unavailable), cache ID, license type, and license status
(valid/invalid).
Displaying Status Of Fluid Cache For DAS License
Command
Description Use this command to review the license status. Information displayed includes the date the
18
fldc --status --license
license was first used, type of license, license status, vendor, product ID, generation, service
tag, and node limit. Also displayed are the license features, including a feature ID, and
description.
Adding Or Applying Fluid Cache For DAS License
Command
Description Use this command to add or apply a new Fluid Cache for DAS license. This command installs a
fldc --apply --license (include the path to the license_file)
specified license file. If a previous license exists, the new license overwrites it.
Exporting Fluid Cache For DAS Configuration To A .TXT File
Command
Description This command exports the current Fluid Cache configuration to a text file. The data format
fldc --export=<file_name>
contained in the file is limited to the Linux device names for the PCIe SSD devices and Fluid
Cache disks currently configured in the cache. Use this command on a server to save its
current caching configuration to be used later to recreate the previous configuration.
NOTE: If you type the fldc --export command without the <file_name argument>, it displays
the file output to the screen.
NOTE: The fldc --export and fldc --import commands are not based on UIDs, but on system
names. On Linux, these system names are subject to “name slippage”. Therefore, before
using the export or import commands, you must ensure that the Linux device names for
cache devices (PCIe SSDs) and virtual disks are equivalent on the two systems which are
involved. For example, if /dev/rssda exists on the exporting host, it must also exist on the
importing host and be available for use by Fluid Cache.
A sample file is given below:
FILE_FORMAT=1.0
TYPE VALUE EXTENSIONS
ssd /dev/sdc
disk /dev/sdd mode=wt
disk /dev/sde mode=wt
disk /dev/sdf mode=wt
Importing Fluid Cache For DAS Configuration From A .TXT File
Command
Description Imports the saved caching configuration generated by the --export command. Use this
fldc --import=<file_name>
command on a server to recreate a working configuration without manually adding the
resources one at a time. This command must be run on an empty cache. If the cache is already
configured, the import command attempts to add the configuration definitions listed in the
import file.
19
NOTE: The fldc --export and fldc --import commands are not based on UIDs but on system
names. On Linux, these system names are subject to “name slippage”. Therefore, before
using the export or import commands, you must ensure that the Linux device names for
cache devices (PCIe SSDs) and virtual disks are equivalent on the two systems which are
involved. For example, if /dev/rssda exists on the exporting host, it must also exist on the
importing host and be available for use by Fluid Cache.
NOTE: The fldc --import command includes a feature that verifies server resources before
the actual import. The import command applies changes only if all of the configuration
definitions in the import file can be applied to the cache. This feature does not allow any
changes that conflict with Fluid Cache for DAS.
Displaying N Events For Fluid Cache
Command
Description Use this command to display a quantity of system events to the screen up to a number (N). The
fldc --events --num=<N=any number>
last 10 events are displayed by default if you only use the command fldc --events without typing
the argument. If you attempt to add a previously used PCIe SSD to the node, the following
message is displayed: Cache device initialization failed because it
belongs to another cache pool
server it came from and discretely remove it or contact support for assistance. You may need
to run the utility fldc_cachedev_wipe to prepare the device for reuse.
. To prepare it for reuse, reinsert the device into the
Displaying All Events For Fluid Cache Based On Time
Command
Description Use this command to review all events recorded in the log file after a particular time.
fldc --events --time=<T=any time>
NOTE: Enter time in the following format: <Day> <Month> <Date> <Hour: Minute: Second>
<Year>. For example: Fri Sep 10 0:0:0 2012 . You must use the Linux system time
format but include the day of the week.
Adding A PCIe SSD To The Cache Pool
Command
fldc --add --ssd=<dev_name or uid>
Description
CAUTION: The command fldc --add --ssd=<dev_name or uid> removes all data from the
PCIe SSD that you are adding to the cache pool.
Use this command to add a PCIe SSD to the cache pool.
Removing A Working PCIe SSD From The Cache Pool
Command
Description Use this command to remove a working PCIe SSD from the cache pool. Use the same
20
fldc --remove --ssd=<ssd_name or uid>[-v verbose option]
command with a UID to remove a failed PCIe SSD.
NOTE: If, when removing a failed PCIe SSD, you are blocked from using the Linux device
name, you can use the UID for the PCIe SSD. To obtain the UID, run the fldc --status --ssd
command.
Use this command to initiate a discrete process that prepares the system for the physical
removal of a PCIe SSD from a localhost. After the command is completed, the following
message is displayed:
SUCCESS:
SSD /dev/rssdb being removed from cache pool
This can be a long process, so make sure the SSD is not listed
in
"--status --ssd" command or check for removed event using "-event" before you physically remove or reuse this SSD
The message from “fldc --event”:
EventID: 2904, MessageID: FLDC2904, severity: Informational,
"The following cache device has been removed."
Cache Dev WWN:(500A075102026041) Cache Dev Path: (/dev/rssdb)
CAUTION: Do not physically remove the PCIe SSD from the localhost until after you
receive the SUCCESS message. Because system operations may continue after you
receive the SUCCESS message, use the fldc --status --ssd command to ensure that the
PCIe SSD is no longer displayed in the list of PCIe SSDs.
Enabling Caching On A Back End PERC Virtual Disk
Command
Description Use this command to enable caching on a back end PERC virtual disk. You must use the --mode
fldc --enable --disk=<disk_name> --mode=<cache_mode> [--mount]
argument and enable a particular caching mode (WB or WT). You can also add the --mount
argument to mount the Fluid Cache disk.
Use the --mount option to ensure that the device you listed in /etc/fstab is actually mounted.
The following conditions must be met for the --mount option to work:
• The backing volume (for example: /dev/sdb) must be unmounted and a commented-out
entry referring to the backing store must exist in /etc/fstab.
• If the backing volume is mounted, the enable caching fails.
• If the backing volume is not mounted, but a commented-out entry in /etc/fstab does not
exist, the enable caching succeeds but the new cache device is not mounted.
The --mount option ensures that the device is in the expected state by verifying if the
information added by the user in /etc/fstab exists.
Disabling Caching On A Back End PERC Virtual Disk
Command
Description Use this command to disable caching on a back end PERC virtual disk and remove all dirty data
fldc --disable --disk=<disk_name>[--discard --mount]
blocks in cache. After the command is completed, a message is displayed indicating a
successful operation. Use the fldc --status --disk command after viewing the confirmation
message to ensure the disk is no longer displayed in the cache. The --umount option ensures
that the device is in the expected state by verifying the information you added in /etc/fstab
exists.
21
CAUTION: If this command is run with the --discard option, the data in cache is destroyed
without being flushed to storage. Use the --discard option with this command only if you
know that destruction of dirty data in cache causes no data integrity issues.
NOTE: Use the --umount argument to ensure that the device you listed in /etc/fstab is
actually unmounted. The --umount option does not need the back end store or cached
device to have an entry in /etc/fstab and, if one exists, it is removed.
Flushing Dirty Data From A Fluid Cache Disk
Command
Description Use this command to flush dirty data on a Fluid Cache disk to storage on a priority basis. After
fldc --flush --disk=<disk_name or uid>
the command is completed, the following message is displayed: Informational, the
following Disk has finished flushing.
CAUTION: All dirty blocks may not be flushed as I/O to the Fluid Cache disk is not stopped
during the operation of the --flush command.
Reactivating A PCIe SSD In A Failed State
Command
Description
fldc --reactivate --ssd=<ssd_name or uid>[-v verbose]
CAUTION: The use of --reactivate command is consequential for data integrity. Before you
try to reactivate the failed PCIe SSD, ensure that you have resolved the issue that caused
the PCIe SSD to fail.
Use this command only if the PCIe SSD is in a failed state.
Reactivating A Failed Back End PERC Virtual Disk
Command
Description Use this command to reactivate a back end PERC virtual disk that has previously failed without
fldc --reactivate --disk=<dev_name or uid>
disabling caching. Use this command only if the disk is in a failed state. Use of this command
allows you to retrieve dirty data that may be lost if you use the disable command.
CAUTION: The use of --reactivate is consequential for data integrity. Before you try to
reactivate the failed back end PERC virtual disk, ensure that you have resolved the issue
that caused the disk to fail.
Displaying Statistics For Configured Cache And Disks
Command
Description Use this command to provide a summary of device statistics from the time the Fluid Cache
22
/opt/dell/fluidcache/bin/fldcstat
service first began. Add the verbose option to the command to create a report that lists all
device statistics for the period. Each subsequent run of the fldcstat command reports statistics
accrued since the last run of the report. Both the summary and verbose reports consist of a
header row followed by a row (or rows) of statistics.
fldcstat Command Options
The following are the options that can be added to the fldcstat command:
fldcstat Command
Syntax With Different
Options
--addr=<host_name>
or localhost
-port=<port_number>
-c|-cache<cahce_na
me>
-r or -w Shows statistics for only read (-r) or only write (-w) operations. For example: To display 5
-s|-samples<nbr_sa
mples>
-t|--time <time> Specifies the amount of time in seconds between each report (default: five). For example: To
--verbose Shows all cache and associated disk statistics.
Description
Shows hostname or IP address of CFM (default: localhost). Example: To display the statistics
for the default host, run: #"fldcstat -a localhost"
TTP listen port of primary CFM (default: 8080).
Shows statistics only for this cache device. For example: To display 5 samples spaced 10
seconds apart for the /dev/fldc0 cache device, run: #"fldcstat -c /dev/fldc0/ -s
5 -t 10"
samples for reads or writes 10 seconds apart, run: #"fldcstat <-r or -w> -s 5 -t
10"
Specifies the number of samples to collect. For example: To display 2 samples each being 5
seconds apart, run: #"fldcstat -s 2"
display continuous samples in 10 second intervals, run: #"fldcstat -t 10". Another
example: To display 5 samples in 10 second intervals, run: #" fldcstat -s 5 -t 10"
--version Shows version number and exits.
--trace Shows debug trace information.
--help Shows help text for the command.
fldcstat Command Summary Report
The fldcstat command summary report provides the following statistical data:
Summary Report
Header
Cache LUN Shows the name of the cache device.
IOPS Shows the number of IOPS for the cache device.
MB/s Shows the read/write in MB/s for the cache device.
%Usage Shows the usage percentage for the cache device.
Description
23
Summary Report
Header
CacheIO-DiskIO Shows the number of bytes read and written to the cache device minus the number of bytes
Avg I/O size Shows the average I/O size for the DAS disk.
Disk Name Shows the name of the DAS disk.
IOPS Shows the number of IOPS for the DAS disk.
MB/s Shows the read/write in MB/s for the DAS disk.
Avg I/O size Shows the average I/O size for the DAS disk.
Description
read and written to the disk. A positive value shows I/O activity is being served mostly by the
cache while a negative value indicates more disk I/O is being performed than cache I/O.
fldcstat Command Verbose Report
The verbose report provides the following additional statistical data:
Verbose Report
Header
ops Shows the actual cache I/O operations per second (not normalized to 1024 bytes per
Thread Shows the percentage of actual cache I/O operations that were read operations.
MB Shows the data in MBs of read/write cache I/O since the last sample.
evictMBs Shows the data in MBs evicted from the cache to allow the cache to be used for caching other
dirtyMB Shows the amount of dirty bytes in the cache.
Ops Shows the actual disk I/O operations per second (not normalized to 1024 byte operations).
%read Shows the percentage of actual disk I/O operations that were read operations.
MB Shows the data in MBs of read/write disk I/O since the last sample.
Description
operation).
I/O.
24
5
Troubleshooting Fluid Cache For DAS
This section contains troubleshooting information for common issues that can occur with Fluid Cache for DAS. Many
symptoms can have more than a single cause. Use the information in this section to eliminate possible problem sources,
determine the root cause of the problem, and correct it. Additional troubleshooting information is available in the
PowerEdge Express Flash PCIe SSD User’s Guide
CAUTION: Many repairs may only be done by a certified service technician. You should only perform
troubleshooting and simple repairs as authorized in your product documentation, or as directed by the online or
telephone service and support team. Damage due to servicing that is not authorized by Dell is not covered by your
warranty. Read and follow the safety instructions that came with the product.
at dell.com/support/manuals.
Basic Troubleshooting Steps
Check for the following conditions for any general issue with the Fluid Cache for DAS software:
• Cables are correctly attached — Ensure that the cables for the power-supply, adapter, enclosure, or backplane
are attached correctly. If the connection is secure, but the system still has issues with read and write
operations, you may need to replace a cable.
• System requirements are met — Ensure that you are using supported devices and packages, including the
correct levels of firmware and drivers.
CAUTION: Driver or firmware updates can sometimes break connections with installed kernel versions. Do
not update the driver or firmware when Fluid Cache for DAS is actively caching data.
• Physical disks or PCIe SSDs are not removed in error — If disks are inadvertently removed, follow the
procedure detailed in the disk-specific documentation to insert it back in the system.
• The license for your software is up to date — System changes can cause a license to appear to be invalid. An
invalid license can degrade or may not support a functionality. For example, if the system date is set later than
the license certification date, you get an invalid license error. To resolve the issue, restart Fluid Cache for DAS.
Dell
Fluid Cache Installation Fails
Probable Cause
Solution
• Too little space on root disk.
• Missing installation prerequisite.
• Installed incorrect package.
• For the space issue, minimum required available disk space for installation is 100 MB.
Free up disk space, then re-run the Fluid Cache for DAS installation.
• For missing installation prerequisites, see the prerequisites in Software and Firmware
Prerequisites For Installing Fluid Cache and identify missing packages to reinstall them.
• To verify if you have installed an incorrect package, check that the operating system
package you are using matches the kernel version. For example: A version 6.2
operating system package cannot be installed on a version 6.1 or version 6.3 kernel.
You also cannot install a SUSE Linux Enterprise Server package over Red Hat
Enterprise Linux or vice versa.
25
Fluid Cache For DAS Fails To Start
Probable Cause Port conflict.
Solution If you have a did not start error for a Mongoose server, check for a port conflict. If
another service or application is trying to use the default fluid_cache service port,
change ports to end the conflict. For more information, see the topic “Fluid Cache For DAS Port
Conflict Daemon Error” in this section.
Fluid Cache For DAS Port Conflict Daemon Error
Probable Cause
Solution
• fldc_cfm is not running and the Mongoose server fails to start.
• DWS login failed. Openwsman is not running.
• If fldc_cfm is not running, query the system to see which ports are in use by running
the command:
displays the sockets all services are listening on.
– If there is a socket listening on port 6772, set fldc-https to an unassigned
– If there is a socket listening on port 8080, set fldc-http to an unassigned
– If there is a socket listening on port 7449, find the service using that socket and
• For Openwsman not running, query the system to see which ports are in use by running
the command: netstat --listening --numeric --tcp. This command
displays the sockets all services are listening on. If there is a socket listening on port
443, find the service using that socket and configure that service to use a different
unassigned port.
netstat --listening --numeric --tcp. This command
port.
port.
configure that service to use a different unassigned port.
Fluid Cache Service For DAS Has Stopped
Probable Cause After starting, the service may shut down and the fluid_cache service is no longer
displayed.
Solution
1. To determine if Fluid Cache has stopped, run the command: fldc --status
2. If status returns a cannot connect message, restart fluid_cache service. Run
the command: service fluid_cache start
PCIe SSD Cannot Be Added As A Cache Device
Probable Cause
26
• The first PCIe SSD cannot be added.
• The second, third, or fourth PCIe SSD cannot be added.
• A PCIe SSD exists, but cannot be added to the cache.
Solution If you have attempted to add the first PCIe SSD by running the fldc --add --ssd command at the
prompt and it did not work, follow the steps below:
1. If the PCIe SSD you are trying to add has no status, check if that specific PCIe SSD is
supported. Only Dell Express Flash PCIe SSDs of 175 GB or 350 GB capacity are
supported as cache media.
2. If the device you are trying to add is supported, determine its status. Run the command:
fldc --list --ssd. If the status is unavailable, the reason listed below helps
determine why the PCIe SSD is not being added, such as whether the PCIe SSD is
partitioned.
If you have an issue while adding the second, third, or fourth PCIe SSD, check for the following
reasons:
• You can only add PCIe SSDs of the same size to Fluid Cache. All PCIe SSDs used in
cache must be either 175 GB or 350 GB. The sizes cannot be mixed in a single
configuration.
• Status information provided by the list command fldc --list --ssd must
disclose PCIe SSD availability for caching and other details you can use to fix the
issue. For example, if the PCIe SSD you are trying to add has a failed status, you cannot
add it to cache.
• The list command may also show that a supported PCIe SSD exists and appears
available or unused for caching. However, after you add it to the cache, the status
changes to failed.
For resolving failed PCIe SSDs, see the
at dell.com/support/manuals and the topic “PCIe SSD Has Failed” in this section.
Dell PowerEdge Express Flash PCIe SSD User's Guide
PCIe SSD Has Failed
Probable Cause
Solution If the PCIe SSD is unplugged, follow the steps below:
• The status of a PCIe SSD in the cache pool is changed to failed because it is
unplugged.
• The status of a PCIe SSD is changed to failed because it is at end-of-life or has an
internal fault.
1. Physically check whether the failed PCIe SSDs is unplugged on your system.
2. If the PCIe SSD is physically present, check if the PCIe SSD is a supported PCIe SSD. If
the PCIe SSD was removed from the system, go to step 5.
3. If the PCIe SSD is supported, get the status of the Fluid Cache service and details on the
state of the cache pool. At the prompt, run any one of the following commands:
– For the status of all PCIe SSDs, run the command: fldc --status
– For the status of a single PCIe SSD, run the command: fldc --status --
ssd=<dev_name>
– For the Fluid Cache status, run the command: fldc --status --cache
The status output shows the PCIe SSD availability. Follow the steps in Configuring Fluid
Cache For DAS Using fldc CLI to reconfigure a supported and available PCIe SSD as a
cache device.
4. If the status continues to show failed, you can examine the events in the log by running
the command: fldc --events
27
5. If the PCIe SSD was removed from the system, plug it back in and reactivate it.
6. Reactivate caching on the back end disk by running the command: fldc --
reactivate --disk=<disk_name>
7. Check that the PCIe SSD is now available. At the prompt, run the command: fldc --
status
If the status of a PCIe SSD is changed to failed, follow the steps below:
1. At the prompt, run either of the following commands: fldc --status --
ssd=<dev_name> or fldc --status --ssd<dev_name>
2. Use Dell OpenManage Server Administrator to determine if the PCIe SSD is at end-of-life.
For more information, see the
dell.com/support/manuals.
3. The PCIe SSD may have an internal fault. Replace the PCIe SSD and follow enable
procedures as outlined in Configuring Fluid Cache For DAS Using fldc CLI.
Dell PowerEdge Express Flash PCIe SSD User's Guide
PCIe SSD Name Changes
Probable Cause Name slippage occurs when one or more PCIe SSDs are inserted, removed, replaced, or
reinstalled. Linux sequentially numbers the PCIe SSDs during a hot swap. However, since Fluid
Cache for DAS uses UIDs to map PCIe SSDs and DAS disks internally, Fluid Cache continues to
identify components successfully regardless of Linux name slippage issue.
Solution This error does not affect the functionality of the program. To display the UID, run the
command: fldc --list --ssd
at
Fluid Cache Disk Does Not Appear
Probable Cause
• The Fluid Cache service did not start when you restarted the system.
• Caching is not enabled.
• The Fluid Cache license has expired.
Solution
• If after a system reboot, the Fluid Cache service did not start (even though fldc0 was
visible prior to rebooting), follow the steps below:
a. To determine the state of the Fluid Cache service, run the command: fldc --
status
b. Attempt to restart the service, run the command: service fluid_cache
start
NOTE: Fluid Cache must restart. You may want to determine whether you can
enable auto-start on boot.
• Fluid Cache for DAS may not restart if caching is not enabled on the disk. The back end
PERC virtual disk may be mounted (using the original device path) instead of the Fluid
Cache disk (using the virtual device path). For example, in the case where the original
Linux reference to the back end PERC virtual disk was /dev/sdd/data1 ext4
defaults 1 2, the Fluid Cache disk reference after --mount must be /dev/
fld0/data1 ext4 defaults 1 2
in the /etc/fstab file and is not commented out, you are mounting the incorrect disk. To
resolve the mounting issue, follow the steps below:
. If the original Linux reference is displayed
28
a. Comment out the back end PERC virtual disk entry so the --mount results shown
in /etc/fstab is displayed as follows:
#/dev/sdd/data1 ext4 defaults 1 2
/dev/fldc0/data1 ext4 defaults 1 2
b. To determine if your edit of the /etc/fstab file has fixed the issue, run the
command: fldc --list --disk. If the system returns available for
fldc0, the issue is resolved.
• An existing Fluid Cache disk cannot be displayed if the Fluid Cache license is missing
or expired. To verify if your Fluid Cache license is valid, run the command: fldc --
status --license. To resolve a licensing issue, contact Dell Sales.
NOTE: If you obtain a new license, you must disable caching before you can apply
the new license.
Fluid Cache Disk Status Has Failed
Probable Cause
Solution
• A supported disk is displayed, but enabling caching fails, because PERC presents a
failed disk.
• The back end PERC virtual disk is already mounted.
• If the back end PERC virtual disk fails, Fluid Cache for DAS can no longer flush dirty
blocks, so I/O's also fail. However, the dirty blocks remain safe in the cache and are
not lost. The following may be the cause(s) of failure:
– The disk enclosure may appear to have no power or appear disconnected.
– The PERC card may have failed.
– Cables may be disconnected.
– The single disk behind the PERC card may have failed, is powered off, or got
disconnected.
– Multiple disks behind the PERC card may have failed.
Fluid Cache reports I/O failures as the result of any of the above issues. To resolve
issues on back end PERC virtual disks behind the PERC card, see the
RAID Controller (PERC) User’s Guide
H710P or H810).
After the problem with the PERC card or a back end PERC virtual disk is resolved,
reactivate caching on the back end PERC virtual disk. Run the command: fldc --
reactivate --disk=<disk_name>. After caching is enabled, Fluid Cache for
DAS flushes the dirty blocks in cache to disk.
• If you cannot reactivate, the disk may already be mounted. Check the disk status by
running the command: fldc --status --disk=<disk_name>. If the status
shows that the disk is already mounted, see the topic “Fluid Cache Disk Does Not
Appear” in this section to resolve any issue with the /etc/fstab file.
for the version of PERC used in your system (H710,
Dell PowerEdge
If the issue persists, contact Dell Technical Support.
Cache Status Is Displayed As Degraded
Probable Cause
• Not enough PCIe SSDs are plugged in to support Write-Back (WB) mode.
29
• A PCIe SSD is marked as failed.
Solution
• Cache status can degrade if you plug in only one PCIe SDD. You must have a minimum
of two PCIe SSDs to run Fluid Cache for DAS in WB mode. You can resolve the issue by
adding more PCIe SSDs to your system.
• Cache status can also degrade because a PCIe SSD has failed. To recover, follow the
steps below:
a. Check the status of the cache. Run the command: fldc --status
b. If the status indicates an SSD failure, see the topic “PCIe SSD Has Failed" in this
section.
Enabling Caching On Disk Has Failed
Probable Cause
Solution
• Disk mounting issue in /etc/fstab.
• The Fluid Cache virtual disk is unavailable.
• Evaluation license has expired after 90 days.
• To resolve the disk mounting issue, see the topic “Fluid Cache Disk Does Not Appear"
in this section.
• A Fluid Cache virtual disk might be unavailable for any of the following reasons:
– The Fluid Cache virtual disk is an unsupported device.
– The virtual disk is currently in use, either because it is already enabled as a
Fluid Cache disk, or it is in use by another application.
– The virtual disk does not exist. You cannot enable back end PERC virtual disks
that do not appear in the PERC card. For example, if you try to enable /dev/sdx,
and it does not exist, verify if it is displayed in the PERC card. To troubleshoot
the issue of the PERC card being unable to see the back end PERC virtual disk,
see the
Dell PowerEdge RAID Controller (PERC) User’s Guide
PERC used in your system (H710, H710P or H810).
– If your PERC version is unsupported, you cannot see any back end PERC virtual
disks.
for the version of
The solution for all of the above issues is the same. Use the fldc --list --disk command
to identify all the supported devices available for caching. Enable caching on a disk
that is both supported and available.
NOTE: Some Logical Volume Managers may pass every test as supported devices.
However, an LVM volume is not displayed in the fldc --list --disk output because
LVMs are not supported by Fluid Cache for DAS. Only supported devices can be
displayed in the fldc --list --disk output.
• Caching cannot be enabled on an existing Fluid Cache virtual disk if the Fluid Cache
license is missing or expired. Verify if your Fluid Cache license is valid. To verify your
license, run the command: fldc --status --license
Cannot Add Virtual Disk
Probable Cause Cannot add the 65th virtual disk to cache.
30
Solution The maximum number of virtual disks that can be added to a cache is 64. If you try to add the
65th virtual disk, the add operation fails.
Disabling Cache Fails
Probable Cause
Solution
• The Fluid Cache disk is in use or mounted.
• The Fluid Cache disk contains dirty data that cannot be flushed to disk.
• For the Fluid Cache disk is in use or mounted issue, you cannot disable caching on a
Fluid Cache disk unless it is first unmounted. To unmount the disk, run the command:
fldc --disable --disk= <disk_name>--umount. The disk is now
disabled and unmounted.
• For the Fluid Cache disk contains dirty data that cannot be flushed to disk issue, you
cannot disable caching on a Fluid Cache disk if it is in a failed state. Resolve any back
end storage issue causing the back end PERC virtual disk to fail and reactivate the disk
before you reattempt to disable it. To disable caching, run the command: fldc --
disable --disk= <disk_name>--umount
restore from a backup. If a backup is restored, you do not need the data in cache. You
can use the disable command and add the --discard flag to destroy all the dirty blocks
in cache.
CAUTION: Do not use the --discard option unless you know that the removal of
dirty data in cache does not cause any data integrity issues. Use of --discard
renders your data unrecoverable.
NOTE: It may take some time for the disable command to complete.
. If reactivation fails, you can
Fluid Cache I/O Errors
Probable Cause The I/O caching errors occur when Fluid Cache for DAS cannot read or write from a back end
virtual disk that the PERC card controls.
Solution If the Fluid Cache layer does not exist, the PERC card sends I/O errors to applications when a
back end virtual disk fails. Since the Fluid Cache for DAS exists, it intercepts I/O errors from the
PERC card and passes them through to any application reading or writing to the back end
virtual disk which the PERC card has identified as missing. For a list of issues that can cause
I/O errors, see the topic “Fluid Cache Disk Status Has Failed” in this section. Some of the
issues that can generate I/O errors include the following:
• A back end PERC virtual disk has failed.
• A data volume is no longer available.
• Multiple PCIe SSDs have failed simultaneously.
Use Dell OpenManage to identify the reason for the I/O error. Use the PERC card controls to
identify the physical device that has failed.
Operating System Fails
If your operating system fails, it may be possible for you to recover your Fluid Cache for DAS
configuration.
31
Follow the next steps to recover your Fluid Cache configuration:
1. Restore your system from a backup.
2. If there is no backup, contact Dell Technical Support.
Fluid Cache For DAS Error Messages And Resolution Steps
If Fluid Cache for DAS displays an error message described in this section, follow the steps suggested to resolve the
issue.
Message ID: 2703| JOURNAL_INSUFFICIENT_MIRRORS_EVENT
The message indicates that enough journal mirrors are not available to operate. To resolve the issue, you must ensure
that there are at least two journal mirrors that are accessible. You must activate either one or more failed cache devices
or use the fldc_restore utility to rebuild the node.
Message ID: 2704| JOURNAL_CLUSTER_ID_MISMATCH
The message indicates that the cluster ID in the journal does not match the cluster ID in the configuration file. To resolve
the issue, contact Dell Technical Support.
Message ID: 2705| JOURNAL_ACCESS_ERROR
The message indicates that the journal cannot be read or written. To resolve the issue, contact Dell Technical Support.
Message ID: 2900| CACHEDEV_FAILURE_EVENT
The message indicates that the cache device has failed. To resolve the issue, replace the PCIe SSD.
Message ID: 2901| STORAGE_DEAD_EVENT
The message indicates that the storage device is either inaccessible or has failed. If the device is inaccessible, restore
connectivity. If the device has failed, replace it.
Message ID: 2909| STORAGE_UNKNOWN_STATE_EVENT
The message indicates that the storage device is in an unknown state. To resolve the issue, contact Dell Technical
Support.
Message ID: 2911| BLOCK_DEVICE_FAILURE
The message indicates that the cached LUN has a failure. To resolve the issue, contact Dell Technical Support.
Message ID: 2915| LICENSE_INVALID_EVENT
The message indicates that no valid license is installed. To resolve the issue, you must install a valid license.
32
Message ID: 2916| LICENSE_EVAL_VALID_EVENT
The message indicates that you are running an evaluation license. The message also indicates the number of days
remaining to run the software in evaluation mode. To resolve the issue, you must install a valid license.
Message ID: 2917| LICENSE_EVAL_EXPIRED_90_EVENT
The message indicates that you are running on an expired evaluation license. No configuration changes are allowed.
The message also indicates the number of days you have run the software with an expired license (90 days maximum).
To resolve the issue, you must install a valid license.
Message ID: 2918| LICENSE_EVAL_EXPIRED_180_EVENT
The message indicates that you are running the software on an expired evaluation license. No configuration changes
are allowed. The message also indicates the number of days you have run the software with an expired license (180
days maximum). To resolve the issue, you must install a valid license.
Message ID: 2919| LICENSE_EXPIRED_NO+CONFIG+CHANGE
The message indicates that you are running on an expired or invalid license. To resolve the issue, you must install a valid
license.
Message ID: 2922| INSUFFICIENT_MEMORY
The message indicates that there is not enough memory to run necessary services. To resolve the issue, you must run
on a system with adequate memory.
Message ID: 2923| CACHEDEV_VIEW_INCOMPLETE
The message indicates that one or more cache devices are missing and the cache is unresponsive. To resolve the issue,
insert the missing cache device. If the cache device was unplugged, reactivate it.
Message ID: 2930| STORAGE_MODE_DEGRADED_TO_WRITE_THROUGH
The message indicates that even though caching was enabled in write-back mode, it is currently operating in writethrough mode. To resolve the issue, add a PCIe SSD to the cache pool.
Message ID: 2931| STORAGE_MODE_DEGRADED_TO_PASS_THROUGH
The message indicates that even though caching was enabled in write-back or write-through mode, it is currently
operating in pass-through mode. To resolve the issue, add one or more PCIe SSDs to the cache pool.
33
34
6
Disaster Recovery Scenarios For Fluid Cache For DAS
To resolve issues with your Fluid Cache For DAS, see Troubleshooting Fluid Cache For DAS. If the Dell Technical Support
contact requests for running fldccollect.pl or fldc_restore, see the sections below.
Data Collection Utility For Fluid Cache For DAS
fldccollect.pl is a Fluid Cache for DAS data-collection utility for use by Dell Technical Support in analyzing
issues.
CAUTION: Do not run this utility except under the direction and supervision of Dell Technical Support.
1. Navigate to the following directory: /opt/dell/fluidcache/bin
2. At the prompt, run the script with super user permissions using sudo, and type: ./fldccollect.pl
While collecting data, fldccollect.pl provides some tracking messages. When finished,
fldccollect.pl creates a tar.gz archive in the /tmp directory.
3. Follow the instructions from Dell Technical Support to send the archive to Dell Technical Support for analysis.
Restoring Fluid Cache For DAS Configuration
Fluid Cache for DAS has a restore utility, fldc_restore, that you can use to restore your caching configuration in the
event of a disaster. Fluid Cache for DAS maintains three journals that are mirror copies of your Fluid Cache for DAS
configuration. Using fldc_restore, you can locate the most recent configuration and restore that configuration to the
new Fluid Cache for DAS installation on a re-built system.
For the restore process to be successful, you must know the UID for at least one of the PCIe SSDs on the system. When
you use the UID, also called WWN, with the restore utility, the utility locates the PCIe SSD. The utility then finds the
journal associated with that PCIe SSD, finds all of the other journals in the configuration, determines which journal is the
latest, and then recreates the missing configuration file that contains the last known Fluid Cache for DAS configuration.
NOTE: To obtain at least one WWN, run the following command: ./dev2wwn /dev/<pcie_ssd_name>
The command returns the UID for the target PCIe SSD.
1. In the event of a disaster, after the system is rebuilt and all the associated software is reinstalled, navigate to the
directory: /opt/dell/fluidcache/bin
2. Run the command: ./dev2wwn /dev/<pcie_ssd_name>
For example: ./dev2wwn /dev/rssda
The command returns the UID for the target PCIe SSD. For example: Path /dev/rssda is WWN
500A07510202EE60
3. Run the command: ./fldc_restore <wwn>
For example: ./fldc_restore 500A07510202EE60
35
NOTE: The fluid_cache service cannot be running when you run the fldc_restore command.
NOTE: The fldc_restore command accepts up to three UIDs on the command line. The more UIDs you include
on the command line, the more likely the utility can locate and use the correct journal for the restore process.
NOTE: For a system with only one PCIe SSD, run the command: ./fldc_restore -f <uid>. For
example: ./fldc_restore WWN 500A07510202EE60 -f . Use the force option (-f) only if you are
sure that the WWN (UID) you are entering is the most recent, pre-disaster running configuration.
The command returns the following output:
[root@devvm14 ~]# /opt/dell/fluidcache/bin/fldc_restore
6E6E6E6E6E6E6E6E6E6E6E6E6E736464 Journal Cluster ID [58f002fb-80f7-4640ac8a-fadfc6deb75b]
Journal mirror locations
location type [Cache Device] - location [6E6E6E6E6E6E6E6E6E6E6E6E6E736464]
location type [Cache Device] - location [6E6E6E6E6E6E6E6E6E6E6E6E6E736465]
location type [Cache Device] - location [6E6E6E6E6E6E6E6E6E6E6E6E6E736467]
Found configuration record at block 5
uncompressing xml to /tmp/<any_name>config.xml
Configuration restored to [/opt/dell/fluidcache/conf/fldc.conf]
You have now restored the Fluid Cache for DAS configuration you had before the disaster.
4. Restart the fluid_cache service.
5. Ensure that the product has started successfully by running the command: fldc --status
This command returns a set of information similar to the following:
Fluid Cache Service
Cache size: 0 GiB
Status: ok
SSDs in Cache: 0
Disk being cached: 0
Write Back: unavailable
Cache ID: <any node>.example.com
License: node – limited to 4 nodes
License Status: valid
SSDs in cache No SSDs are configured in the cache
pool
Disks being cached No Disks are being cached
Caching now continues as before the disaster occurred.
36
Frequently Asked Questions
Question How do I contact Fluid Cache Technical Support?
Answer See Contacting Dell.
Question What is the lifecycle of my Dell Express Flash PCIe SSD?
Answer Dell warrants the Express Flash PCIe SSD to a maximum amount of data written to the PCIe
SSD in total bytes. The PCIe SSD monitors these cycles, and Dell software management
applications notify you when the warranty limits are reached. For more details, see the
PowerEdge Express Flash PCIe SSD User’s Guide
Question How can I safely remove or replace a PCIe SSD?
Answer Use the fldc --remove command to notify Fluid Cache that you intend to remove a SSD. After
notification, you can physically remove a SSD from the system. See the topic “Removing
Working PCIe SSD From Cache Pool” in the section fldc CLI Commands.
Question How do I recover from removing the wrong PCIe SSD?
Answer Use the fldc --reactivate command to recover from a single PCIe SSD failure or removal. For
more information on a single PCIe SSD failure, see the topic “PCIe SSD Has Failed” in the
section Troubleshooting Fluid Cache For DAS.
at dell.com/support/manuals.
Dell
7
Question What is the maximum cache size of Fluid Cache for DAS?
Answer The supported PCIe SSD capacities are 175 GB and 350 GB and you can install up to four PCIe
SSDs in your PowerEdge system. PCIe SSDs of the same capacities (either 175 GB or 350 GB)
can be combined to create flash cache capacity ranging from a minimum of 175 GB to a
maximum of 1400 GB.
NOTE: You cannot mix PCIe SSDs of different capacities for running Fluid Cache for DAS.
Question How does Fluid Cache for DAS ensure writes are not lost?
Answer In write-through cache mode, writes are sent both to the PCIe SSD and to the backing store.
This is slower than write-back cache mode, but it ensures that writes are not lost. In writeback cache mode, writes are replicated across multiple PCIe SSDs and in the event of a PCIe
SSD or operating system failure, Fluid Cache for DAS can successfully recover the data.
Question What file system does Fluid Cache for DAS support?
Answer Any file system supported by the operating system(s) used by Fluid Cache for DAS is
supported. For more details on supported operating systems, see Software and Firmware
Prerequisites For Installing Fluid Cache.
Question What controls the cache flush rate?
37
Answer The cache flush rate is controlled by an internal timer in Fluid Cache for DAS.
Question What is the cache eviction policy?
Answer The cache eviction policy is set to remove the least recently used (LRU) cache block. The
eviction rate can be reviewed using the fldcstat command. To view a sample report, see the
topic “fldcstat Command Options” in the section fldc CLI Commands.
Question How can I measure Fluid Cache for DAS performance?
Answer Fluid Cache for DAS uses the fldcstat command to create a report that includes performance
metrics. The report includes Performance Score, IOPS, MBs, CacheIO-DiskIO, evictMBs,
dirtyMB, and Ops among other statistics. To view a sample report, see the topic “fldcstat
Command Options” in the section fldc CLI Commands.
Question How do I identify the firmware version that is installed on my system?
Answer See How to Update Firmware and Drivers on Dell PowerEdge 12G Servers.
Question How can I view resident blocks that contain dirty data?
Answer Run the fldcstat command. To view a sample report, see the topic “fldcstat Command Options”
in the section fldc CLI Commands.
Question How can I upgrade from 175 GB PCIe SSDs to 350 GB PCIe SSDs?
Answer See the
NOTE: You cannot mix PCIe SSDs of different capacities for running Fluid Cache for DAS.
Dell PowerEdge Express Flash PCIe SSD User’s Guide
at dell.com/support/manuals.
38
Loading...