Dell DX6000G, DX Storage Cluster File Server Setup And Configuration Manual

Download

DX Storage Cluster File Server (CFS)

Setup and Configuration Guide

Version 2.6

DX Storage Cluster File Server (CFS) Setup and Configuration Guide: Version 2.6

No part of this document may be reproduced, transmitted, or transcribed without the written consent of Caringo, Inc.

Table of Contents

1. Welcome to DX Storage CFS ............................................................................................... 1

1.1. Overview of DX Storage CFS .................................................................................... 1

1.2. About this Document ................................................................................................. 1

1.2.1. Audience ....................................................................................................... 1

1.2.2. Scope ............................................................................................................ 1

2. Installation and Guided Configuration .................................................................................... 2

2.1. Requirements ........................................................................................................... 2

2.1.1. Operating System ........................................................................................... 2

2.1.2. Recommended Network Infrastructure ............................................................. 2

2.1.3. Protecting Local Disk Resources ..................................................................... 2

2.2. Installation Steps ...................................................................................................... 3

2.2.1. Installing the DX Storage CFS Server .............................................................. 3

2.2.2. Cluster Name Space (CNS) ............................................................................ 3

2.2.3. Configuring DX Storage CFS and its Mountpoints ............................................ 6

2.2.4. Post-configuration File System Options ............................................................ 8

2.3. Deleting a Mountpoint ............................................................................................... 9

2.4. Upgrading CFS ......................................................................................................... 9

2.4.1. Upgrading from Versions 1.x ........................................................................... 9

2.4.2. Upgrading from Version 2.0 ............................................................................ 9

2.4.3. Upgrading from Version 2.5 .......................................................................... 10

2.5. Removing CFS and CNS ......................................................................................... 10

2.6. Restoring CFS ........................................................................................................ 11

3. Running and Managing DX Storage CFS ............................................................................ 12

3.1. Starting DX Storage CFS and CNS .......................................................................... 12

3.2. DX Storage CFS and CNS Shutdown ...................................................................... 12

3.3. Timescapes ............................................................................................................ 13

3.3.1. Deleting a Timescape Mount ......................................................................... 14

3.3.2. Mounting a Timescape at an Alternate Time .................................................. 14

3.4. File Revisions and DX Storage Deletion ................................................................... 14

3.5. Support and Management Utilities ............................................................................ 14

3.5.1. cns-reporter .................................................................................................. 14

3.5.2. cfs-grab ....................................................................................................... 15

3.5.3. Configuration Modifications with cns-admin and cfs-admin ............................... 15

3.5.4. cfs-loglevel ................................................................................................... 16

3.6. DX Storage Metadata and Policies ........................................................................... 16

3.6.1. Custom Metadata ......................................................................................... 17

3.6.2. DX Storage Lifecycle Policies ........................................................................ 18

A. Temp and Logging Space Configuration ............................................................................. 20

Version 2.6

February 2011

Chapter 1. Welcome to DX Storage CFS

1.1. Overview of DX Storage CFS

DX Storage Cluster File Server (CFS) is a native DX Storage application that presents a DX Storage cluster as a standard Linux file system, enabling the use of DX Storage by software that uses a file system. You can also layer a network file service over DX Storage CFS using Samba or NFS. CFS separates file data stored directly in DX Storage from file metadata stored in DX Storage via the Cluster Name Space (CNS).

To enhance performance, the CFS process uses the local file system as a spool/cache for files. This local cache is where user files are written before they get spooled to DX Storage and where they will be accessed on subsequent reads if locally available. The cache/spool size is managed by evicting files based on a least recently used basis when space needs to be reclaimed.

The Cluster Name Space (CNS) also uses DX Storage to store name space records for the file system structure and metadata (owner, permissions, etc.), fulfilling the role typically performed by a database but without the scalability and management concerns. CNS uses a RAM-based cache to deliver metadata objects that are in high demand. Name space modifications and updates are continuously flushed to DX Storage via a background process.

Recoverability for both CFS and CNS are enhanced by using the Linux machine's local hard disk or shared storage for journaling of in process events. In the event of a power failure or system crash, the processes are restarted automatically and any changes not recorded in the namespace or spooled to DX Storage are replayed from the journal to prevent data loss. A process monitor runs at all times to restart a CFS or CNS process in the unlikely event of a process crash. All process monitoring and recovery activities appear in the syslog for management purposes.

1.2. About this Document

1.2.1. Audience

This document is intended for people in the following roles.

1. Storage system administrators

2. Network administrators

3. Technical architects Throughout this document, the storage system administrator and network administrator roles will be

referred to as the administrator. The administrators are normally responsible for allocating storage, managing capacity, monitoring storage system health, replacing malfunctioning hardware, and adding additional capacity when needed.

1.2.2. Scope

This document covers the steps needed to deploy and configure DX Storage CFS. The reader is expected to have a background in networking, basic knowledge of Linux operating systems, and optional experience with NFS, CIFS, SAMBA or other file server clients.

Version 2.6

February 2011

Chapter 2. Installation and Guided Configuration

2.1. Requirements

2.1.1. Operating System

DX Storage CFS has been developed and tested with 64-bit Red Hat Enterprise Linux 6.0; other RHEL versions or Linux distributions are not currently supported. Subsequent installation instructions will assume a pre-installed RHEL environment with either internet connectivity or an alternately configured RHEL yum repository for use in installing required 3rd party packages.

Warning

The recently added Red Hat 'hugepages' feature has been found to be incompatible with CFS and other applications, often resulting in kernel hangs particularly under high stress scenarios. As a result it is highly recommended that huge pages be disabled as follows prior to installing and running CFS:

echo 'never' > /proc/sys/kernel/mm/redhat_transparent_hugepage/enabled If the feature is not disabled, both CFS and CNS will log a warning at startup as an

additional reminder. For the changes to remain in effect with the next reboot, you must add 'transparent_hugepage=never' to the existing kernel parameters in the /boot/grub/ menu.lst file prior to rebooting the CFS server.

Note

For optimal performance, it is highly recommended that the 'deadline' I/O scheduler be utilized on the installed CFS server. The following command demonstrates how to enable the deadline scheduler on an 'sda' disk:

echo deadline | sudo tee /sys/block/sda/queue/scheduler

For the changes to remain in effect with the next reboot, you must add 'elevator=deadline' to the existing kernel parameters in the /boot/grub/menu.lst file prior to rebooting the CFS server.

2.1.2. Recommended Network Infrastructure

A configured Network Time Protocol (NTP) server is strongly recommended to provide clock synchronization services to DX Storage CFS, the name space server and the DX Storage cluster. A common sense of time is especially important between CFS servers and the name space server if the latter is configured remotely. To install a functional NTP configuration on RHEL, run the following command as the root user:

# yum install ntp

Gigabit Ethernet or better is the recommended connection speed between DX Storage CFS and DX Storage nodes.

2.1.3. Protecting Local Disk Resources

Red Hat 6 does not guarantee immediate write of data to disk with their default ext4 file system. In order to ensure data security in the event of a system or power failure, it is therefore highly recommended that administrators add the 'nodelalloc' mount option to several critical local

Version 2.6

February 2011

directories critical to recovering from a failure. Specifically, the following 3 directories should be mounted with 'nodelalloc':

mount /dev/sdxx /var/lib/dht -o rw,nodelalloc,user_xattr

mount /dev/sdxx /var/cache/cns -o rw,nodelalloc,user_xattr

mount /dev/sdxx /var/spool/cfs -o rw,nodelalloc,user_xattr

2.2. Installation Steps

The following steps are necessary to install DX Storage CFS.

• Install DX Storage CFS

• Configure DX Storage CNS

• Configure DX Storage CFS On an Red Hat Enterprise Linux system with a local name space, the following commands must be

run as root to install and setup the necessary components:

$ sudo su # ./installCFS.sh # cns-admin # cfs-admin mkfs --add # /etc/init.d/caringo-cfs start

There are many valid ways to install each of these components and services. This is not a comprehensive guide to NTP or other 3rd party package installation. For complete instructions on installing, please see the documentation included with the operating system, from the software project web sites, and other materials. The following is a list of examples with steps highlighted for your reference. Please make sure that you understand the details of configuring these packages and customize these steps to fit your IT environment.

For information on configuring CFS for High Availability, please reference the Linux-HA project.

2.2.1. Installing the DX Storage CFS Server

DX Storage CFS distribution is available as a Red Hat rpm package that is installed with a shell script. The package and its dependencies must be installed as the 'root' user with the following steps:

1. Install caringo-cfs package by running the following:

$ sudo su # ./installCFS.sh

2.2.2. Cluster Name Space (CNS)

The file system structure, metadata and DX Storage UUIDs are stored in a journaling name space that persists file metadata into DX Storage automatically.

After installing the DX Storage CFS server, you will need to configure a new name space. This is only required once at installation time, subsequent CFS share definitions will not require a new name space. You can initiate the configuration process by running the following command as the root user:

Version 2.6

February 2011

# cns-admin

Note: Please make sure DX Storage is online and reachable prior to configuring the new name space. cns-admin will prompt for the following values:

1. Log facility. Enter the logging facility to use. The default value is syslog. a. Log filename. If the file logging facility was selected, enter the filename and location of

the logfile. By default the installer will use a /var/log/caringo/cns.log file. An additional file in /var/log/caringo/cnsaudit.log will log the UUIDs for all successful DX Storage deletes for audit purposes. With file logging, default log rotation for both files will be configured to keep up to 8 log files, rotating weekly or at a max file size of 512 MB.

b. Syslog facility. If the syslog option was selected, enter the facility to log to. The default

value is local4. Configuration of a remote syslog host must be done in the syslogd configuration file.

2. Log level. Enter the minimum level of log messages to record. The default value is info. Each log level includes the following:

• info: errors, warnings and information messages like system start up and stop

• verbose: errors, warnings, and info messages plus high-level operational functions

• debug: errors, warnings, info and verbose messages plus lower level operational functions

• trace: all possible log messages

Note

Debug or trace logging are not recommended unless explicitly requested as part of a support investigation due to the amount of disk space consumed. If debug logging is required for any length of time, administrators must monitor space availability carefully and are strongly encouraged to allocate a dedicated partition for logging to protect other system services.

3. CNS host: The location of the name space server for the name space you are configuring. The default is localhost (127.0.0.1), which will configure a name space on the local server. Enter an external IP address if using a remote server separate from the CFS server for the name space. You may also enter a 0.0.0.0 address if the name space should be created locally but will need to connect to CFS shares on both the local server and remote servers.

4. DX Storage: use Zeroconf. This option allows you to specify whether or not Zeroconf should be used to discover the list of nodes in the DX Storage cluster. The CNS server must be in the same subnet with DX Storage in order to use Zeroconf. CFS 2.6 is compatible with DX Storage versions 3.0.5 and later.

a. If 'No' is selected:

Cluster primary node address: Enter the IP address or hostname for a DX Storage node in the target cluster. The target cluster must be the same as the one configured for all CFS mounts.

Primary node SCSP port: Enter the port the DX Storage node uses for SCSP communications. The default is '80'.

Version 2.6

February 2011

Cluster secondary node address: Enter the IP address or hostname for a second DX Storage node in the target cluster for redundancy. The target cluster must be the same as the one configured for all CFS mounts.

Secondary node SCSP port: Enter the port the secondary DX Storage node uses for SCSP communications. The default is '80'.

b. If 'Yes' is selected:

Cluster name Enter the name of the DX Storage cluster. This must match the value of the DX Storage "cluster" parameter in the node.cfg file and be the same target cluster as the one configured for all CFS mounts. Note that if using Zeroconf for discovering the list of nodes and using a cluster name with periods in it, the periods should be replaced with underscores to ensure proper node discovery.

5. Do you want me to email you the Root UUID for CNS? Whether or not you would like the generated root UUID for the name space in DX Storage emailed to you. The root id is critical to recovering the name space in the event of a name space server failure and should be saved in a safe location. The email function will only work with SMTP servers that do not require authentication.

a. If you answer yes, you will be prompted to enter email server and address information:

i. SMTP email server: The SMTP email server that should be used to send the email.

Ex: mail.host.com. A mail account of 'admin' at the specified mail server is assumed (i.e. admin@mail.host.com) as the sending account. Mail servers without an admin account will not be able to send the root uuid email.

ii. Your email address: The email address to which the root id should be sent. Ex:

name@company.com

Note

Administrators who experience email send errors should consult /var/log/ caringo/smtp.log.

b. If you answer no, the utility will confirm that a new root id for the name space has been

created with a message similar to the following: Generated New Root Anchor [f223011d1fa56f67cf79a87a5de45901]

Warning

If the configuration utility detects that a root id already exists it will ask you if you would like to create a new one. Answering 'yes' to this question will create a new root and delete all local CFS state information (spool, cache, etc.), as you will not be able to access the information in the shares once the name space root has changed. The configuration files for all shares will remain intact. You must stop CFS and CNS prior to attempting to create a new root id.

The script will ask if you would like CNS to be started once you confirm the root id. 'Yes' is recommended as a functioning namespace is required for CFS configuration. The created configuration file will be located in: /opt/caringo/fabric/etc/cns.conf. Manual editing of this file is not recommended unless under direct advice from your technical support contact.

Version 2.6

February 2011

Note

The name space must remain online at all times to ensure CFS can obtain the needed file system information. If CFS cannot connect to the name space, it will start throwing permission denied errors and block new file writes.

2.2.3. Configuring DX Storage CFS and its Mountpoints

After installing the DX Storage CFS server and creating the name space via cns-admin, you will need to configure a mountpoint, logging and DX Storage cluster information. You can initiate the configuration process by running the following command as the root user:

# cfs-admin mkfs --add

This will prompt you for the minimum required parameters needed to run DX Storage CFS. Most fields have a default value with the exception of fields that require your input which will be blank. A description of each of the configuration prompts is included below. To stop the configuration process at any time, use Ctrl-C; no configuration files will be created until the utility completes successfully. The utility can be run multiple times if you are creating multiple DX Storage CFS mountpoints. Each mountpoint must have its own unique name, configuration file and spooler partition. All created configuration files will be located in: /opt/caringo/fabric/etc/<mountid>.conf. Manual editing of this file is not recommended unless under direct advice from your technical support contact.

1. ID to be used for this mount. Enter a name for the mountpoint. It must be unique on each server and consist of letters and numbers without punctuation or spaces. For example: "acctShare" or "archiveFS".

2. Mount directory path. Enter the location of the mountpoint. By default, the utility will use the previously provided mount ID in the /mnt directory. Spaces in mountpoint paths are not supported.

3. Spooler directory path. Enter the location where the spooler/cache directory will reside. The location must be unique for each distinct read/write mount ID or the filesystem mount will fail. By default, the installer will use the previously provided mount ID in the /var/spool/cfs directory. Extended attributes must be enabled on the specified spooler partition to store the lifepoints and custom and system metadata associated with each unspooled item.

Warning

A dedicated spooler partition is required for each share to ensure space monitoring is accurate and the local cache is properly maintained.

4. Mount on boot. This determines whether or not this DX Storage CFS file system should be mounted automatically when the server reboots. The default and recommended value is Yes.

5. Mount read-only. This value determines whether or not this DX Storage CFS file system should be mounted in read-only mode. The default value is No.

6. Log facility. Enter the logging facility to use. The default value is syslog. a. Log filename. If the file logging facility was selected, enter the filename and location of the

logfile. By default the installer will use a <mountid>.log file in the /var/log/caringo directory. With file logging, default log rotation will be configured to keep up to 8 log files, rotating weekly or at a max file size of 512 MB.

Version 2.6

February 2011

b. Syslog facility. If the syslog option was selected, enter the facility to log to. The default

value is local4. Configuration of a remote syslog host must be done in the syslogd configuration file.

7. Log level. Enter the minimum level of log messages to record. The default value is info. Debug is not recommended unless explicitly requested as part of a support investigation due to the amount of disk space consumed that will not then be available for core processes. Each log level includes the following:

• info: errors, warnings and information messages like system start up and stop

• verbose: errors, warnings, and info messages plus high-level operational functions

• debug: errors, warnings, info and verbose messages plus lower level operational functions

• trace: all possible log messages

Note

8. CNS Host: The IP address or hostname of the name space server that will store the filesystem metadata for the mount you are configuring. The default is localhost (127.0.0.1), which will configure the share to use a name space on the local server. To use a name space configured on a remote server, enter the external IP address for the previously created name space.

Note

If configuring a multi-server environment where more than one CFS server is sharing a single CNS instance, you must enter the same CNS host information that was entered in the cns-admin utility for each share. cfs-admin will generate a world unique serial number for every created CFS share for internal use when requesting exclusive file locks across all servers in the name space. This serial number is stored with the CFS configuration files so they must not be copied from one server to another to ensure the CFS share serial number is not duplicated.

9. CFS Time Horizon (Default=0d means no Timescapes): The time that deleted file system objects will be kept in order to be available for a timescape mount of the file system at a time in the past. Times are accepted in hours (i.e. 2h), days (i.e. 1d), months (i.e. 2m), or years (i.e. 1y). After the time horizon, deleted files are garbage collected from DX Storage and can no longer be accessed.

10. Use Zeroconf for DX Storage configuration?. This option allows you to specify whether or not Zeroconf should be used to discover the list of nodes in the DX Storage cluster. The CFS server must be in the same subnet with DX Storage in order to use Zeroconf. CFS 2.6 is compatible with DX Storage versions 3.0.5 and later.

a. If 'No' is selected:

Version 2.6

February 2011

Cluster primary node address: Enter the IP address or hostname of a DX Storage node in the target cluster. The target cluster must be the same as the one configured for the Cluster Name Space (CNS).

Primary node SCSP port: Enter the port the DX Storage node uses for SCSP communications. The default is '80'.

Cluster secondary node address: Enter the IP address or hostname of a second DX Storage node (different from the primary) in the target cluster for redundancy. If subclusters are configured, a node in a sub-cluster that is not the same as the primary's is recommended. The target cluster must be the same as the one configured for the Cluster Name Space (CNS).

Secondary node SCSP port: Enter the port the secondary DX Storage node uses for SCSP communications. The default is '80'.

b. If 'Yes' is selected:

Cluster name: Enter the name of the DX Storage cluster. This must match the value of the DX Storage "cluster" parameter in the node.cfg file and be the same as the one configured for the Cluster Name Space (CNS). Note that if using Zeroconf for discovering the list of nodes and using a cluster name with periods in it, the periods should be replaced with underscores to ensure proper node discovery.

11. If the Spooler/cache and mount directories do not already exist, you will be asked if you would like them to be created. The recommended response is 'Yes', as DX Storage CFS cannot start without these directories. You will also be asked if you would like to mount the configured mountpoint immediately.

12. After installing DX Storage CFS initially, running the following command will start the CFS monitoring process. This will be started automatically on subsequent reboots.

# /etc/init.d/caringo-cfs start

Note

If you attempt to start CFS without first starting the Cluster Name Space, the initialization will fail after 60 seconds. This may leave the filesystem mounted without a name space. To unmount the filesystem, execute the following command and then start the name space and CFS sequentially:

fusermount -uz /mnt/mountname

2.2.4. Post-configuration File System Options

In addition to the required parameters gathered by the cfs-admin mkfs utility, there are some optional mountpoint settings that can be set in the /etc/fstab file to modify certain file system behaviors. The cfs-admin mkfs utility will automatically create an entry in /etc/fstab each time the utility is run for a unique mount ID. Prior to committing any changes to this file, creating a backup of the previous version of the file is highly recommended. To modify this file, simply open it in a text editor like vi and find the entry for the mountpoint you would like to update. For instance, the 'testMount1' mountpoint point would display as follows using the default parameters from the cfsadmin mkfs utility:

testMount1 /mnt/testMount1 fuse.cfs config=testMount1.conf,noatime

Version 2.6

February 2011

2.2.4.1. ACLs

The cfs-admin mkfs utility does not by default enable Access Control Lists or ACLs. ACLs provide more fine-grained file and directory access control than the basic user, group, owner permissions and can be very useful in complex permission scenarios. For more information on ACLs functionality and syntax, please refer to the ACL manpage. To enable ACLs for a CFS mountpoint, you will need to edit as root the /etc/fstab file and add 'acl' to the mount options as shown in the following example:

testMount1 /mnt/testMount1 fuse.cfs config=testMount1.conf,noatime,acl

2.2.4.2. Access Time Updates

The cfs-admin mkfs utility by default disables updates to the last access time for both a file and a directory by adding the 'noatime' options to the fstab entry for each mountpoint. Access time or atime can have a significant impact on file system performance and is often not needed. This setting does not impact update times when a file or directory is modified. If this information is required for a particular mountpoint, the options can be modified to reenable access time by removing the 'noatime' value since access time updates are enabled by default. An updated fstab entry for the 'testMount1' mountpoint above would appear as:

testMount1 /mnt/testMount1 fuse.cfs config=testMount1.conf

2.3. Deleting a Mountpoint

In the event that you need to delete a mountpoint, the 'cfs-admin mkfs' utility can also be used to specify the mountpoint and issue the delete. Prior to deleting the mountpoint, you should delete all contents from the mountpoint and then unmount it to be sure there are no in process activities (see Section 3.2). As a root user, type the following command (where 'testMount1' equals the name of the mountpoint to be deleted):

# cfs-admin mkfs --delete --cfs-id=testMount1

The delete command should be used only when the mountpoint is no longer needed both currently and historically. It may take some time to delete all the files and resources associated with a mountpoint. This will not remove the name space nor will it remove the files' streams from the DX Storage cluster.

2.4. Upgrading CFS

2.4.1. Upgrading from Versions 1.x

The 2.6 release now supports data migration from the previous 1.x versions of CFS that utilized a MySQL database. Please contact your support resource for migration instructions.

2.4.2. Upgrading from Version 2.0

2.0 Ubuntu-based installations can migrate to 2.6 by installing a new Red Hat server with

configuration data and shares that exactly mirror the current installation and then switching out the newly created CNS rootID with the previous Ubuntu-based one. The same DX Storage cluster must be used for both the old and the new installation. Please reference the 'Restoring CFS' section of the Setup and Configuration Guide for complete details.

Version 2.6

February 2011

2.4.3. Upgrading from Version 2.5

Version 2.5 Red Hat installations can upgrade to 2.6 by stopping all services, installing the new version and then restarting services. After copying the 2.6 distribution file to the server and unzipping it, run the following commands as root:

Note

Any configuration file backups previously taken for disaster recovery purposes must be discarded and a new backup taken, as previous configuration files are not compatible with the new software version.

1. Stop all client activity on the server. Stopping remote Samba or NFS access to the share is an easy way to do this.

2. Monitor the local spooler cache draining to zero by using the following script:

./cfs-checkspool

The script will monitor the spool cache and any active writes for all configured shares in a loop and print a statement when all activity has completed and it is safe to shutdown CFS.

3. Stop CFS: /etc/init.d/caringo-cfs stop

4. Stop CNS: /etc/init.d/caringo-cns stop

5. Install the new software: ./installCFS.sh

6. Start CNS: /etc/init.d/caringo-cns start

7. Start CFS: /etc/init.d/caringo-cfs start

2.5. Removing CFS and CNS

The following commands should be run to remove CNS and CFS if the data and metadata are no longer needed:

1. Prior to removing CFS, the timescape horizon for all shares should be set to 0 days and then all files and folders should be manually deleted from all mountpoints to ensure content is not orphaned in DX Storage. This will trigger the namespace garbage collection process to delete the associated content from DX Storage. When the CNS log at debug level consistently shows that no files have been deleted as part of the garbage collection process, it is safe to proceed with removal of the software.

2. Stop CFS: /etc/init.d/caringo-cfs stop

3. Stop CNS: /etc/init.d/caringo-cns stop

4. To remove all mounts, do the following for each mount: cfs-admin mkfs --delete -- cfs-id=testMount1 (where 'testMount1' equals the name of the mountpoint to be deleted)

5. Purge CFS code: yum remove caringo-cfs

Note

Removing CFS without first unmounting will not succeed and will print a scriptlet error to the command line. Due to a defect in the Red Hat 6 package manager (yum) a more

Version 2.6

February 2011

descriptive error is not passed through to the client. Check the syslog for a full description of the possible errors if the uninstall is unsuccessful.

2.6. Restoring CFS

Should a CFS server ever fail or need to be repurposed, all persisted data can be restored or moved onto a new CFS server by configuring the server as you would a new server but then replacing the CNS rootUUID with the previously configured one. Recovering with the rootUUID will not restore files that had not yet been spooled into DX Storage or that were written to the namespace in the 60 seconds prior to a server failure. For planned moves, administrators should ensure that writes are quiesced for a period of time prior to the restore.

To re-install or move CFS onto a new server, execute the following steps:

1. If migrating from version 2.0, install Red Hat 6.0 on a new server.

2. Install and configure CNS as it was configured previously using the installation instructions above. The DX Storage cluster MUST be the same as the previous configuration in order to retrieve the previously stored data.

3. Stop all client activity on the old server. Stopping remote Samba or NFS access to the share is an easy way to do this.

4. Monitor the local spooler cache draining to zero by using the following script:

./cfs-checkspool

The script will monitor the spool cache and any active writes for all configured shares in a loop and print a statement when all activity has completed and it is safe to shutdown CFS.

5. Stop CFS on the old server: /etc/init.d/caringo-cfs stop

6. Stop CNS on the old server: /etc/init.d/caringo-cns stop

7. Edit the CNS configuration file on the new server (nano /opt/caringo/fabric/etc/ cns.conf) and replace the existing root_anchor parameter with the previous server's rootUUID.

8. Recreate the CFS shares using the cfs-admin mkfs script. The DX Storage cluster and the mount IDs for the shares MUST be same as the ones prior to the upgrade.

9. Start CNS and CFS as normal.

If you are restoring a system after a failure, the process is the same except steps 2-5 above will not be necessary.

Version 2.6

February 2011

Chapter 3. Running and Managing DX Storage CFS

3.1. Starting DX Storage CFS and CNS

The Cluster Name Space should always be started prior to CFS. The name space can be started using the following command:

$ /etc/init.d/caringo-cns start

CNS will fail to start and print an appropriate error message if one of several critical configuration parameters is missing or invalid. Once the configuration is corrected via the cns-admin script, CNS should start correctly.

DX Storage CFS will start automatically when the DX Storage CFS server boots if the "mount on boot" option was selected during the configuration process for each mountpoint. If the process was stopped for any reason it can be manually started with a standard mount command. To mount all configured DX Storage CFS mountpoints at once, type in the following command:

$ sudo /etc/init.d/caringo-cfs start

To mount a single mountpoint that was previously defined using the cfs-admin mkfs script, run a command similar to the following (where /mnt/testMount1 is the mounted destination for the desired mountpoint)

$ sudo mount /mnt/testMount1

CFS will fail to mount a share and will print an appropriate error message if one of several critical configuration parameters is missing or invalid. Once the configuration is corrected via the cfs-admin mkfs script, the share should mount correctly.

Before writing any data to a mountpoint, run a 'mount' command with no options to ensure the desired mountpoints started correctly and are present in the mounted list. If the mountpoint is not in the mounted list, the install was not successful and you should not write any data to the mountpoint.

3.2. DX Storage CFS and CNS Shutdown

To stop DX Storage CFS and unmount all configured shares, use the following command:

$ sudo /etc/init.d/caringo-cfs stop

To stop and/or unmount a specific configured share, use a command similar to the following (where /mnt/testMount1 is the mounted destination for the mountpoint):

$ sudo umount /mnt/testMount1

For file systems shared via Samba, NFS, etc, administrators may need to stop associated file system services before they can cleanly unmount CFS.

Note

If DX Storage CFS is stopped using a kill command, a 'fusermount -u /mnt/mountpoint' command must be executed before restarting to ensure the mountpoint is properly released and remounted. If the remount option is utilized, the mountpoint will be unmounted and then immediately mounted.

Version 2.6

February 2011

After CFS has been stopped, CNS may also be stopped using the following command:

$ /etc/init.d/caringo-cns stop

3.3. Timescapes

The DX Storage CFS Timescape continuous snapshot feature allows an administrator to mount a historical, read-only view of a file system at any past time, providing a mechanism to both view and/ or copy previous versions of files that might otherwise be lost.

To utilize the feature, you must first create a timescape mount at the point in time you would like to view, using the cfs-admin mktimescape script. The script can be initiated by running the following command:

sudo cfs-admin mktimescape --add

This will bring up a wizard similar to the ones used to initially create a CFS share that will collect the information necessary to create the timescape mount. It includes the following prompts:

1. TCFSID to be used for this timescape mount: Enter a name for the timescape mountpoint. It must be unique on each server and consist of letters and numbers without punctuation or spaces. For example: "acctShareT20100826" or "TimescapearchiveFS20100826".

2. CFSID of the CFS share you wish to make a timescape mount for: Enter the name of the local CFS share for which you are creating a timescape mount. i.e. "acctShare" or "archiveFS" from the initial installation example. The share must be on the same server as the timescape mount you are creating to ensure the configuration data can be shared. The share must be in existence for the entire time the timescape share is mounted for the timescape to work.

3. CFS mount directory path: Enter the location of the timescape mountpoint. By default, the utility will use the previously provided TCFS ID in the /mnt directory. Spaces in mountpoint paths are not supported.

4. Spooler directory path: Enter the location where the spooler/cache directory will reside. The location must be unique for each distinct read/write TCFS ID or the filesystem mount will fail. By default, the installer will use the previously provided TCFS ID in the /var/spool/cfs directory. Extended attributes must be enabled on the specified spooler partition to store the lifepoints and custom and system metadata associated with each unspooled item.

Warning

A dedicated spooler partition is required for each share to ensure space monitoring is accurate and the local cache is properly maintained.

5. CFS Timescape (ISO8601 Format ex. 2010-08-26T13:47:42): Enter the time at which to mount the filesystem specified in ISO 8601 format (ex. 9:38 am US/Pacific on 14 June, 2010 would be written 2010-06-14T09:38:00, with the date prior to T and the time following). The local time zone offset of the DX Storage CFS server will be used. The wizard will return an error if you attempt to configure a time period either earlier than the current time minus the configured timescape horizon or in the future. A timescape share mounted when the configured horizon passes may result in inconsistent results and an inability to access files that have been garbage collected from the namespace while the share is mounted.

6. If the Spooler/cache and mount directories do not already exist, you will be asked if you would like them to be created. The recommended response is 'Yes', as DX Storage CFS cannot

Version 2.6

February 2011

start without these directories. You will also be asked if you would like to mount the configured mountpoint immediately.

Once mounted, you will be able to view the specified CFS share at the time in the past specified by your configured Timescape time. All timescape mounts are read-only but you may copy a deleted file from a timescape mount to an active, current mount to recreate the file in the present.

Timescape mounts log to the same configured logging facility as the CFS share they are based on. By default, timescape shares are configured to require less RAM, as a read-only file system requires less resources than a current file system.

3.3.1. Deleting a Timescape Mount

Similar to an active mount, the 'cfs-admin mktimescape' utility can also be used to specify a mountpoint to be deleted. As a root user, type the following command (where 'ttestMount1' equals the name of the timescape mountpoint to be deleted):

# cfs-admin mktimescape --delete --tcfs-id=ttestMount1

Removing the timescape mount will not remove the name space nor will it remove the files' streams from the DX Storage cluster.

3.3.2. Mounting a Timescape at an Alternate Time

Administrators needing to quickly mount a different time period for the same CFS share may do so by manually mounting the timescape share with the time overridden by a timescape mount option similar to the following:

mount -o remount,timescape=2010-09-15T12:34:20 /mnt/ttestMount1/

where the timescape value is an ISO 8601 time. This time is not persisted into the timescape share configuration file, but prevents creation of a multiple timescape shares with different time periods for the same CFS share. This mount option only applies to timescape mounts and cannot be used for active, current mounts.

3.4. File Revisions and DX Storage Deletion

Each modification to a file stored in CFS creates a new object in DX Storage, with CNS keeping track of the modification revision that is current for each file. Old revisions are deleted from DX Storage via a background garbage collection process based on the configured time horizon. Files that are explicitly deleted from the CFS mount are removed from DX Storage via the same background process. After the time horizon, deleted files are no longer accessible via a timescape mount.

3.5. Support and Management Utilities

3.5.1. cns-reporter

The cns-reporter utility provides basic capacity and content information for the Cluster Name Space. It takes the following arguments:

Argument Definition and usage

-a ADDRESS or --address=ADDRESS Address of a DX Storage node used by the CFS share

Version 2.6

February 2011

Argument Definition and usage

-p PORT or --port=PORT SCSP Port for the configured DX Storage node (default=80)

-r ROOTUUID or --uuid=ROOTUUID The CNS rootID you received during initial configuration

-c CFSID or --cfsid=CFSID The CFSID of the share for which you want to report

-l list or --list=LIST Option that returns the list of CFSIDs in the designated rootUUID. Output returns the Namespace ID, CFSID, Time Horizon, Creation Time and Anchor Stream of the toplevel directory.

-s SUBTREE or --subtree=SUBTREE Relative path of the subtree for which you want to report ( default is the entire share )

-t HISTORY or --timescape=HISTORY Whether or not timescape data should be included in the report. 0=no(default) 1=yes. This argument does not work with the subtree option.

The following shows an example command for a CFS share named 'Accounting' with a subtree named '2010/taxes':

cns-reporter -a 192.168.1.220 -r f4f5388cb8b21ebb917c332625f75fca -c Accounting -s "2010/taxes"

Example output:

Namespace Capacity Report for [Accounting/2010/taxes] created in elapsed: 330.703432 s Total Folders: 11258 Total Files: 6026 Total Data Capacity Used: 10.73 Gb or [11521613767] bytes Total Metadata objects: 22552 Total Metadata Capacity Used: 0.029100 Gb or [31245858] bytes Total CAStor UUID's found:6026 Total Folder attributes: 90530 Total File attributes: 54234

3.5.2. cfs-grab

cfs-grab is a command-line tool that collects runtime environment information to assist with support ticket submissions. From a command line as a root user, simply type 'cfs-grab' to utilize the tool. The tool will generate a gzipped tar file with the current date timestamp (example: grab-1272553621.tgz ) with a collection of relevant data that can be attached to a support ticket or email.

3.5.3. Configuration Modifications with cns-admin and cfs-admin

3.5.3.1. cfs-admin config

Administrators who need to update a configuration parameter for a previously configured CFS share can do so using the command:

Version 2.6

February 2011

sudo cfs-admin config The script will prompt for the CFSID you wish to modify and then display the same prompts from

initial configuration but with the current configuration displayed for each prompt. If you do not wish to modify a particular parameter, simple press enter and the current value will remain in place. Some parameters like the configured namespace that might strand or corrupt data are view only.

Note

The share you are attempting to edit must have been mounted at least once prior to running cfs-admin config to ensure all configuration data is completely populated.

3.5.3.2. cns-admin

If the cns-admin script is re-run after initial configuration, it will prompt for the same questions again but pre-populate the previously provided values and allow for modifications. The script can be run using the same command as initial configuration:

sudo cns-admin At the end, the script will prompt to restart CNS for the new parameters to take effect.

3.5.4. cfs-loglevel

cfs-loglevel is a tool that allows easy access to update the log level for all CFS and CNS logs. From a command line as a root user, type 'cfs-loglevel', then select the desired log level from the dialog box and click 'Ok'. The log level will be updated in the configuration files for both CFS and CNS and will take effect immediately with no required process restart.

3.6. DX Storage Metadata and Policies

Along with the standard file system attributes that are associated with a file stored in DX Storage CFS, additional metadata and lifecycle policies can be stored with files when they are stored in DX Storage. This metadata can assist with storage management and data movement through various life cycles. All metadata and policies are specific to DX Storage and are not visible from within the file system.

Metadata and policies are specific to each mountpoint so it is advisable to configure separate mountpoints in cases where distinct metadata values would be helpful. For instance, to distribute content based on the branch office a file originated from, set up a mountpoint for each branch office and add a custom metadata attribute with the branch office name that would be added to each file stored. Similarly, to store more replicas of files needed by the Media department due to frequent read activity, setup a separate mountpoint for the Media department with a Lifecycle policy for additional replicas and leave all other departments utilizing a mountpoint with a lifepoint policy for fewer replicas.

As a baseline for file metadata, DX Storage CFS will automatically add the following system attributes on all streams stored in DX Storage:

• Castor-CFS-Server: the name of the server from which the file was originally stored

• Castor-CFS-Version: the version of the CFS software that originally stored the file

• Content-Type: the file's mimetype

• CAStor-CFS-CFSID: the name of the mountpoint id through which the file was originally stored

Version 2.6

February 2011

• Castor-CFS-FileId: the filesystem id for the file in the CFS name space

• Castor-CFS-Uid: the file's user id at the time it was stored

• Castor-CFS-Gid: the file's group id at the time it was stored

• Castor-CFS-Mode: the file's permission mode in decimal at the time it was stored The mechanisms to add both custom metadata and life cycle retention policies to files stored in DX

Storage via DX Storage CFS are outlined in the sections below.

3.6.1. Custom Metadata

The 'cfs-admin metadata' utility allows a root user to administer the custom metadata that is attached to newly created files when they are ready to be stored in DX Storage (last file close plus 5 seconds). Custom metadata applies to all files stored in a particular mountpoint and, like all DX Storage metadata, is immutable after the file is stored in DX Storage. Thus, changing the custom metadata for a CFS file system will affect only new or modified files and will not change metadata for files that were previously written. To add metadata to a mountpoint, execute the following command as the root user:

$ sudo su # cfs-admin metadata --set [--name=NAME] [--value=VALUE] <mount-root>

where <mount-root> is the full path for the CFS mountpoint to which the metadata should be attached. For example, to add an attribute of 'Department' with a value of 'Legal' to all files in the legal mountpoint, you would execute the following command:

# cfs-admin metadata --set --name=Department --value=Legal /mnt/legal

Metadata names must be unique within a mountpoint and may contain up to 100 characters. A '-set' command that uses a pre-existing metadata name with new or different values will overwrite the previous values. By default, metadata is cached for 60 seconds so attachment of new or updated metadata to new files written to the mountpoint will be delayed until the cache is refreshed.

To add multiple values for a name, simply append them with a separator. For instance, a mountpoint that had files that applied to both the legal department and the accounting department might have a metadata value of '--value=Legal, Accounting'. Metadata values may contain up to 1536 characters. A "no space left on device" error will be returned if the size limit is exceeded. ASCII control characters are not supported and any preceding or trailing whitespace will be discarded from both the name and the value.

To view all the existing metadata name/value pairs for a mountpoint, the 'get' action for the cfsadmin metadata utility can be utilized. For instance, to view all metadata for the legal mountpoint the following command would be executed:

# cfs-admin metadata --get /mnt/legal

Similarly, if you wanted to see the value for a particular metadata name, you would include the name in the command. For example:

# cfs-admin metadata --get --name=Department /mnt/legal

To delete a metadata attribute for a mountpoint, the 'delete' action for the cfs-admin metadata utility can be executed. For instance, to delete the Department=Legal attribute example added to the legal mountpoint above, the following command could be executed:

Version 2.6

February 2011

# cfs-admin metadata --delete --name=Department --value=Legal /mnt/legal

3.6.2. DX Storage Lifecycle Policies

In addition to custom metadata, the 'cfs-admin policy' utility allows a root user to administer the content storage constraint policy for newly created files that are ready to be stored in DX Storage (last file close plus 5 seconds). To add a new policy to a mountpoint, the following command can be executed:

$ sudo su # cfs-admin policy --add [action-options]<mount-root>

where <mount-root> is the full path for the CFS mountpoint to which the policy should be attached. The available action-options for a policy are:

• --reps=# : The number of file replicas that should be created in DX Storage. Valid entries are between 1 and 16.

• --del=yes|no: Whether or not the file should be deletable within DX Storage. Value can be either "yes" or "no". Immutable DX Storage objects can not be deleted via the background CFS delete cleanup process (garbage collection).

• --lifepoint=#: The number of the lifepoint to affect with a particular action. This number is not indicative of the order in which a lifepoint will be considered but simply a naming mechanism to distinguish one lifepoint from another. You can use the 'print' command to determine what lifepoint number is associated with a particular policy.

• --span=TIME: When, measured from when the file is stored, the lifepoint will cease to be in effect, specified as a number followed by the unit of measure where allowable units are: d = day, w = week, m = month, y = year (e.g. "2y" for two years). This option should be omitted for lifepoints that should not have an end date. There can be only one such lifepoint.

All times are measured from the date the file is created in DX Storage and are not relative to each other when using multiple lifepoints. For instance a lifecycle policy that states a file should have 3 replicas for the first year and 2 replicas for a year after that would require two lifepoints: the first with a span=1y and the second with a span=2y, representing the cumulative time from the point the file was stored.

To provide a specific example, a single policy that set all files as not deletable with 2 replicas for one year for the legal mountpoint would be implemented as follows:

# cfs-admin policy --add --reps=2 --del=no --span=1y

--lifepoint=1 /mnt/legal

Confirmation of a policy will be printed at the end of every addition but you can view the policies in effect at any time by using the print command, as follows:

# cfs-admin policy --print /mnt/legal

To modify an existing lifepoint, use the same action options detailed above for the 'add' command and just include the change to the updated attribute. The lifepoint option is required with a modify command to ensure the correct lifepoint is updated when more than one is in effect. If the specified lifepoint does not exist, the system will return an 'IndexError'. Existing attributes not specified in the modify command will remain the same as previously stored. For instance, to lengthen only the span for the lifepoint set in the example above but leave the replica and delete policy the same, the following command would be executed:

Version 2.6

February 2011

# cfs-admin policy --modify --span=2y --lifepoint=1 /mnt/legal

Finally, to delete a policy, the 'delete' action can be executed as follows by specifying which lifepoint to delete. If the specified lifepoint does not exist, the system will return a 'No such lifepoint' error. If more than one lifepoint is in effect, you should be sure to validate the remaining lifepoints printed after the delete to ensure the end-to-end lifecycle policy is as intended. If you delete a lifepoint in the middle of a series of lifepoints, you may need to adjust other lifepoints to rearrange their order.

# cfs-admin policy --delete --lifepoint=1 /mnt/legal

Version 2.6

February 2011

Appendix A. Temp and Logging Space Configuration

Every file that is written to or read from the DX Storage cluster will be cached in the location specified for the spooler directory during the installation process. The spooler directory is integral to the internal function of DX Storage CFS and should never be manually manipulated, particularly while CFS is actively running. Consequently, it is important that the directory be properly configured prior to using CFS. The spool directory's file system should be configured with enough space to keep the default maximum of 100,000 files and should be no smaller than 50GB. It must also be large enough to hold 2X the largest data file being written. When allocating disk space, be sure to allow for adequate logging space if you selected to log to a file (max of 8 log files with 512MB each).

In addition to the spool and logging space required for CFS, CNS maintains a lookup file in /var/lib. The size of this file will grow approximately 1Gb per 10 million files stored in the name space. To ensure there is adequate space for swapping of this file, 2Gb of free space must be available per 10 million files written to the name space.

If files in your system will be large, verify that the file system type will support sufficiently large files and extended attributes. The mount option 'user_xattr' must be included in the /etc/fstab entry for the spooler partition. Ext3 is recommended, although there are plenty of valid choices. A good feature comparison is available on the Wikipedia at the following URL: http://en.wikipedia.org/wiki/

Comparison_of_file_systems

Note

If you are using an Ext3 file system, the 'dir_index' option should be enabled by default but should be verified for performance, particularly in situations where there are a large number of files per directory. To determine whether the option is already enabled, run the following command:

sudo debugfs -R feature /dev/sda1

where '/dev/sda1' represents the disk partition where the temp space resides. If the returned list contains a 'dir_index' option, no further action is needed. Otherwise you can enable this option using the following command, again replacing /dev/sda1 with the disk partition where the temp space resides:

sudo tunefs -O dir_index /dev/sda1

If the available space on the partition where the spool/cache directories reside gets too low or the file count exceeds 100,000, cached files with the oldest access dates will be flushed until adequate space has been regained. If more than 65% of the available space for the configured partition is utilized, the least recently used cached files are flushed until the cache is reduced by 10%. When 80% of available spool space is consumed, CFS will start returning out of space errors. The evaluation of available space considers all usage for the partition, not just CFS usage, and includes any reserved space you have explicitly dedicated for other purposes. For this reason, best practice is to configure a dedicated partition for the CFS spool/cache with a minimum of 50GB of available space. Environments writing large files under high throughput conditions may need to tune their cache eviction percentages to prevent rapid consumption of spool space and should contact their support resource for further assistance.

Version 2.6

February 2011

Dell DX6000G, DX Storage Cluster File Server Setup And Configuration Manual

Specifications and Main Features

Frequently Asked Questions

User Manual