Dell DX6000, DX6004S, DX6012S, DX Storage Cluster Services Node 2.0 Installation And Configuration Manual

Cluster Services Node (CSN)

Installation and Configuration Guide

Version 2.0

Cluster Services Node (CSN) Installation and Configuration Guide: Version 2.0

No part of this document may be reproduced, transmitted, or transcribed without the written consent of Caringo, Inc.

Table of Contents

1. Introduction to the DX Object Storage Cluster Services Node ................................................. 1

1.1. DX Object Storage Cluster Services Node Overview ................................................... 1

1.2. Components ............................................................................................................. 1

1.3. About this Document ................................................................................................. 2

1.3.1. Audience ....................................................................................................... 2

1.3.2. Scope ............................................................................................................ 2

2. Installation and Configuration ............................................................................................... 3

2.1. Requirements ........................................................................................................... 3

2.1.1. Operating System ........................................................................................... 3

2.1.2. Network ......................................................................................................... 3

2.2. Primary vs. Secondary .............................................................................................. 3

2.3. Installation Steps ...................................................................................................... 4

2.3.1. Primary CSN Configuration ............................................................................. 4

2.3.2. Secondary CSN Configuration ......................................................................... 6

2.3.3. Configuring Object Storage Cluster Licensing ................................................... 6

2.4. Booting DX Storage Nodes ....................................................................................... 9

2.5. CSN Hardware Validation ........................................................................................ 10

2.6. Upgrading the CSN ................................................................................................. 10

2.6.1. Backup Compatibility Across Software Versions ............................................. 11

2.7. CSN-Reset ............................................................................................................. 11

3. Running and Managing the CSN ........................................................................................ 12

3.1. Modifying CSN Network Configuration ...................................................................... 12

3.1.1. Updating the Internal Interface ...................................................................... 13

3.2. Modifying SNMP Access ......................................................................................... 14

3.3. Updating the DX Storage software version ............................................................... 14

3.4. Editing User Access ................................................................................................ 15

3.5. Configuration File Backup and Restore ..................................................................... 16

3.5.1. Backup Manifest ........................................................................................... 17

3.5.2. Automatic Backups ....................................................................................... 18

3.5.3. Manual Backups ........................................................................................... 18

3.5.4. Restoring a Backup ...................................................................................... 18

3.5.5. Failover ........................................................................................................ 18

3.6. Configuring the SCSP Proxy .................................................................................... 20

3.7. Syslog Configuration ............................................................................................... 22

3.8. Password Management ........................................................................................... 22

4. Running and Managing DX Storage .................................................................................... 23

4.1. Adding DX Storage Nodes ....................................................................................... 23

4.2. Updating Configuration Settings ............................................................................... 23

4.3. DX Storage User Access ......................................................................................... 24

4.4. DX Storage Admin Console ..................................................................................... 25

4.5. Gathering SNMP Statistics ...................................................................................... 26

4.5.1. SNMP Command Examples .......................................................................... 26

5. Configuring and Managing DX Content Router .................................................................... 27

5.1. Configuring DX Content Router ................................................................................ 27

5.1.1. Publisher ...................................................................................................... 27

5.1.2. Replicator ..................................................................................................... 28

5.2. Managing DX Content Router .................................................................................. 30

5.2.1. Publisher Console ........................................................................................ 30

5.2.2. Gathering SNMP Statistics ............................................................................ 30

A. Open Source Software ...................................................................................................... 32

Version 2.0

December 2010

Chapter 1. Introduction to the DX Object Storage Cluster Services Node

1.1. DX Object Storage Cluster Services Node Overview

The DX Storage Cluster Services Node (CSN) is an integrated services node that centralizes installation and configuration of both the network services required to run a DX Storage cluster and the software used to interface with it.

1.2. Components

The DX Storage CSN infrastructure is made up of the following components:

1. A collection of required network services configured to support a DX Storage cluster, including DHCP, PXE network boot, TFTP, syslog, NTP and several others

2. An integrated PXE network boot and configuration server for DX Storage nodes booted onto the DX Storage CSN's internal network

3. Several SNMP MIBs with operational and status information for all nodes in the associated DX Storage cluster as well as the DX Content Router Publisher and Replicator services

4. Integrated installation and configuration of DX Content Router and the SCSP Proxy

5. An administrative web console that provides easy configuration of common settings and parameters as well as access to useful utilities like updating DX Storage software versions and backup and restore of configuration files

Version 2.0

December 2010

1.3. About this Document

1.3.1. Audience

This document is intended for people in the following roles.

1. Storage system administrators

2. Network administrators

3. Technical architects

Throughout this document, the storage system administrator and network administrator roles will be referred to as the administrator. DX Storage CSN administrators are normally responsible for installation and configuration, license management, monitoring storage system health, software upgrades, backup/recovery, and capacity management.

1.3.2. Scope

This document covers the steps needed to install and configure the DX Storage CSN. The reader is expected to have a background in RHEL system administration, TCP/IP networking, and monitoring via SNMP.

Version 2.0

December 2010

Chapter 2. Installation and Configuration

2.1. Requirements

2.1.1. Operating System

DX Storage CSN has been developed and tested with 64-bit RHEL 5.5; other RHEL versions or Linux distributions are not currently supported. Subsequent installation instructions will assume a pre-installed RHEL Linux environment with either internet connectivity or an alternately configured RHEL yum repository for use in installing required 3rd party packages.

2.1.2. Network

The CSN requires access to both the external network as well as a dedicated internal network. The internal private network ensures the DX Storage cluster traffic is protected from unauthorized access and also that the external network is isolated from both the PXE network boot server (including DHCP) and cluster multicast traffic. Allocation of the address space in the internal network is broken down as follows, depending on the network size selected during initial configuration (small or large):

Network Size CSN 3rd Party DHCP DX Storage

Netboot

small (/24) x.y.z.0-16 x.y.z.17-32 x.y.z.33-48 x.y.z.49-254 large (/16) x.y.0.0-254 x.y.1.0-254 x.y.2.0-254 x.y.3.0-

x.y.255.254

The CSN range provides IPs for the various services on the Primary and Secondary CSNs. The 3rd Party range is provided for 3rd party applications that need to run on the internal network to interface with the DX Storage cluster. The DHCP range provides an initial IP to DX Storage nodes during their initial boot until permanent addresses can be assigned to each DX Storage process by the CSN. Other applications using the CSN's DHCP server on the internal network will reduce the number of DX Storage that can be booted at the same time. The Netboot range is used to provide the permanent IPs for all DX Storage processes.

From a network configuration standpoint, the CSN will automatically allocate half of the detected NICs to the internal network and half of detected NICs to the external network. All NICs allocated to a network are bonded into a bond interface using Linux mode 6, or balance-alb, bonding. In configurations where there is both an onboard NIC card as well as an add-on card and the hardware supports detection of the difference between the two, the internal/external network allocation will be disbursed across both cards for redundancy. The CSN NIC assignments may not match the physical NIC ports.

Note

Network switch configuration is outside the scope of the CSN software. Switches must be properly configured by an experienced system administrator to ensure correct connectivity and bonding mode support. In particular, administrators must be careful to not connect the configured internal network ports to the external network.

2.2. Primary vs. Secondary

With two CSNs on an internal network, there must be one CSN designated as the primary. The primary CSN is responsible for responding to DHCP requests for the internal network and also

Version 2.0

December 2010

listening for all communication on the well-known IP addresses for the internal and external network. When a secondary CSN joins the internal network, it registers with the primary CSN via a privileged SSH channel using the primary root password entered during the initial configuration process. This allows the primary to sync needed information to the secondary, specifically the Primary's Backup Manifest UUID. The secondary CSN provides redundancy for all services on the primary CSN in the event of a disaster and also provides scalability for incoming SCSP requests to the SCSP Proxy.

2.3. Installation Steps

The CSN distribution is available as a collection of RPM packages that are installed with a shell script. The packages and their dependencies must be installed as the 'root' user from an attached monitor and keyboard with at least one NIC connected to the external network using the following steps. Please reference the Upgrade section for additional steps required for upgrading the software:

1. Copy the distributed zip file to your Cluster Services Node and unzip it into the directory of your choice. The following example command would unzip the version of the software which was previously copied to the /tmp/csninstall directory:

$ sudo su # cd /tmp/csninstall # unzip caringo-csn-2.0.0.zip

Note

The full zip name may include additional descriptive labeling related to versions or distribution channel depending on how the CSN software release is delivered.

2. Install the CSN by running the self-extracting script from the directory location where the shell script was unzipped. For instance, continuing the example above from the /tmp/csninstall directory:

# cd caringo-csn-2.0.0 # ./caringo-csn-bundle-install.sh

This command will initiate installation of the CSN and its dependent packages. When the installation is complete, the following prompt will display:

Would you like to proceed with CSN network configuration? (yes/no):

Answer 'yes' to proceed with configuring the CSN. If you answer 'no', you may return to the configuration screen at a later time by running the command: /opt/caringo/csn/bin/ firstcsnboot.

If you run the configuration at a later time, it must still be from an attached monitor and keyboard.

2.3.1. Primary CSN Configuration

After installing the CSN, you will automatically be prompted to enter some minimal configuration data to configure the server on the overall network. Network settings are central to all CSN services and should be planned with care in advance by an administrator knowledgeable about the environment. The initial configuration process is only required once after the initial installation. Any necessary subsequent updates to the initial configuration parameters can be made from the CSN Console.

Version 2.0

December 2010

Several prompts will suggest a default value in brackets that can be accepted by simply pressing enter.

Is this the Primary CSN (yes/no)? [yes]:: This prompt allows specification of whether or not the CSN is a primary or secondary CSN. The default value is yes. Administrators should take care to ensure that only a single primary CSN is configured on the internal network to prevent conflicts with both DHCP and DX Storage netboot configuration. The primary server must be configured prior to configuration of a secondary. DHCP is not started on the secondary CSN.

Half of the NIC ports on this system will be bonded and assigned to the external network. The following questions configure the external network: Enter the CSN IP address []:

This parameter requires entry of an external IP address for the CSN node. The entered address must be in a valid w.x.y.z format and must not already be in use on the network.

Enter the cluster IP address. This IP address will remain with the Primary CSN in the event of a CSN failover []: This parameter requires entry of an external IP

address for the CSN cluster address. This well-known address remains with the primary CSN in the event of a failover, meaning the cluster can always be reached at this address. The entered address must be in a valid w.x.y.z format and must not already be in use on the network.

Enter the subnet mask [255.255.255.0]: This parameter requires entry of the subnet mask for the external network that corresponds with the entered IP address. The default is

255.255.255.0.

Enter the gateway IP address []: This parameter requires entry of the gateway associated with the entered IP address for the external interface.

Half of the NIC ports on this system will be bonded and assigned to the internal network. The following questions configure the internal network: Enter the network address, e.g. 192.168.100.0 (small network),

192.168.0.0, 172.20.0.0 (large network) []:

This parameter allows specification of the network interface that should be used for the internal network. Enter an interface in the format of 192.168.100.0 to use a small interface that will support 128 DX Storage nodes. Enter an interface in the format of 192.168.0.0 or 172.20.0.0 to use a larger network that will support much larger DX Storage clusters. The entered interface will be divided between the CSN(s) and privileged applications on the internal network and the DX Storage nodes. The initial configuration process automatically creates multiple alias IP addresses on the internal network for use by various system services and reserves similar IP addresses for a Secondary CSN.

Enter a list of IP addresses (separated by spaces) for external name servers [8.8.8.8 8.8.4.4]: This parameter allows specification of one or more DNS servers

for the external interface. Entries must be separated by spaces. Publicly available name servers have been defaulted.

Enter a list of IP addresses or server names (separated by a space) for external time servers [0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org]: This

parameter allows specification of one or more NTP servers for the external interface. The defaults are public NTP servers.

Enter a unique storage cluster name. This name cannot be changed once assigned. A fully qualified domain name is recommended []: This parameter

Version 2.0

December 2010

is used to populate the name of the storage cluster on the admin console as well as in stream metadata for all streams written to the local cluster. The CSN also uses this name to detect all the nodes participating in the cluster. For all of these purposes, the name must be unique. An IANA fully qualified domain name is recommended.

Are these values correct (yes/no)? This last step allows you to review the values entered for all prompts before submitting them. Answering yes will allow the initial configuration process to proceed with network and service configuration, resulting in a fully functional CSN. Answering no to the final initial configuration prompt will restart the initial configuration script at the first prompt, with the previously entered values populated.

At the completion of a successful initial configuration, the CSN will immediately reboot the server to initialize all services. When the node comes back up all network services will be configured and available including SNMP, syslog, DHCP, DNS, NTP, and firewall. Additionally, the CSN Console will be available, the SCSP Proxy will be configured and started and the DX Content Router Publisher will be configured and started.

2.3.2. Secondary CSN Configuration

Once a Primary CSN has been configured on the network, a Secondary CSN may also be configured. The Secondary only requires entry of a single unique external IP address and identification of the internal interface that is already defined on the Primary. The Secondary will then pull much of its network configuration data from the Primary. To facilitate this, a one-time use of the Primary's root password is required as follows:

Additional information about the network will be obtained from the primary CSN Please enter the primary csn root password []: Please re-enter the primary csn root password []:

Taken from the confirmation at the end of the initial configuration script, the Secondary configuration parameters are simply:

Primary: no External CSN IP address: 192.168.66.11 Internal network interface: 172.20.20.0 Are these values correct (yes/no)?

2.3.3. Configuring Object Storage Cluster Licensing

Subsequent to rebooting after configuration and prior to booting any storage cluster nodes, an aggregate license file comprised of any capacity keys included with your hardware or sent by your hardware manufacturer must be entered and published via the administrative CSN Console. The console allows web-based configuration of all CSN services after the initial network configuration. Please reference subsequent chapters for a full overview of console capabilities.

To access the console initially for license publication, enter the following address:http://

<CSNExternalIP>:8090 . You will be required to authenticate prior to being granted access. The

username for the console is 'admin' and the default password is 'dell'. Once authenticated, click on the Licensing link under the Content Storage tab to access the licensing interface.

Version 2.0

December 2010

After installation, a license shell is defaulted into the licensing interface. If published as is, this license will provide 0 TB of capacity for storing content. To correctly publish a license the following high-level steps are required:

1. Enter License Contact and Storage Information

2. Enter capacity keys one at a time

3. After all keys have been entered, publish the license

Version 2.0

December 2010

2.3.3.1. Contact and Storage Information

The license information is displayed once the license is published as part of the license details popup on the DX Storage admin console. To add or update this information, click the 'Edit' button under the License Contact and Storage Information section of the interface and enter the desired company name, address and cluster description information. Only the company name is required but populating the address and description fields is recommended to assist with easily distinguishing licenses from multiple locations. When all the desired information has been entered, click the 'Update' button to save the changes.

2.3.3.2. Entering License Keys

The license interface supports entry of two different types of license keys: hardware keys and capacity keys. Hardware keys are used to determine the validity of the host hardware for each storage node at boot. Unauthorized hardware will log a critical error followed by a forced shutdown. Hardware keys are updated infrequently and, in most installs, pre-populated so additional entry is not required. Once entered hardware keys cannot be deleted. Administrators who receive hardware validation errors for storage nodes that are believed to be authorized should contact their support representative.

Capacity keys are more commonly entered than hardware keys. Printed capacity keys are delivered in conjunction with a hardware shipment. The capacity increment for an individual capacity key usually corresponds to the capacity of an individual storage node. Each key represents a preallocated amount of license space from 3TB to 480 TB. The sum of all entered capacity keys determines the total amount of available licensed capacity for the cluster as a whole. The total capacity for all keys is displayed at the bottom of the license key grid for easy validation.

Both types of license keys are represented by an encoded string broken down into 6 groups of 5 characters each, separated by dashes (XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX). To enter a new key, click the 'Add License Key' button on the main licensing page and enter the capacity key exactly as printed. When complete, click the 'Add' button to submit the key. Invalid or mistyped keys will result in an error on the screen. To proceed, either correct the key entry or click 'Cancel'

Version 2.0

December 2010

+ 24 hidden pages

Dell DX6000, DX6004S, DX6012S, DX Storage Cluster Services Node 2.0 Installation And Configuration Manual

Specifications and Main Features

Frequently Asked Questions

User Manual