How it Works
HP
Loading...
IBRIX X9730

HP IBRIX X9720, IBRIX X9730 System Administrator Manual

HP IBRIX X9720/X9730 Network Storage System Administrator Guide
Abstract
This guide describes tasks related to cluster configuration and monitoring, system upgrade and recovery, hardware component replacement, and troubleshooting. It does not document X9000 file system features or standard Linux administrative tools and commands. For information about configuring and using X9000 software file system features, see the HP IBRIX X9000 Network Storage System File System User Guide.
support/manuals. In the storage section, select NAS Systems and then select HP X9000 Network Storage Systems from the
IBRIX Storage Systems section.
HP Part Number: AW549-96035 Published: June 2012 Edition: 9
© Copyright 2009, 2012 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.
Acknowledgments
Microsoft® and Windows® are U.S. registered trademarks of Microsoft Corporation.
UNIX® is a registered trademark of The Open Group.
Warranty
WARRANTY STATEMENT: To obtain a copy of the warranty for this product, see the warranty information website:
http://www.hp.com/go/storagewarranty
Revision History
DescriptionSoftware
Version
DateEdition
Initial release of the X9720 Network Storage System.5.3.1December 20091
Added network management and Support ticket.5.4April 20102
Added Fusion Manager backup, migration to an agile Fusion Manager configuration, software upgrade procedures, and system recovery procedures.
5.4.1August 20103
Revised upgrade procedure.5.4.1August 20104
Added information about NDMP backups and configuring virtual interfaces, and updated cluster procedures.
5.5December 20105
Updated segment evacuation information.5.5March 20116
Revised upgrade procedure.5.6April 20117
Added or updated information about the agile Fusion Manager, Statistics tool, Ibrix Collect, event notification, capacity block installation, NTP servers, upgrades.
6.0September 20118
Added or updated information about X9730 systems, hardware monitoring, segment evacuation, HP Insight Remote Support, software upgrades, events, Statistics tool.
6.1June 20129
Contents
1 Product description...................................................................................11
System features.......................................................................................................................11
System components.................................................................................................................11
HP X9000 software features.....................................................................................................11
High availability and redundancy.............................................................................................12
2 Getting started.........................................................................................13
Setting up the X9720/X9730 Network Storage System................................................................13
Installation steps................................................................................................................13
Additional configuration steps.............................................................................................13
Logging in to the system..........................................................................................................14
Using the network..............................................................................................................14
Using the TFT keyboard/monitor..........................................................................................14
Using the serial link on the Onboard Administrator.................................................................15
Booting the system and individual server blades.........................................................................15
Management interfaces...........................................................................................................15
Using the GUI...................................................................................................................15
Customizing the GUI..........................................................................................................19
Adding user accounts for GUI access...................................................................................19
Using the CLI.....................................................................................................................20
Starting the array management software...............................................................................20
X9000 client interfaces.......................................................................................................20
X9000 software manpages......................................................................................................21
Changing passwords..............................................................................................................21
Configuring ports for a firewall.................................................................................................21
Configuring NTP servers..........................................................................................................22
Configuring HP Insight Remote Support on X9000 systems...........................................................23
Configuring the X9000 cluster for Insight Remote Support.......................................................23
Configuring Insight Remote Support for HP SIM 7.1 and IRS 5.7...............................................27
Configuring Insight Remote Support for HP SIM 6.3 and IRS 5.6..............................................29
Testing the Insight Remote Support configuration....................................................................32
Updating the Phone Home configuration...............................................................................32
Disabling Phone Home.......................................................................................................32
Troubleshooting Insight Remote Support................................................................................32
3 Configuring virtual interfaces for client access..............................................34
Network and VIF guidelines.....................................................................................................34
Creating a bonded VIF............................................................................................................34
Configuring standby backup nodes...........................................................................................34
Configuring NIC failover.........................................................................................................35
Configuring automated failover................................................................................................35
Example configuration.............................................................................................................35
Specifying VIFs in the client configuration...................................................................................36
Support for link state monitoring...............................................................................................36
4 Configuring failover..................................................................................37
Agile management consoles....................................................................................................37
Agile Fusion Manager modes..............................................................................................37
Agile Fusion Manager and failover......................................................................................37
Viewing information about Fusion Managers.........................................................................38
Cluster high availability...........................................................................................................38
Failover modes..................................................................................................................38
What happens during a failover..........................................................................................38
Contents 3
Setting up automated failover..............................................................................................39
Configuring standby pairs..............................................................................................39
Identifying power sources...............................................................................................39
Turning automated failover on and off..............................................................................40
Manually failing over a file serving node..............................................................................40
Failing back a file serving node...........................................................................................41
Using network interface monitoring......................................................................................41
Setting up HBA monitoring..................................................................................................43
Discovering HBAs..........................................................................................................43
Identifying standby-paired HBA ports...............................................................................44
Turning HBA monitoring on or off....................................................................................44
Deleting standby port pairings........................................................................................44
Deleting HBAs from the configuration database.................................................................44
Displaying HBA information............................................................................................44
Checking the High Availability configuration.........................................................................45
5 Configuring cluster event notification...........................................................47
Cluster events.........................................................................................................................47
Setting up email notification of cluster events..............................................................................47
Associating events and email addresses................................................................................47
Configuring email notification settings..................................................................................48
Dissociating events and email addresses...............................................................................48
Testing email addresses......................................................................................................48
Viewing email notification settings........................................................................................48
Setting up SNMP notifications..................................................................................................49
Configuring the SNMP agent...............................................................................................49
Configuring trapsink settings................................................................................................50
Associating events and trapsinks..........................................................................................50
Deleting elements of the SNMP configuration........................................................................50
Listing SNMP configuration information.................................................................................50
6 Configuring system backups.......................................................................51
Backing up the Fusion Manager configuration............................................................................51
Using NDMP backup applications............................................................................................51
Configuring NDMP parameters on the cluster........................................................................52
NDMP process management...............................................................................................52
Viewing or canceling NDMP sessions..............................................................................52
Starting, stopping, or restarting an NDMP Server..............................................................53
Viewing or rescanning tape and media changer devices.........................................................53
NDMP events....................................................................................................................54
7 Creating hostgroups for X9000 clients.........................................................55
How hostgroups work..............................................................................................................55
Creating a hostgroup tree........................................................................................................55
Adding an X9000 client to a hostgroup.....................................................................................56
Adding a domain rule to a hostgroup........................................................................................56
Viewing hostgroups.................................................................................................................56
Deleting hostgroups................................................................................................................56
Other hostgroup operations.....................................................................................................57
8 Monitoring cluster operations.....................................................................58
Monitoring the system status.....................................................................................................58
Monitoring intervals...........................................................................................................58
Viewing storage monitoring output.......................................................................................58
Monitoring X9720/X9730 hardware.........................................................................................58
Monitoring servers and chassis............................................................................................58
Monitoring chassis and chassis components.....................................................................60
4 Contents
Monitoring storage and storage components.........................................................................61
Monitoring the status of file serving nodes..................................................................................64
Monitoring cluster events.........................................................................................................65
Viewing events..................................................................................................................65
Removing events from the events database table....................................................................66
Monitoring cluster health.........................................................................................................66
Health checks....................................................................................................................66
Health check reports..........................................................................................................67
Viewing logs..........................................................................................................................69
Viewing and clearing the Integrated Management Log (IML).........................................................69
Viewing operating statistics for file serving nodes........................................................................69
9 Using the Statistics tool..............................................................................71
Installing and configuring the Statistics tool................................................................................71
Installing the Statistics tool...................................................................................................71
Enabling collection and synchronization................................................................................71
Upgrading the Statistics tool from X9000 software 6.0................................................................72
Using the Historical Reports GUI...............................................................................................72
Generating reports.............................................................................................................73
Deleting reports.................................................................................................................74
Maintaining the Statistics tool...................................................................................................74
Space requirements............................................................................................................74
Updating the Statistics tool configuration...............................................................................74
Changing the Statistics tool configuration..............................................................................75
Fusion Manager failover and the Statistics tool configuration...................................................75
Checking the status of Statistics tool processes.......................................................................76
Controlling Statistics tool processes.......................................................................................76
Troubleshooting the Statistics tool..............................................................................................76
Log files.................................................................................................................................77
Uninstalling the Statistics tool...................................................................................................77
10 Maintaining the system............................................................................78
Shutting down the system.........................................................................................................78
Shutting down the X9000 software......................................................................................78
Powering off the system hardware........................................................................................79
Starting up the system.............................................................................................................80
Powering on the system hardware........................................................................................80
Powering on after a power failure........................................................................................80
Starting the X9000 software................................................................................................80
Powering file serving nodes on or off.........................................................................................80
Performing a rolling reboot......................................................................................................81
Starting and stopping processes...............................................................................................81
Tuning file serving nodes and X9000 clients...............................................................................81
Migrating segments................................................................................................................83
Removing a node from the cluster.............................................................................................83
Removing storage from the cluster.............................................................................................83
Maintaining networks..............................................................................................................86
Cluster and user network interfaces......................................................................................86
Adding user network interfaces............................................................................................86
Setting network interface options in the configuration database................................................87
Preferring network interfaces................................................................................................87
Unpreferring network interfaces...........................................................................................89
Making network changes....................................................................................................89
Changing the IP address for a Linux X9000 client..............................................................89
Changing the cluster interface.........................................................................................89
Managing routing table entries.......................................................................................89
Contents 5
Deleting a network interface...........................................................................................90
Viewing network interface information..................................................................................90
11 Migrating to an agile Fusion Manager configuration....................................91
Backing up the configuration....................................................................................................91
Performing the migration..........................................................................................................91
Testing failover and failback of the agile Fusion Manager............................................................93
Converting the original management console node to a file serving node hosting the agile Fusion
Manager...............................................................................................................................94
12 Upgrading the X9000 software to the 6.1 release.......................................95
Online upgrades for X9000 software 6.0 to 6.1.........................................................................95
Preparing for the upgrade...................................................................................................95
Performing the upgrade......................................................................................................96
After the upgrade..............................................................................................................96
Offline upgrades for X9000 software 5.6.x or 6.0.x to 6.1..........................................................97
Preparing for the upgrade...................................................................................................97
Performing the upgrade......................................................................................................98
After the upgrade..............................................................................................................98
Upgrading Linux X9000 clients.................................................................................................99
Installing a minor kernel update on Linux clients...................................................................100
Upgrading Windows X9000 clients........................................................................................100
Upgrading pre-6.0 file systems for software snapshots...............................................................100
Troubleshooting upgrade issues..............................................................................................102
Automatic upgrade..........................................................................................................102
Manual upgrade.............................................................................................................102
Offline upgrade fails because iLO firmware is out of date......................................................103
Node is not registered with the cluster network ...................................................................103
File system unmount issues.................................................................................................103
Moving the Fusion Manager VIF to bond1..........................................................................104
13 Upgrading the X9000 software to the 5.6 release.....................................106
Automatic upgrades..............................................................................................................106
Manual upgrades.................................................................................................................107
Preparing for the upgrade.................................................................................................107
Saving the node configuration...........................................................................................107
Performing the upgrade....................................................................................................108
Restoring the node configuration........................................................................................108
Completing the upgrade...................................................................................................108
Troubleshooting upgrade issues..............................................................................................109
Automatic upgrade..........................................................................................................109
Manual upgrade.............................................................................................................110
14 Upgrading the X9000 software to the 5.5 release.....................................111
Automatic upgrades..............................................................................................................111
Manual upgrades.................................................................................................................112
Standard upgrade for clusters with a dedicated Management Server machine or blade............112
Standard online upgrade.............................................................................................112
Standard offline upgrade.............................................................................................114
Agile upgrade for clusters with an agile management console configuration............................116
Agile online upgrade...................................................................................................116
Agile offline upgrade...................................................................................................120
Troubleshooting upgrade issues..............................................................................................123
15 Licensing.............................................................................................124
Viewing license terms............................................................................................................124
Retrieving a license key.........................................................................................................124
6 Contents
Using AutoPass to retrieve and install permanent license keys......................................................124
16 Upgrading the system hardware and firmware..........................................125
Upgrading firmware..............................................................................................................125
Adding performance modules on X9730 systems......................................................................125
Adding new server blades on X9720 systems...........................................................................125
Adding capacity blocks on X9720 systems...............................................................................127
Where to install the capacity blocks...................................................................................128
Installation procedure.......................................................................................................129
Enabling monitoring for the new storage.............................................................................134
Setting the chassis name of the new capacity block..............................................................134
Removing server blades.........................................................................................................135
Removing capacity blocks......................................................................................................135
17 Troubleshooting....................................................................................136
Collecting information for HP Support with Ibrix Collect.............................................................136
Collecting logs................................................................................................................136
Deleting the archive file....................................................................................................137
Downloading the archive file.............................................................................................137
Configuring Ibrix Collect...................................................................................................138
Viewing data collection information....................................................................................139
Viewing data collection configuration information................................................................139
Adding/deleting commands or logs in the XML file..............................................................139
Troubleshooting X9720 systems..............................................................................................139
Escalating issues..............................................................................................................139
Useful utilities and processes.............................................................................................140
exds_stdiag utility........................................................................................................140
exds_netdiag utility.....................................................................................................141
exds_netperf utility......................................................................................................141
Accessing the Onboard Administrator.....................................................................................142
Accessing the OA through the network...............................................................................142
Access the OA Web-based administration interface.........................................................142
Accessing the OA through the serial port............................................................................143
Accessing the OA through the service port..........................................................................143
Using hpacucli – Array Configuration Utility (ACU)...............................................................143
POST error messages............................................................................................................143
X9730 controller error messages.............................................................................................143
X9720 LUN layout................................................................................................................146
X9720 component monitoring................................................................................................146
Identifying failed I/O modules on an X9700cx chassis..............................................................146
Failure indications............................................................................................................147
Identifying the failed component........................................................................................147
Re-seating an X9700c controller........................................................................................150
Viewing software version numbers..........................................................................................151
Troubleshooting specific issues................................................................................................151
Software services.............................................................................................................151
Failover..........................................................................................................................151
Windows X9000 clients...................................................................................................152
Mode 1 or mode 6 bonding.............................................................................................152
Onboard Administrator is unresponsive...............................................................................153
X9000 RPC call to host failed............................................................................................153
Degrade server blade/Power PIC.......................................................................................153
LUN status is failed..........................................................................................................153
Apparent failure of HP P700m...........................................................................................154
X9700c enclosure front panel fault ID LED is amber..............................................................155
Spare disk drive not illuminated green when in use..............................................................155
Contents 7
Replacement disk drive LED is not illuminated green.............................................................155
X9700cx GSI LED is amber...............................................................................................155
X9700cx drive LEDs are amber after firmware is flashed.......................................................155
Configuring the Virtual Connect domain..................................................................................155
Synchronizing information on file serving nodes and the configuration database...........................156
18 Recovering the X9720/X9730 Network Storage System.............................158
Obtaining the latest IBRIX X9000 software release....................................................................158
Preparing for the recovery......................................................................................................158
Recovering an X9720 or X9730 file serving node.....................................................................159
Completing the restore .........................................................................................................165
Troubleshooting....................................................................................................................167
iLO remote console does not respond to keystrokes...............................................................167
19 Support and other resources...................................................................168
Contacting HP......................................................................................................................168
Related information...............................................................................................................168
HP websites.........................................................................................................................169
Rack stability........................................................................................................................169
Product warranties................................................................................................................169
Subscription service..............................................................................................................169
20 Documentation feedback.......................................................................170
A X9730 component and cabling diagrams..................................................171
Back view of the main rack....................................................................................................171
Back view of the expansion rack.............................................................................................172
X9730 CX I/O modules and SAS port connectors.....................................................................172
X9730 CX 1 connections to the SAS switches...........................................................................173
X9730 CX 2 connections to the SAS switches...........................................................................174
X9730 CX 3 connections to the SAS switches...........................................................................175
X9730 CX 7 connections to the SAS switches in the expansion rack............................................176
B X9730 spare parts list ............................................................................177
HP IBRIX X9730 Performance Chassis (QZ729A)......................................................................177
HP IBRIX X9730 140 TB MLStorage 2xBL Performance Module (QZ730A)....................................177
HP IBRIX X9730 210 TB ML Storage 2xBL Performance Module (QZ731A)....................................178
(QZ732A)...........................................................................................................................178
(QZ733A)...........................................................................................................................179
C X9720 component and cabling diagrams.................................................180
Base and expansion cabinets.................................................................................................180
Front view of a base cabinet..............................................................................................180
Back view of a base cabinet with one capacity block...........................................................181
Front view of a full base cabinet.........................................................................................182
Back view of a full base cabinet.........................................................................................183
Front view of an expansion cabinet ...................................................................................184
Back view of an expansion cabinet with four capacity blocks.................................................185
Performance blocks (c-Class Blade enclosure)............................................................................185
Front view of a c-Class Blade enclosure...............................................................................185
Rear view of a c-Class Blade enclosure...............................................................................186
Flex-10 networks...............................................................................................................186
Capacity blocks...................................................................................................................187
X9700c (array controller with 12 disk drives).......................................................................188
Front view of an X9700c..............................................................................................188
Rear view of an X9700c..............................................................................................188
X9700cx (dense JBOD with 70 disk drives)..........................................................................188
Front view of an X9700cx............................................................................................189
8 Contents
Rear view of an X9700cx.............................................................................................189
Cabling diagrams................................................................................................................189
Capacity block cabling—Base and expansion cabinets........................................................189
Virtual Connect Flex-10 Ethernet module cabling—Base cabinet.............................................190
SAS switch cabling—Base cabinet.....................................................................................191
SAS switch cabling—Expansion cabinet..............................................................................191
D X9720 spare parts list ............................................................................193
X9720 Network Storage System Base (AW548A).....................................................................193
X9700 Expansion Rack (AQ552A)..........................................................................................193
X9700 Server Chassis (AW549A)...........................................................................................194
X9700 Blade Server (AW550A).............................................................................................194
X9700 82TB Capacity Block (X9700c and X9700cx) (AQ551A).................................................195
X9700 164TB Capacity Block (X9700c and X9700cx) (AW598B)...............................................196
E Warnings and precautions.......................................................................198
Electrostatic discharge information..........................................................................................198
Preventing electrostatic discharge.......................................................................................198
Grounding methods.....................................................................................................198
Grounding methods.........................................................................................................198
Equipment symbols...............................................................................................................199
Weight warning...................................................................................................................199
Rack warnings and precautions..............................................................................................199
Device warnings and precautions...........................................................................................200
F Regulatory compliance notices..................................................................202
Regulatory compliance identification numbers..........................................................................202
Federal Communications Commission notice............................................................................202
FCC rating label..............................................................................................................202
Class A equipment......................................................................................................202
Class B equipment......................................................................................................202
Modification...................................................................................................................203
Cables...........................................................................................................................203
Canadian notice (Avis Canadien)...........................................................................................203
Class A equipment...........................................................................................................203
Class B equipment...........................................................................................................203
European Union notice..........................................................................................................203
Japanese notices..................................................................................................................204
Japanese VCCI-A notice....................................................................................................204
Japanese VCCI-B notice....................................................................................................204
Japanese VCCI marking...................................................................................................204
Japanese power cord statement.........................................................................................204
Korean notices.....................................................................................................................204
Class A equipment...........................................................................................................204
Class B equipment...........................................................................................................204
Taiwanese notices.................................................................................................................205
BSMI Class A notice.........................................................................................................205
Taiwan battery recycle statement........................................................................................205
Turkish recycling notice..........................................................................................................205
Vietnamese Information Technology and Communications compliance marking.............................205
Laser compliance notices.......................................................................................................205
English laser notice..........................................................................................................205
Dutch laser notice............................................................................................................206
French laser notice...........................................................................................................206
German laser notice.........................................................................................................206
Italian laser notice............................................................................................................207
Contents 9
Japanese laser notice.......................................................................................................207
Spanish laser notice.........................................................................................................207
Recycling notices..................................................................................................................208
English recycling notice....................................................................................................208
Bulgarian recycling notice.................................................................................................208
Czech recycling notice......................................................................................................208
Danish recycling notice.....................................................................................................208
Dutch recycling notice.......................................................................................................208
Estonian recycling notice...................................................................................................209
Finnish recycling notice.....................................................................................................209
French recycling notice.....................................................................................................209
German recycling notice...................................................................................................209
Greek recycling notice......................................................................................................209
Hungarian recycling notice...............................................................................................209
Italian recycling notice......................................................................................................210
Latvian recycling notice.....................................................................................................210
Lithuanian recycling notice................................................................................................210
Polish recycling notice.......................................................................................................210
Portuguese recycling notice...............................................................................................210
Romanian recycling notice................................................................................................211
Slovak recycling notice.....................................................................................................211
Spanish recycling notice...................................................................................................211
Swedish recycling notice...................................................................................................211
Battery replacement notices...................................................................................................212
Dutch battery notice.........................................................................................................212
French battery notice........................................................................................................212
German battery notice......................................................................................................213
Italian battery notice........................................................................................................213
Japanese battery notice....................................................................................................214
Spanish battery notice......................................................................................................214
Glossary..................................................................................................215
Index.......................................................................................................217
10 Contents
1 Product description
HP X9720 and X9730 Network Storage Systems are a scalable, network-attached storage (NAS) product. The system combines HP X9000 File Serving Software with HP server and storage hardware to create a cluster of file serving nodes.
System features
The X9720 and X9730 Network Storage Systems provide the following features:
• Segmented, scalable file system under a single namespace
• NFS, CIFS, FTP, and HTTP support for accessing file system data
• Centralized CLI and GUI for cluster management
• Policy management
• Continuous remote replication
• Dual redundant paths to all storage components
• Gigabytes-per-second of throughput
IMPORTANT: It is important to keep regular backups of the cluster configuration. See “Backing
up the Fusion Manager configuration” (page 51) for more information.
System components
IMPORTANT: All software included with the X9720/X9730 Network Storage System is for the
sole purpose of operating the system. Do not add, remove, or change any software unless instructed to do so by HP-authorized personnel.
For information about X9730 system components and cabling, see “X9730 component and cabling
diagrams” (page 171).
For information about X9720 system components and cabling, see “X9720 component and cabling
diagrams” (page 180).
For a complete list of system components, see the HP X9000 Network Storage System QuickSpecs, which are available at:
http://www.hp.com/go/X9000
HP X9000 software features
HP X9000 software is a scale-out, network-attached storage solution including a parallel file system for clusters, an integrated volume manager, high-availability features such as automatic failover of multiple components, and a centralized management interface. X9000 software can scale to thousands of nodes.
Based on a segmented file system architecture, X9000 software integrates I/O and storage systems into a single clustered environment that can be shared across multiple applications and managed from a central Fusion Manager.
X9000 software is designed to operate with high-performance computing applications that require high I/O bandwidth, high IOPS throughput, and scalable configurations.
Some of the key features and benefits are as follows:
• Scalable configuration. You can add servers to scale performance and add storage devices
to scale capacity.
• Single namespace. All directories and files are contained in the same namespace.
System features 11
• Multiple environments. Operates in both the SAN and DAS environments.
• High availability. The high-availability software protects servers.
• Tuning capability. The system can be tuned for large or small-block I/O.
• Flexible configuration. Segments can be migrated dynamically for rebalancing and data
tiering.
High availability and redundancy
The segmented architecture is the basis for fault resilience—loss of access to one or more segments does not render the entire file system inaccessible. Individual segments can be taken offline temporarily for maintenance operations and then returned to the file system.
To ensure continuous data access, X9000 software provides manual and automated failover protection at various points:
• Server. A failed node is powered down and a designated standby server assumes all of its
segment management duties.
• Segment. Ownership of each segment on a failed node is transferred to a designated standby
server.
• Network interface. The IP address of a failed network interface is transferred to a standby
network interface until the original network interface is operational again.
• Storage connection. For servers with HBA-protected Fibre Channel access, failure of the HBA
triggers failover of the node to a designated standby server.
12 Product description
2 Getting started
This chapter describes how to log in to the system, boot the system and individual server blades, change passwords, and back up the Fusion Manager configuration. It also describes the X9000 software management interfaces.
IMPORTANT: Follow these guidelines when using your system:
• Do not modify any parameters of the operating system or kernel, or update any part of the
X9720/X9730 Network Storage System unless instructed to do so by HP; otherwise, the system could fail to operate properly.
• File serving nodes are tuned for file serving operations. With the exception of supported
backup programs, do not run other applications directly on the nodes.
Setting up the X9720/X9730 Network Storage System
An HP service specialist sets up the system at your site, including the following tasks:
Installation steps
• Before starting the installation, ensure that the product components are in the location where
they will be installed. Remove the product from the shipping cartons, confirm the contents of each carton against the list of included items, check for any physical damage to the exterior of the product, and connect the product to the power and network provided by you.
• Review your server, network, and storage environment relevant to the HP Enterprise NAS
product implementation to validate that prerequisites have been met.
• Validate that your file system performance, availability, and manageability requirements have
not changed since the service planning phase. Finalize the HP Enterprise NAS product implementation plan and software configuration.
• Implement the documented and agreed-upon configuration based on the information you
provided on the pre-delivery checklist.
• Document configuration details.
Additional configuration steps
When your system is up and running, you can continue configuring the cluster and file systems. The Management Console GUI and CLI are used to perform most operations. (Some features described here may be configured for you as part of the system installation.)
Cluster. Configure the following as needed:
• Firewall ports. See “Configuring ports for a firewall” (page 21)
• HP Insight Remote Support and Phone Home. See “Configuring HP Insight Remote Support
on X9000 systems” (page 23).
• Virtual interfaces for client access. See “Configuring virtual interfaces for client access”
(page 34).
• Cluster event notification through email or SNMP. See “Configuring cluster event notification”
(page 47).
• Fusion Manager backups. See “Backing up the Fusion Manager configuration” (page 51).
• NDMP backups. See “Using NDMP backup applications” (page 51).
• Statistics tool. See “Using the Statistics tool” (page 71).
• Ibrix Collect. See “Collecting information for HP Support with Ibrix Collect” (page 136).
Setting up the X9720/X9730 Network Storage System 13
File systems. Set up the following features as needed:
• NFS, CIFS, FTP, or HTTP. Configure the methods you will use to access file system data.
• Quotas. Configure user, group, and directory tree quotas as needed.
• Remote replication. Use this feature to replicate changes in a source file system on one cluster
to a target file system on either the same cluster or a second cluster.
• Data retention and validation. Use this feature to manage WORM and retained files.
• Antivirus support. This feature is used with supported Antivirus software, allowing you to scan
files on an X9000 file system.
• X9000 software snapshots. This feature allows you to capture a point-in-time copy of a file
system or directory for online backup purposes and to simplify recovery of files from accidental deletion. Users can access the filesystem or directory as it appeared at the instant of the snapshot.
• File allocation. Use this feature to specify the manner in which segments are selected for storing
new files and directories.
• Data tiering. Use this feature to move files to specific tiers based on file attributes.
For more information about these file system features, see the HP IBRIX X9000 Network Storage System File System User Guide.
Localization support
Red Hat Enterprise Linux 5 uses the UTF-8 (8-bit Unicode Transformation Format) encoding for supported locales. This allows you to create, edit and view documents written in different locales using UTF-8. X9000 software supports modifying the /etc/sysconfig/i18n configuration file for your locale. The following example sets the LANG and SUPPORTED variables for multiple character sets:
LANG="ko_KR.utf8" SUPPORTED="en_US.utf8:en_US:en:ko_KR.utf8:ko_KR:ko:zh_CN.utf8:zh_CN:zh" SYSFONT="lat0-sun16" SYSFONTACM="iso15"
Logging in to the system
Using the network
Use ssh to log in remotely from another host. You can log in to any server using any configured site network interface (eth1, eth2, or bond1).
With ssh and the root user, after you log in to any server, your .ssh/known_hosts file will work with any server in the cluster.
The original server blades in your cluster are configured to support password-less ssh. After you have connected to one server, you can connect to the other servers without specifying the root password again. To enable the same support for other server blades, or to access the system itself without specifying a password, add the keys of the other servers to .ssh/authorized keys on each server blade.
Using the TFT keyboard/monitor
If the site network is down, you can log in to the console as follows:
1. Pull out the keyboard monitor (See “Front view of a base cabinet” (page 180)).
2. Access the on-screen display (OSD) main dialog box by pressing Print Scrn or by pressing Ctrl twice within one second.
3. Double-click the first server name.
14 Getting started
4. Log in as normal.
NOTE: By default, the first port is connected with the dongle to the front of blade 1 (that is, server
1). If server 1 is down, move the dongle to another blade.
Using the serial link on the Onboard Administrator
If you are connected to a terminal server, you can log in through the serial link on the Onboard Administrator.
Booting the system and individual server blades
Before booting the system, ensure that all of the system components other than the server blades—the capacity blocks or performance modules and so on—are turned on. By default, server blades boot whenever power is applied to the system performance chassis (c-Class Blade enclosure). If all server blades are powered off, you can boot the system as follows:
1. Press the power button on server blade 1.
2. Log in as root to server 1.
3. Power on the remaining server blades:
ibrix_server -P on -h <hostname>
NOTE: Alternatively, press the power button on all of the remaining servers. There is no
need to wait for the first server blade to boot.
Management interfaces
Cluster operations are managed through the X9000 Fusion Manager, which provides both a GUI and a CLI. Most operations can be performed from either the GUI or the CLI.
The following operations can be performed only from the CLI:
• SNMP configuration (ibrix_snmpagent, ibrix_snmpgroup, ibrix_snmptrap,
ibrix_snmpuser, ibrix_snmpview)
• Health checks (ibrix_haconfig, ibrix_health, ibrix_healthconfig)
• Raw storage management (ibrix_pv, ibrix_vg, ibrix_lv)
• Fusion Manager operations (ibrix_fm) and Fusion Manager tuning (ibrix_fm_tune)
• File system checks (ibrix_fsck)
• Kernel profiling (ibrix_profile)
• Cluster configuration (ibrix_clusterconfig)
• Configuration database consistency (ibrix_dbck)
• Shell task management (ibrix_shell)
The following operations can be performed only from the GUI:
• Scheduling recurring data validation scans
• Scheduling recurring software snapshots
Using the GUI
The GUI is a browser-based interface to the Fusion Manager. See the release notes for the supported browsers and other software required to view charts on the dashboard. You can open multiple GUI windows as necessary.
If you are using HTTP to access the GUI, open a web browser and navigate to the following location, specifying port 80:
Booting the system and individual server blades 15
http://<management_console_IP>:80/fusion
If you are using HTTPS to access the GUI, navigate to the following location, specifying port 443:
https://<management_console_IP>:443/fusion
In these URLs, <management_console_IP> is the IP address of the Fusion Manager user VIF. The GUI prompts for your user name and password. The default administrative user is ibrix.
Enter the password that was assigned to this user when the system was installed. (You can change the password using the Linux passwd command.) To allow other users to access the GUI, see
“Adding user accounts for GUI access” (page 19).
Upon login, the GUI dashboard opens, allowing you to monitor the entire cluster. (See the online help for information about all GUI displays and operations.) There are three parts to the dashboard: System Status, Cluster Overview, and the Navigator.
16 Getting started
System Status
The System Status section lists the number of cluster events that have occurred in the last 24 hours. There are three types of events:
Alerts. Disruptive events that can result in loss of access to file system data. Examples are a segment that is unavailable or a server that cannot be accessed.
Warnings. Potentially disruptive conditions where file system access is not lost, but if the situation is not addressed, it can escalate to an alert condition. Examples are a very high server CPU utilization level or a quota limit close to the maximum.
Information. Normal events that change the cluster. Examples are mounting a file system or creating a segment.
Cluster Overview
The Cluster Overview provides the following information:
Capacity
The amount of cluster storage space that is currently free or in use.
Filesystems
The current health status of the file systems in the cluster. The overview reports the number of file systems in each state (healthy, experiencing a warning, experiencing an alert, or unknown).
Segment Servers
The current health status of the file serving nodes in the cluster. The overview reports the number of nodes in each state (healthy, experiencing a warning, experiencing an alert, or unknown).
Management interfaces 17
Services
Whether the specified file system services are currently running:
One or more tasks are running.
No tasks are running.
Statistics
Historical performance graphs for the following items:
• Network I/O (MB/s)
• Disk I/O (MB/s)
• CPU usage (%)
• Memory usage (%)
On each graph, the X-axis represents time and the Y-axis represents performance. Use the Statistics menu to select the servers to monitor (up to two), to change the maximum
value for the Y-axis, and to show or hide resource usage distribution for CPU and memory.
Recent Events
The most recent cluster events. Use the Recent Events menu to select the type of events to display.
You can also access certain menu items directly from the Cluster Overview. Mouse over the Capacity, Filesystems or Segment Server indicators to see the available options.
Navigator
The Navigator appears on the left side of the window and displays the cluster hierarchy. You can use the Navigator to drill down in the cluster configuration to add, view, or change cluster objects such as file systems or storage, and to initiate or view tasks such as snapshots or replication. When you select an object, a details page shows a summary for that object. The lower Navigator allows you to view details for the selected object, or to initiate a task. In the following example, we selected Filesystems in the upper Navigator and Mountpoints in the lower Navigator to see details about the mounts for file system ifs1.
18 Getting started
NOTE: When you perform an operation on the GUI, a spinning finger is displayed until the
operation is complete. However, if you use Windows Remote Desktop to access the GUI, the spinning finger is not displayed.
Customizing the GUI
For most tables in the GUI, you can specify the columns that you want to display and the sort order of each column. When this feature is available, mousing over a column causes the label to change color and a pointer to appear. Click the pointer to see the available options. In the following example, you can sort the contents of the Mountpoint column in ascending or descending order, and you can select the columns that you want to appear in the display.
Adding user accounts for GUI access
X9000 software supports administrative and user roles. When users log in under the administrative role, they can configure the cluster and initiate operations such as remote replication or snapshots. When users log in under the user role, they can view the cluster configuration and status, but cannot make configuration changes or initiate operations. The default administrative user name is ibrix. The default regular username is ibrixuser.
Usernames for the administrative and user roles are defined in the /etc/group file. Administrative users are specified in the ibrix-admin group, and regular users are specified in the ibrix-user
Management interfaces 19
group. These groups are created when X9000 software is installed. The following entries in the
/etc/group file show the default users in these groups:
ibrix-admin:x:501:root,ibrix ibrix-user:x:502:ibrix,ibrixUser,ibrixuser
You can add other users to these groups as needed, using Linux procedures. For example:
adduser -G ibrix-<groupname> <username>
When using the adduser command, be sure to include the -G option.
Using the CLI
The administrative commands described in this guide must be executed on the Fusion Manager host and require root privileges. The commands are located in $IBRIXHOME⁄bin. For complete information about the commands, see the HP IBRIX X9000 Network Storage System CLI Reference Guide.
When using ssh to access the machine hosting the Fusion Manager, specify the IP address of the Fusion Manager user VIF.
Starting the array management software
Depending on the array type, you can launch the array management software from the GUI. In the Navigator, select Vendor Storage, select your array from the Vendor Storage page, and click Launch Storage Management.
X9000 client interfaces
X9000 clients can access the Fusion Manager as follows:
• Linux clients. Use Linux client commands for tasks such as mounting or unmounting file systems
and displaying statistics. See the HP IBRIX X9000 Network Storage System CLI Reference Guide for details about these commands.
• Windows clients. Use the Windows client GUI for tasks such as mounting or unmounting file
systems and registering Windows clients.
Using the Windows X9000 client GUI
The Windows X9000 client GUI is the client interface to the Fusion Manager. To open the GUI, double-click the desktop icon or select the IBRIX Client program from the Start menu on the client. The client program contains tabs organized by function.
NOTE: The Windows X9000 client GUI can be started only by users with Administrative privileges.
• Status. Shows the client’s Fusion Manager registration status and mounted file systems, and
provides access to the IAD log for troubleshooting.
• Registration. Registers the client with the Fusion Manager, as described in the HP IBRIX X9000
Network Storage System Installation Guide.
• Mount. Mounts a file system. Select the Cluster Name from the list (the cluster name is the
Fusion Manager name), enter the name of the file system to mount, select a drive, and then click Mount. (If you are using Remote Desktop to access the client and the drive letter does not appear, log out and log in again.)
• Umount. Unmounts a file system.
• Tune Host. Tunable parameters include the NIC to prefer (the client uses the cluster interface
by default unless a different network interface is preferred for it), the communications protocol (UDP or TCP), and the number of server threads to use.
• Active Directory Settings. Displays current Active Directory settings.
20 Getting started
For more information, see the client GUI online help.
X9000 software manpages
X9000 software provides manpages for most of its commands. To view the manpages, set the MANPATH variable to include the path to the manpages and then export it. The manpages are in the $IBRIXHOME/man directory. For example, if $IBRIXHOME is /usr/local/ibrix (the default), set the MANPATH variable as follows and then export the variable:
MANPATH=$MANPATH:/usr/local/ibrix/man
Changing passwords
IMPORTANT: The hpspAdmin user account is added during the IBRIX software installation and
is used internally. Do not remove this account or change its password. You can change the following passwords on your system:
• Hardware passwords. See the documentation for the specific hardware for more information.
• Root password. Use the passwd(8) command on each server.
• X9000 software user password. This password is created during installation and is used to
log in to the GUI. The default is ibrix. You can change the password using the Linux passwd command.
# passwd ibrix
You will be prompted to enter the new password.
Configuring ports for a firewall
IMPORTANT: To avoid unintended consequences, HP recommends that you configure the firewall
during scheduled maintenance times. When configuring a firewall, you should be aware of the following:
• SELinux should be disabled.
• By default, NFS uses random port numbers for operations such as mounting and locking.
These ports must be fixed so that they can be listed as exceptions in a firewall configuration file. For example, you will need to lock specific ports for rpc.statd, rpc.lockd, rpc.mountd, and rpc.quotad.
• It is best to allow all ICMP types on all networks; however, you can limit ICMP to types 0, 3,
8, and 11 if necessary.
Be sure to open the ports listed in the following table.
DescriptionPort
SSH22/tcp
SSH for Onboard Administrator (OA); only for X9720/X9730 blades9022/tcp
NTP123/tcp, 123/upd
Multicast DNS, 224.0.0.2515353/udp
netperf tool12865/tcp
Fusion Manager to file serving nodes80/tcp
443/tcp
Fusion Manager and X9000 file system5432/tcp
X9000 software manpages 21
DescriptionPort
8008/tcp 9002/tcp 9005/tcp 9008/tcp 9009/tcp 9200/tcp
Between file serving nodes and NFS clients (user network) NFS
2049/tcp, 2049/udp 111/tcp, 111/udp
RPC
875/tcp, 875/udp
quota
32803/tcp
lockmanager
32769/udp
lockmanager
892/tcp, 892/udp
mount daemon
662/tcp, 662/udp
stat
2020/tcp, 2020/udp
stat outgoing
4000:4003/tcp
reserved for use by a custom application (CMU) and can be disabled if not used
Between file serving nodes and CIFS clients (user network)137/udp 138/udp 139/tcp 445/tcp
Between file serving nodes and X9000 clients (user network)9000:9002/tcp 9000:9200/udp
Between file serving nodes and FTP clients (user network)20/tcp, 20/udp 21/tcp, 21/udp
Between GUI and clients that need to access the GUI7777/tcp 8080/tcp
Dataprotector5555/tcp, 5555/udp
Internet Printing Protocol (IPP)631/tcp, 631/udp
ICAP1344/tcp, 1344/udp
Configuring NTP servers
When the cluster is initially set up, primary and secondary NTP servers are configured to provide time synchronization with an external time source. The list of NTP servers is stored in the Fusion Manager configuration. The active Fusion Manager node synchronizes its time with the external source. The other file serving nodes synchronize their time with the active Fusion Manager node. In the absence of an external time source, the local hardware clock on the agile Fusion Manager node is used as the time source. This configuration method ensures that the time is synchronized on all cluster nodes, even in the absence of an external time source.
On X9000 clients, the time is not synchronized with the cluster nodes. You will need to configure NTP servers on X9000 clients.
List the currently configured NTP servers:
ibrix_clusterconfig -i -N
Specify a new list of NTP servers:
22 Getting started
ibrix_clusterconfig -c -N SERVER1[,...,SERVERn]
Configuring HP Insight Remote Support on X9000 systems
IMPORTANT: In the X9000 software 6.1 release, the default port for the X9000 SNMP agent
changed from 5061 to 161. This port number cannot be changed.
Prerequisites
The required components for supporting X9000 systems are preinstalled on the file serving nodes. You must install HP Insight Remote Support on a separate Windows system termed the Central Management Server (CMS):
• HP Insight Manager (HP SIM). This software manages HP systems and is the easiest and least
expensive way to maximize system uptime and health.
• Insight Remote Support Advanced (IRSA). This version is integrated with HP Systems Insight
Manager (SIM). It provides comprehensive remote monitoring, notification/advisories, dispatch, and proactive service support. IRSA and HP SIM together are referred to as the CMS.
The following versions of the software are supported.
• HP SIM 6.3 and IRSA 5.6
• HP SIM 7.1 and IRSA 5.7
IMPORTANT: Insight Remote Support Standard (IRSS ) is not supported with X9000 software 6.1
and later. For product descriptions and information about downloading the software, see the HP Insight
Remote Support Software web page:
http://www.hp.com/go/insightremotesupport
For information about HP SIM:
http://www.hp.com/products/systeminsightmanager
For IRSA documentation:
http://www.hp.com/go/insightremoteadvanced-docs
Limitations
Note the following:
• For X9000 systems, the HP Insight Remote Support implementation is limited to hardware
events.
• The X9720 CX storage device is not supported for HP Insight Remote Support.
Configuring the X9000 cluster for Insight Remote Support
To enable X9720/X9730 systems for remote support, first enable Phone Home on the cluster, and then configure Phone Home settings. All nodes in the cluster should be up when you perform this step.
Enabling Phone Home on the cluster
To enable Phone Home, run the following command:
ibrix_phonehome -F
Configuring HP Insight Remote Support on X9000 systems 23
NOTE: Enabling Phone Home removes any previous X9000 snmp configuration details and
populates the snmp configuration with Phone Home configuration details. When Phone Home is enabled, you cannot use ibrix_snmpagent to edit or change the X9000 snmp agent configuration. However, you can use ibrix_snmptrap to add trapsink IPs and you can use ibrix_event to associate events to the trapsink IPs.
Registering Onboard Administrator
The Onboard Administrator is registered automatically.
Configuring the Virtual SAS Manager
On X9730 systems, the SNMP service is disabled by default on the SAS switches. To enable the SNMP service manually and provide the trapsink IP on all SAS switches, complete these steps:
1. Open the Virtual SAS Manager from the OA. Select OA IP > Interconnet Bays > SAS Switch > Management Console.
2. On the Virtual SAS Manager, open the Maintain tab, click SAS Blade Switch, and select SNMP Settings. On the dialog box, enable the SNMP service and supply the information needed for alerts.
Configuring the Virtual Connect Manager
To configure the Virtual Connect Manager on an X9720/X9730 system, complete the following steps:
1. From the Onboard Administrator, select OA IP > Interconnet Bays > HP VC Flex-10 > Management Console.
2. On the HP Virtual Connect Manager, open the SNMP Configuration tab.
3. Configure the SNMP Trap Destination:
• Enter the Destination Name and IP Address (the CMS IP).
• Select SNMPv1 as the SNMP Trap Format.
• Specify public as the Community String.
4. Select all trap categories, VCM traps, and trap severities.
24 Getting started
Configuring Phone Home settings
To configure Phone Home on the GUI, select Cluster Configuration in the upper Navigator and then select Phone Home in the lower Navigator. The Phone Home Setup panel shows the current configuration.
Configuring HP Insight Remote Support on X9000 systems 25
Click Enable to configure the settings on the Phone Home Settings dialog box. Skip the Software Entitlement ID field; it is not currently used.
The time required to enable Phone Home depends on the number of devices in the cluster, with larger clusters requiring more time.
To configure Phone Home settings from the CLI, use the following command:
ibrix_phonehome -c -i <IP Address of the Central Management Server> [-z Software Entitlement Id] [-r Read Community] [-w Write Community] [-t System Contact] [-n System Name] [-o System Location]
For example:
26 Getting started
ibrix_phonehome -c -i 99.2.4.75 -P US -r public -w private -t Admin -n SYS01.US -o Colorado
Next, configure Insight Remote Support for the version of HP SIM you are using:
• HP SIM 7.1 and IRS 5.7. See “Configuring Insight Remote Support for HP SIM 7.1 and IRS
5.7” (page 27).
• HP SIM 6.3 and IRS 5.6. See “Configuring Insight Remote Support for HP SIM 6.3 and IRS
5.6” (page 29).
Configuring Insight Remote Support for HP SIM 7.1 and IRS 5.7
To configure Insight Remote Support, complete these steps:
1. Configure Entitlements for the servers and chassis in your system.
2. Discover devices on HP SIM.
Configuring Entitlements for servers and chassis
Expand Phone Home in the lower Navigator. When you select Chassis or Servers, the GUI displays the current Entitlements for that type of device. The following example shows Entitlements for the servers in the cluster.
To configure Entitlements, select a device and click Modify to open the dialog box for that type of device. The following example shows the Server Entitlement dialog box. The customer-entered serial number and product number are used for warranty checks at HP Support.
Use the following commands to entitle devices from the CLI. The commands must be run for each device present in the cluster.
Entitle a server:
ibrix_phonehome -e -h <Host Name> -b <Customer Entered Serial Number>
-g <Customer Entered Product Number>
Configuring HP Insight Remote Support on X9000 systems 27
Enter the Host Name parameter exactly as it is listed by the ibrix_fm -l command.
Entitle a chassis:
ibrix_phonehome -e -C <OA IP Address of the Chassis> -b <Customer Entered Serial Number> -g <Customer Entered Product Number>
NOTE: The Phone Home > Storage selection on the GUI does not apply to X9720/X9730 systems.
Discovering devices on HP SIM
HP Systems Insight Manager (SIM) uses the SNMP protocol to discover and identify X9000 systems automatically. On HP SIM, open Options > Discovery > New. Select Discover a group of systems, and then enter the discovery name and the Fusion Manager IP address on the New Discovery dialog box.
Enter the read community string on the Credentials > SMTP tab. This string should match the Phone Home read community string. If the strings are not identical, the Fusion Manager IP might be discovered as “Unknown.”
Devices are discovered as described in the following table.
Discovered asDevice
Fusion Manager
nl
System Type:
nl
Fusion Manager IP
X9000
nl
System Subtype:
nl
28 Getting started
Discovered asDevice
Product Model: HP X9000 Solution
Storage Device
nl
System Type:
nl
File serving nodes
X9000, Storage, HP ProLiant
nl
System Subtype:
nl
HP X9720 NetStor FSN(ProLiant BL460 G6)
nl
Product Model:
nl
HP X9720 NetStor FSN(ProLiant BL460 G6)
nl
HP X9730 NetStor FSN(ProLiant BL460 G7)
nl
HP X9730 NetStor FSN(ProLiant BL460 G7)
The following example shows discovered devices on HP SIM 7.1.
File serving nodes and the OA IP are associated with the Fusion Manager IP address. In HP SIM, select Fusion Manager and open the Systems tab. Then select Associations to view the devices.
You can view all X9000 devices under Systems by Type > Storage System > Scalable Storage Solutions > All X9000 Systems
Configuring Insight Remote Support for HP SIM 6.3 and IRS 5.6
Discovering devices in HP SIM
HP Systems Insight Manager (SIM) uses the SNMP protocol to discover and identify X9000 systems automatically. On HP SIM, open Options > Discovery > New, and then select Discover a group of systems. On the Edit Discovery dialog box, enter the discovery name and the IP addresses of the devices to be monitored. For more information, see the HP Sim 6.3 documentation.
NOTE: Each device in the cluster should be discovered separately.
Configuring HP Insight Remote Support on X9000 systems 29
Enter the read community string on the Credentials > SMTP tab. This string should match the Phone Home read community string. If the strings are not identical, the device will be discovered as “Unknown.”
The following example shows discovered devices on HP SIM 6.3. File serving nodes are discovered as ProLiant server.
Configuring device Entitlements
Configure the CMS software to enable remote support for X9000 systems. For more information, see "Using the Remote Support Setting Tab to Update Your Client and CMS Information” and “Adding Individual Managed Systems” in the HP Insight Remote Support Advanced A.05.50 Operations Guide.
Enter the following custom field settings in HP SIM:
• Custom field settings for X9720/X9730 Onboard Administrator
The Onboard Administrator (OA) is discovered with OA IP addresses. When the OA is discovered, edit the system properties on the HP Systems Insight Manager. Locate the Entitlement Information section of the Contract and Warranty Information page and update the following:
◦ Enter the X9000 enclosure product number as the Customer-Entered product number
◦ Enter X9000 as the Custom Delivery ID
◦ Select the System Country Code
◦ Enter the appropriate Customer Contact and Site Information details
• Contract and Warranty Information
Under Entitlement Information, specify the Customer-Entered serial number, Customer-Entered product number, System Country code, and Custom Delivery ID.
30 Getting started
Loading...
+ 190 hidden pages
Buy Points How it Works FAQ Contact Us Questions and Suggestions Privacy Policy Cookie Policy Terms of Use