HP NC570C, NC571C User Manual

HP Dual-port 4x Fabric Adapter

User Guide

November 2004 (Second Edition)

Part Number 377704-002

The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

HP Dual-port 4x Fabric Adapter User Guide

November 2004 (Second Edition)

Part Number 377704-002

Table of Contents

Regulatory Model Number ..........................................................................................................................v

Federal Communications Commission Notice ............................................................................................v

Declaration of Conformity for Products marked with the FCC Logo, United States Only....................... vi

Modifications ............................................................................................................................................. vi

Cables......................................................................................................................................................... vi

Canadian Notice (Avis Canadien) ............................................................................................................. vi

European Union Regulatory Notice........................................................................................................... vi

Japanese Notice......................................................................................................................................... vii

Korean Notice ........................................................................................................................................... vii

BSMI Notice ............................................................................................................................................. vii

Electrostatic Discharge ............................................................................................................................ viii

Preventing Electrostatic Damage................................................................................................ viii

Grounding Methods To Prevent Electrostatic Damage .............................................................. viii

Contact Information................................................................................................................................. viii

1: About the Host Channel Adapter (HCA) .................... 1

HP Dual-port 4x Fabric Adapters ................................................................................................................1

Supported Protocols........................................................................................................................ 1

HCA Package Contents .................................................................................................................. 2

About the HCA Drivers ...............................................................................................................................2

IPoIB............................................................................................................................................... 2

Socket Direct Protocol (SDP) ......................................................................................................... 2

uDAPL ............................................................................................................................................ 2

SCSI RDMA (SRP) ........................................................................................................................ 2

MPI ................................................................................................................................................. 2

Linux Kernels ................................................................................................................................. 3

About Boot Over InfiniBand Functionality .................................................................................................3

How Boot Over IB Works .............................................................................................................. 3

Value of Boot over IB..................................................................................................................... 3

2: Installing the Host Channel Adapter (HCA) ............... 5

Requirements ...............................................................................................................................................5

Installation Overview...................................................................................................................................5

Selecting the Host Connector ......................................................................................................................6

Selecting PCI-X Connector(s) ........................................................................................................ 6

Selecting PCI-Express Connector(s) .............................................................................................. 7

Warnings......................................................................................................................................... 7

Selecting the Type of Installation ................................................................................................................8

Installing a High Profile HCA in a PCI-X Connector .................................................................... 8

Installing a Low Profile HCA in a PCI-X Connector..................................................................... 9

Installing Two HCAs in One Host with PCI-X Connectors......................................................... 10

Installing HCA(s) in a 1U Host with PCI-Express Connectors.................................................... 11

Installing HCA(s) in a 2U+ Host with PCI-Express Connectors.................................................. 12

Connecting the InfiniBand Cables.............................................................................................................12

3: Installing the HCA Drivers........................................ 15

About the Installation.................................................................................................................................15

Installing HCA Host Drivers .....................................................................................................................15

Verify the HCA and Driver Installation.....................................................................................................18

Check the HCA............................................................................................................................. 18

Verify the HCA and Server Communication................................................................................ 19

Check the Modules ....................................................................................................................... 19

Verify the HCA Initialization ....................................................................................................... 20

Upgrading the Firmware on the HCA........................................................................................................20

Determine the Card Type.............................................................................................................. 21

Upgrade the Firmware .................................................................................................................. 21

4: Configuring IPoIB Drivers ........................................ 23

Assign Interfaces to HCAs.........................................................................................................................23

About Assigning Interface for Single HCAs ................................................................................ 23

About Assigning Interfaces for Multiple HCAs ........................................................................... 23

View all the Interfaces .................................................................................................................. 24

Create Interface Partitions..........................................................................................................................25

About Dividing an Interface ......................................................................................................... 25

Configuring a Subinterface........................................................................................................... 26

Verify IPoIB Connectivity............................................................................................................ 26

Deleting an Interface Partition...................................................................................................... 27

Run an IPoIB Performance Test ................................................................................................................27

5: Configuring MPI Drivers........................................... 29

Configure MPI ...........................................................................................................................................29

Configure SSH.............................................................................................................................. 30

Edit PATH Variable...................................................................................................................... 31

Perform Bandwidth Test............................................................................................................... 32

Perform Latency Test.................................................................................................................... 32

6: Configuring SDP Drivers.......................................... 35

Configure IPoIB Interfaces........................................................................................................................35

Specify Connection Overrides ...................................................................................................................35

Convert Sockets-Based Applications.........................................................................................................35

Converting Sockets-Based Applications to Use SDP................................................................... 36

Run a Performance Test on SDP ...............................................................................................................37

Sample Configuration - OracleNet™ Over SDP for Oracle 9i..................................................................38

Performance Acceleration............................................................................................................. 38

Overview....................................................................................................................................... 38

Sample Topology.......................................................................................................................... 38

Configure the Application Server ................................................................................................. 39

Configure the Database Server ..................................................................................................... 39

Set Up Non-IB Connections ......................................................................................................... 40

Troubleshoot the Configuration.................................................................................................... 40

7: Configuring SRP Drivers.......................................... 41

Auto-Mount SRP Devices .........................................................................................................................41

Verify Configurations from the Host.........................................................................................................41

Verify the SCSI Devices from the Host........................................................................................ 41

Special Considerations...............................................................................................................................44

Scenario ........................................................................................................................................ 44

SRP Sample Configuration........................................................................................................................45

Sample SRP/Storage Topology .................................................................................................... 45

Viewing the Storage Configuration .............................................................................................. 45

Viewing the SRP Host .................................................................................................................. 46

View the Topology ....................................................................................................................... 47

Configure the Fibre Channel Gateway ......................................................................................... 48

Verify Configurations from the Host............................................................................................ 51

Verify SRP Functionality.............................................................................................................. 52

Configure the SRP Target............................................................................................................. 54

iii

8: Configuring uDAPL Drivers...................................... 59

About the uDAPL Configuration...............................................................................................................59

Building uDAPL Applications...................................................................................................................59

Run a uDAPL Performance Test ...............................................................................................................60

Run a uDAPL Throughput Test.................................................................................................... 60

Run a uDAPL Latency Test.......................................................................................................... 61

9: Troubleshooting the HCA Installation ...................... 63

Interpret HCA LEDs..................................................................................................................................63

Check the InfiniBand Cable.......................................................................................................................64

Check the InfiniBand Network Interfaces .................................................................................................64

Run the HCA Self-Test..............................................................................................................................65

10: Sample Test Plan................................................... 67

Overview....................................................................................................................................................67

Requirements .............................................................................................................................................67

Prerequisites.................................................................................................................................. 67

Hardware and Applications .......................................................................................................... 67

Network Topology.....................................................................................................................................68

Host and Switch Setup...............................................................................................................................68

IPoIB Setup................................................................................................................................................69

About IPoIB.................................................................................................................................. 69

Configuring IPoIB ........................................................................................................................ 69

IPoIB Performance vs Ethernet Using netperf...........................................................................................70

Perform a Throughput Test........................................................................................................... 70

Perform a Latency Test................................................................................................................. 71

SDP Performance vs IPoIB Using netperf.................................................................................................71

About SDP .................................................................................................................................... 71

Configuring SDP........................................................................................................................... 71

Perform a Throughput Test........................................................................................................... 72

Perform a Latency Test................................................................................................................. 72

Regulatory Notices

Regulatory Model Number

For the purpose of regulatory compliance certifications and identification, this product has been assigned a unique regulatory model number. The regulatory model number can be found on the product nameplate label, along with all required approval markings and information. When requesting compliance information for this product, always refer to this regulatory model number. The regulatory model number is not the marketing name or model number of the product.

Federal Communications Commission Notice

This equipment has been tested and found to comply with the limits for a Class B digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference in a residential installation. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instructions, may cause harmful interference to radio communications. However, there is no guarantee that interference will not occur in a particular installation. If this equipment does cause harmful interference to radio or television reception, which can be determined by turning the equipment off and on, the user is encouraged to try to correct the interference by one or more of the following measures:

• Reorient or relocate the receiving antenna.

• Increase the separation between the equipment and receiver.

• Connect the equipment into an outlet on a circuit that is different from that to which the receiver is connected.

• Consult the dealer or an experienced radio or television technician for help.

Declaration of Conformity for Products marked with the FCC Logo, United States Only

This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation.

For questions regarding your product, contact us by mail or telephone:

Hewlett-Packard Company

P. O. Box 692000, Mail Stop 530113

Houston, Texas 77269-2000

1-800-652-6672 (For continuous quality improvement, calls may be recorded or monitored.)

For questions regarding this FCC declaration, contact us by mail or telephone:

Hewlett-Packard Company

P. O. Box 692000, Mail Stop 510101

Houston, Texas 77269-2000

1-281-514-3333

To identify this product, refer to the part, series, or model number found on the product.

Modifications

The FCC requires the user to be notified that any changes or modifications made to this device that are not expressly approved by Hewlett-Packard Company may void the user’s authority to operate the equipment.

Cables

Connections to this device must be made with shielded cables with metallic RFI/EMI connector hoods in order to maintain compliance with FCC Rules and Regulations.

Canadian Notice (Avis Canadien)

This Class B digital apparatus meets all requirements of the Canadian Interference-Causing Equipment Regulations

Cet appareil numérique de la classe B respecte toutes les exigences du Règlement sur le matériel brouilleur du Canada

European Union Regulatory Notice

This product complies with the following EU Directives:

•Low Voltage Directive 73/23/EEC

•EMC Directive 89/336/EEC

Compliance with these directives implies conformity to applicable harmonized European standards (European Norms) which are listed on the EU Declaration of Conformity issued by Hewlett-Packard for this product or product family. This compliance is indicated by the following conformity marking placed on the product:

vii

This marking is valid for non-Telecom products

and EU harmonized Telecom products (e.g. Bluetooth).

Japanese Notice

Korean Notice

This marking is valid for EU non-harmonized Telecom products .

*Notified body number (used only if applicable - refer to the product label)

xxxx*

BSMI Notice

viii

Electrostatic Discharge

Preventing Electrostatic Damage

A discharge of static electricity from a finger or other conductor may damage system boards or other static-sensitive devices. This type of damage may reduce the life expectancy of the device.

To prevent electrostatic damage when setting up the system or handling parts:

• Avoid hand contact by transporting and storing products in static-safe containers.

• Keep electrostatic-sensitive parts in their containers until they arrive at static-free workstations.

• Place parts on a grounded surface before removing them from their containers.

• Avoid touching pins, leads, or circuitry.

• Handle parts by edges only.

• Avoid contact between the parts and clothing (for example, a wool sweater) . Wrist straps only protect parts of the body from ESD voltages.

• Do not wear jewelry.

• Always be properly grounded when touching a static-sensitive component or assembly.

Grounding Methods To Prevent Electrostatic Damage

There are several methods for grounding. Use one or more of the following methods when handling or installing electrostatic-sensitive parts:

• Use a wrist strap connected by a ground cord to a grounded workstation or computer chassis. Wrist straps are flexible straps with a minimum of 1 megohm ± 10 percent resistance in the ground cords. To provide proper ground, wear the strap snug against the skin.

• Use heel straps, toe straps, or boot straps at standing workstations. Wear the straps on both feet when standing on conductive floors or dissipating floor mats.

• Use conductive field service tools.

• Use a portable field service kit with a folding static-dissipating work mat.

If you do not have any of the suggested equipment for proper grounding, have an authorized reseller install the part.

For more information on static electricity, or assistance with product installation, contact your authorized reseller.

Contact Information

Table 2-1: Customer Contact Information

For the name of your nearest authorized HP reseller:

In the United States, call 1-800-345-1518.

In Canada, call 1-800-263-5868.

Table 2-1: Customer Contact Information

For HP technical support: In the United States and Canada, call 1-800-HP-INVENT

(1-800-474-6836). This service is available 24 hours a day, 7 days a week. For continuous quality improvement, calls may be recorded or monitored.

Outside the United States and Canada, refer to

www.hp.com

About the Host Channel Adapter (HCA)

This document provides the following information:

• “HP Dual-port 4x Fabric Adapters” on page 1

• “About the HCA Drivers” on page 2

• “About Boot Over InfiniBand Functionality” on page 3

HP Dual-port 4x Fabric Adapters

This document describes the following HCAs:

• HP NC570C PCI-X Dual-port 4x Fabric Adapter

• HP NC571C PCI Express Dual-port 4x Fabric Adapter

Both HCAs provide 4x InfiniBand each direction. Each HCA and associated protocol drivers are designed to run in conjunction with an HP Dual-port 4x Fabric Adapter. The HP Dual-port 4x Fabric Adapters feature a full suite of upper-layer protocols and APIs.

Supported Protocols

• IPoIB - Internet Protocol over InfiniBand. Refer to “IPoIB” on page 2 or “Configuring IPoIB

Drivers” on page 23.

• SDP - Socket Direct Protocol. Refer to “Socket Direct Protocol (SDP)” on page 2 or “Configuring

SDP Drivers” on page 35.

• uDAPL - User Direct Access Programming Library. Refer to “uDAPL” on page 2 or “Configuring

uDAPL Drivers” on page 59

• SRP - SCSI RDMA Protocol. Refer to “SCSI RDMA (SRP)” on page 2 or “Configuring SRP

Drivers” on page 41.

• MPI - Message Passing Interface. Refer to page 2 or “Configuring MPI Drivers” on page 29

™ copper connectors which provide 10Gbps connections per port in

HCA Package Contents

Inspect all items for shipping damage. If anything appears to be damaged, or if you encounter problems when installing or configuring your system, contact a customer service representative.

The HP Dual-port 4x Fabric Adapters ship with the following components:

• One HP Dual-port 4x Fabric Adapter

• HP Dual-port 4x Fabric Adapter Quick Setup Instructions

• Limited Warranty and Material Limitations Documentation

About the HCA Drivers

The HP Dual-port 4x Fabric Adapters provide a full suite of upper-layer protocols, including IPoIB, SD P, SRP, M PI a n d uD A P L.

IPoIB

IPoIB is a required protocol; it allows the IP network to utilize the InfiniBand fabric. It is used by SDP and uDAPL to resolve IP addresses. IPoIB is configured like a normal Ethernet interface. During the installation process, ib interface names are automatically added to the network configuration. These correspond to the ports on the HCA.

Socket Direct Protocol (SDP)

The Socket Direct Protocol (SDP) is a high-performance, zero-copy data-transfer protocol used for stream-socket networking over an InfiniBand fabric. The driver can be configured to automatically translate TCP to SDP based on source IP, destination, or program name.

uDAPL

The User Direct Access Programming Library (uDAPL) defines a set of APIs that exploits RDMA capabilities. uDAPL is installed transparently with the driver library. Your application must explicitly support uDAPL. uDAPL is transparently installed and requires no further configuration. However, if your application supports uDAPL, it may require additional configuration changes. Please refer to your application documentation for more information.

SCSI RDMA (SRP)

The SCSCI RDMA (SRP) protocol runs SCSI commands across RDMA-capable networks for InfiniBand hosts to communicate with Fibre Channel storage devices. This information is used to assign devices and mount file-systems so that the data on those file-systems is accessible to the host.

The SRP driver is installed as part of the driver package, and is loaded automatically upon host reboot.

Use of this protocol requires that a Fibre Channel gateway be present in the chassis.

MPI

The MPI protocol is bundled with the Upper Layer Protocol (ULP) suite. Topspin has taken the Ohio State University’s (OSU’s) MVAPICH and created Topspin’s version of this release. However, in addition, the HCAs also run using other popular InfiniBand MPI implementations.

Alternative MPI Implementations

Topspin customers have also deployed a variety of MPIs that use Mellanox’s VAPI layer. This includes OSU, LAM-MPI, Verari Systems Software, Inc’s MPI/Pro (formerly Softech’s ), and LANL MPI. Topspin products have also been used successfully with SCALI MPI, which is based on uDAPL.

Differences Between Topspin and Standard MPI

There are significant differences between the version of MPI provided, and OSU’s MPI.

• There is no restriction on which HCA port is used (OSU only supports Port 1)

• Support for Opteron 64 bit operation is provided

• Bug fixes have been provided for the purpose of improving stability

Linux Kernels

Check the HP Support website at: http://support.hp.com/ website for the latest list of supported kernels and system architectures.

About Boot Over InfiniBand Functionality

The Host Channel Adapter has the capability of running bootable firmware, which allows you to use Boot Over InfiniBand functionality.

How Boot Over IB Works

When the InfiniBand host boots, it initializes the HCA and executes the HCA Boot over IB firmware image. The HCA firmware communicates with the connected Server Switch to load the operating system (OS) from Fibre Channel storage that the Server Switch accesses through the Fibre Channel gateway. Once the host loads the image from the target FC storage, it boots the OS.

Value of Boot over IB

The Boot over IB feature serves as a manageability tool to help you more easily and centrally administer your network. With this feature, you can:

• Quickly and easily change the image that hosts run.

• Centrally localize images.

• Easily reallocate hosts based on your immediate needs.

• Eliminate any need for local storage.

• Reduce the amount of power that your servers consume.

• Increase the mean time between failure of your servers.

• Replace old hardware with new hardware and boot the existing image and configuration.

With the Boot over IB feature, you can change storage mappings during production, then reboot servers

from different storage to change the functions of the servers.

Installing the Host Channel Adapter (HCA)

This chapter provides the following information:

• “Requirements” on page 5

• “Installation Overview” on page 5

• “Selecting the Host Connector” on page 6

• “Selecting the Type of Installation” on page 8

Requirements

• HCA cards support 64-bit PCI variants. 32-bit slots are not compatible.

• A maximum of 3.3V power is required. The HCA(s) should be installed in those slots that are keyed to provide 3.3V. Note: low-profile HCAs require 1 watt less power.

• For maximum performance, 133 MHz PCI-X or PCI-Express is required. 100 MHz is the minimum that can be utilized, but is not recommended.

Installation Overview

The following steps are required when performing the HCA installation procedure:

• “Selecting the Host Connector” on page 6

• “Selecting the Type of Installation” on page 8

• “Installing a High Profile HCA in a PCI-X Connector” on page 8

• “Installing a Low Profile HCA in a PCI-X Connector” on page 9

• “Installing Two HCAs in One Host with PCI-X Connectors” on page 10

• “Installing HCA(s) in a 1U Host with PCI-Express Connectors” on page 11

• “Installing HCA(s) in a 2U+ Host with PCI-Express Connectors” on page 12

• “Installing HCA Host Drivers” on page 15

Selecting the Host Connector

The following types of connectors are supported:

• “Selecting PCI-X Connector(s)” on page 6

• “Selecting PCI-Express Connector(s)” on page 7

Selecting PCI-X Connector(s)

The HCA requires that specific PCI-X slots be used.

When determining which PCI-X slot to use, inspect the server chassis and keep the following in mind:

Consider the Speed of the Slot

Locate the 133MHz PCI-X (64-bit, 3.3V) or 100MHz PCI-X (64-bit, 3.3V) slots.

A conventional PCI 64-bit connector is not recommended as the first option, but is supported.

Systems with 66 MHz PCI-X connectors are supported.

Consider Other Devices on the Bus

It is recommended that you select a connector that is the only one on that particular PCI-X bus. This is most often the case for the 133MHz connectors.

Use the mother board (server) documentation in order to get a block diagram of all the available PCI-X/PCI buses. This will help you determine which connectors belong to which bus. If this is not obvious from the documentation you may need to contact the server vendor technical support.

If there are two connectors (or more) on the same PCI-X bus, make sure to remove all other devices from this bus. It is highly undesirable to have another device on the same PCI-X bus, as performance will most likely be affected. However, if performance is not a concern and the frequency of the PCI-X bus is 100MHz, it is permissible to have two devices (for example, an IB HCA and GE NIC) on the same bus.

If the bus is 133MHz, it is mandatory that you remove any other devices so that the InfiniBand HCA is the only device on that bus.

Consider Cooling

Most HCAs have totally passive cooling, which means there are no extra fans installed on the board.

It is mandatory that you arrange for suitable airflow to go around the HCA head sink. This may mean choosing PCI-X slots that do not place the HCA too close to another card.

In addition, some server chassis vendors provide extra fan assemblies, and you should make sure to have them installed.

Consider the Physical Stability of the Installation

When selecting the PCI-X slot, consider whether the HCA(s) can be installed in such a way that they are absolutely secure. It is possible to stress the HCA connectors while arranging the cables. A poorly secured HCA could also damage the PCI-X connector mechanically.

Consider the PCI-X Frequency Configuration

It is important that you verify the PCI-X frequency configuration.

Some motherboards have jumper configurations for the PCI-X frequency. Check the mother board documentation and verify that the frequencies are set to 133MHz or 100MHz.

Some mother boards are PCI-X frequency-configurable via the CMOS BIOS setup, and some provide jumpers and CMOS configuration.

Consider Dual HCA Installation Requirements

• For dual HCA installation in a single host, it is required to have two completely isolated PCI-X buses to avoid any performance degradation.

• If the host has only one PCI-X 100 or 133MHz bus (regardless of the number of connectors), then this mother board should not be used for a dual HCA installation.

• It is acceptable to have one of the PCI-X slots operate at 133MHz and the other at 100MHz. However, the best case is to have two 133MHz individual connectors on two completely isolated PCI-X buses.

• Systems with one 133MHz connector, and one 66MHz connector are suitable for dual-HCA installations.

Selecting PCI-Express Connector(s)

Consider the Type of PCI-E Connector

Only PCI-Express 8x connectors should be used to install an HCA. PCI-Express 1x, 4x, 16x should not be used, even if it is possible mechanically.

Before selecting a connector, you should verify with the motherboard documentation that the connector is actually 8x, and is supported by the BIOS as 8x. This is important because some vendors use 8x connectors for 4x.

Consider the following general rules:

• If there are three 8x PCI-Express connectors in your server, it is almost a guarantee that one of them is actually 4x.

• If there are 16x and 8x connectors in your server, its very possible the 8x connector is actually 4x. Verify with the motherboard documentation that the connector is actually 8x and is supported by the BIOS as 8x.

• Some early version of the PCI Express motherboards had issues on one of the PCI Express connectors. If you encounter problems when using the HCA in one of the PCI-Express connectors, it might help to move the HCA to a different PCI-Express connector.

Consider Cooling

Most HCAs have totally passive cooling, which means there are no extra fans installed on the board.

It is mandatory that you arrange for suitable airflow to go around the HCA head sink. This may mean choosing slots that do not place the HCA too close to another card.

In addition, some server chassis vendors provide extra fan assemblies, and you should make sure to have them installed.

Consider the Physical Stability of the Installation

When selecting the PCI-Express slot, consider whether the HCA(s) can be installed in such a way that they are absolutely secure. It is possible to stress the HCA connectors while arranging the cables. A poorly secured HCA could also damage the PCI-E connector mechanically.

Warnings

When installing the HCA in the server, observe the following:

• To avoid the risk of personal injury or damage to the equipment, consult the User's Documentation provided with your equipment before attempting the installation.

• Many computers are capable of producing energy levels that are considered hazardous. Users should not remove enclosures nor should they bypass the interlocks provided to protect one from these hazardous conditions.

• Installation of this HCA should be performed by individuals who are both qualified in the servicing of computer equipment, and trained in the hazards associated with products capable of producing hazardous energy levels.

• To reduce the risk of personal injury from hot surfaces, allow the internal system components to cool before touching.

Selecting the Type of Installation

There are a variety of HCA installations with slight differences, depending on the type of HCA you have, the type of PCI connector your host has, and the number of HCAs you are installing:

• “Installing a High Profile HCA in a PCI-X Connector” on page 8

• “Installing a Low Profile HCA in a PCI-X Connector” on page 9

• “Installing Two HCAs in One Host with PCI-X Connectors” on page 10

• “Installing HCA(s) in a 1U Host with PCI-Express Connectors” on page 11

• “Installing HCA(s) in a 2U+ Host with PCI-Express Connectors” on page 12

Installing a High Profile HCA in a PCI-X Connector

The HCA comes preconfigured. You do not have to set any jumpers or connectors.

To install the HCA:

1. Note the Global Unique ID (GUID) numbers from the hardware. You will need this number when performing configurations.

Optionally, you can run vstat (a utility that is available after host driver installation) to view the Global ID (GID). The GUID is the last 8-bytes of the GID.

The GUID will look something like this: 00:05:ad:00:00:00:02:40

2. Log on to the host system as the root user.

3. Power-down the host system.

4. Disconnect the power cable.

Note: This is an important step, as serious damage could be caused by the standby power accidently being powered on during the HCA installation.

5. Ground yourself appropriately to the host chassis.

6. Remove the host-system cover to access the PCI-X slots.

7. Select a PCI-X or PCI slot in which to insert HCA, if you have not already done so. Refer to

“Selecting the Host Connector” on page 6.

8. (Optional) If it is not necessary for you to remove the riser from the server, slide the HCA edge-connector into the PCI-X slot now. If you need to remove the riser, refer to Step 9.

a. Slip the IB ports into the back of the open slot.

b. Slide the edge-connector of the HCA into the PCI-X slot. Make sure that the card is fully

seated by pushing the card gently into the slot until the connectors are no longer visible.

9. Optional) Remove the riser from the host, if necessary. It may not be possible to fit the edge-connector into the slot without removing the riser.

a. Unscrew the riser and lift it from the host chassis. This step will vary depending on your server.

b. Slide the HCA edge-connector into the PCI-X slot while the riser is out of the server.

c. Make sure that the edge-connectors are fully seated in the slot. Push the card gently until the

connectors are no longer visible.

10. Screw the HCA to the host mounting-rail.

11. Replace the host-system access cover.

12. Power-up the host system.

13. Install the host drivers as described in page 15.

14. Connect the InfiniBand cables, as described in “Connecting the InfiniBand Cables” on page 12.

Installing a Low Profile HCA in a PCI-X Connector

The HCA comes preconfigured. You do not have to set any jumpers or connectors.

To install the HCA:

1. Note the Global Unique ID (GUID) numbers from the hardware. You will need this number when performing configurations. Optionally, you can run vstat (a utility that is available after host driver installation) to view the Global ID (GID). The GUID is the last 8-bytes of the GID. The GUID will look something like this: 00:05:ad:00:00:00:02:40

2. Log on to the host system as the root user.

3. Power-down the host system.

4. Disconnect the power cable.

Note: This is an important step, as serious damage could be caused by the standby power accidently being powered on during the HCA installation.

5. Ground yourself appropriately to the host chassis.

6. Remove the host-system cover to access the PCI-X slots.

7. Select a PCI-X or PCI slot in which to insert HCA, if you have not already done so. Refer to

“Selecting the Host Connector” on page 6.

The low-profile HCA comes with a high-profile bracket.

8. (Optional) If it is not necessary for you to remove the riser from the server, slide the HCA edge-connector into the PCI-X slot now. If you need to remove the riser, refer to Step 9.

a. Slip the IB ports into the back of the open slot.

b. Slide the edge-connector of the HCA into the PCI-X slot. Make sure that the card is fully

seated by pushing the card gently into the slot until the connectors are no longer visible.

9. (Optional) Remove the riser from the host, if necessary. It may not be possible to fit the edge-connector into the slot without removing the riser.

a. Unscrew the riser and lift it from the host chassis. This step will vary depending on your server.

b. Slide the HCA edge-connector into the PCI-X slot while the riser is out of the server.

c. Make sure that the edge-connectors are fully seated in the slot. Push the card gently until the

connectors are no longer visible.

10. Screw the HCA to the host mounting-rail.

11. Replace the host-system access cover.

12. Power-up the host system.

13. Install the host drivers as described in page 15.

14. Connect the InfiniBand cables as described in “Connecting the InfiniBand Cables” on page 12.

Installing Two HCAs in One Host with PCI-X Connectors

The HCA comes preconfigured. You do not have to set any jumpers or connectors.

To install the HCAs:

2. Log on to the host system as the root user.

3. Power-down the host system.

4. Disconnect the power cable.

Note: This is an important step, as serious damage could be caused by the standby power accidently being powered on during the HCA installation.

5. Ground yourself appropriately to the host chassis.

6. Remove the host-system cover to access the PCI-X slots.

7. Select two PCI-X or PCI slots in which to install the HCAs, if you have not already done so. When installing two HCAs in a single host, it is particularly important to select the appropriate slots.

Refer to “Selecting the Host Connector” on page 6.

8. (Optional) If it is not necessary for you to remove the riser from the server, slide the HCA edge-connector into the PCI-X slot now. If you need to remove the riser, refer to Step 9.

a. Slip the IB ports into the back of the open slot.

b. Slide the edge-connector of the HCA into the PCI-X slot. Make sure that the card is fully

seated by pushing the card gently into the slot until the connectors are no longer visible.

c. Repeat for the second HCA.

9. (Optional) Remove the riser from the host, if necessary. It may not be possible to fit the edge-connector into the slot without removing the riser.

a. Unscrew the riser and lift it from the host chassis. This step will vary depending on your server.

b. Slide the HCA edge-connector into the PCI-X slot while the riser is out of the server.

c. Make sure that the edge-connectors are fully seated in the slot. Push the card gently until the

connectors are no longer visible.

d. Repeat on a second PCI-X slot.

10. Screw the HCAs to the host mounting-rail.

11. Replace the host-system access cover.

12. Power-up the host system.

13. Install the host drivers as described in page 15.

14. Connect the InfiniBand cables as described in “Connecting the InfiniBand Cables” on page 12.

Installing HCA(s) in a 1U Host with PCI-Express Connectors

The HCA comes preconfigured. You do not have to set any jumpers or connectors.

To install the HCA:

2. Log on to the host system as the root user.

3. Power-down the host system.

4. Disconnect the power cable.

Note: This is an important step, as serious damage could be caused by the standby power accidently being powered on during the HCA installation.

5. Ground yourself appropriately to the host chassis.

Remove the host-system cover to access the PCI-Express slots.

6. Select the PCI-Express slot in which to install the HCA, if you have not already done so.

Refer to “Selecting PCI-Express Connector(s)” on page 7.

7. Slide the HCA into the PCI-Express slot.

8. Gently push the HCA until it is fully seated in the slot.

9. Press the fastener on the host closed.

10. (Optional) Install a second HCA in the host.

11. Gently push the HCA into place.

12. Snap the fastener on the host closed.

13. Make sure that the HCA installation is secure before connecting any InfiniBand cables.

14. Reinstall the host system cover.

Installing HCA(s) in a 2U+ Host with PCI-Express Connectors

The HCA comes preconfigured. You do not have to set any jumpers or connectors.

To install the HCA:

2. Log on to the host system as the root user.

3. Power-down the host system.

4. Disconnect the power cable.

Note: This is an important step, as serious damage could be caused by the standby power accidently being powered on during the HCA installation.

5. Ground yourself appropriately to the host chassis.

Remove the host-system cover to access the PCI-Express slots.

6. Insert the HCA into a PCI-Express slot, and make sure the InfiniBand ports extend out of the opening.

7. Screw the bracket to the host when the bracket is flush.

8. (Optional) Add a second HCA to the host.

9. Screw the bracket to the host when the bracket is flush.

10. Replace the system cover on the host.

Connecting the InfiniBand Cables

To connect the InfiniBand host to the InfiniBand switch, standard 4x InfiniBand cables are required. InfiniBand cables can be used to connect any two InfiniBand devices, whether switch or host.

Figure 2-1: Example of InfiniBand Ports

1. Plug InfiniBand cables from the host to the InfiniBand switch.

a. To plug in an InfiniBand cable, push the connector into the interface until you hear/feel a click.

Figure 2-2: Fully Installed IB Cable with Pinch Connector

NOTE: If your host does not provide an ample amount of free space around a given IB port, double-check that your IB cable connector engages fully. Wiggle your connector back and forth to be sure that both sides of the connector have locked firmly into place.

b. To remove a cable with a pinch connector, pinch both sides of the back of the connector and

pull the connector away from the port.

Figure 2-3: Removing a Pinch Connector

c. To remove a cable with a pull connector, grasp the connector with one hand and push it toward

the port, then pull the latch away from the port with your other hand and gently wiggle the connector away from the port.

1. push connector toward port

2. pull latch away from port

3. pull connector away from port

Figure 2-4: Removing a Pull Connector

Installing the HCA Drivers

This chapter provides the following information:

• “Installing HCA Host Drivers” on page 15.

• “Verify the HCA and Driver Installation” on page 18

• “Upgrading the Firmware on the HCA” on page 20

About the Installation

The driver suite is architected to work optimally as a group of drivers. Due to inter-driver dependencies, it is recommended that you install all the drivers. If you use tsinstall as described, all drivers are installed.

After the installation, you can move on to configuring the drivers of your choice.

• “Configuring IPoIB Drivers” on page 23

• “Configuring SDP Drivers” on page 35

• “Configuring MPI Drivers” on page 29

• “Configuring SRP Drivers” on page 41

• “Configuring uDAPL Drivers” on page 59

Installing HCA Host Drivers

To install HCA software:

1. Go to http://support.hp.com/

2. Select “Software & Driver downloads.”

3. On the Software & Driver Downloads page, enter your product name, then click the double arrow.

+ 60 hidden pages

HP NC570C, NC571C User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual