Q-Logic IB0056101-00 G.02, OFED User Manual

Loading...

QLogic HCA and QLogic OFED Software

Install Guide

Draft

QLogic OFED Version 1.4

IB0056101-00 G.02

Preliminary

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

S

Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no representation nor warranty that such applications are suitable for the specified use without further testing or modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document.

No part of this document may be copied nor reproduced by any means, nor translated nor transmitted to any magnetic medium without the express written consent of QLogic Corporation. In accordance with the terms of their valid QLogic agreements, customers are permitted to make electronic and paper copies of this document for their own exclusive use.

The QHT7040, QHT7140, QLE7140, QLE7240, and QLE7280 QLogic Host Channel Adapters are covered by the following patent: 7308535.

Document Revision History

Rev. 1.0, 8/20/2005

Draft

 

Rev. 1.1, 11/15/05

 

Rev. 1.2, 02/15/06

Rev. 1.3, Beta 1, 4/15/06

Rev 1.3, 6/15/06

Rev. 2.0 Beta, 8/15/06, QLogic Version Number IB0056101-00 A

Rev. 2.0 Beta 2 10/15/06, QLogic Version Number IB0056101-00 B

Rev. 2.0 11/30/06, QLogic Version Number IB0056101-00 C

Rev. 2.0 3/23/07, QLogic Version Number IB0056101-00 D

Rev. 2.1 8/24/07, QLogic Version Number IB0056101-00 E

Rev. 2.2 5/27/08, QLogic Version Number IB0056101-00 F

Rev. 2.2 9/5/08, QLogic Version Number IB0056101-00 G

Rev. QLogic OFED 1.4 4/7/09, QLogic Version Number IB0056101-00 G.02

Changes

Sections Affected

 

 

 

 

Product name changed from InfiniPath to QLogic

All

OFED. Version number is set to 1.4. Instances of

 

InfiniPath changed where appropriate; some file-

 

names and output messages keep old name.

 

 

 

Updated Contact Information.

“Contact Information” on page 1-5

 

 

Removed InfiniPath from 1st paragraph; replaced

“Hardware Installation” on page 4-1

with QLogic adapters, etc.

 

 

 

Changed section title from What’s New in this

“Features” on page 2-1

Release to Features.

 

 

 

Page ii

Preliminary

IB0056101-00 G.02

S

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

 

 

 

 

 

 

 

 

 

 

 

 

 

Replace EM64T with 64-bit Intel Xeon.

“Supported Linux Distributions” on page 5-1

 

 

 

 

 

 

Added more information on VNIC interface.

“OpenSM” on page 6-3

 

 

 

 

 

 

Split the Install section into three separate sec-

“Software Installation” on page 5-1

 

 

tions.

 

“Configuring Drivers and Services” on page 6-1

 

 

 

 

 

 

 

 

“Installation Verification and Additional Settings” on

 

 

 

 

page 7-1

 

 

 

 

 

 

Modified installation checklist. Clarified which driv-

“Software Installation” on page 3-2

 

 

ers are configured, which are optional. Added two

 

 

 

 

methods of installation to the list.

 

 

 

 

 

 

 

 

Remove sections on configuring ipath_ether.

Was “Configuring the ipath_ether Network Inter-

 

 

ipath_ether now deprecated.

face” on page 5-12

 

 

 

 

 

 

 

 

Draft

 

 

Updated the supported distributions information.

Table 5-1 on page 5-2

 

 

Distribution identifiers are now RHEL4, RHEL5,

Table 5-2 on page 5-2

 

 

and SLES10.

 

 

 

 

 

Delete reference to ipath ether.

“Removing Software Packages” on page 5-24

 

 

Updated Lustre information; patches are no longer

“Installing Lustre” on page 5-23

 

 

needed.

 

 

 

 

 

Updated compiler support information.

“Compiler Support” on page 5-3

 

 

Updated information on software components.

“Software Components” on page 2-4

 

 

Removed paragraph about this release adding

“Features” on page 2-1

 

 

support for QLE7240/7280, since this is outdated.

 

 

 

 

Added new “Configuration Issues” to Trouble-

ibsrpdm Command Hangs When Two HCAs are

 

 

shooting.

 

Installed but Only Unit 1 is Connected to the

 

 

 

 

Switch” on page A-5

 

 

 

 

 

 

Deleted Troubleshooting item:

Appendix A

 

 

ifup on ipath_ether on SLES 10 Reports

 

 

 

 

"unknown device"

 

 

 

 

 

 

 

 

 

Deleted entries pertaining to ipath ether in this

Table C-1 on page C-1

 

 

table.

 

 

 

 

IB0056101-00 G.02

Preliminary

Page iii

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

S

 

 

 

 

 

 

 

 

Sections rearranged, renamed, and expanded due

“Choose the Appropriate Download Files” on

to multiple install methods. Previous sections:

page 5-4

Installing the InfiniPath and Open fabrics RPMs

“Install QLogicIB-Basic with the Installer Tool” on

 

 

 

Downloading and Unpacking the InfiniPath and

page 5-7

OpenFabrics Software

“About rpm Installation” on page 5-13

 

 

 

 

 

 

“Install QLogic OFED Using Rocks” on page 5-20

 

 

 

“Install QLogic OFED Using a Platform OCS Kit”

 

 

 

on page 5-22

 

 

 

“Using rpm to Install InfiniPath and OpenFabrics”

 

 

 

on page 5-14

 

 

Added new section about installing using the

“Install QLogicIB-Basic with the Installer Tool” on

Installer Tool.

 

 

page 5-7

 

 

Moved introductory information about RPM instal-

“About rpm Installation” on page 5-13

lation before sections on installing using the two

 

different methods.

 

 

 

 

 

Removed paragraph about kernel module support

Was in old section “Installing the InfiniPath and

being part of InfiniPath RPMs; was relevant to pre-

Open fabrics RPMs”

vious release.

 

 

 

 

 

Added note about need to use prefixed install if

“Using rpm to Install InfiniPath and OpenFabrics”

using QLogic MPI with mpi-selector.

on page 5-14

 

 

Modified module for installed layout. New location

“Installed Layout” on page 5-23

for ipath modules. Location of other OFED mod-

 

ules listed here.

 

 

 

 

 

Modified module for driver overview.

“InfiniPath and OpenFabrics Driver Overview” on

 

 

 

page 6-1

 

 

Modified module for configuring IPoIB. Only restart

“Configuring the IPoIB Network Interface” on

information has been changed.

page 6-2

 

 

Modified module for configuring VNIC interface.

“SRP stands for SCSI RDMA Protocol. It was origi-

Changed infinipath start/stop commands, and

nally intended to allow the SCSI protocol to run

added some introductory material.

over InfiniBand for Storage Area Network (SAN)

 

 

 

usage. SRP interfaces directly to the Linux file sys-

 

 

 

tem through the SRP Upper Layer Protocol. SRP

 

Draft

 

 

 

storage can be treated as another device.” on

 

 

 

page 6-4

Modified module for configuring SRP.

“SRP” on page 6-4

 

 

Modified module for uDAPL configuration.

“MPI over uDAPL” on page 6-13

 

 

Deleted reference to MTRR BIOS setting.

“Configuring the BIOS” on page 4-4

 

 

 

“BIOS Settings” on page A-2

 

 

 

 

Page iv

Preliminary

IB0056101-00 G.02

S

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

 

 

 

 

 

 

 

 

 

 

New Appendix for Write Combining. MTRR set-

“Write Combining” on page B-1

 

tings information moved here.

 

 

 

 

 

 

Minor wording change to introduction section.

“Software Installation” on page 5-1

 

 

 

 

Added new issue in Troubleshooting.

openmpi_gcc Fails to Install Because of Depen-

 

 

 

dency on gfortran (RHEL 4)” on page A-4

 

Added new issue in Troubleshooting.

“Outdated ipath_ether Configuration Setup

 

 

 

Generates Error” on page A-5

 

 

 

 

Configuration file qlogic_vnic.cfg changed to

Table C-1 on page C-1

 

qlgc_vnic.cfg.

 

 

 

 

Added new section.

 

“Uninstalling Software with Rocks or Platform

 

 

 

OCS” on page 5-25

 

 

 

 

 

Added new section.

 

“Install QLogic OFED Using Rocks” on page 5-20

 

 

 

 

Change name of Appendix RPM Descriptions to

“Package Descriptions” on page D-1

 

Package Descriptions. Updated information.

 

 

 

 

 

 

Removed phrase “and Transmission Control

“Interoperability” on page 1-3

 

Protocol (TCP)” from introduction; related to

 

 

 

now-obsolete ipath_ether.

 

 

 

 

 

 

Removed this issue from the Troubleshooting sec-

Was “OpenFabrics Library Dependencies”

 

tion.

 

 

 

 

 

 

 

Combined What’s New in This Release and Fea-

“Feature Overview” on page 2-1

 

tures section.

 

 

 

 

 

 

 

Deleted Note about Fedora Core 6 not supported

This was noted in InfiniPath 2.2.1 release notes.

 

in this release.

 

 

 

 

 

 

 

Removed this line “Please check the QLogic

“Software Components” on page 2-4

 

web site for updated information on supported

 

 

 

compilers.” This information is not on the web

 

 

 

site.

 

 

 

 

 

 

 

Added more details about setting the switch MTU

“Other Configuration: Changing the MTU Size” on

 

default to 4K.

 

page 6-14

 

 

Draft

 

Deleted the section “Installing QLogic MPI in an

“Using rpm to Install InfiniPath and OpenFabrics”

 

Alternate Location with rpm”; merged instructions

on page 5-14

 

into rpm install section.

 

 

 

 

 

 

 

Minor text changes to this section. Modified title

“Install Additional Software” on page 5-23

 

from Additional Installation Instructions to Install

 

 

 

Additional Software. Moved this section to be with

 

 

 

the rest of the installation instructions.

 

 

 

 

 

 

 

 

IB0056101-00 G.02

Preliminary

Page v

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

S

 

 

 

 

 

 

 

 

 

 

 

Changed title of Troubleshooting issue. Was

“Outdated ipath_ether Configuration Setup

 

ipath_ether Configuration Setup Gener-

Generates Error” on page A-5

 

ates Error”

 

 

 

 

 

 

 

Change main configuration file from /etc/sys-

Table C-1 on page C-1 and “SRP” on page 6-4.

 

config/infinipath to /etc/infini-

 

 

band/openib.conf. The infinipath file is

 

 

still to enable or disable the ipath_mtrr script.

 

 

 

 

 

Updated note about setting datagram mode.

“Configuring the IPoIB Network Interface” on

 

 

 

 

page 6-2

 

 

 

 

Deleted Troubleshooting Issue OpenFabrics

“Software Installation Issues” on page A-2

 

Dependencies.

 

 

 

 

 

 

 

Deleted Troubleshooting Issue Version Number

“Software Installation Issues” on page A-2

 

 

Draft

 

Conflict with opensm-* on RHEL5 Systems

 

 

Two new tables showing OS package require-

Table on page 5-4 and Table on page 5-4

 

ments. Replaces some bullet points in “Distribution

 

 

Identifiers” on page 5-2

 

 

 

 

Removed section “Check for Missing Files or

Was “Check for Missing files or RPMs” on

 

RPMs”

 

 

page 5-5

 

Merged the two sections Downloading the QLogic

“Choose the Appropriate Download Files” on

 

OFED Software and Choose the Appropriate

page 5-4

 

Download Files

 

 

 

 

Removed Supported Compilers and Supported

2 Feature Overview

 

Linux Distributions from the 2 Feature Overview

 

 

chapter. Information is still in Software Installation

 

 

chapter.

 

 

 

 

 

 

 

Moved table “InfiniPath and OpenFabrics

“Package Descriptions” on page D-1

 

RPMs to Use for Each Node in a Cluster” from

 

 

Software Installation section to Appendix

 

 

 

 

 

Consolidated Configuring the InfiniPath Drivers

6 Configuring Drivers and Services

 

and InfiniPath and OpenFabrics Driver Overview

 

 

 

 

 

opensm is now off by default

Various

 

 

 

 

Merged “Install on an Unsupported Distribution

Now “Rebuilding the kernel-ib Driver on an

 

with the rpm Command” with the Rebuilding...sec-

Unsupported Distribution or an Unsupported Distri-

 

tion.

 

 

bution/Kernel Pair” on page 5-19

 

 

 

 

Removed $ or # at beginning of lines indicating

All. Also Table 1-1

 

commands. The #, if cut and pasted from the docu-

 

 

ment, may be interpreted as a shell command.

 

 

 

 

 

 

 

Updated footnote b.

 

 

Table D-11

 

 

 

 

 

Page vi

Preliminary

IB0056101-00 G.02

S

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

 

 

 

 

 

 

 

 

 

 

 

 

 

Adapter Settings section name changed. Also

“Adapter and Other Settings” on page 7-1

 

 

added new bullet point “Check the PCIe bus width”

 

 

 

 

 

 

 

 

Updated tables of RPMs. Moved all documentation

All tables in D Package Descriptions

 

 

RPMs together. New table for OpenSM-Devel

 

 

 

 

RPMs. Updated OtherMPIs.

 

 

 

 

 

 

 

 

Deleted table 5-7 RPMs to Install, as information is

“Using rpm to Install InfiniPath and OpenFabrics”

 

 

repeated in “Using rpm to Install InfiniPath and

on page 5-14

 

 

OpenFabrics” on page 5-14

 

 

 

 

 

 

 

 

Moved RPM Organization section under RPM

“RPM Organization” on page 5-16

 

 

Installation.

 

 

 

 

 

 

 

 

 

Merge two sections Uninstalling InfiniPath and

“Uninstalling InfiniPath and OpenFabrics RPMs”

 

 

OpenFabrics RPMs and Uninstalling OFED 1.3

on page 5-24

 

 

Software.

Draft

 

 

 

 

 

 

 

 

 

 

 

Reference to this location

“Form Factors” on page 4-2

 

 

/sys/bus/pci/drivers/ib ipath/00/

 

 

 

 

changed to

 

 

 

 

 

/sys/class/infiniband/ipath*/device/

 

 

 

 

Removed note: “OpenFabrics programs (32-bit)

“Software Components” on page 2-4

 

 

using the Verbs interfaces are not supported

 

 

 

 

in this release, but may be supported in a

 

 

 

 

future release.”

 

 

 

 

 

 

 

 

 

 

IB0056101-00 G.02

Preliminary

Page vii

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

S

Notes

Draft

Page viii

Preliminary

IB0056101-00 G.02

1

2

3

4

Table of Contents

Introduction

Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-1

How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-1

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-2

Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-3

Conventions Used in this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-3

Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-4

Contact Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-5

Feature Overview

Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2-1

Other Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2-2

Continued Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2-2

Software Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2-4

Step-by-Step Installation Checklist

Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3-1

Software Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3-2

Hardware Installation

Hardware Installation Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-1

Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-1

Form Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-2

Cabling and Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-3

Optical Fibre Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-4

Configuring the BIOS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-4

Safety withDraftElectricity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-5

Unpacking Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-5

Verify the Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-5

List of the Package Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-5

Unpacking the QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-8

Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-9

IB0056101-00 G.02

Preliminary

ix

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

S

Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Dual Adapter Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Installation Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hardware Installation for QHT7140 with HTX Riser . . . . . . . . . . . . . .

Hardware Installation for QLE7240, QLE7280, and QLE7140 Without a PCI Express Riser. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hardware Installation for the QHT7140 Without an HTX Riser . . . . . .

Switch Configuration and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Cabling the Adapter to the InfiniBand Switch . . . . . . . . . . . . . . . . . . . . . . . .

Completing the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5 Software Installation

Draft

Cluster Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Types of Nodes in a Cluster Environment . . . . . . . . . . . . . . . . . . . . . .

Supported Linux Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Distribution Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Compiler Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Setting Up Your Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Choose the Appropriate Download Files . . . . . . . . . . . . . . . . . . . . . . . . . . .

Install QLogicIB-Basic with the Installer Tool . . . . . . . . . . . . . . . . . . . . . . . .

About rpm Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Using rpm to Install InfiniPath and OpenFabrics . . . . . . . . . . . . . . . . . . . . .

RPM Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Install QLogic OFED User-level Software with the rpm Command . . . . . . .

Rebuilding or Reinstalling the kernel-ib Driver with rpm After a

Kernel Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Rebuilding the kernel-ib Driver on an Unsupported Distribution or an Unsupported Distribution/Kernel Pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Install QLogic OFED Using Rocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Install Frontend and Compute Nodes . . . . . . . . . . . . . . . . . . . . . . . . .

Rocks Installation on an Existing Frontend Node . . . . . . . . . . . . . . . .

Install QLogic OFED Using a Platform OCS Kit . . . . . . . . . . . . . . . . . . . . . .

Install FastFabric Software CD/ISO Image . . . . . . . . . . . . . . . . . . . . . . . . .

Install Additional Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Installing Lustre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Installed Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Removing Software Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Uninstalling Using the Installer Tool. . . . . . . . . . . . . . . . . . . . . . . . . . .

Uninstalling InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . .

4-9

4-9

4-9

4-12

4-15

4-16

4-17

4-17

4-18

5-1

5-1

5-1

5-2

5-3

5-3

5-4

5-7

5-13

5-14

5-16

5-17

5-19

5-19

5-20

5-20

5-21

5-22

5-22

5-23

5-23

5-23

5-24

5-24

5-24

x

Preliminary

IB0056101-00 G.02

A

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

6

7

A

 

 

 

. . . . . . . . . . . . . . .Uninstalling Software with Rocks or Platform OCS

5-25

Downgrading RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-25

Configuring Drivers and Services

InfiniPath and OpenFabrics Driver Overview . . . . . . . . . . . . . . . . . . . . . . . .

6-1

OpenFabrics Drivers and Services Configuration and Startup. . . . . . . . . . .

6-1

Configuring the IPoIB Network Interface . . . . . . . . . . . . . . . . . . . . . . .

6-2

OpenSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-3

SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-4

Using QLogic SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-4

Using OFED SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-4

Configuring and Administering the VNIC Interface . . . . . . . . . . . . . . .

6-6

MPI over uDAPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-13

Draft

6-14

Other Configuration: Changing the MTU Size . . . . . . . . . . . . . . . . . . . . . . .

Managing the InfiniPath Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-15

Configure InfiniPath Driver State . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-16

Start, Stop or Restart InfiniPath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-16

Unloading the Driver/Modules Manually . . . . . . . . . . . . . . . . . . . . . . .

6-17

Further Information on Configuring and Loading Drivers . . . . . . . . . . . . . . .

6-17

Installation Verification and Additional Settings

 

LED Link and Data Indicators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7-1

Adapter and Other Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7-1

Customer Acceptance Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7-2

Installation Troubleshooting

 

Hardware Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-1

Node Spontaneously Reboots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-1

Some HTX Motherboards May Need Two or More CPUs in Use . . . .

A-1

BIOS Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-2

Enable Advanced Configuration and Power Interface (ACPI). . . . . . .

A-2

Issue with Supermicro® H8DCE-HTe and QHT7040 . . . . . . . . . . . . .

A-2

Software Installation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-2

Missing Kernel RPM Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-2

Resolving Conflicts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-4

openmpi_gcc Fails to Install Because of Dependency on gfortran

 

(RHEL 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-4

mpirun Installation Requires 32-bit Support . . . . . . . . . . . . . . . . . . .

A-4

Lockable Memory Error on Initial Installation of InfiniPath. . . . . . . . . .

A-5

Configuration Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-5

IB0056101-00 G.02

Preliminary

xi

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

S

ibsrpdm Command Hangs When Two HCAs are Installed but Only Unit 1 is

Connected to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-5

Outdated ipath_ether Configuration Setup Generates Error . . . . .

A-5

B

Write Combining

 

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

Verify Write Combining is Working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

PAT and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

MTRR Mapping and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

Edit BIOS Settings to Fix MTRR Issues . . . . . . . . . . . . . . . . . . . . . . .

 

Use the ipath_mtrr Script to Fix MTRR Issues. . . . . . . . . . . . . . . .

C

Configuration Files

D

Draft

Package Descriptions

 

Package Names with the QLogicIB-Basic Download. . . . . . . . . . . . . . . . . .

 

Different Nodes May Use Different RPMs . . . . . . . . . . . . . . . . . . . . . . . . . .

 

InfiniPath RPM Version Numbers and Identifiers . . . . . . . . . . . . . . . . . . . . .

 

OpenFabrics RPM Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

Documentation RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

InfiniPath RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

Other HCAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

 

Other MPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B-1

B-1

B-2

B-2

B-2

B-3

D-1

D-1

D-2

D-2

D-2

D-3

D-3

D-4

D-9

D-10

xii

Preliminary

IB0056101-00 G.02

A

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

List of Figures

Figure

4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-2 QLogic QLE7140 Card with Riser, Top View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-3 QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View . . . .

4-4 PCIe Slot in a Typical Motherboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-5 QLogic PCIe HCA Assembly with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-6 Assembled PCIe HCA with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-7 HTX Slot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-8 QLogic QHT7140 Adapter with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-9 Assembled QHT7140 with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-10 QHT7140 Without Riser Installed in a 3U Chassis . . . . . . . . . . . . . . . . . . . . . . . . .

Table

List of Tables

 

1-1

Typographical Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2-1

QLogic Adapter Model Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-1

Adapter Models and Related Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-2

QLogic InfiniBand Cables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-1

InfiniPath/OpenFabrics Supported Distributions and Kernels . . . . . . . . . . . . . . . . .

5-2

Distribution Identifiers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-3

Required OS Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-4

Specific Component Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-5

Available Packages for QLogic OFED 1.4 Release . . . . . . . . . . . . . . . . . . . . . . . . .

5-6

INSTALL Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7-1

ipath_checkout Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C-1

Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-1

Documentation/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-2

InfiniPath/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-3

InfiniPath-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-4

InfiniPath-MPI/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-5

OpenFabrics/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-6

OpenFabrics-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-7

OpenSM/RPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-8

OpenSM-Devel/RPMDraft. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-9

Other HCAs/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-10

Other HCAs-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D-11

OtherMPIs/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Page

4-7

4-7

4-8

4-10

4-11

4-12

4-13

4-14

4-15

4-17

Page

1-3

2-3

4-1

4-3

5-2

5-2

5-4

5-4

5-5

5-12

7-3 C-1 D-3 D-3 D-4 D-4 D-4 D-7 D-8 D-9 D-9 D-9 D-10

IB0056101-00 G.02

Preliminary

xiii

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

S

Notes

Draft

xiv

Preliminary

IB0056101-00 G.02

1 Introduction

This chapter describes the contents, intended audience, and organization of the

QLogic HCA and QLogic OFED Software Install Guide.

The QLogic HCA and QLogic OFED Software Install Guide contains instructions for installing the QLogic Host Channel Adapters (HCAs) and the QLogic InfiniPath and OpenFabrics software. The following adapters are covered in this guide:

QLE7140 PCI Express® (PCIe)

Who ShouldDraftRead this Guide

QLE7240 PCI Express

QLE7280 PCI Express

QHT7040/QHT7140 HyperTransport Expansion (HTX™)

This installation guide is intended for cluster administrators responsible for installing the QLogic QLE7140, QLE7240, QLE7280 or QHT7040/QHT7140 adapter and QLogic InfiniPath software on their Linux® cluster. Additional detailed installation information and instructions for administering the QLogic cluster can be found in the QLogic HCA and QLogic OFED Software Users Guide.

The QLogic HCA and QLogic OFED Software Install Guide assumes that you are familiar with both cluster networking and the specific hardware that you plan to use. Before installing the HCA, you should have basic knowledge of your host and target operating systems, and working knowledge of message passing concepts.

This document does not contain all the information you need to use basic Linux commands or to perform all system administration tasks. For this information, see the software documentation you received with your system.

How this Guide is Organized

The QLogic HCA and QLogic OFED Software Install Guide is organized into these sections:

Section 1, Introduction, contains an overview of the HCAs and software, describes interoperability with other products, lists all related documentation, and provides QLogic contact information.

Section 2, Feature Overview, contains features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components.

IB0056101-00 G.02

Preliminary

1-1

Introduction

Overview

S

 

 

 

 

Section 3, Step-by-Step Installation Checklist, provides a high-level

 

overview of the hardware and software installation procedures.

 

Section 4, Hardware Installation, includes instructions for installing the

 

QLogic QLE7140, QLE7240, QLE7280, QHT7040, and QHT7140 HCAs.

 

Section 5, Software Installation, includes instructions for installing the

 

QLogic InfiniPath and OpenFabrics software.

 

Section 6, Configuring Drivers and Services, includes instructions for

 

configuring the QLogic InfiniPath and OpenFabrics drivers and services.

 

Section 7, Installation Verification and Additional Settings, describes tools

 

for verifying the installation, and adapter settings for best performance.

 

Appendix A, Installation Troubleshooting contains troubleshooting

 

information about issues that may occur during installation.

 

Draft

 

Appendix B, Write Combining contains information about settings that will

 

ensure better performance.

 

Appendix C, Configuration Files contains descriptions of the configuration

 

used by the QLogic InfiniPath and OpenFabrics software.

 

Appendix D, Package Descriptions contains RPM Descriptions.

 

Index lists major subjects and concepts with page numbers for easy

reference.

Overview

is

and the

rate SDR

The QLogic adapters utilize standard, off-the-shelf InfiniBand 4X switches and cabling. The QLogic interconnect is designed to work with all InfiniBand-compliant switches.

NOTE:

If you are using the QLE7240 or QLE7280, and want to use DDR mode, then DDR-capable switches must be used.

1-2

Preliminary

IB0056101-00 G.02

A

Introduction

Interoperability

QLogic OFED OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases. There are several options for subnet management in your cluster:

Use the embedded Subnet Manager (SM) in one or more managed switches supplied by your InfiniBand switch vendor.

Use a host-based Subnet Manager. QLogic provides one, HSM, as a part of the InfiniBand fabric Suite download.

Use the Open source Subnet Manager (OpenSM) component of OpenFabrics.

Interoperability

 

Draft

 

InfiniPath OpenFabrics (including Internet Protocol over InfiniBand (IPoIB))

 

is interoperable with other vendors’ InfiniBand HCAs running compatible

 

OpenFabrics releases.

 

 

The QLogic MPI stack is not interoperable with other InfiniBand HCAs and

 

Target Channel Adapters (TCAs). Instead, it uses an InfiniBand-compliant,

 

vendor-specific protocol that is highly optimized for QLogic MPI and for MPI

 

over verbs.

 

Conventions

 

 

 

 

 

 

Convention

Meaning

 

 

 

command Fixed-space font is used for literal items such as commands, functions, programs, files and pathnames, and program output.

variable Italic fixed-space font is used for variable names in programs and command lines.

IB0056101-00 G.02

Preliminary

1-3

Introduction

Documentation

S

Table 1-1. Typographical Conventions (Continued)

Convention

Meaning

 

 

concept Italic font is used for emphasis and concepts, as well as for documentation names/titles.

user input Bold fixed-space font is used for literal items in commands or constructs that you type.

$

Indicates a command line prompt.

#Indicates a command line prompt as root.

[ ]

Brackets enclose optional elements of a command or program con-

 

struct.

 

Draft

...

Ellipses indicate that a preceding element can be repeated.

>

A right caret identifies the cascading path of menu commands used

 

in a procedure.

QLogic OFED

The current version number of the software is included within this

1.4

documentation.

NOTE:

Indicates important information.

Documentation

The product documentation includes:

The QLogic HCA and QLogic OFED Software Install GuideThe QLogic HCA and QLogic OFED Software Users GuideThe QLogic InfiniBand Software Install Guide

The QLogic ULP and Tools Reference Guide (OFED+ Users Guide)

Release NotesQuick Start GuideReadme file

For more information on system administration, using the QLogic Message-Passing Interface (MPI), and troubleshooting adapter hardware and software, see the QLogic HCA and QLogic OFED Software Users Guide.

1-4

Preliminary

IB0056101-00 G.02

A

Introduction

Contact Information

Contact Information

Support Headquarters

QLogic Corporation

 

4601 Dean Lakes Blvd

 

Shakopee, MN 55379

 

USA

 

 

QLogic Web Site

www.qlogic.com

Technical Support Web Site

support.qlogic.com

Technical Support Email

support@qlogic.com

Technical Training Email

tech.training@qlogic.com

 

 

 

 

Draft

 

Additional contact information is available from the Contact Support area of the Technical Support Web Site.

IB0056101-00 G.02

Preliminary

1-5

Introduction

Contact Information

S

Notes

Draft

1-6

Preliminary

IB0056101-00 G.02

2 Feature Overview

This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components.

Features

Draft

 

 

Installation improvements. Provides a single software load for InfiniBand

 

HCAs from QLogic and other vendors supported by OFED. the software is

 

available packaged in the following ways:

 

Text User Interface (TUI) installer available (with the QLogicIB-Basic*

 

download). TUI is used for install on smaller clusters. Software can be

 

installed either standalone or via FastFabric (if the QLogic InfiniBand

 

Fabric Suite is purchased).

 

Software packaged for use with rpm install method.

 

A subset of the software (the accelerated MPI stack, precompiled

 

versions of MVAPICH and Open MPI, and other user-level tools) can

 

be installed on top of stock OFED or on an IB-enabled distribution.

 

Software packaged for Rocks installation method.

 

Software packaged for Platform OCS installation method

 

Write-combining (WC) mappings for the PIO buffers is now configured by

 

default using the x86 Page Attribute Table (PAT) mechanism.

IB0056101-00 G.02

Preliminary

2-1

Feature Overview

Features

S

MVAPICH and OpenMPI compiled with newer versions of each of four different compilers (GCC, PGI, Intel and PathScale) are available.

The QLogic InfiniBand Fabric Suite (IFS) is available separately for purchase. It includes FastFabric, the QLogic Host Subnet Manager (HSM), and the Fabric Viewer, and the InfiniServ Host Software.The QLogic OFED 1.4 software is supported by IFS.

Support for newer compiler versions (PathScale 3.x, PGI 7.x, PGI 8.x, Intel 10.x, Intel 11.x)

Support for newer Linux distributions, including RHEL 4 U7Performance enhancements and bug fixes

Other

Changes

 

 

Draft

 

 

ipath ether Ethernet emulation has been removed; IPoIB-CM may be

used instead.

The /etc/init.d/infinipath command to start the InfiniPath service has been replaced by the /etc/init.d/openibd command.

The infinipath-kernel RPM no longer exists: it has been integrated into the kernel-ib RPM.

Continued

Multiple high-performance native PSM Message Passing Interface (MPI) implementations. (PSM is QLogic’s accelerated library for high performance MPIs). In addition to QLogic MPI, the currently supported MPI implementations are HP-MPI, Open MPI, MVAPICH, and Scali (Platform). Open MPI provides MPI-2 functionality, including one-sided operations and dynamic processes. These all offer the same high performance as QLogic MPI.

Dual PCIe QLogic adapters per node.

QLogic MPI supports running exclusively on a single node without the installation of the HCA hardware.

4K Maximum Transfer Unit (MTU) is supported and is on by default. To take advantage of 4KB MTU, use a switch that supports 4KB MTU. QLogic also supports 2KB switches, and 4KB MTU switches configured for 2KB MTU.

QLogic switches with firmware version 4.2.x or later are recommended.

2-2

Preliminary

IB0056101-00 G.02

A

Feature Overview

Features

This version of the QLogic OFED software provides support for all of the QLogic HCAs in Table 2-1.

 

Table 2-1. QLogic Adapter Model Numbers

 

 

QLogic Model

Description

Number

 

 

 

 

 

 

QHT7040

 

Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For

 

 

systems with HTX expansion slots.

 

 

 

QHT7140 a

 

Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For

 

 

systems with HTX expansion slots.

 

 

 

QLE7140

 

Single port 10Gbps SDR 4X InfiniBand to PCI Express x8

 

 

adapter. Supported on systems with PCI Express (PCIe) x8 or

Draft

 

 

x16 slots.

QLE7240

 

Single port 20Gbps DDR 4X InfiniBand to PCI Express x8

 

 

adapter. Supported on systems with PCI Express x8 or

 

 

x16 slots.

QLE7280

 

Single port 20Gbps DDR 4X InfiniBand to PCI Express x16

 

 

adapter. Supported on systems with PCI Express x16 slots.

 

 

The QLE7280 is backward compatible; it can also be used with

 

 

PCIe adapters that connect to x8 slots.

Table Notes

 

 

PCIe is Gen 1

 

 

a The QHT7140 has a smaller form factor than the QHT7040, but is otherwise the same. Throughout this document, the QHT7040 and QHT7140 will be collectively referred to as the QHT7140 unless otherwise noted.

Additional up-to-date information can be found on the QLogic web site, specifically:

The high performance computing page at www.qlogic.com/Products/HPC products landingpage.aspx

The InfiniBand HCA page at www.qlogic.com/Products/HPC products infipathhcas.aspx

IB0056101-00 G.02

Preliminary

2-3

Feature Overview

Software Components

S

Software Components

This release includes all of OFED 1.4 with enhancements (QLogic OFED 1.4), including a new version of the VNIC tools and driver, and support for the QHT7xxx and QLE7xxx adapters. The software includes the QLogic InfiniPath HCA driver, libraries, QLogic MPI, Subnet Management Agent, and associated utilities. Included components are:

InfiniPath driver

InfiniPath libraries, InfiniPath utilities, configuration, and support tools, including ipath_checkout, ipath_control, ipath_pkt_test, and ipathstats

QLogic MPI

PSMDraftsupport for accelerated MPI

OpenMPI and MVAPICH (with PSM support) built with the GNU, PGI, PathScale, and Intel compilers, with corresponding mpitests and mpi-selector

QLogic MPI benchmarks and utilities

OpenFabrics protocols

OpenFabrics libraries and utilities

QLogic VNIC module

FastFabric Enablement tools services:

OFED SRP

QLogic SRP

No support is provided for Reliable Datagram (RD).

More details about the hardware and software can be found in Section 4 and Section 5.

2-4

Preliminary

IB0056101-00 G.02

3 Step-by-Step

Installation Checklist

This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 Hardware Installation” and Section 5 Software Installation”.

Hardware

Installation

1.

Check that the adapter hardware is appropriate for your platform. See

 

Table 4-1.

2.

Check to see that you have the appropriate cables and switches, as

 

described in “Cabling and Switches” on page 4-3.

3.

Check to see that you are running a supported Linux distribution/kernel. See

 

Table 5-1.

4.

Verify that the BIOS for your system is configured for use with the QLogic

 

adapter. See “Configuring the BIOS” on page 4-4.

5.

Following the safety instructions in “Safety with Electricity” on page 4-5.

 

Unpack the adapter (“Unpacking Information” on page 4-5) and verify the

 

package contents.

6.

Install the adapter by following the instructions in “Hardware Installation” on

 

page 4-9.

7.

Cable the adapter to the switch, as described in “Cabling the Adapter to the

 

InfiniBand Switch” on page 4-17. Check that all InfiniBand switches are

 

configured.

8.

FollowDraftthe steps in “Completing the Installation” on page 4-18 to finish the

installation.

IB0056101-00 G.02

Preliminary

3-1

Step-by-Step Installation Checklist

Software Installation

S

Software Installation

The following steps summarize the basic QLogic OFED 1.4 software installation and startup. These steps must be performed on each node in the cluster:

1.Make sure that the HCA hardware installation has been completed according to the instructions in “Hardware Installation” on page 4-1.

2.Verify that the Linux kernel software is installed on each node in the cluster. The required kernels and supported Linux distributions for both QLogic InfiniPath and OpenFabrics are defined in Table 5-1.

3.Make sure that your environment has been set up as described in “Setting Up Your Environment” on page 5-3.

4.Download your version of the QLogic InfiniPath/OpenFabrics software from theDraftQLogic web site to a local server directory. See “Choose the Appropriate Download Files” on page 5-4.

5.Install the selected packages on each cluster node using the corresponding method as described in one of the following: “Install QLogicIB-Basic with the Installer Tool” on page 5-7, “Using rpm to Install InfiniPath and OpenFabrics” on page 5-14, “Install QLogic OFED User-level Software with the rpm Command” on page 5-17, “Install QLogic OFED Using Rocks” on page 5-20, “Install QLogic OFED Using a Platform OCS Kit” on page 5-22, “Install FastFabric Software CD/ISO Image” on page 5-22.

6.The system can be rebooted after all the software has been installed.

7.The configuration file for the ib ipath driver is set up correctly at installation and is loaded automatically during system boot once the RPMs have been installed. However, if you wish to change the configuration file, see “Managing the InfiniPath Driver” on page 6-15.

8.If you want to configure the optional OpenFabrics driver ipoib, and you have not configured it yet with the Install tool, or if you have used the rpm install method, configure it as described in “Configuring the IPoIB Network Interface” on page 6-2.

9.If you want to use the optional OpenFabrics services (opensm, srp,or VNIC), configure them as described in “OpenSM” on page 6-3, “SRP” on page 6-4, or “SRP stands for SCSI RDMA Protocol. It was originally intended to allow the SCSI protocol to run over InfiniBand for Storage Area Network (SAN) usage. SRP interfaces directly to the Linux file system through the SRP Upper Layer Protocol. SRP storage can be treated as another device.” on page 6-4.

10.Check the system state by observing the LEDs. See “LED Link and Data Indicators” on page 7-1.

3-2

Preliminary

IB0056101-00 G.02

A

Step-by-Step Installation Checklist

Software Installation

11. You can optimize your system and adapter for the best performance. See “Adapter and Other Settings” on page 7-1. Also see the Performance Settings and Management Tips section in the QLogic HCA and QLogic OFED Software Users Guide.

12. Perform the recommended health checks. See “Customer Acceptance Utility” on page 7-2.

13. After installing the QLogic InfiniPath and OpenFabrics software, refer to the

QLogic HCA and QLogic OFED Software Users Guide for more information about using QLogic InfiniPath, QLogic MPI, and OpenFabrics products. Refer to the QLogic ULP and Tools Reference Guide for more information about configuring and using QLogic SRP, QLogic VNIC, and the Enablement Tools. The InfiniBand Software Installation Guide also has information on

installing the QLogic InfiniBand Fabric Suite. Draft

IB0056101-00 G.02

Preliminary

3-3

Step-by-Step Installation Checklist

Software Installation

S

Notes

Draft

3-4

Preliminary

IB0056101-00 G.02

4 Hardware Installation

This section lists the requirements and provides instructions for installing the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic PCIe adapter and PCIe riser card, QLE7140; and the QHT7040 or QHT7140 adapter hardware and HTX riser card. These components are collectively referred to as the adapter and the riser card in the remainder of this document.

data rate of 20Gbps (data rate of 16Gbps) for the QLE7240 and QLE7280; and 10Gbps (data rate of 8Gbps) for the QLE7140, QHT7040, and QHT7140.

The adapter is a low-latency, high-bandwidth, high message rate cluster interconnectDraftfor InfiniBand. The QLogic interconnect is InfiniBand 4X, with a raw

OpenFabrics is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases.

Hardware Installation Requirements

This section lists hardware and software environment requirements for installing the QLogic QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140.

Hardware

QLogic interconnect adapters are for use with UL listed computers. The following statement is true for all the adapters:

This device complies with part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) This device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operations.

Different adapter cards work on different platforms. Table 4-1 shows the relationship between the adapter model and different types of motherboards.

Table 4-1. Adapter Models and Related Platforms

QLogic

Platform

Plugs Into

Model

Number

 

 

 

 

 

QLE7240

PCI Express systems

Standard PCI Express x8 or x16 slot

 

 

 

QLE7280

PCI Express systems

Standard PCI Express x16 slot

 

 

 

IB0056101-00 G.02

Preliminary

4-1

Hardware Installation

Hardware Installation Requirements

S

Table 4-1. Adapter Models and Related Platforms (Continued)

QLogic

Platform

Plugs Into

Model

Number

 

 

 

 

 

QLE7140

PCI Express systems

Standard PCI Express x8 or x16 slot

 

 

 

QHT7040

Motherboards with HTX connectors

HyperTransport HTX slot

 

 

 

QHT7140

Motherboards with HTX connectors

HyperTransport HTX slot

 

 

 

Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of a PCI Express (PCIe) slot in a typical motherboard. See Figure 4-7 for an illustrationDraftof an HTX slot for a typical Opteron motherboard.

The motherboard vendor is the optimal source for information on the layout and use of HyperTransport and PCI Express-enabled expansion slots on supported motherboards.

Form Factors

The QLE7240, QLE7280, and QLE7140 are the model numbers for the adapters that ship in the standard PCI Express half-height, short-form factor. These adapters can be used with either full-height or low-profile face plates.

The QHT7040 is the model number for the adapter that shipped in the HTX full-height factor. The HTX low-profile form factor is referred to as the QHT7140. It is the same as the QHT7040, except for its more compact size. In either case, the adapter is backward and forward compatible for the motherboards in which it is supported. The QHT7040 and QHT7140 HTX adapters are collectively referred to as the QHT7140 unless otherwise stated.

When the QHT7040 or QHT7140 adapter is installed with the riser card, it may prevent some or all of the other PCI expansion slots from being used, depending on the form factor of the adapter and motherboard.

Run ipath control -i to see information on which form adapter is installed. The file /sys/class/infiniband/ipath0/device/boardversion contains the same information. For more information, see the Useful Programs and Files appendix in the QLogic HCA and QLogic OFED Software Users Guide.

4-2

Preliminary

IB0056101-00 G.02

A

Hardware Installation

Hardware Installation Requirements

Cabling and Switches

The cable installation uses a standard InfiniBand (IB) 4X cable. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cable that QLogic has currently qualified is 20 meters. For DDR-capable adapters and switches, the DDR-capable passive copper cables cannot be longer than 10 meters. Active cables can eliminate some of the cable length restrictions.

InfiniBand switches are available through QLogic.

NOTE:

If you are using the QLE7240 or QLE7280 and want to use DDR mode, then

DDR-capable switches must be used.

7104-12MDraft-Cable 4x-4x cable—12 meters (SDR only)

The copper cables listed in Table 4-2 are available from QLogic:

Table 4-2. QLogic InfiniBand Cables

Product Number

Description

7104-1M-Cable

4x-4x cable—1 meter

7104-2M-Cable

4x-4x cable—2 meters

7104-3M-Cable

4x-4x cable—3 meters

7104-4M-Cable

4x-4x cable—4 meters

7104-5M-Cable

4x-4x cable—5 meters

 

 

7104-6M-Cable

4x-4x cable—6 meters

 

 

7104-7M-Cable

4x-4x cable—7 meters

 

 

7104-8M-Cable

4x-4x cable—8 meters

 

 

7104-9M-Cable

4x-4x Cable—9 meters

 

 

7104-10M-Cable

4x-4x cable—10 meters

 

 

 

 

7104-14M-Cable

4x-4x cable—14 meters (SDR only)

 

 

7104-16M-Cable

4x-4x cable—16 meters (SDR only)

 

 

7104-18M-Cable

4x-4x cable—18 meters (SDR only)

 

 

For cabling instructions, see “Cabling the Adapter to the InfiniBand Switch” on page 4-17.

IB0056101-00 G.02

Preliminary

4-3

Hardware Installation

Hardware Installation Requirements

S

Optical Fibre Option

The QLogic adapter also supports connection to the switch by means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these types of convertors. For more information on the EMCORE convertor, see www.emcore.com.

Intel® and Zarlink™ also offer optical cable solutions. See www.intel.com and www.zarlink.com for more information.

Configuring the BIOS

To achieve the best performance with QLogic adapters, you need to configure your BIOS with specific settings. The BIOS settings, which are stored in non-volatile memory, contain certain parameters characterizing the system. These

parameters may include date and time, configuration settings, and information

 

Draft

 

Advanced Configuration and Power Interface (ACPI) BIOS option must be

enabled.

Interface

documentation

is

information,

4-4

Preliminary

IB0056101-00 G.02

A

Hardware Installation Safety with Electricity

Safety with Electricity

Observe these guidelines and safety precautions when working around computer hardware and electrical equipment:

Locate the power source shutoff for the computer room or lab where you are working. This is where you will turn OFF the power in the event of an emergency or accident. Never assume that power has been disconnected for a circuit; always check first.

Do not wear loose clothing. Fasten your tie or scarf, remove jewelry, and roll up your sleeves. Wear safety glasses when working under any conditions that might be hazardous to your eyes.

Shut down and disconnect the system’s power supply from AC service beforeDraftyou begin work, to insure that standby power is not active. Power off all attached devices such as monitors, printers, and external components. Note that many motherboards and power supplies maintain standby power at all times. Inserting or removing components while standby is active can damage them.

Use normal precautions to prevent electrostatic discharge, which can damage integrated circuits.

Unpacking

Verify the

List of the

QLogic QLE7240

Additional short bracket

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or motherboard vendor.

IB0056101-00 G.02

Preliminary

4-5

Hardware Installation

Unpacking Information

S

The package contents for the QLE7280 adapter are:

QLogic QLE7280

Additional short bracket

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or motherboard vendor.

The package contents for the QLE7140 adapter are:

QLogic QLE7140

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or

motherboard vendor. The contents are illustrated in Figure 4-2.

 

Draft

The package contents for the QHT7140 adapter are:

 

QLogic QHT7140

HTX riser card for use in 1U or 2U chassis

 

Quick Start Guide

The contents are illustrated in Figure 4-3.

The IBA6120, IBA6110, and IBA7220 are the QLogic ASICs, which are the central components of the interconnect. The location of the IBA7220 ASIC on the adapter is shown in Figure 4-1. The location of the IBA6120 ASIC on the adapter is shown in Figure 4-2. The location of the IBA6110 ASIC on the adapter is shown in Figure 4-3.

4-6

Preliminary

IB0056101-00 G.02

A

Hardware Installation

Unpacking Information

PCI Express

Edge Connectors

InfiniBand

Connector

Draft

Face Plate

 

 

 

 

IBA7220 ASIC

 

Figure 4-1. QLogic QLE7280 with IBA7220 ASIC

 

 

PCI Express Riser Card. Not

 

 

supplied; shown for reference.

PCI Express

 

 

InfiniBand

Edge Connectors

 

 

 

 

 

Connector

 

IBA6120 ASIC

 

Face Plate

 

 

 

Figure 4-2. QLogic QLE7140 Card with Riser, Top View

IB0056101-00 G.02

Preliminary

4-7

Hardware Installation

Unpacking Information

S

HTX Riser Card

HTX Edge

 

InfiniBand

Connectors

 

 

Connector

 

 

 

 

 

QHT7140 Low

 

Figure 4-3. QLogic

Draft

 

 

 

Unpacking

 

 

 

1.

When unpacking, ground yourself before removing the QLogic adapter from

 

the anti-static bag.

 

2.

Grasping the QLogic adapter by its face plate, pull the adapter out of the

 

anti-static bag. Handle the adapter only by its edges or the face plate. Do not

 

allow the adapter or any of its components to touch any metal parts.

3.

After checking for visual damage, store the adapter and the riser card in their

 

anti-static bags until you are ready to install them.

 

4-8

Preliminary

IB0056101-00 G.02

A

Hardware Installation

Hardware Installation

Hardware Installation

This section contains hardware installation instructions for the QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140.

Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser

Installation for the QLE7240, QLE7280, and QLE7140 is similar. The following instructions are for the QLE7140, but can be used for any of these three adapters.

Most installations will be in 1U and 2U chassis, using a PCIe right angle riser card.

This results in an installation of the adapter that is parallel to the motherboard.

This type of installation is described first. Installation in a 3U chassis is described

in “Hardware Installation for the QHT7140 Without an HTX Riser” on page 4-16.

Dual Adapter

Draft

 

Installation Steps

 

1.

The BIOS should already be configured properly by the motherboard

 

manufacturer. However, if any additional BIOS configuration is required, it

 

will usually need to be done before installing the QLogic adapter. See

 

“Configuring the BIOS” on page 4-4.

2.

Shut down the power supply to the system into which you will install the

 

QLogic adapter.

3.

Take precautions to avoid electrostatic damage (ESD) to the cards by

 

properly grounding yourself or touching the metal chassis to discharge static

 

electricity before handling the cards.

IB0056101-00 G.02

Preliminary

4-9

Hardware Installation

Hardware Installation

S

4.Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system.

5.Locate the PCIe slot on your motherboard. Note that the PCIe slot has two separate sections, with the smaller slot opening located towards the front (see Figure 4-4). These two sections correspond to the shorter and longer connector edges of the adapter and riser.

6.DetermineDraftif a blanking panel is installed in your chassis. If it is, remove it so that the InfiniBand connector will be accessible. Refer to your system vendor instructions for how to remove the blanking panel.

7.Remove the QLogic adapter from the anti-static bag.

8.Locate the face plate on the connector edge of the card.

4-10

Preliminary

IB0056101-00 G.02

+ 86 hidden pages