Q-Logic IB0056101-00 G.02, OFED User Manual

QLogic HCA and QLogic OFED Software

Install Guide

Draft

QLogic OFED Version 1.4

IB0056101-00 G.02

Preliminary

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no representation nor warranty that such applications are suitable for the specified use without further testing or modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document.

No part of this document may be copied nor reproduced by any means, nor translated nor transmitted to any magnetic medium without the express written consent of QLogic Corporation. In accordance with the terms of their valid QLogic agreements, customers are permitted to make electronic and paper copies of this document for their own exclusive use.

The QHT7040, QHT7140, QLE7140, QLE7240, and QLE7280 QLogic Host Channel Adapters are covered by the following patent: 7308535.

Document Revision History

Rev. 1.0, 8/20/2005	Draft
Rev. 1.0, 8/20/2005
Rev. 1.1, 11/15/05

Rev. 1.2, 02/15/06

Rev. 1.3, Beta 1, 4/15/06

Rev 1.3, 6/15/06

Rev. 2.0 Beta, 8/15/06, QLogic Version Number IB0056101-00 A

Rev. 2.0 Beta 2 10/15/06, QLogic Version Number IB0056101-00 B

Rev. 2.0 11/30/06, QLogic Version Number IB0056101-00 C

Rev. 2.0 3/23/07, QLogic Version Number IB0056101-00 D

Rev. 2.1 8/24/07, QLogic Version Number IB0056101-00 E

Rev. 2.2 5/27/08, QLogic Version Number IB0056101-00 F

Rev. 2.2 9/5/08, QLogic Version Number IB0056101-00 G

Rev. QLogic OFED 1.4 4/7/09, QLogic Version Number IB0056101-00 G.02

Changes	Sections Affected


Product name changed from InfiniPath to QLogic	All
OFED. Version number is set to 1.4. Instances of
InfiniPath changed where appropriate; some file-
names and output messages keep old name.

Updated Contact Information.	“Contact Information” on page 1-5

Removed InfiniPath from 1st paragraph; replaced	“Hardware Installation” on page 4-1
with QLogic adapters, etc.

Changed section title from What’s New in this	“Features” on page 2-1
Release to Features.

Page ii

Preliminary

IB0056101-00 G.02

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4



Replace EM64T with 64-bit Intel Xeon.		“Supported Linux Distributions” on page 5-1

Added more information on VNIC interface.		“OpenSM” on page 6-3

Split the Install section into three separate sec-		“Software Installation” on page 5-1
tions.		“Configuring Drivers and Services” on page 6-1

		“Installation Verification and Additional Settings” on
		page 7-1

Modified installation checklist. Clarified which driv-		“Software Installation” on page 3-2
ers are configured, which are optional. Added two
methods of installation to the list.

Remove sections on configuring ipath_ether.		Was “Configuring the ipath_ether Network Inter-
ipath_ether now deprecated.		face” on page 5-12

	Draft
Updated the supported distributions information.		Table 5-1 on page 5-2
Distribution identifiers are now RHEL4, RHEL5,		Table 5-2 on page 5-2
and SLES10.
Delete reference to ipath ether.		“Removing Software Packages” on page 5-24
Updated Lustre information; patches are no longer		“Installing Lustre” on page 5-23
needed.
Updated compiler support information.		“Compiler Support” on page 5-3
Updated information on software components.		“Software Components” on page 2-4
Removed paragraph about this release adding		“Features” on page 2-1
support for QLE7240/7280, since this is outdated.
Added new “Configuration Issues” to Trouble-		“ibsrpdm Command Hangs When Two HCAs are
shooting.		Installed but Only Unit 1 is Connected to the
		Switch” on page A-5

Deleted Troubleshooting item:		Appendix A
ifup on ipath_ether on SLES 10 Reports
"unknown device"

Deleted entries pertaining to ipath ether in this		Table C-1 on page C-1
table.

IB0056101-00 G.02

Preliminary

Page iii

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4



Sections rearranged, renamed, and expanded due		“Choose the Appropriate Download Files” on
to multiple install methods. Previous sections:		page 5-4
Installing the InfiniPath and Open fabrics RPMs		“Install QLogicIB-Basic with the Installer Tool” on
		“Install QLogicIB-Basic with the Installer Tool” on
Downloading and Unpacking the InfiniPath and		page 5-7
OpenFabrics Software		“About rpm Installation” on page 5-13
		“About rpm Installation” on page 5-13
		“Install QLogic OFED Using Rocks” on page 5-20
		“Install QLogic OFED Using a Platform OCS Kit”
		on page 5-22
		“Using rpm to Install InfiniPath and OpenFabrics”
		on page 5-14

Added new section about installing using the		“Install QLogicIB-Basic with the Installer Tool” on
Installer Tool.		page 5-7

Moved introductory information about RPM instal-		“About rpm Installation” on page 5-13
lation before sections on installing using the two
different methods.

Removed paragraph about kernel module support		Was in old section “Installing the InfiniPath and
being part of InfiniPath RPMs; was relevant to pre-		Open fabrics RPMs”
vious release.

Added note about need to use prefixed install if		“Using rpm to Install InfiniPath and OpenFabrics”
using QLogic MPI with mpi-selector.		on page 5-14

Modified module for installed layout. New location		“Installed Layout” on page 5-23
for ipath modules. Location of other OFED mod-
ules listed here.

Modified module for driver overview.		“InfiniPath and OpenFabrics Driver Overview” on
		page 6-1

Modified module for configuring IPoIB. Only restart		“Configuring the IPoIB Network Interface” on
information has been changed.		page 6-2

Modified module for configuring VNIC interface.		“SRP stands for SCSI RDMA Protocol. It was origi-
Changed infinipath start/stop commands, and		nally intended to allow the SCSI protocol to run
added some introductory material.		over InfiniBand for Storage Area Network (SAN)
		usage. SRP interfaces directly to the Linux file sys-
		tem through the SRP Upper Layer Protocol. SRP
	Draft
		storage can be treated as another device.” on
		page 6-4
Modified module for configuring SRP.		“SRP” on page 6-4

Modified module for uDAPL configuration.		“MPI over uDAPL” on page 6-13

Deleted reference to MTRR BIOS setting.		“Configuring the BIOS” on page 4-4
		“BIOS Settings” on page A-2

Page iv

Preliminary

IB0056101-00 G.02

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4



New Appendix for Write Combining. MTRR set-		“Write Combining” on page B-1
tings information moved here.

Minor wording change to introduction section.		“Software Installation” on page 5-1

Added new issue in Troubleshooting.		“openmpi_gcc Fails to Install Because of Depen-
		dency on gfortran (RHEL 4)” on page A-4
Added new issue in Troubleshooting.		“Outdated ipath_ether Configuration Setup
		Generates Error” on page A-5

Configuration file qlogic_vnic.cfg changed to		Table C-1 on page C-1
qlgc_vnic.cfg.
Added new section.		“Uninstalling Software with Rocks or Platform
		OCS” on page 5-25

Added new section.		“Install QLogic OFED Using Rocks” on page 5-20

Change name of Appendix RPM Descriptions to		“Package Descriptions” on page D-1
Package Descriptions. Updated information.

Removed phrase “and Transmission Control		“Interoperability” on page 1-3
Protocol (TCP)” from introduction; related to
now-obsolete ipath_ether.

Removed this issue from the Troubleshooting sec-		Was “OpenFabrics Library Dependencies”
tion.

Combined What’s New in This Release and Fea-		“Feature Overview” on page 2-1
tures section.

Deleted Note about Fedora Core 6 not supported		This was noted in InfiniPath 2.2.1 release notes.
in this release.

Removed this line “Please check the QLogic		“Software Components” on page 2-4
web site for updated information on supported
compilers.” This information is not on the web
site.

Added more details about setting the switch MTU		“Other Configuration: Changing the MTU Size” on
default to 4K.		page 6-14
	Draft
Deleted the section “Installing QLogic MPI in an		“Using rpm to Install InfiniPath and OpenFabrics”
Alternate Location with rpm”; merged instructions		on page 5-14
into rpm install section.

Minor text changes to this section. Modified title		“Install Additional Software” on page 5-23
from Additional Installation Instructions to Install
Additional Software. Moved this section to be with
the rest of the installation instructions.

IB0056101-00 G.02

Preliminary

Page v

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4



Changed title of Troubleshooting issue. Was		“Outdated ipath_ether Configuration Setup
“ipath_ether Configuration Setup Gener-		Generates Error” on page A-5
ates Error”

Change main configuration file from /etc/sys-		Table C-1 on page C-1 and “SRP” on page 6-4.
config/infinipath to /etc/infini-
band/openib.conf. The infinipath file is
still to enable or disable the ipath_mtrr script.

Updated note about setting datagram mode.		“Configuring the IPoIB Network Interface” on
		page 6-2

Deleted Troubleshooting Issue OpenFabrics		“Software Installation Issues” on page A-2
Dependencies.

Deleted Troubleshooting Issue Version Number		“Software Installation Issues” on page A-2
	Draft
Conflict with opensm-* on RHEL5 Systems
Two new tables showing OS package require-		Table on page 5-4 and Table on page 5-4
ments. Replaces some bullet points in “Distribution
Identifiers” on page 5-2
Removed section “Check for Missing Files or		Was “Check for Missing files or RPMs” on
RPMs”		page 5-5
Merged the two sections Downloading the QLogic		“Choose the Appropriate Download Files” on
OFED Software and Choose the Appropriate		page 5-4
Download Files
Removed Supported Compilers and Supported		2 Feature Overview
Linux Distributions from the 2 Feature Overview
chapter. Information is still in Software Installation
chapter.

Moved table “InfiniPath and OpenFabrics		“Package Descriptions” on page D-1
RPMs to Use for Each Node in a Cluster” from
Software Installation section to Appendix

Consolidated Configuring the InfiniPath Drivers		6 Configuring Drivers and Services
and InfiniPath and OpenFabrics Driver Overview

opensm is now off by default		Various

Merged “Install on an Unsupported Distribution		Now “Rebuilding the kernel-ib Driver on an
with the rpm Command” with the Rebuilding...sec-		Unsupported Distribution or an Unsupported Distri-
tion.		bution/Kernel Pair” on page 5-19

Removed $ or # at beginning of lines indicating		All. Also Table 1-1
commands. The #, if cut and pasted from the docu-
ment, may be interpreted as a shell command.

Updated footnote b.		Table D-11

Page vi

Preliminary

IB0056101-00 G.02

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4



Adapter Settings section name changed. Also		“Adapter and Other Settings” on page 7-1
added new bullet point “Check the PCIe bus width”

Updated tables of RPMs. Moved all documentation		All tables in D Package Descriptions
RPMs together. New table for OpenSM-Devel
RPMs. Updated OtherMPIs.

Deleted table 5-7 RPMs to Install, as information is		“Using rpm to Install InfiniPath and OpenFabrics”
repeated in “Using rpm to Install InfiniPath and		on page 5-14
OpenFabrics” on page 5-14

Moved RPM Organization section under RPM		“RPM Organization” on page 5-16
Installation.

Merge two sections Uninstalling InfiniPath and		“Uninstalling InfiniPath and OpenFabrics RPMs”
OpenFabrics RPMs and Uninstalling OFED 1.3		on page 5-24
Software.	Draft
Software.

Reference to this location		“Form Factors” on page 4-2
/sys/bus/pci/drivers/ib ipath/00/
changed to
/sys/class/infiniband/ipath*/device/
Removed note: “OpenFabrics programs (32-bit)		“Software Components” on page 2-4
using the Verbs interfaces are not supported
in this release, but may be supported in a
future release.”

IB0056101-00 G.02

Preliminary

Page vii

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

Notes

Draft

Page viii

Preliminary

IB0056101-00 G.02

Introduction

Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-1
How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-1
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-2
Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-3
Conventions Used in this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-3
Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-4
Contact Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-5

Feature Overview

Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-1
Other Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-2
Continued Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-2
Software Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-4

Step-by-Step Installation Checklist

Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	3-1
Software Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	3-2

Hardware Installation

Hardware Installation Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-1
Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-1
Form Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-2
Cabling and Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-3
Optical Fibre Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-4
Configuring the BIOS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-4
Safety withDraftElectricity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-5
Unpacking Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-5
Verify the Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-5
List of the Package Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-5
Unpacking the QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-8
Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-9

IB0056101-00 G.02

Preliminary

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Dual Adapter Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Installation Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hardware Installation for QHT7140 with HTX Riser . . . . . . . . . . . . . .

Hardware Installation for QLE7240, QLE7280, and QLE7140 Without a PCI Express Riser. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hardware Installation for the QHT7140 Without an HTX Riser . . . . . .

Switch Configuration and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Cabling the Adapter to the InfiniBand Switch . . . . . . . . . . . . . . . . . . . . . . . .

Completing the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5 Software Installation

Draft

Cluster Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Types of Nodes in a Cluster Environment . . . . . . . . . . . . . . . . . . . . . .

Supported Linux Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Distribution Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Compiler Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Setting Up Your Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Choose the Appropriate Download Files . . . . . . . . . . . . . . . . . . . . . . . . . . .

Install QLogicIB-Basic with the Installer Tool . . . . . . . . . . . . . . . . . . . . . . . .

About rpm Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Using rpm to Install InfiniPath and OpenFabrics . . . . . . . . . . . . . . . . . . . . .

RPM Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Install QLogic OFED User-level Software with the rpm Command . . . . . . .

Rebuilding or Reinstalling the kernel-ib Driver with rpm After a

Kernel Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Rebuilding the kernel-ib Driver on an Unsupported Distribution or an Unsupported Distribution/Kernel Pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Install QLogic OFED Using Rocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Install Frontend and Compute Nodes . . . . . . . . . . . . . . . . . . . . . . . . .

Rocks Installation on an Existing Frontend Node . . . . . . . . . . . . . . . .

Install QLogic OFED Using a Platform OCS Kit . . . . . . . . . . . . . . . . . . . . . .

Install FastFabric Software CD/ISO Image . . . . . . . . . . . . . . . . . . . . . . . . .

Install Additional Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Installing Lustre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Installed Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Removing Software Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Uninstalling Using the Installer Tool. . . . . . . . . . . . . . . . . . . . . . . . . . .

Uninstalling InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . .

4-9

4-12

4-15

4-16

4-17

4-18

5-1

5-2

5-3

5-4

5-7

5-13

5-14

5-16

5-17

5-19

5-20

5-21

5-22

5-23

5-24

Preliminary

IB0056101-00 G.02

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4


. . . . . . . . . . . . . . .Uninstalling Software with Rocks or Platform OCS		5-25
Downgrading RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-25

Configuring Drivers and Services

InfiniPath and OpenFabrics Driver Overview . . . . . . . . . . . . . . . . . . . . . . . .	6-1
OpenFabrics Drivers and Services Configuration and Startup. . . . . . . . . . .	6-1
Configuring the IPoIB Network Interface . . . . . . . . . . . . . . . . . . . . . . .	6-2
OpenSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	6-3
SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	6-4
Using QLogic SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	6-4
Using OFED SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	6-4
Configuring and Administering the VNIC Interface . . . . . . . . . . . . . . .	6-6
MPI over uDAPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	6-13
Draft	6-14
Other Configuration: Changing the MTU Size . . . . . . . . . . . . . . . . . . . . . . .	6-14
Managing the InfiniPath Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	6-15
Configure InfiniPath Driver State . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	6-16
Start, Stop or Restart InfiniPath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	6-16
Unloading the Driver/Modules Manually . . . . . . . . . . . . . . . . . . . . . . .	6-17
Further Information on Configuring and Loading Drivers . . . . . . . . . . . . . . .	6-17
Installation Verification and Additional Settings
LED Link and Data Indicators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	7-1
Adapter and Other Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	7-1
Customer Acceptance Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	7-2
Installation Troubleshooting
Hardware Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-1
Node Spontaneously Reboots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-1
Some HTX Motherboards May Need Two or More CPUs in Use . . . .	A-1
BIOS Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-2
Enable Advanced Configuration and Power Interface (ACPI). . . . . . .	A-2
Issue with Supermicro® H8DCE-HTe and QHT7040 . . . . . . . . . . . . .	A-2
Software Installation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-2
Missing Kernel RPM Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-2
Resolving Conflicts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-4
openmpi_gcc Fails to Install Because of Dependency on gfortran
(RHEL 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-4
mpirun Installation Requires 32-bit Support . . . . . . . . . . . . . . . . . . .	A-4
Lockable Memory Error on Initial Installation of InfiniPath. . . . . . . . . .	A-5
Configuration Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-5

IB0056101-00 G.02

Preliminary

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

ibsrpdm Command Hangs When Two HCAs are Installed but Only Unit 1 is

Connected to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	A-5
Outdated ipath_ether Configuration Setup Generates Error . . . . .	A-5

B	Write Combining
	Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	Verify Write Combining is Working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	PAT and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	MTRR Mapping and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	Edit BIOS Settings to Fix MTRR Issues . . . . . . . . . . . . . . . . . . . . . . .
	Use the ipath_mtrr Script to Fix MTRR Issues. . . . . . . . . . . . . . . .
C	Configuration Files
D	Draft
D	Package Descriptions
	Package Names with the QLogicIB-Basic Download. . . . . . . . . . . . . . . . . .
	Different Nodes May Use Different RPMs . . . . . . . . . . . . . . . . . . . . . . . . . .
	InfiniPath RPM Version Numbers and Identifiers . . . . . . . . . . . . . . . . . . . . .
	OpenFabrics RPM Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	Documentation RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	InfiniPath RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	Other HCAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	Other MPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B-1

B-2

B-3

D-1

D-2

D-3

D-4

D-9

D-10

xii

Preliminary

IB0056101-00 G.02

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

List of Figures

Figure

4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-2 QLogic QLE7140 Card with Riser, Top View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-3 QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View . . . .

4-4 PCIe Slot in a Typical Motherboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-5 QLogic PCIe HCA Assembly with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-6 Assembled PCIe HCA with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-7 HTX Slot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-8 QLogic QHT7140 Adapter with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-9 Assembled QHT7140 with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-10 QHT7140 Without Riser Installed in a 3U Chassis . . . . . . . . . . . . . . . . . . . . . . . . .

Table	List of Tables
Table
1-1	Typographical Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2-1	QLogic Adapter Model Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-1	Adapter Models and Related Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-2	QLogic InfiniBand Cables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-1	InfiniPath/OpenFabrics Supported Distributions and Kernels . . . . . . . . . . . . . . . . .
5-2	Distribution Identifiers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-3	Required OS Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-4	Specific Component Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-5	Available Packages for QLogic OFED 1.4 Release . . . . . . . . . . . . . . . . . . . . . . . . .
5-6	INSTALL Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7-1	ipath_checkout Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C-1	Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-1	Documentation/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-2	InfiniPath/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-3	InfiniPath-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-4	InfiniPath-MPI/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-5	OpenFabrics/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-6	OpenFabrics-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-7	OpenSM/RPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-8	OpenSM-Devel/RPMDraft. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-9	Other HCAs/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-10	Other HCAs-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D-11	OtherMPIs/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Page

4-7

4-8

4-10

4-11

4-12

4-13

4-14

4-15

4-17

Page

1-3

2-3

4-1

4-3

5-2

5-4

5-5

5-12

7-3 C-1 D-3 D-3 D-4 D-4 D-4 D-7 D-8 D-9 D-9 D-9 D-10

IB0056101-00 G.02

Preliminary

xiii

QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4

Notes

Draft

xiv

Preliminary

IB0056101-00 G.02

1 Introduction

This chapter describes the contents, intended audience, and organization of the

QLogic HCA and QLogic OFED Software Install Guide.

The QLogic HCA and QLogic OFED Software Install Guide contains instructions for installing the QLogic Host Channel Adapters (HCAs) and the QLogic InfiniPath and OpenFabrics software. The following adapters are covered in this guide:

QLE7140 PCI Express® (PCIe)

Who ShouldDraftRead this Guide

QLE7240 PCI Express

QLE7280 PCI Express

QHT7040/QHT7140 HyperTransport Expansion (HTX™)

This installation guide is intended for cluster administrators responsible for installing the QLogic QLE7140, QLE7240, QLE7280 or QHT7040/QHT7140 adapter and QLogic InfiniPath software on their Linux® cluster. Additional detailed installation information and instructions for administering the QLogic cluster can be found in the QLogic HCA and QLogic OFED Software Users Guide.

The QLogic HCA and QLogic OFED Software Install Guide assumes that you are familiar with both cluster networking and the specific hardware that you plan to use. Before installing the HCA, you should have basic knowledge of your host and target operating systems, and working knowledge of message passing concepts.

This document does not contain all the information you need to use basic Linux commands or to perform all system administration tasks. For this information, see the software documentation you received with your system.

How this Guide is Organized

The QLogic HCA and QLogic OFED Software Install Guide is organized into these sections:

Section 1, Introduction, contains an overview of the HCAs and software, describes interoperability with other products, lists all related documentation, and provides QLogic contact information.

Section 2, Feature Overview, contains features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components.

IB0056101-00 G.02

Preliminary

1-1

Introduction

Overview


	Section 3, Step-by-Step Installation Checklist, provides a high-level
	overview of the hardware and software installation procedures.
	Section 4, Hardware Installation, includes instructions for installing the
	QLogic QLE7140, QLE7240, QLE7280, QHT7040, and QHT7140 HCAs.
	Section 5, Software Installation, includes instructions for installing the
	QLogic InfiniPath and OpenFabrics software.
	Section 6, Configuring Drivers and Services, includes instructions for
	configuring the QLogic InfiniPath and OpenFabrics drivers and services.
	Section 7, Installation Verification and Additional Settings, describes tools
	for verifying the installation, and adapter settings for best performance.
	Appendix A, Installation Troubleshooting contains troubleshooting
	information about issues that may occur during installation.
	Draft
	Appendix B, Write Combining contains information about settings that will
	ensure better performance.
	Appendix C, Configuration Files contains descriptions of the configuration
	used by the QLogic InfiniPath and OpenFabrics software.
	Appendix D, Package Descriptions contains RPM Descriptions.
	Index lists major subjects and concepts with page numbers for easy

reference.

Overview

and the

rate SDR

The QLogic adapters utilize standard, off-the-shelf InfiniBand 4X switches and cabling. The QLogic interconnect is designed to work with all InfiniBand-compliant switches.

NOTE:

If you are using the QLE7240 or QLE7280, and want to use DDR mode, then DDR-capable switches must be used.

1-2

Preliminary

IB0056101-00 G.02

Introduction

Interoperability

QLogic OFED OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases. There are several options for subnet management in your cluster:

Use the embedded Subnet Manager (SM) in one or more managed switches supplied by your InfiniBand switch vendor.

Use a host-based Subnet Manager. QLogic provides one, HSM, as a part of the InfiniBand fabric Suite download.

Use the Open source Subnet Manager (OpenSM) component of OpenFabrics.

Interoperability

	Draft
	InfiniPath OpenFabrics (including Internet Protocol over InfiniBand (IPoIB))
	is interoperable with other vendors’ InfiniBand HCAs running compatible
	OpenFabrics releases.
	The QLogic MPI stack is not interoperable with other InfiniBand HCAs and
	Target Channel Adapters (TCAs). Instead, it uses an InfiniBand-compliant,
	vendor-specific protocol that is highly optimized for QLogic MPI and for MPI
	over verbs.
Conventions

	Convention	Meaning

command Fixed-space font is used for literal items such as commands, functions, programs, files and pathnames, and program output.

variable Italic fixed-space font is used for variable names in programs and command lines.

IB0056101-00 G.02

Preliminary

1-3

Introduction

Documentation

Table 1-1. Typographical Conventions (Continued)

Convention	Meaning

concept Italic font is used for emphasis and concepts, as well as for documentation names/titles.

user input Bold fixed-space font is used for literal items in commands or constructs that you type.

$	Indicates a command line prompt.

#Indicates a command line prompt as root.

[ ]	Brackets enclose optional elements of a command or program con-
	struct.

Draft
...	Ellipses indicate that a preceding element can be repeated.
>	A right caret identifies the cascading path of menu commands used
	in a procedure.
QLogic OFED	The current version number of the software is included within this
1.4	documentation.
NOTE:	Indicates important information.

Documentation

The product documentation includes:

The QLogic HCA and QLogic OFED Software Install GuideThe QLogic HCA and QLogic OFED Software Users GuideThe QLogic InfiniBand Software Install Guide

The QLogic ULP and Tools Reference Guide (OFED+ Users Guide)

Release NotesQuick Start GuideReadme file

For more information on system administration, using the QLogic Message-Passing Interface (MPI), and troubleshooting adapter hardware and software, see the QLogic HCA and QLogic OFED Software Users Guide.

1-4

Preliminary

IB0056101-00 G.02

Introduction

Contact Information

Support Headquarters	QLogic Corporation
	4601 Dean Lakes Blvd
	Shakopee, MN 55379
	USA

QLogic Web Site	www.qlogic.com
Technical Support Web Site	support.qlogic.com
Technical Support Email	support@qlogic.com
Technical Training Email	tech.training@qlogic.com


Draft

Additional contact information is available from the Contact Support area of the Technical Support Web Site.

IB0056101-00 G.02

Preliminary

1-5

Introduction

Contact Information

Notes

Draft

1-6

Preliminary

IB0056101-00 G.02

2 Feature Overview

This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components.

Features	Draft
	Draft
	Installation improvements. Provides a single software load for InfiniBand
	HCAs from QLogic and other vendors supported by OFED. the software is
	available packaged in the following ways:
	Text User Interface (TUI) installer available (with the QLogicIB-Basic*
	download). TUI is used for install on smaller clusters. Software can be
	installed either standalone or via FastFabric (if the QLogic InfiniBand
	Fabric Suite is purchased).
	Software packaged for use with rpm install method.
	A subset of the software (the accelerated MPI stack, precompiled
	versions of MVAPICH and Open MPI, and other user-level tools) can
	be installed on top of stock OFED or on an IB-enabled distribution.
	Software packaged for Rocks installation method.
	Software packaged for Platform OCS installation method
	Write-combining (WC) mappings for the PIO buffers is now configured by
	default using the x86 Page Attribute Table (PAT) mechanism.

IB0056101-00 G.02

Preliminary

2-1

Feature Overview

Features

MVAPICH and OpenMPI compiled with newer versions of each of four different compilers (GCC, PGI, Intel and PathScale) are available.

The QLogic InfiniBand Fabric Suite (IFS) is available separately for purchase. It includes FastFabric, the QLogic Host Subnet Manager (HSM), and the Fabric Viewer, and the InfiniServ Host Software.The QLogic OFED 1.4 software is supported by IFS.

Support for newer compiler versions (PathScale 3.x, PGI 7.x, PGI 8.x, Intel 10.x, Intel 11.x)

Support for newer Linux distributions, including RHEL 4 U7Performance enhancements and bug fixes

Other	Changes
		Draft
		ipath ether Ethernet emulation has been removed; IPoIB-CM may be

used instead.

The /etc/init.d/infinipath command to start the InfiniPath service has been replaced by the /etc/init.d/openibd command.

The infinipath-kernel RPM no longer exists: it has been integrated into the kernel-ib RPM.

Continued

Multiple high-performance native PSM Message Passing Interface (MPI) implementations. (PSM is QLogic’s accelerated library for high performance MPIs). In addition to QLogic MPI, the currently supported MPI implementations are HP-MPI, Open MPI, MVAPICH, and Scali (Platform). Open MPI provides MPI-2 functionality, including one-sided operations and dynamic processes. These all offer the same high performance as QLogic MPI.

Dual PCIe QLogic adapters per node.

QLogic MPI supports running exclusively on a single node without the installation of the HCA hardware.

4K Maximum Transfer Unit (MTU) is supported and is on by default. To take advantage of 4KB MTU, use a switch that supports 4KB MTU. QLogic also supports 2KB switches, and 4KB MTU switches configured for 2KB MTU.

QLogic switches with firmware version 4.2.x or later are recommended.

2-2

Preliminary

IB0056101-00 G.02

Feature Overview

Features

This version of the QLogic OFED software provides support for all of the QLogic HCAs in Table 2-1.

	Table 2-1. QLogic Adapter Model Numbers

QLogic Model		Description
Number		Description
Number

QHT7040		Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For
		systems with HTX expansion slots.

QHT7140 a		Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For
		systems with HTX expansion slots.

QLE7140		Single port 10Gbps SDR 4X InfiniBand to PCI Express x8
		adapter. Supported on systems with PCI Express (PCIe) x8 or
Draft
		x16 slots.
QLE7240		Single port 20Gbps DDR 4X InfiniBand to PCI Express x8
		adapter. Supported on systems with PCI Express x8 or
		x16 slots.
QLE7280		Single port 20Gbps DDR 4X InfiniBand to PCI Express x16
		adapter. Supported on systems with PCI Express x16 slots.
		The QLE7280 is backward compatible; it can also be used with
		PCIe adapters that connect to x8 slots.
Table Notes
PCIe is Gen 1

a The QHT7140 has a smaller form factor than the QHT7040, but is otherwise the same. Throughout this document, the QHT7040 and QHT7140 will be collectively referred to as the QHT7140 unless otherwise noted.

Additional up-to-date information can be found on the QLogic web site, specifically:

The high performance computing page at www.qlogic.com/Products/HPC products landingpage.aspx

The InfiniBand HCA page at www.qlogic.com/Products/HPC products infipathhcas.aspx

IB0056101-00 G.02

Preliminary

2-3

Feature Overview

Software Components

This release includes all of OFED 1.4 with enhancements (QLogic OFED 1.4), including a new version of the VNIC tools and driver, and support for the QHT7xxx and QLE7xxx adapters. The software includes the QLogic InfiniPath HCA driver, libraries, QLogic MPI, Subnet Management Agent, and associated utilities. Included components are:

InfiniPath driver

InfiniPath libraries, InfiniPath utilities, configuration, and support tools, including ipath_checkout, ipath_control, ipath_pkt_test, and ipathstats

QLogic MPI

PSMDraftsupport for accelerated MPI

OpenMPI and MVAPICH (with PSM support) built with the GNU, PGI, PathScale, and Intel compilers, with corresponding mpitests and mpi-selector

QLogic MPI benchmarks and utilities

OpenFabrics protocols

OpenFabrics libraries and utilities

QLogic VNIC module

FastFabric Enablement tools services:

OFED SRP

QLogic SRP

No support is provided for Reliable Datagram (RD).

More details about the hardware and software can be found in Section 4 and Section 5.

2-4

Preliminary

IB0056101-00 G.02

3 Step-by-Step

Installation Checklist

This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 “Hardware Installation” and Section 5 “Software Installation”.

Hardware	Installation
1.	Check that the adapter hardware is appropriate for your platform. See
	Table 4-1.
2.	Check to see that you have the appropriate cables and switches, as
	described in “Cabling and Switches” on page 4-3.
3.	Check to see that you are running a supported Linux distribution/kernel. See
	Table 5-1.
4.	Verify that the BIOS for your system is configured for use with the QLogic
	adapter. See “Configuring the BIOS” on page 4-4.
5.	Following the safety instructions in “Safety with Electricity” on page 4-5.
	Unpack the adapter (“Unpacking Information” on page 4-5) and verify the
	package contents.
6.	Install the adapter by following the instructions in “Hardware Installation” on
	page 4-9.
7.	Cable the adapter to the switch, as described in “Cabling the Adapter to the
	InfiniBand Switch” on page 4-17. Check that all InfiniBand switches are
	configured.
8.	FollowDraftthe steps in “Completing the Installation” on page 4-18 to finish the

installation.

IB0056101-00 G.02

Preliminary

3-1

Step-by-Step Installation Checklist

Software Installation

The following steps summarize the basic QLogic OFED 1.4 software installation and startup. These steps must be performed on each node in the cluster:

1.Make sure that the HCA hardware installation has been completed according to the instructions in “Hardware Installation” on page 4-1.

2.Verify that the Linux kernel software is installed on each node in the cluster. The required kernels and supported Linux distributions for both QLogic InfiniPath and OpenFabrics are defined in Table 5-1.

3.Make sure that your environment has been set up as described in “Setting Up Your Environment” on page 5-3.

4.Download your version of the QLogic InfiniPath/OpenFabrics software from theDraftQLogic web site to a local server directory. See “Choose the Appropriate Download Files” on page 5-4.

5.Install the selected packages on each cluster node using the corresponding method as described in one of the following: “Install QLogicIB-Basic with the Installer Tool” on page 5-7, “Using rpm to Install InfiniPath and OpenFabrics” on page 5-14, “Install QLogic OFED User-level Software with the rpm Command” on page 5-17, “Install QLogic OFED Using Rocks” on page 5-20, “Install QLogic OFED Using a Platform OCS Kit” on page 5-22, “Install FastFabric Software CD/ISO Image” on page 5-22.

6.The system can be rebooted after all the software has been installed.

7.The configuration file for the ib ipath driver is set up correctly at installation and is loaded automatically during system boot once the RPMs have been installed. However, if you wish to change the configuration file, see “Managing the InfiniPath Driver” on page 6-15.

8.If you want to configure the optional OpenFabrics driver ipoib, and you have not configured it yet with the Install tool, or if you have used the rpm install method, configure it as described in “Configuring the IPoIB Network Interface” on page 6-2.

9.If you want to use the optional OpenFabrics services (opensm, srp,or VNIC), configure them as described in “OpenSM” on page 6-3, “SRP” on page 6-4, or “SRP stands for SCSI RDMA Protocol. It was originally intended to allow the SCSI protocol to run over InfiniBand for Storage Area Network (SAN) usage. SRP interfaces directly to the Linux file system through the SRP Upper Layer Protocol. SRP storage can be treated as another device.” on page 6-4.

10.Check the system state by observing the LEDs. See “LED Link and Data Indicators” on page 7-1.

3-2

Preliminary

IB0056101-00 G.02

Step-by-Step Installation Checklist

Software Installation

11. You can optimize your system and adapter for the best performance. See “Adapter and Other Settings” on page 7-1. Also see the Performance Settings and Management Tips section in the QLogic HCA and QLogic OFED Software Users Guide.

12. Perform the recommended health checks. See “Customer Acceptance Utility” on page 7-2.

13. After installing the QLogic InfiniPath and OpenFabrics software, refer to the

QLogic HCA and QLogic OFED Software Users Guide for more information about using QLogic InfiniPath, QLogic MPI, and OpenFabrics products. Refer to the QLogic ULP and Tools Reference Guide for more information about configuring and using QLogic SRP, QLogic VNIC, and the Enablement Tools. The InfiniBand Software Installation Guide also has information on

installing the QLogic InfiniBand Fabric Suite. Draft

IB0056101-00 G.02

Preliminary

3-3

Step-by-Step Installation Checklist

Software Installation

Notes

Draft

3-4

Preliminary

IB0056101-00 G.02

4 Hardware Installation

This section lists the requirements and provides instructions for installing the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic PCIe adapter and PCIe riser card, QLE7140; and the QHT7040 or QHT7140 adapter hardware and HTX riser card. These components are collectively referred to as the adapter and the riser card in the remainder of this document.

data rate of 20Gbps (data rate of 16Gbps) for the QLE7240 and QLE7280; and 10Gbps (data rate of 8Gbps) for the QLE7140, QHT7040, and QHT7140.

The adapter is a low-latency, high-bandwidth, high message rate cluster interconnectDraftfor InfiniBand. The QLogic interconnect is InfiniBand 4X, with a raw

OpenFabrics is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases.

Hardware Installation Requirements

This section lists hardware and software environment requirements for installing the QLogic QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140.

Hardware

QLogic interconnect adapters are for use with UL listed computers. The following statement is true for all the adapters:

This device complies with part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) This device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operations.

Different adapter cards work on different platforms. Table 4-1 shows the relationship between the adapter model and different types of motherboards.

Table 4-1. Adapter Models and Related Platforms

QLogic	Platform	Plugs Into
Model	Platform	Plugs Into
Number

QLE7240	PCI Express systems	Standard PCI Express x8 or x16 slot

QLE7280	PCI Express systems	Standard PCI Express x16 slot

IB0056101-00 G.02

Preliminary

4-1

Hardware Installation

Hardware Installation Requirements

Table 4-1. Adapter Models and Related Platforms (Continued)

QLogic	Platform	Plugs Into
Model	Platform	Plugs Into
Number

QLE7140	PCI Express systems	Standard PCI Express x8 or x16 slot

QHT7040	Motherboards with HTX connectors	HyperTransport HTX slot

QHT7140	Motherboards with HTX connectors	HyperTransport HTX slot

Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of a PCI Express (PCIe) slot in a typical motherboard. See Figure 4-7 for an illustrationDraftof an HTX slot for a typical Opteron motherboard.

The motherboard vendor is the optimal source for information on the layout and use of HyperTransport and PCI Express-enabled expansion slots on supported motherboards.

Form Factors

The QLE7240, QLE7280, and QLE7140 are the model numbers for the adapters that ship in the standard PCI Express half-height, short-form factor. These adapters can be used with either full-height or low-profile face plates.

The QHT7040 is the model number for the adapter that shipped in the HTX full-height factor. The HTX low-profile form factor is referred to as the QHT7140. It is the same as the QHT7040, except for its more compact size. In either case, the adapter is backward and forward compatible for the motherboards in which it is supported. The QHT7040 and QHT7140 HTX adapters are collectively referred to as the QHT7140 unless otherwise stated.

When the QHT7040 or QHT7140 adapter is installed with the riser card, it may prevent some or all of the other PCI expansion slots from being used, depending on the form factor of the adapter and motherboard.

Run ipath control -i to see information on which form adapter is installed. The file /sys/class/infiniband/ipath0/device/boardversion contains the same information. For more information, see the Useful Programs and Files appendix in the QLogic HCA and QLogic OFED Software Users Guide.

4-2

Preliminary

IB0056101-00 G.02

Hardware Installation

Hardware Installation Requirements

Cabling and Switches

The cable installation uses a standard InfiniBand (IB) 4X cable. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cable that QLogic has currently qualified is 20 meters. For DDR-capable adapters and switches, the DDR-capable passive copper cables cannot be longer than 10 meters. Active cables can eliminate some of the cable length restrictions.

InfiniBand switches are available through QLogic.

NOTE:

If you are using the QLE7240 or QLE7280 and want to use DDR mode, then
DDR-capable switches must be used.
7104-12MDraft-Cable 4x-4x cable—12 meters (SDR only)
The copper cables listed in Table 4-2 are available from QLogic:
Table 4-2. QLogic InfiniBand Cables
Product Number	Description
7104-1M-Cable	4x-4x cable—1 meter
7104-2M-Cable	4x-4x cable—2 meters
7104-3M-Cable	4x-4x cable—3 meters
7104-4M-Cable	4x-4x cable—4 meters
7104-5M-Cable	4x-4x cable—5 meters

7104-6M-Cable	4x-4x cable—6 meters

7104-7M-Cable	4x-4x cable—7 meters

7104-8M-Cable	4x-4x cable—8 meters

7104-9M-Cable	4x-4x Cable—9 meters

7104-10M-Cable	4x-4x cable—10 meters


7104-14M-Cable	4x-4x cable—14 meters (SDR only)

7104-16M-Cable	4x-4x cable—16 meters (SDR only)

7104-18M-Cable	4x-4x cable—18 meters (SDR only)

For cabling instructions, see “Cabling the Adapter to the InfiniBand Switch” on page 4-17.

IB0056101-00 G.02

Preliminary

4-3

Hardware Installation

Hardware Installation Requirements

Optical Fibre Option

The QLogic adapter also supports connection to the switch by means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these types of convertors. For more information on the EMCORE convertor, see www.emcore.com.

Intel® and Zarlink™ also offer optical cable solutions. See www.intel.com and www.zarlink.com for more information.

Configuring the BIOS

To achieve the best performance with QLogic adapters, you need to configure your BIOS with specific settings. The BIOS settings, which are stored in non-volatile memory, contain certain parameters characterizing the system. These

parameters may include date and time, configuration settings, and information
	Draft
	Advanced Configuration and Power Interface (ACPI) BIOS option must be

enabled.

Interface

documentation

information,

4-4

Preliminary

IB0056101-00 G.02

Hardware Installation Safety with Electricity

Safety with Electricity

Observe these guidelines and safety precautions when working around computer hardware and electrical equipment:

Locate the power source shutoff for the computer room or lab where you are working. This is where you will turn OFF the power in the event of an emergency or accident. Never assume that power has been disconnected for a circuit; always check first.

Do not wear loose clothing. Fasten your tie or scarf, remove jewelry, and roll up your sleeves. Wear safety glasses when working under any conditions that might be hazardous to your eyes.

Shut down and disconnect the system’s power supply from AC service beforeDraftyou begin work, to insure that standby power is not active. Power off all attached devices such as monitors, printers, and external components. Note that many motherboards and power supplies maintain standby power at all times. Inserting or removing components while standby is active can damage them.

Use normal precautions to prevent electrostatic discharge, which can damage integrated circuits.

Unpacking

Verify the

List of the

QLogic QLE7240

Additional short bracket

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or motherboard vendor.

IB0056101-00 G.02

Preliminary

4-5

Hardware Installation

Unpacking Information

The package contents for the QLE7280 adapter are:

QLogic QLE7280

Additional short bracket

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or motherboard vendor.

The package contents for the QLE7140 adapter are:

QLogic QLE7140

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or
motherboard vendor. The contents are illustrated in Figure 4-2.
	Draft
The package contents for the QHT7140 adapter are:
	QLogic QHT7140
HTX riser card for use in 1U or 2U chassis
	Quick Start Guide

The contents are illustrated in Figure 4-3.

The IBA6120, IBA6110, and IBA7220 are the QLogic ASICs, which are the central components of the interconnect. The location of the IBA7220 ASIC on the adapter is shown in Figure 4-1. The location of the IBA6120 ASIC on the adapter is shown in Figure 4-2. The location of the IBA6110 ASIC on the adapter is shown in Figure 4-3.

4-6

Preliminary

IB0056101-00 G.02

Hardware Installation

Unpacking Information

PCI Express

Edge Connectors

InfiniBand

Connector

Draft			Face Plate
			Face Plate
	IBA7220 ASIC
Figure 4-1. QLogic QLE7280 with IBA7220 ASIC
		PCI Express Riser Card. Not
		supplied; shown for reference.
PCI Express			InfiniBand
Edge Connectors			InfiniBand
			Connector
	IBA6120 ASIC		Face Plate
	IBA6120 ASIC

Figure 4-2. QLogic QLE7140 Card with Riser, Top View

IB0056101-00 G.02

Preliminary

4-7

Hardware Installation

Unpacking Information

HTX Riser Card

HTX Edge			InfiniBand
Connectors
			Connector

		QHT7140 Low
Figure 4-3. QLogic		Draft

Unpacking
1.	When unpacking, ground yourself before removing the QLogic adapter from
	the anti-static bag.
2.	Grasping the QLogic adapter by its face plate, pull the adapter out of the
	anti-static bag. Handle the adapter only by its edges or the face plate. Do not
	allow the adapter or any of its components to touch any metal parts.
3.	After checking for visual damage, store the adapter and the riser card in their
	anti-static bags until you are ready to install them.

4-8

Preliminary

IB0056101-00 G.02

Hardware Installation

This section contains hardware installation instructions for the QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140.

Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser

Installation for the QLE7240, QLE7280, and QLE7140 is similar. The following instructions are for the QLE7140, but can be used for any of these three adapters.

Most installations will be in 1U and 2U chassis, using a PCIe right angle riser card.

This results in an installation of the adapter that is parallel to the motherboard.
This type of installation is described first. Installation in a 3U chassis is described
in “Hardware Installation for the QHT7140 Without an HTX Riser” on page 4-16.
Dual Adapter	Draft
Dual Adapter
Installation Steps
1.	The BIOS should already be configured properly by the motherboard
	manufacturer. However, if any additional BIOS configuration is required, it
	will usually need to be done before installing the QLogic adapter. See
	“Configuring the BIOS” on page 4-4.
2.	Shut down the power supply to the system into which you will install the
	QLogic adapter.
3.	Take precautions to avoid electrostatic damage (ESD) to the cards by
	properly grounding yourself or touching the metal chassis to discharge static
	electricity before handling the cards.

IB0056101-00 G.02

Preliminary

4-9

Hardware Installation

4.Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system.

5.Locate the PCIe slot on your motherboard. Note that the PCIe slot has two separate sections, with the smaller slot opening located towards the front (see Figure 4-4). These two sections correspond to the shorter and longer connector edges of the adapter and riser.

6.DetermineDraftif a blanking panel is installed in your chassis. If it is, remove it so that the InfiniBand connector will be accessible. Refer to your system vendor instructions for how to remove the blanking panel.

7.Remove the QLogic adapter from the anti-static bag.

8.Locate the face plate on the connector edge of the card.

4-10

Preliminary

IB0056101-00 G.02

+ 86 hidden pages

Q-Logic IB0056101-00 G.02, OFED User Manual

Table of Contents

1 Introduction

How this Guide is Organized

Overview

Interoperability

Documentation

Contact Information

2 Feature Overview

Features

Table 2-1. QLogic Adapter Model Numbers

Software Components

This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 “Hardware Installation” and Section 5 “Software Installation”.

Software Installation

4 Hardware Installation

Hardware Installation Requirements

Hardware

Table 4-1. Adapter Models and Related Platforms

Form Factors

Cabling and Switches

Table 4-2. QLogic InfiniBand Cables

Optical Fibre Option

Configuring the BIOS

Safety with Electricity

Figure 4-1. QLogic QLE7280 with IBA7220 ASIC

Figure 4-2. QLogic QLE7140 Card with Riser, Top View

Hardware Installation

Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser

Installation Steps