Draft
QLogic HCA and QLogic OFED Software
Install Guide
QLogic OFED Version 1.4
IB0056101-00 G.02
Preliminary
Draft
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no
responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its
use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications
described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no
representation nor warranty that such applications are suitable for the specified use without further testing or
modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document.
No part of this document may be copied nor reproduced by any means, nor translated nor transmitted to any magnetic
medium without the express written consent of QLogic Corporation. In accordance with the terms of their valid QLogic
agreements, customers are permitted to make electronic and paper copies of this document for their own exclusive
use.
The QHT7040, QHT7140, QLE7140, QLE7240, and QLE7280 QLogic Host Channel Adapters are covered by the
following patent: 7308535.
Document Revision History
Rev. 1.0, 8/20/2005
Rev. 1.1, 11/15/05
Rev. 1.2, 02/15/06
S
Rev. 1.3, Beta 1, 4/15/06
Rev 1.3, 6/15/06
Rev. 2.0 Beta, 8/15/06, QLogic Version Number IB0056101-00 A
Rev. 2.0 Beta 2 10/15/06, QLogic Version Number IB0056101-00 B
Rev. 2.0 11/30/06, QLogic Version Number IB0056101-00 C
Rev. 2.0 3/23/07, QLogic Version Number IB0056101-00 D
Rev. 2.1 8/24/07, QLogic Version Number IB0056101-00 E
Rev. 2.2 5/27/08, QLogic Version Number IB0056101-00 F
Rev. 2.2 9/5/08, QLogic Version Number IB0056101-00 G
Rev. QLogic OFED 1.4 4/7/09, QLogic Version Number IB0056101-00 G.02
Changes Sections Affected
Product name changed from InfiniPath to QLogic
OFED. Version number is set to 1.4. Instances of
InfiniPath changed where appropriate; some filenames and output messages keep old name.
Updated Contact Information. “Contact Information” on page 1-5
All
Removed InfiniPath from 1st paragraph; replaced
with QLogic adapters, etc.
Changed section title from What’s New in this
Release to Features.
Page ii Preliminary IB0056101-00 G.02
“Hardware Installation” on page 4-1
“Features” on page 2-1
Draft
QLogic HCA and QLogic OFED Software Install Guide
S
Replace EM64T with 64-bit Intel Xeon. “Supported Linux Distributions” on page 5-1
Added more information on VNIC interface. “OpenSM” on page 6-3
QLogic OFED Version 1.4
Split the Install section into three separate sections.
Modified installation checklist. Clarified which drivers are configured, which are optional. Added two
methods of installation to the list.
Remove sections on configuring ipath_ether.
ipath_ether now deprecated.
Updated the supported distributions information. Table 5-1 on page 5-2
Distribution identifiers are now RHEL4, RHEL5,
and SLES10.
Delete reference to ipath_ether. “Removing Software Packages” on page 5-24
Updated Lustre information; patches are no longer
needed.
Updated compiler support information. “Compiler Support” on page 5-3
Updated information on software components. “Software Components” on page 2-4
Removed paragraph about this release adding
support for QLE7240/7280, since this is outdated.
“Software Installation” on page 5-1
“Configuring Drivers and Services” on page 6-1
“Installation Verification and Additional Settings” on
page 7-1
“Software Installation” on page 3-2
Was “Configuring the ipath_ether Network Interface” on page 5-12
Table 5-2 on page 5-2
“Installing Lustre” on page 5-23
“Features” on page 2-1
Added new “Configuration Issues” to Troubleshooting.
Deleted Troubleshooting item:
ifup on ipath_ether on SLES 10 Reports
"unknown device"
Deleted entries pertaining to ipath_ether in this
table.
IB0056101-00 G.02 Preliminary Page iii
“ibsrpdm Command Hangs When Two HCAs are
Installed but Only Unit 1 is Connected to the
Switch” on page A-5
Appendix A
Table C-1 on page C-1
Draft
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
S
Sections rearranged, renamed, and expanded due
to multiple install methods. Previous sections:
Installing the InfiniPath and Open fabrics RPMs
Downloading and Unpacking the InfiniPath and
OpenFabrics Software
Added new section about installing using the
Installer Tool.
Moved introductory information about RPM installation before sections on installing using the two
different methods.
Removed paragraph about kernel module support
being part of InfiniPath RPMs; was relevant to previous release.
Added note about need to use prefixed install if
using QLogic MPI with mpi-selector.
“Choose the Appropriate Download Files” on
page 5-4
“Install QLogicIB-Basic with the Installer Tool” on
page 5-7
“About rpm Installation” on page 5-13
“Install QLogic OFED Using Rocks” on page 5-20
“Install QLogic OFED Using a Platform OCS Kit”
on page 5-22
“Using rpm to Install InfiniPath and OpenFabrics”
on page 5-14
“Install QLogicIB-Basic with the Installer Tool” on
page 5-7
“About rpm Installation” on page 5-13
Was in old section “Installing the InfiniPath and
Open fabrics RPMs”
“Using rpm to Install InfiniPath and OpenFabrics”
on page 5-14
Modified module for installed layout. New location
for ipath modules. Location of other OFED modules listed here.
Modified module for driver overview. “InfiniPath and OpenFabrics Driver Overview” on
Modified module for configuring IPoIB. Only restart
information has been changed.
Modified module for configuring VNIC interface.
Changed infinipath start/stop commands, and
added some introductory material.
Modified module for configuring SRP. “SRP” on page 6-4
Modified module for uDAPL configuration. “MPI over uDAPL” on page 6-13
Deleted reference to MTRR BIOS setting. “Configuring the BIOS” on page 4-4
“Installed Layout” on page 5-23
page 6-1
“Configuring the IPoIB Network Interface” on
page 6-2
“SRP stands for SCSI RDMA Protocol. It was originally intended to allow the SCSI protocol to run
over InfiniBand for Storage Area Network (SAN)
usage. SRP interfaces directly to the Linux file system through the SRP Upper Layer Protocol. SRP
storage can be treated as another device.” on
page 6-4
“BIOS Settings” on page A-2
Page iv Preliminary IB0056101-00 G.02
Draft
S
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
New Appendix for Write Combining. MTRR settings information moved here.
Minor wording change to introduction section. “Software Installation” on page 5-1
Added new issue in Troubleshooting. “openmpi_gcc Fails to Install Because of Depen-
Added new issue in Troubleshooting. “Outdated ipath_ether Configuration Setup
Configuration file qlogic_vnic.cfg changed to
qlgc_vnic.cfg.
Added new section. “Uninstalling Software with Rocks or Platform
Added new section. “Install QLogic OFED Using Rocks” on page 5-20
Change name of Appendix RPM Descriptions to
Package Descriptions. Updated information.
Removed phrase “and Transmission Control
Protocol (TCP)” from introduction; related to
now-obsolete ipath_ether.
Removed this issue from the Troubleshooting section.
“Write Combining” on page B-1
dency on gfortran (RHEL 4)” on page A-4
Generates Error” on page A-5
Table C-1 on page C-1
OCS” on page 5-25
“Package Descriptions” on page D-1
“Interoperability” on page 1-3
Was “OpenFabrics Library Dependencies”
Combined What’s New in This Release and Features section.
Deleted Note about Fedora Core 6 not supported
in this release.
Removed this line “
web site for updated information on supported
compilers.” This information is not on the web
site.
Added more details about setting the switch MTU
default to 4K.
Deleted the section “Installing QLogic MPI in an
Alternate Location with rpm”; merged instructions
into rpm install section.
Minor text changes to this section. Modified title
from Additional Installation Instructions to Install
Additional Software. Moved this section to be with
the rest of the installation instructions.
Please check the QLogic
“Feature Overview” on page 2-1
This was noted in InfiniPath 2.2.1 release notes.
“Software Components” on page 2-4
“Other Configuration: Changing the MTU Size” on
page 6-14
“Using rpm to Install InfiniPath and OpenFabrics”
on page 5-14
“Install Additional Software” on page 5-23
IB0056101-00 G.02 Preliminary Page v
Draft
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
S
Changed title of Troubleshooting issue. Was
ipath_ether Configuration Setup Gener-
“
ates Error”
Change main configuration file from /etc/sys-
config/infinipath to /etc/infiniband/openib.conf. The infinipath file is
still to enable or disable the ipath_mtrr script.
Updated note about setting datagram mode. “Configuring the IPoIB Network Interface” on
Deleted Troubleshooting Issue OpenFabrics
Dependencies.
Deleted Troubleshooting Issue Version Number
Conflict with opensm-* on RHEL5 Systems
Two new tables showing OS package requirements. Replaces some bullet points in “Distribution
Identifiers” on page 5-2
Removed section “Check for Missing Files or
RPMs”
Merged the two sections Downloading the QLogic
OFED Software and Choose the Appropriate
Download Files
“Outdated ipath_ether Configuration Setup
Generates Error” on page A-5
Table C-1 on page C-1 and “SRP” on page 6-4.
page 6-2
“Software Installation Issues” on page A-2
“Software Installation Issues” on page A-2
Table on page 5-4 and Table on page 5-4
Was “Check for Missing files or RPMs” on
page 5-5
“Choose the Appropriate Download Files” on
page 5-4
Removed Supported Compilers and Supported
Linux Distributions from the 2 Feature Overview
chapter. Information is still in Software Installation
chapter.
Moved table “
RPMs to Use for Each Node in a Cluster” from
Software Installation section to Appendix
Consolidated Configuring the InfiniPath Drivers
and InfiniPath and OpenFabrics Driver Overview
opensm is now off by default Various
Merged “Install on an Unsupported Distribution
with the rpm Command” with the Rebuilding...section.
Removed $ or # at beginning of lines indicating
commands. The #, if cut and pasted from the document, may be interpreted as a shell command.
Updated footnote b. Tab le D -11
InfiniPath and OpenFabrics
2 Feature Overview
“Package Descriptions” on page D-1
6 Configuring Drivers and Services
Now “Rebuilding the kernel-ib Driver on an
Unsupported Distribution or an Unsupported Distribution/Kernel Pair” on page 5-19
All. Also Ta bl e 1 -1
Page vi Preliminary IB0056101-00 G.02
Draft
S
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
Adapter Settings section name changed. Also
added new bullet point “Check the PCIe bus width”
Updated tables of RPMs. Moved all documentation
RPMs together. New table for OpenSM-Devel
RPMs. Updated OtherMPIs.
Deleted table 5-7 RPMs to Install, as information is
repeated in “Using rpm to Install InfiniPath and
OpenFabrics” on page 5-14
Moved RPM Organization section under RPM
Installation.
Merge two sections Uninstalling InfiniPath and
OpenFabrics RPMs and Uninstalling OFED 1.3
Software.
Reference to this location
/sys/bus/pci/drivers/ib_ipath/00/
changed to
/sys/class/infiniband/ipath*/device/
Removed note: “
using the Verbs interfaces are not supported
in this release, but may be supported in a
future release.”
OpenFabrics programs (32-bit)
“Adapter and Other Settings” on page 7-1
All tables in D Package Descriptions
“Using rpm to Install InfiniPath and OpenFabrics”
on page 5-14
“RPM Organization” on page 5-16
“Uninstalling InfiniPath and OpenFabrics RPMs”
on page 5-24
“Form Factors” on page 4-2
“Software Components” on page 2-4
IB0056101-00 G.02 Preliminary Page vii
Draft
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
Notes
S
Page viii Preliminary IB0056101-00 G.02
Draft
Table of Contents
1 Introduction
Who Should Read this Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
Conventions Used in this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
Contact Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
2 Feature Overview
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Other Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Continued Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Software Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4
3 Step-by-Step Installation Checklist
Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Software Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
4 Hardware Installation
Hardware Installation Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
Form Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Cabling and Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Optical Fibre Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4
Configuring the BIOS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4
Safety with Electricity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
Unpacking Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
Verify the Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
List of the Package Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
Unpacking the QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9
IB0056101-00 G.02 Preliminary ix
Draft
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI
Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9
Dual Adapter Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9
Installation Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9
Hardware Installation for QHT7140 with HTX Riser . . . . . . . . . . . . . . 4-12
Hardware Installation for QLE7240, QLE7280, and QLE7140
Without a PCI Express Riser. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-15
Hardware Installation for the QHT7140 Without an HTX Riser . . . . . . 4-16
Switch Configuration and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-17
Cabling the Adapter to the InfiniBand Switch. . . . . . . . . . . . . . . . . . . . . . . . 4-17
Completing the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-18
5 Software Installation
Cluster Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1
Types of Nodes in a Cluster Environment . . . . . . . . . . . . . . . . . . . . . . 5-1
Supported Linux Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1
Distribution Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Compiler Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3
Setting Up Your Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3
Choose the Appropriate Download Files . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-4
Install QLogicIB-Basic with the Installer Tool . . . . . . . . . . . . . . . . . . . . . . . . 5-7
About rpm Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-13
Using rpm to Install InfiniPath and OpenFabrics . . . . . . . . . . . . . . . . . . . . . 5-14
RPM Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-16
Install QLogic OFED User-level Software with the rpm Command . . . . . . . 5-17
Rebuilding or Reinstalling the kernel-ib Driver with rpm After a
Kernel Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-19
Rebuilding the kernel-ib Driver on an Unsupported Distribution or an
Unsupported Distribution/Kernel Pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-19
Install QLogic OFED Using Rocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-20
Install Frontend and Compute Nodes . . . . . . . . . . . . . . . . . . . . . . . . . 5-20
Rocks Installation on an Existing Frontend Node . . . . . . . . . . . . . . . . 5-21
Install QLogic OFED Using a Platform OCS Kit. . . . . . . . . . . . . . . . . . . . . . 5-22
Install FastFabric Software CD/ISO Image . . . . . . . . . . . . . . . . . . . . . . . . . 5-22
Install Additional Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-23
Installing Lustre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-23
Installed Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-23
Removing Software Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-24
Uninstalling Using the Installer Tool. . . . . . . . . . . . . . . . . . . . . . . . . . . 5-24
Uninstalling InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . 5-24
S
x Preliminary IB0056101-00 G.02
Draft
QLogic HCA and QLogic OFED Software Install Guide
A
Uninstalling Software with Rocks or Platform OCS . . . . . . . . . . . . . . . 5-25
Downgrading RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-25
6 Configuring Drivers and Services
InfiniPath and OpenFabrics Driver Overview . . . . . . . . . . . . . . . . . . . . . . . . 6-1
OpenFabrics Drivers and Services Configuration and Startup . . . . . . . . . . . 6-1
Configuring the IPoIB Network Interface . . . . . . . . . . . . . . . . . . . . . . . 6-2
OpenSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-3
SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-4
Using QLogic SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-4
Using OFED SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-4
Configuring and Administering the VNIC Interface . . . . . . . . . . . . . . . 6-6
MPI over uDAPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13
Other Configuration: Changing the MTU Size . . . . . . . . . . . . . . . . . . . . . . . 6-14
Managing the InfiniPath Driver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-15
Configure InfiniPath Driver State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-16
Start, Stop or Restart InfiniPath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-16
Unloading the Driver/Modules Manually . . . . . . . . . . . . . . . . . . . . . . . 6-17
Further Information on Configuring and Loading Drivers . . . . . . . . . . . . . . . 6-17
QLogic OFED Version 1.4
7 Installation Verification and Additional Settings
LED Link and Data Indicators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1
Adapter and Other Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1
Customer Acceptance Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-2
A Installation Troubleshooting
Hardware Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1
Node Spontaneously Reboots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1
Some HTX Motherboards May Need Two or More CPUs in Use . . . . A-1
BIOS Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2
Enable Advanced Configuration and Power Interface (ACPI). . . . . . . A-2
Issue with Supermicro® H8DCE-HTe and QHT7040 . . . . . . . . . . . . . A-2
Software Installation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2
Missing Kernel RPM Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2
Resolving Conflicts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-4
openmpi_gcc Fails to Install Because of Dependency on gfortran
(RHEL 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-4
mpirun Installation Requires 32-bit Support . . . . . . . . . . . . . . . . . . . A-4
Lockable Memory Error on Initial Installation of InfiniPath. . . . . . . . . . A-5
Configuration Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5
IB0056101-00 G.02 Preliminary xi
Draft
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
ibsrpdm Command Hangs When Two HCAs are Installed but Only Unit 1 is
Connected to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5
Outdated ipath_ether Configuration Setup Generates Error . . . . . A-5
B Write Combining
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1
Verify Write Combining is Working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1
PAT and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-2
MTRR Mapping and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-2
Edit BIOS Settings to Fix MTRR Issues . . . . . . . . . . . . . . . . . . . . . . . B-2
Use the ipath_mtrr Script to Fix MTRR Issues. . . . . . . . . . . . . . . . B-3
C Configuration Files
D Package Descriptions
Package Names with the QLogicIB-Basic Download. . . . . . . . . . . . . . . . . . D-1
Different Nodes May Use Different RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . D-1
InfiniPath RPM Version Numbers and Identifiers . . . . . . . . . . . . . . . . . . . . . D-2
OpenFabrics RPM Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2
InfiniPath and OpenFabrics RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2
Documentation RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-3
InfiniPath RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-3
OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-4
Other HCAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-9
Other MPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-10
S
xii Preliminary IB0056101-00 G.02
Draft
QLogic HCA and QLogic OFED Software Install Guide
A
List of Figures
Figure Page
4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7
4-2 QLogic QLE7140 Card with Riser, Top View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7
4-3 QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View . . . . 4-8
4-4 PCIe Slot in a Typical Motherboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10
4-5 QLogic PCIe HCA Assembly with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-11
4-6 Assembled PCIe HCA with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-12
4-7 HTX Slot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13
4-8 QLogic QHT7140 Adapter with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-14
4-9 Assembled QHT7140 with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-15
4-10 QHT7140 Without Riser Installed in a 3U Chassis . . . . . . . . . . . . . . . . . . . . . . . . . 4-17
List of Tables
Table Page
1-1 Typographical Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
2-1 QLogic Adapter Model Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3
4-1 Adapter Models and Related Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
4-2 QLogic InfiniBand Cables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
5-1 InfiniPath/OpenFabrics Supported Distributions and Kernels . . . . . . . . . . . . . . . . . 5-2
5-2 Distribution Identifiers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
5-3 Required OS Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-4
5-4 Specific Component Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-4
5-5 Available Packages for QLogic OFED 1.4 Release . . . . . . . . . . . . . . . . . . . . . . . . . 5-5
5-6 INSTALL Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-12
7-1 ipath_checkout Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-3
C-1 Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1
D-1 Documentation/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-3
D-2 InfiniPath/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-3
D-3 InfiniPath-Devel/RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-4
D-4 InfiniPath-MPI/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-4
D-5 OpenFabrics/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-4
D-6 OpenFabrics-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-7
D-7 OpenSM/RPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-8
D-8 OpenSM-Devel/RPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-9
D-9 Other HCAs/RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-9
D-10 Other HCAs-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-9
D-11 OtherMPIs/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-10
QLogic OFED Version 1.4
IB0056101-00 G.02 Preliminary xiii
Draft
QLogic HCA and QLogic OFED Software Install Guide
QLogic OFED Version 1.4
Notes
S
xiv Preliminary IB0056101-00 G.02
Draft
1 Introduction
This chapter describes the contents, intended audience, and organization of the
QLogic HCA and QLogic OFED Software Install Guide.
The QLogic HCA and QLogic OFED Software Install Guide contains instructions
for installing the QLogic Host Channel Adapters (HCAs) and the QLogic InfiniPath
and OpenFabrics software. The following adapters are covered in this guide:
®
QLE7140 PCI Express
QLE7240 PCI Express
QLE7280 PCI Express
QHT7040/QHT7140 HyperTransport Expansion (HTX™)
Who Should Read this Guide
This installation guide is intended for cluster administrators responsible for
installing the QLogic QLE7140, QLE7240, QLE7280 or QHT7040/QHT7140
adapter and QLogic InfiniPath software on their Linux
installation information and instructions for administering the QLogic cluster can
be found in the QLogic HCA and QLogic OFED Software Users Guide.
(PCIe)
®
cluster. Additional detailed
The QLogic HCA and QLogic OFED Software Install Guide assumes that you are
familiar with both cluster networking and the specific hardware that you plan to
use. Before installing the HCA, you should have basic knowledge of your host and
target operating systems, and working knowledge of message passing concepts.
This document does not contain all the information you need to use basic Linux
commands or to perform all system administration tasks. For this information, see
the software documentation you received with your system.
How this Guide is Organized
The QLogic HCA and QLogic OFED Software Install Guide is organized into these
sections:
Section 1, Introduction, contains an overview of the HCAs and software,
describes interoperability with other products, lists all related documentation,
and provides QLogic contact information.
Section 2, Feature Overview, contains features for this release, the
supported QLogic adapter models, supported distributions and kernels, and
a list of the software components.
IB0056101-00 G.02 Preliminary 1-1
Draft
Introduction
Overview
S
Section 3, Step-by-Step Installation Checklist, provides a high-level
overview of the hardware and software installation procedures.
Section 4, Hardware Installation, includes instructions for installing the
QLogic QLE7140, QLE7240, QLE7280, QHT7040, and QHT7140 HCAs.
Section 5, Software Installation, includes instructions for installing the
QLogic InfiniPath and OpenFabrics software.
Section 6, Configuring Drivers and Services, includes instructions for
configuring the QLogic InfiniPath and OpenFabrics drivers and services.
Section 7, Installation Verification and Additional Settings, describes tools
for verifying the installation, and adapter settings for best performance.
Appendix A, Installation Troubleshooting contains troubleshooting
information about issues that may occur during installation.
Appendix B, Write Combining contains information about settings that will
ensure better performance.
Appendix C, Configuration Files contains descriptions of the configuration
used by the QLogic InfiniPath and OpenFabrics software.
Appendix D, Package Descriptions contains RPM Descriptions.
Index lists major subjects and concepts with page numbers for easy
Overview
The material in this documentation pertains to a QLogic OFED cluster. A cluster is
defined as a collection of nodes, each attached to an InfiniBand™-based fabric
through the QLogic interconnect. The nodes are Linux-based computers, each
having up to 16 processors.
The QLogic HCAs are InfiniBand 4X. The Double Data Rate (DDR) QLE7240 and
QLE7280 adapters have a raw data rate of 20Gbps (data rate of 16Gbps). For the
Single Data Rate (SDR) adapters, the QLE7140 and QHT7140, the raw data rate
is 10Gbps (data rate of 8Gbps). The QLE7240 and QLE7280 can also run in SDR
mode.
The QLogic adapters utilize standard, off-the-shelf InfiniBand 4X switches and
cabling. The QLogic interconnect is designed to work with all InfiniBand-compliant
switches.
NOTE:
reference.
If you are using the QLE7240 or QLE7280, and want to use DDR mode,
then DDR-capable switches must be used.
1-2 Preliminary IB0056101-00 G.02
Draft
A
QLogic OFED OpenFabrics software is interoperable with other vendors’
InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics
releases. There are several options for subnet management in your cluster:
Use the embedded Subnet Manager (SM) in one or more managed switches
supplied by your InfiniBand switch vendor.
Use a host-based Subnet Manager. QLogic provides one, HSM, as a part of
the InfiniBand fabric Suite download.
Use the Open source Subnet Manager (OpenSM) component of
OpenFabrics.
Interoperability
QLogic InfiniPath participates in the standard InfiniBand subnet management
protocols for configuration and monitoring. Note that:
InfiniPath OpenFabrics (including Internet Protocol over InfiniBand (IPoIB))
is interoperable with other vendors’ InfiniBand HCAs running compatible
OpenFabrics releases.
Introduction
Interoperability
The QLogic MPI stack is not interoperable with other InfiniBand HCAs and
Target Channel Adapters (TCAs). Instead, it uses an InfiniBand-compliant,
vendor-specific protocol that is highly optimized for QLogic MPI and for MPI
over verbs.
NOTE:
See the OpenFabrics web site at www.openfabrics.org
on the OpenFabrics Alliance.
Conventions Used in this Guide
This guide uses the typographical conventions listed in Tab le 1 -1 .
Table 1-1. Typographical Conventions
Convention Meaning
command Fixed-space font is used for literal items such as commands, func-
tions, programs, files and pathnames, and program output.
variable Italic fixed-space font is used for variable names in programs and
command lines.
for more information
IB0056101-00 G.02 Preliminary 1-3
Draft
Introduction
Documentation
S
Table 1-1. Typographical Conventions (Continued)
Convention Meaning
concept Italic font is used for emphasis and concepts, as well as for docu-
mentation names/titles.
user input Bold fixed-space font is used for literal items in commands or con-
structs that you type.
$ Indicates a command line prompt.
# Indicates a command line prompt as root.
[] Brackets enclose optional elements of a command or program con-
struct.
... Ellipses indicate that a preceding element can be repeated.
> A right caret identifies the cascading path of menu commands used
in a procedure.
QLogic OFED
1.4
NOTE: Indicates important information.
Documentation
The product documentation includes:
The QLogic HCA and QLogic OFED Software Install Guide
The QLogic HCA and QLogic OFED Software Users Guide
The QLogic InfiniBand Software Install Guide
The QLogic ULP and Tools Reference Guide (OFED+ Users Guide)
Release Notes
Quick Start Guide
Readme file
For more information on system administration, using the QLogic
Message-Passing Interface (MPI), and troubleshooting adapter hardware and
software, see the QLogic HCA and QLogic OFED Software Users Guide.
The current version number of the software is included within this
documentation.
1-4 Preliminary IB0056101-00 G.02
Draft
A
Contact Information
Support Headquarters QLogic Corporation
QLogic Web Site www.qlogic.com
Technical Support Web Site support.qlogic.com
Technical Support Email support@qlogic.com
Technical Training Email tech.training@qlogic.com
Additional contact information is available from the Contact Support area of the Technical Support Web Site.
Introduction
Contact Information
4601 Dean Lakes Blvd
Shakopee, MN 55379
USA
IB0056101-00 G.02 Preliminary 1-5
Draft
Introduction
Contact Information
Notes
S
1-6 Preliminary IB0056101-00 G.02
Draft
2 Feature Overview
This section contains the features for this release, the supported QLogic adapter
models, supported distributions and kernels, and a list of the software
components.
Features
The QLogic OFED 1.4 software release contains the complete OFED 1.4, plus
additional QLogic improvements, including an enhanced QLogic HCA driver. The
InfiniPath 2.3 components (libraries, QLogic MPI/PSM, and utilities) are also
included. QLogic also supplies MVAPICH and OpenMPI compiled with newer
versions of each of four different compilers (GCC, PGI, Intel and PathScale).
The following features and enhancements are included in the QLogic OFED 1.4
release:
Installation improvements. Provides a single software load for InfiniBand
HCAs from QLogic and other vendors supported by OFED. the software is
available packaged in the following ways:
Text User Interface (TUI) installer available (with the QLogicIB-Basic*
download). TUI is used for install on smaller clusters. Software can be
installed either standalone or via FastFabric (if the QLogic InfiniBand
Fabric Suite is purchased).
Software packaged for use with rpm install method.
A subset of the software (the accelerated MPI stack, precompiled
versions of MVAPICH and Open MPI, and other user-level tools) can
be installed on top of stock OFED or on an IB-enabled distribution.
Software packaged for Rocks installation method.
Software packaged for Platform OCS installation method
Write-combining (WC) mappings for the PIO buffers is now configured by
default using the x86 Page Attribute Table (PAT) mechanism.
IB0056101-00 G.02 Preliminary 2-1
Draft
Feature Overview
Features
MVAPICH and OpenMPI compiled with newer versions of each of four
different compilers (GCC, PGI, Intel and PathScale) are available.
The QLogic InfiniBand Fabric Suite (IFS) is available separately for
purchase. It includes FastFabric, the QLogic Host Subnet Manager (HSM),
and the Fabric Viewer, and the InfiniServ Host Software.The QLogic OFED
1.4 software is supported by IFS.
Support for newer compiler versions (PathScale 3.x, PGI 7.x, PGI 8.x, Intel
10.x, Intel 11.x)
Support for newer Linux distributions, including RHEL 4 U7
Performance enhancements and bug fixes
Other Changes
ipath_ether Ethernet emulation has been removed; IPoIB-CM may be
used instead.
The /etc/init.d/infinipath command to start the InfiniPath service
has been replaced by the /etc/init.d/openibd command.
S
The infinipath-kernel RPM no longer exists: it has been integrated
into the kernel-ib RPM.
Continued Support
Multiple high-performance native PSM Message Passing Interface (MPI)
implementations. (PSM is QLogic’s accelerated library for high performance
MPIs). In addition to QLogic MPI, the currently supported MPI
implementations are HP-MPI, Open MPI, MVAPICH, and Scali (Platform).
Open MPI provides MPI-2 functionality, including one-sided operations and
dynamic processes. These all offer the same high performance as QLogic
MPI.
Dual PCIe QLogic adapters per node.
QLogic MPI supports running exclusively on a single node without the
installation of the HCA hardware.
4K Maximum Transfer Unit (MTU) is supported and is on by default. To take
advantage of 4KB MTU, use a switch that supports 4KB MTU. QLogic also
supports 2KB switches, and 4KB MTU switches configured for 2KB MTU.
QLogic switches with firmware version 4.2.x or later are recommended.
2-2 Preliminary IB0056101-00 G.02
Draft
A
Feature Overview
Features
This version of the QLogic OFED software provides support for all of the QLogic
HCAs in Tab le 2- 1.
Table 2-1. QLogic Adapter Model Numbers
QLogic Model
Number
QHT7040 Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For
systems with HTX expansion slots.
QHT7140
QLE7140 Single port 10Gbps SDR 4X InfiniBand to PCI Express x8
QLE7240 Single port 20Gbps DDR 4X InfiniBand to PCI Express x8
QLE7280 Single port 20Gbps DDR 4X InfiniBand to PCI Express x16
Table Notes
PCIe is Gen 1
a
The QHT7140 has a smaller form factor than the QHT7040, but is otherwise the same. Throughout
this document, the QHT7040 and QHT7140 will be collectively referred to as the QHT7140 unless
otherwise noted.
a
Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For
systems with HTX expansion slots.
adapter. Supported on systems with PCI Express (PCIe) x8 or
x16 slots.
adapter. Supported on systems with PCI Express x8 or
x16 slots.
adapter. Supported on systems with PCI Express x16 slots.
The QLE7280 is backward compatible; it can also be used with
PCIe adapters that connect to x8 slots.
Description
Additional up-to-date information can be found on the QLogic web site,
specifically:
The high performance computing page at
www.qlogic.com/Products/HPC_products_landingpage.aspx
The InfiniBand HCA page at
www.qlogic.com/Products/HPC_products_infipathhcas.aspx
IB0056101-00 G.02 Preliminary 2-3
Draft
Feature Overview
Software Components
Software Components
This release includes all of OFED 1.4 with enhancements (QLogic OFED 1.4),
including a new version of the VNIC tools and driver, and support for the QHT7xxx
and QLE7xxx adapters. The software includes the QLogic InfiniPath HCA driver,
libraries, QLogic MPI, Subnet Management Agent, and associated utilities.
Included components are:
InfiniPath driver
InfiniPath libraries, InfiniPath utilities, configuration, and support tools,
including ipath_checkout, ipath_control, ipath_pkt_test, and
ipathstats
QLogic MPI
PSM support for accelerated MPI
OpenMPI and MVAPICH (with PSM support) built with the GNU, PGI,
PathScale, and Intel compilers, with corresponding mpitests and
mpi-selector
S
QLogic MPI benchmarks and utilities
OpenFabrics protocols
OpenFabrics libraries and utilities
QLogic VNIC module
FastFabric Enablement tools
This release provides support for the following protocols and transport services:
IPoIB (TCP/IP networking in either Connected or Datagram mode)
Sockets Direct Protocol (SDP)
Open source Subnet Manager (OpenSM)
Reliable Datagram Sockets (RDS)
iSCSI Extensions for RDMA (iSER)
This release supports two versions of SCSI RDMA Protocol (SRP):
OFED SRP
QLogic SRP
No support is provided for Reliable Datagram (RD).
More details about the hardware and software can be found in Section 4 and
Section 5.
2-4 Preliminary IB0056101-00 G.02
Draft
3 Step-by-Step
Installation Checklist
This section provides an overview of the hardware and software installation
procedures. Detailed steps are found in Section 4 “Hardware Installation” and
Section 5 “Software Installation”.
Hardware Installation
The following steps summarize the basic hardware installation procedure:
1. Check that the adapter hardware is appropriate for your platform. See
Table 4-1.
2. Check to see that you have the appropriate cables and switches, as
described in “Cabling and Switches” on page 4-3.
3. Check to see that you are running a supported Linux distribution/kernel. See
Table 5-1.
4. Verify that the BIOS for your system is configured for use with the QLogic
adapter. See “Configuring the BIOS” on page 4-4.
5. Following the safety instructions in “Safety with Electricity” on page 4-5.
Unpack the adapter (“Unpacking Information” on page 4-5) and verify the
package contents.
6. Install the adapter by following the instructions in “Hardware Installation” on
page 4-9.
7. Cable the adapter to the switch, as described in “Cabling the Adapter to the
InfiniBand Switch” on page 4-17. Check that all InfiniBand switches are
configured.
8. Follow the steps in “Completing the Installation” on page 4-18 to finish the
installation.
IB0056101-00 G.02 Preliminary 3-1
Draft
Step-by-Step Installation Checklist
Software Installation
Software Installation
The following steps summarize the basic QLogic OFED 1.4 software installation
and startup. These steps must be performed on each node in the cluster:
1. Make sure that the HCA hardware installation has been completed
according to the instructions in “Hardware Installation” on page 4-1.
2. Verify that the Linux kernel software is installed on each node in the cluster.
The required kernels and supported Linux distributions for both QLogic
InfiniPath and OpenFabrics are defined in Table 5-1.
3. Make sure that your environment has been set up as described in “Setting
Up Your Environment” on page 5-3.
4. Download your version of the QLogic InfiniPath/OpenFabrics software from
the QLogic web site to a local server directory. See “Choose the Appropriate
Download Files” on page 5-4.
5. Install the selected packages on each cluster node using the corresponding
method as described in one of the following: “Install QLogicIB-Basic with the
Installer Tool” on page 5-7, “Using rpm to Install InfiniPath and OpenFabrics”
on page 5-14, “Install QLogic OFED User-level Software with the rpm
Command” on page 5-17, “Install QLogic OFED Using Rocks” on page 5-20,
“Install QLogic OFED Using a Platform OCS Kit” on page 5-22, “Install
FastFabric Software CD/ISO Image” on page 5-22.
S
6. The system can be rebooted after all the software has been installed.
7. The configuration file for the ib_ipath driver is set up correctly at
installation and is loaded automatically during system boot once the RPMs
have been installed. However, if you wish to change the configuration file,
see “Managing the InfiniPath Driver” on page 6-15.
8. If you want to configure the optional OpenFabrics driver ipoib, and you
have not configured it yet with the Install tool, or if you have used the rpm
install method, configure it as described in “Configuring the IPoIB Network
Interface” on page 6-2.
9. If you want to use the optional OpenFabrics services (opensm, srp,or
VNIC), configure them as described in “OpenSM” on page 6-3, “SRP” on
page 6-4, or “SRP stands for SCSI RDMA Protocol. It was originally
intended to allow the SCSI protocol to run over InfiniBand for Storage Area
Network (SAN) usage. SRP interfaces directly to the Linux file system
through the SRP Upper Layer Protocol. SRP storage can be treated as
another device.” on page 6-4.
10. Check the system state by observing the LEDs. See “LED Link and Data
Indicators” on page 7-1.
3-2 Preliminary IB0056101-00 G.02
Draft
A
Step-by-Step Installation Checklist
Software Installation
11. You can optimize your system and adapter for the best performance. See
“Adapter and Other Settings” on page 7-1. Also see the Performance
Settings and Management Tips section in the QLogic HCA and QLogic
OFED Software Users Guide.
12. Perform the recommended health checks. See “Customer Acceptance
Utility” on page 7-2.
13. After installing the QLogic InfiniPath and OpenFabrics software, refer to the
QLogic HCA and QLogic OFED Software Users Guide for more information
about using QLogic InfiniPath, QLogic MPI, and OpenFabrics products.
Refer to the QLogic ULP and Tools Reference Guide for more information
about configuring and using QLogic SRP, QLogic VNIC, and the Enablement
Tools. The InfiniBand Software Installation Guide also has information on
installing the QLogic InfiniBand Fabric Suite.
IB0056101-00 G.02 Preliminary 3-3
Draft
Step-by-Step Installation Checklist
Software Installation
Notes
S
3-4 Preliminary IB0056101-00 G.02
Draft
4 Hardware Installation
This section lists the requirements and provides instructions for installing the
QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic
DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic PCIe
adapter and PCIe riser card, QLE7140; and the QHT7040 or QHT7140 adapter
hardware and HTX riser card. These components are collectively referred to as
the adapter and the riser card in the remainder of this document.
The adapter is a low-latency, high-bandwidth, high message rate cluster
interconnect for InfiniBand. The QLogic interconnect is InfiniBand 4X, with a raw
data rate of 20Gbps (data rate of 16Gbps) for the QLE7240 and QLE7280; and
10Gbps (data rate of 8Gbps) for the QLE7140, QHT7040, and QHT7140.
OpenFabrics is interoperable with other vendors’ InfiniBand Host Channel
Adapters (HCAs) running compatible OpenFabrics releases.
Hardware Installation Requirements
This section lists hardware and software environment requirements for installing
the QLogic QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140.
Hardware
QLogic interconnect adapters are for use with UL listed computers. The following
statement is true for all the adapters:
This device complies with part 15 of the FCC Rules. Operation is subject to
the following two conditions: (1) This device may not cause harmful
interference, and (2) this device must accept any interference received,
including interference that may cause undesired operations.
Different adapter cards work on different platforms. Ta bl e 4 -1 shows the
relationship between the adapter model and different types of motherboards.
Table 4-1. Adapter Models and Related Platforms
QLogic
Model
Number
QLE7240 PCI Express systems Standard PCI Express x8 or x16 slot
QLE7280 PCI Express systems Standard PCI Express x16 slot
Platform Plugs Into
IB0056101-00 G.02 Preliminary 4-1
Draft
Hardware Installation
Hardware Installation Requirements
Table 4-1. Adapter Models and Related Platforms (Continued)
QLogic
Model
Number
QLE7140 PCI Express systems Standard PCI Express x8 or x16 slot
S
Platform Plugs Into
Form Factors
QHT7040
QHT7140
Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U
or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of
a PCI Express (PCIe) slot in a typical motherboard. See Figure 4-7 for an
illustration of an HTX slot for a typical Opteron motherboard.
The motherboard vendor is the optimal source for information on the layout and
use of HyperTransport and PCI Express-enabled expansion slots on supported
motherboards.
The QLE7240, QLE7280, and QLE7140 are the model numbers for the adapters
that ship in the standard PCI Express half-height, short-form factor. These
adapters can be used with either full-height or low-profile face plates.
The QHT7040 is the model number for the adapter that shipped in the HTX
full-height factor. The HTX low-profile form factor is referred to as the QHT7140. It
is the same as the QHT7040, except for its more compact size. In either case, the
adapter is backward and forward compatible for the motherboards in which it is
supported. The QHT7040 and QHT7140 HTX adapters are collectively referred to
as the QHT7140 unless otherwise stated.
Motherboards with HTX connectors
Motherboards with HTX connectors
HyperTransport HTX slot
HyperTransport HTX slot
When the QHT7040 or QHT7140 adapter is installed with the riser card, it may
prevent some or all of the other PCI expansion slots from being used, depending
on the form factor of the adapter and motherboard.
Run i
path_control -i to see information on which form adapter is installed.
The file /sys/class/infiniband/ipath0/device/boardversion
contains the same information. For more information, see the Useful Programs
and Files appendix in the QLogic HCA and QLogic OFED Software Users Guide.
4-2 Preliminary IB0056101-00 G.02
Loading...