Q-Logic IB0056101-00 G User Manual

QLogic HCA and InfiniPath® Software

Install Guide

Version 2.2

IB0056101-00 G

QLogic HCA and InfiniPath® Software Install Guide

Version 2.2

S

Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no representation nor warranty that such applications are suitable for the specified use without further testing or modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document.

No part of this document may be copied nor reproduced by any means, nor translated nor transmitted to any magnetic medium without the express written consent of QLogic Corporation. In accordance with the terms of their valid QLogic agreements, customers are permitted to make electronic and paper copies of this document for their own exclusive use.

The QHT7040, QHT7140, QLE7140, QLE7240, and QLE7280 QLogic Host Channel Adapters are covered by the following patent: 7308535.

Document Revision History

Rev. 1.0, 8/20/2005

Rev. 1.1, 11/15/05

Rev. 1.2, 02/15/06

Rev. 1.3, Beta 1, 4/15/06

Rev 1.3, 6/15/06

Rev. 2.0 Beta, 8/15/06, QLogic Version Number IB0056101-00 A

Rev. 2.0 Beta 2 10/15/06, QLogic Version Number IB0056101-00 B

Rev. 2.0 11/30/06, QLogic Version Number IB0056101-00 C

Rev. 2.0 3/23/07, QLogic Version Number IB0056101-00 D

Rev. 2.1 8/24/07, QLogic Version Number IB0056101-00 E

Rev. 2.2 5/27/08, QLogic Version Number IB0056101-00 F

Rev. 2.2 9/5/08, QLogic Version Number IB0056101-00 G

Changes	Sections Affected
Re-format to new template (no change to content)	All

Page ii

IB0056101-00 G

Table of Contents

1	Introduction
	Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-1
	How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-1
	Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-2
	Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-3
	Conventions Used in this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-3
	Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-4
	Contact Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-5

2	Feature Overview
	What’s New in this Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-1
	New Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-2
	Supported Distributions and Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-3
	Compiler Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-4
	Software Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-4

3	Step-by-Step Installation Checklist
	Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	3-1
	Software Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	3-2
4	Hardware Installation
	Hardware Installation Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-1
	Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-1
	Form Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-2
	Cabling and Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-3
	Optical Fibre Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-4
	Configuring the BIOS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-4
	Safety with Electricity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-5
	Unpacking Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-5
	Verify the Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-5
	List of the Package Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-5
	Unpacking the QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-8
	Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-9

IB0056101-00 G

Page iii

QLogic HCA and InfiniPath® Software Install Guide

Version 2.2

S

	Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI
	Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		4-9
	Dual Adapter Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		4-9
	Installation Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		4-9
	Hardware Installation for QHT7140 with HTX Riser . . . . . . . . . . . . . .		4-12
	Hardware Installation for QLE7240, QLE7280, and QLE7140 Without a PCI
	Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		4-15
	Hardware Installation for the QHT7140 Without an HTX Riser . . . . . .		4-16
	Switch Configuration and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		4-17
	Cabling the Adapter to the InfiniBand Switch . . . . . . . . . . . . . . . . . . . . . . . .		4-17
	Completing the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		4-18
5	Software Installation
	Cluster Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-1
	Types of Nodes in a Cluster Environment . . . . . . . . . . . . . . . . . . . . . .		5-1
	Supported Linux Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-2
	Setting Up Your Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-2
	Downloading and Unpacking the InfiniPath and OpenFabrics Software . . .		5-3
	Choosing the Appropriate Download Files . . . . . . . . . . . . . . . . . . . . .		5-3
	Unpacking the InfiniPath tar File . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-4
	Check for Missing Files or RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-5
	Installing the InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . .		5-6
	Choosing the RPMs to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-7
	Using rpm to Install InfiniPath and OpenFabrics. . . . . . . . . . . . . . . . .		5-8
	Installed Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-9
	Starting the InfiniPath Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-10
	InfiniPath and OpenFabrics Driver Overview . . . . . . . . . . . . . . . . . . . . . . . .		5-11
	Configuring the InfiniPath Drivers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-11
	Configuring the ib_ipath Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-12
	Configuring the ipath_ether Network Interface . . . . . . . . . . . . . . .		5-12
	ipath_ether Configuration on Red Hat . . . . . . . . . . . . . . . . .		5-12
	ipath_ether Configuration on SLES . . . . . . . . . . . . . . . . . . .		5-14
	OpenFabrics Drivers and Services Configuration and Startup. . . . . . . . . . .		5-16
	Configuring the IPoIB Network Interface . . . . . . . . . . . . . . . . . . . . . . .		5-17
	Configuring and Administering the VNIC Interface . . . . . . . . . . . . . . .		5-18
	OpenSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-23
	SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-24
	Using OFED SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-24
	MPI over uDAPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-26
	Other Configuration: Changing the MTU Size . . . . . . . . . . . . . . . . . . . . . . .		5-26

Page iv

IB0056101-00 G

A

QLogic HCA and InfiniPath® Software Install Guide

Version 2.2

	. . . . . . . . . . . . . . . . . . . . . . .Starting and Stopping the InfiniPath Software		5-26
	Rebuilding or Reinstalling Drivers After a Kernel Upgrade . . . . . . . . . . . . .		5-28
	Rebuilding or Reinstalling Drivers if a Different Kernel is Installed. . . . . . . .		5-29
	Further Information on Configuring and Loading Drivers . . . . . . . . . . . . . . .		5-29
	LED Link and Data Indicators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-29
	Adapter Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-30
	Customer Acceptance Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-31
	Removing Software Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-33
	Uninstalling InfiniPath and OpenFabrics RPMs. . . . . . . . . . . . . . . . . .		5-33
	Uninstalling OFED 1.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-33
	Downgrading RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-33
	Additional Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-33
	Installing Lustre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		5-34
	Installing QLogic MPI in an Alternate Location . . . . . . . . . . . . . . . . . .		5-34
	Installing on an Unsupported Distribution . . . . . . . . . . . . . . . . . . . . . .		5-35
	Managing and Installing Software Using Rocks . . . . . . . . . . . . . . . . .		5-36
	Installing Rocks and InfiniPath RPMs. . . . . . . . . . . . . . . . . . . . .		5-37
	Further Information on Rocks and kickstart . . . . . . . . . . . . . . . .		5-41
A	Installation Troubleshooting
	Hardware Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		A-1
	Node Spontaneously Reboots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		A-1
	Some HTX Motherboards May Need Two or More CPUs in Use . . . .		A-1
	BIOS Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		A-1
	MTRR Mapping and Write Combining. . . . . . . . . . . . . . . . . . . . . . . . .		A-2
	Edit BIOS Settings to Fix MTRR Issues . . . . . . . . . . . . . . . . . . .		A-2
	Use the ipath_mtrr Script to Fix MTRR Issues . . . . . . . . . . .		A-3
	Issue with Supermicro® H8DCE-HTe and QHT7040 . . . . . . . . . . . . .		A-3
	Software Installation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		A-3
	Version Number Conflict with opensm-* on RHEL5 Systems. . . . . . . . . . .		A-4
	OpenFabrics Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		A-4
	OpenFabrics Library Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . .		A-4
	Missing Kernel RPM Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		A-5
	Resolving Conflicts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		A-6
	mpirun Installation Requires 32-bit Support . . . . . . . . . . . . . . . . . . .		A-6
	ifup on ipath_ether on SLES 10 Reports "unknown device" . . . .		A-7
	Lockable Memory Error on Initial Installation of InfiniPath. . . . . . . . . .		A-7

IB0056101-00 G

Page v

QLogic HCA and InfiniPath® Software Install Guide

Version 2.2

S

B	Configuration Files
C	RPM Descriptions
	InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		C-1
	Different Nodes May Use Different RPMs . . . . . . . . . . . . . . . . . . . . . . . . . .		C-1
	RPM Version Numbers and Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		C-1
	RPM Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		C-2
	Documentation and InfiniPath RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		C-2
	OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		C-4

Page vi

IB0056101-00 G

A

QLogic HCA and InfiniPath® Software Install Guide

Version 2.2

List of Figures

Figure		Page
4-1	QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-7
4-2	QLogic QLE7140 Card with Riser, Top View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-7
4-3	QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View . . . .	4-8
4-4	PCIe Slot in a Typical Motherboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-10
4-5	QLogic PCIe HCA Assembly with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-11
4-6	Assembled PCIe HCA with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-12
4-7	HTX Slot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-13
4-8	QLogic QHT7140 Adapter with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-14
4-9	Assembled QHT7140 with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-15
4-10	QHT7140 Without Riser Installed in a 3U Chassis . . . . . . . . . . . . . . . . . . . . . . . . .	4-17
5-1	Relationship Between InfiniPath and OpenFabrics Software. . . . . . . . . . . . . . . . . .	5-11

List of Tables

Table		Page
1-1	Typographical Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	1-3
2-1	QLogic Adapter Model Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	2-1
2-2	InfiniPath/OpenFabrics Supported Distributions and Kernels . . . . . . . . . . . . . . . . .	2-3
4-1	Adapter Models and Related Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-1
4-2	QLogic InfiniBand Cables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	4-3
5-1	InfiniPath/OpenFabrics Supported Distributions and Kernels . . . . . . . . . . . . . . . . .	5-2
5-2	Available Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	5-3
5-3	InfiniPath and OpenFabrics RPMs to Use for Each Node in a Cluster . . . . . . . . . .	5-6
5-4	RPMS to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	5-7
5-5	LED Link and Data Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	5-29
5-6	ipath_checkout Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	5-32
B-1	Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	B-1
C-1	InfiniPath Documentation/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-2
C-2	InfiniPath/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-3
C-3	InfiniPath-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-3
C-4	InfiniPath-MPI/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-4
C-5	OpenFabrics Documentation/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-4
C-6	OpenFabrics/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-4
C-7	OpenFabrics-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-7
C-8	OpenSM/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-8
C-9	Other HCAs/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-8
C-10	Other HCAs-Devel/RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-9
C-11	OtherMPIs/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .	C-9

IB0056101-00 G

Page vii

QLogic HCA and InfiniPath® Software Install Guide

Version 2.2

Notes

S

Page viii

IB0056101-00 G

1 Introduction

This chapter describes the contents, intended audience, and organization of the QLogic HCA and InfiniPath Software Install Guide.

The QLogic HCA and InfiniPath Software Install Guide contains instructions for installing the QLogic Host Channel Adapters (HCAs) and the InfiniPath and OpenFabrics software. The following adapters are covered in this guide:

QLE7140 PCI Express® (PCIe)

QLE7240 PCI Express

QLE7280 PCI Express

QHT7040/QHT7140 HyperTransport Expansion (HTX™)

Who Should Read this Guide

This installation guide is intended for cluster administrators responsible for installing the QLogic QLE7140, QLE7240, QLE7280 or QHT7040/QHT7140 adapter and InfiniPath software on their Linux® cluster. Additional detailed installation information and instructions for administering the QLogic cluster can be found in the QLogic HCA and InfiniPath Software User Guide.

The QLogic HCA and InfiniPath Software Install Guide assumes that you are familiar with both cluster networking and the specific hardware that you plan to use. Before installing the HCA, you should have basic knowledge of your host and target operating systems, and working knowledge of message passing concepts.

This document does not contain all the information you need to use basic Linux commands or to perform all system administration tasks. For this information, see the software documentation you received with your system.

How this Guide is Organized

The QLogic HCA and InfiniPath Software Install Guide is organized into these sections:

Section 1, Introduction, contains an overview of the HCAs and software, describes interoperability with other products, lists all related documentation, and provides QLogic contact information.

Section 2, Feature Overview, contains features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components.

IB0056101-00 G

1-1

1 – Introduction

Overview

S

Section 3, Step-by-Step Installation Checklist, provides a high-level overview of the hardware and software installation procedures.

Section 4, Hardware Installation, includes instructions for installing the QLogic QLE7140, QLE7240, QLE7280, QHT7040, and QHT7140 HCAs.

Section 5, Software Installation, includes instructions for installing the QLogic InfiniPath and OpenFabrics software.

Appendix A, Installation Troubleshooting, contains information about issues that may occur during installation.

Appendix B, Configuration Files, contains descriptions of the configuration and configuration template files used by the InfiniPath and OpenFabrics software.

Appendix C, RPM Descriptions

Index, lists major subjects and concepts with page numbers for easy reference.

Overview

The material in this documentation pertains to an InfiniPath cluster. A cluster is defined as a collection of nodes, each attached to an InfiniBand™-based fabric through the QLogic interconnect. The nodes are Linux-based computers, each having up to 16 processors.

The QLogic HCAs are InfiniBand 4X. The Double Data Rate (DDR) QLE7240 and QLE7280 adapters have a raw data rate of 20Gbps (data rate of 16Gbps). For the Single Data Rate (SDR) adapters, the QLE7140 and QHT7140, the raw data rate is 10Gbps (data rate of 8Gbps). The QLE7240 and QLE7280 can also run in SDR mode.

The QLogic adapters utilize standard, off-the-shelf InfiniBand 4X switches and cabling. The QLogic interconnect is designed to work with all InfiniBand-compliant switches.

NOTE:

If you are using the QLE7240 or QLE7280, and want to use DDR mode, then DDR-capable switches must be used.

1-2	IB0056101-00 G

A

1 – Introduction

Interoperability

InfiniPath OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases. There are several options for subnet management in your cluster:

Use the embedded Subnet Manager (SM) in one or more managed switches supplied by your InfiniBand switch vendor.

Use the Open source Subnet Manager (OpenSM) component of OpenFabrics.

Use a host-based Subnet Manager.

Interoperability

QLogic InfiniPath participates in the standard InfiniBand subnet management protocols for configuration and monitoring. Note that:

InfiniPath OpenFabrics (including Internet Protocol over InfiniBand (IPoIB)) is interoperable with other vendors’ InfiniBand HCAs running compatible OpenFabrics releases.

The QLogic MPI and Ethernet emulation stacks (ipath_ether) are not interoperable with other InfiniBand HCAs and Target Channel Adapters (TCAs). Instead, InfiniPath uses an InfiniBand-compliant, vendor-specific protocol that is highly optimized for MPI and Transmission Control Protocol (TCP) between InfiniPath-equipped hosts.

NOTE:

See the OpenFabrics web site at www.openfabrics.org for more information on the OpenFabrics Alliance.

Conventions Used in this Guide

This guide uses the typographical conventions listed in Table 1-1.

Table 1-1. Typographical Conventions

Convention	Meaning

command Fixed-space font is used for literal items such as commands, functions, programs, files and pathnames, and program output.

variable Italic fixed-space font is used for variable names in programs and command lines.

concept Italic font is used for emphasis and concepts, as well as for documentation names/titles.

IB0056101-00 G

1-3

1 – Introduction

Documentation

S

Table 1-1. Typographical Conventions (Continued)

Convention	Meaning
user input	Bold fixed-space font is used for literal items in commands or con-
	structs that you type.
$	Indicates a command line prompt.
#	Indicates a command line prompt as root when using bash or sh.
[ ]	Brackets enclose optional elements of a command or program con-
	struct.
...	Ellipses indicate that a preceding element can be repeated.
>	A right caret identifies the cascading path of menu commands used
	in a procedure.
2.2	The current version number of the software is included in the RPM
	names and within this documentation.
NOTE:	Indicates important information.

Documentation

The product documentation includes:

The QLogic HCA and InfiniPath Software Install Guide

The QLogic HCA and InfiniPath Software User Guide

The QLogic FastFabric Users Guide (for information on QLogic InfiniServ Tools)

The OFED+ Users Guide (for information on QLogic VNIC and QLogic SRP)

Release Notes

Quick Start Guide

Readme file

For more information on system administration, using the QLogic Message-Passing Interface (MPI), and troubleshooting adapter hardware and software, see the QLogic HCA and InfiniPath Software User Guide.

1-4	IB0056101-00 G

A

1 – Introduction

Contact Information

Contact Information

Support Headquarters	QLogic Corporation
	4601 Dean Lakes Blvd
	Shakopee, MN 55379
	USA
QLogic Web Site	www.qlogic.com
Technical Support Web Site	support.qlogic.com
Technical Support Email	[e-mail]
Technical Training Email	[e-mail]
North American Region
Email	[e-mail]
Phone	[phone]
Fax	[phone]
All other regions of the world
QLogic Web Site	www.qlogic.com

IB0056101-00 G

1-5

1 – Introduction

Contact Information

Notes

S

1-6	IB0056101-00 G

2 Feature Overview

This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components.

What’s New in this Release

This release adds support for the QLE7240 and QLE7280 InfiniBand DDR Host

Channel Adapters (HCAs), which offer twice the link bandwidth of SDR HCAs. The extra bandwidth improves performance for both latency-sensitive and bandwidth-intensive applications.

This version of the InfiniPath software provides support for all of the QLogic HCAs in Table 2-1.

	Table 2-1. QLogic Adapter Model Numbers
QLogic Model		Description
Number
QHT7040		Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For
		systems with HTX expansion slots.
QHT7140 a		Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For
		systems with HTX expansion slots.
QLE7140		Single port 10Gbps SDR 4X InfiniBand to PCI Express x8
		adapter. Supported on systems with PCI Express (PCIe) x8 or
		x16 slots.
QLE7240		Single port 20Gbps DDR 4X InfiniBand to PCI Express x8
		adapter. Supported on systems with PCI Express x8 or
		x16 slots.

IB0056101-00 G

2-1

2 – Feature Overview

New Features

S

Table 2-1. QLogic Adapter Model Numbers (Continued)

	QLogic Model	Description
	Number
	QLE7280	Single port 20Gbps DDR 4X InfiniBand to PCI Express x16
		adapter. Supported on systems with PCI Express x16 slots.
		The QLE7280 is backward compatible; it can also be used with
		PCIe adapters that connect to x8 slots.

Table Notes

PCIe is Gen 1

aThe QHT7140 has a smaller form factor than the QHT7040, but is otherwise the same. Throughout this document, the QHT7040 and QHT7140 will be collectively referred to as the QHT7140 unless otherwise noted.

New Features

The following features are new to the 2.2 release:

Expanded MPI scalability enhancements for PCI Express have been added. On the QLE7240 and QLE7280, up to 16 dedicated hardware contexts per node are available. The QHT7140 has eight per node. The QLE7140 has four per node.

A total of 64 processes on the QLE7240 and QLE7280 are supported when context sharing is enabled. The QHT7040 and QHT7140 support a total of 32 processes per adapter. The QLE7140 supports a total of 16 processes per adapter.

This release continues support for multiple high-performance native PSM Message Passing Interface (MPI) implementations added in the 2.1 release. (PSM is QLogic’s accelerated library for high performance MPIs). In addition to QLogic MPI, the currently supported MPI implementations are HP-MPI, Open MPI, MVAPICH, and Scali. Open MPI provides MPI-2 functionality, including one-sided operations and dynamic processes. These all offer the same high performance as QLogic MPI.

Dual PCIe QLogic adapters per node are supported.

Driver support for the QLogic Virtual Network Interface Controller (VNIC) is provided in this release. The VNIC Upper Layer Protocol (ULP) works in concert with firmware running on Virtual Input/Output (VIO) hardware such as the SilverStorm™ Ethernet Virtual I/O Controller (EVIC), providing virtual Ethernet connectivity for Linux operating systems.

The QLogic InfiniBand Fabric Suite CD is available separately for purchase. The CD includes FastFabric, the QLogic Subnet Manager (SM), and the Fabric Viewer GUI for the QLogic SM.

2-2	IB0056101-00 G

A

2 – Feature Overview

Supported Distributions and Kernels

A subset of the QLogic InfiniBand Fabric Suite, the enablement tools, are offered with this release.

Two separate SCSI RDMA Protocol (SRP) modules are provided: the standard OpenFabrics (OFED) SRP, and QLogic SRP.

QLogic MPI supports running exclusively on a single node without the installation of the HCA hardware.

OpenMPI and MVAPICH libraries built with the GNU, PGI, PathScale™, and Intel® compilers are available, with corresponding mpitests RPMs. You can use mpi-selector to choose which MPI you want. These all run over PSM.

4K Maximum Transfer Unit (MTU) is supported and is on by default. To take advantage of 4KB MTU, use a switch that supports 4KB MTU. QLogic also supports 2KB switches, and 4KB MTU switches configured for 2KB MTU.

QLogic switches with firmware version 4.1 or later are recommended.

The Lustre® cluster filesystem is supported.

Additional up-to-date information can be found on the QLogic web site, specifically:

The high performance computing page at www.qlogic.com/Products/HPC_products_landingpage.aspx

The InfiniBand HCA page at www.qlogic.com/Products/HPC_products_infipathhcas.aspx

Supported Distributions and Kernels

The QLogic interconnect runs on AMD™ Opteron™ and Intel EM64T systems running Linux®. The currently supported distributions and associated Linux kernel versions for InfiniPath and OpenFabrics are listed in Table 2-2. The kernels are the ones that shipped with the distributions.

Table 2-2. InfiniPath/OpenFabrics Supported Distributions

and Kernels

Distribution	InfiniPath/OpenFabrics Supported
		Kernels
Fedora Core 6 (FC6)	2.6.22 (x86_64)
Red Hat® Enterprise Linux® 4.4, 4.5, 4.6	2.6.9-42	(U4), 2.6.9-55 (U5)
(RHEL4.4, 4.5, 4.6)	2.6.9-67	(U6) (x86_64)
CentOS 4.4, 4.5 (Rocks 4.4, 4.5, 4.6)	2.6.9-42, 2.6.9.55, 2.6.9-67 (x86_64)
Scientific Linux 4.4, 4.5, 4.6	2.6.9-42, 2.6.9.55, 2.6.9-67 (x86_64)

IB0056101-00 G

2-3

2 – Feature Overview

Compiler Support

S

Table 2-2. InfiniPath/OpenFabrics Supported Distributions

and Kernels (Continued)

	Distribution	InfiniPath/OpenFabrics Supported
		Kernels
	Red Hat Enterprise Linux 5.0 (RHEL 5.0),	2.6.18-8, 2.6.18-53 (x86_64)
	RHEL 5.1
	CentOS 5.0, 5.1 (Rocks 5.0, 5.1)	2.6.18, 2.6.18-53 (x86_64)
	Scientific Linux 5.0, 5.1	2.6.18, 2.6.18-53 (x86_64)
	SUSE® Linux Enterprise Server	2.6.16.21, 2.6.16.46 (x86_64)
	(SLES 10 GM, SP 1)

NOTE:

Fedora Core 4 and Fedora Core 5 are not supported in the InfiniPath 2.2 release.

Compiler Support

QLogic MPI supports a number of compilers. These include:

PathScale Compiler Suite 3.0 and 3.1

PGI 5.2 , 6.0, and 7.1

Intel 9.x and 10.1

GNU gcc 3.3.x, 3.4.x, 4.0 and 4.1 compiler suites

gfortran

The PathScale Compiler Suite Version 3.x is now supported on systems that have the GNU 4.0 and 4.1 compilers and compiler environment (header files and libraries).

Please check the QLogic web site for updated information on supported compilers.

Software Components

The software includes the InfiniPath HCA driver, QLogic MPI, standard networking over Ethernet emulation, InfiniPath Subnet Management Agent and associated utilities, and OFED for InfiniPath.

This release includes a full set of OFED 1.3 usermode RPMs, with some enhancements, including a new version of the VNIC tools and driver, and support for the QLE7240 and QLE7280 adapters.

2-4	IB0056101-00 G

A

2 – Feature Overview

Software Components

Included components are:

InfiniPath driver

InfiniPath Ethernet emulation (ipath_ether)

InfiniPath libraries

InfiniPath utilities, configuration, and support tools, including ipath_checkout, ipath_control, ipath_pkt_test, and ipathstats

QLogic MPI

QLogic MPI benchmarks and utilities

OpenMPI and MVAPICH libraries built with the GNU, PGI, PathScale, and Intel compilers, with corresponding mpitests RPMs and mpi-selector

OpenFabrics protocols, including Subnet Management Agent

OpenFabrics libraries and utilities

QLogic VNIC module

Enablement tools

This release provides support for the following protocols and transport services:

IPoIB (TCP/IP networking in either Connected or Datagram mode)

Sockets Direct Protocol (SDP)

Open source Subnet Manager (OpenSM)

Unreliable Datagram (UD)

Reliable Connection (RC)

Unreliable Connection (UC)

Shared Receive Queue (SRQ)

Reliable Datagram Sockets (RDS)

iSCSI Extensions for RDMA (iSER)

This release supports two versions of SCSI RDMA Protocol (SRP):

OFED SRP

QLogic SRP

No support is provided for Reliable Datagram (RD).

NOTE:

OpenFabrics programs (32-bit) using the verbs interfaces are not supported in this InfiniPath release, but may be supported in a future release.

IB0056101-00 G

2-5

2 – Feature Overview

Software Components

S

More details about the hardware and software can be found in Section 4 and Section 5.

2-6	IB0056101-00 G

3 Step-by-Step

Installation Checklist

This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 “Hardware Installation” and Section 5 “Software Installation”.

Hardware Installation

The following steps summarize the basic hardware installation procedure:

1.Check that the adapter hardware is appropriate for your platform. See Table 4-1.

2.Check to see that you have the appropriate cables and switches, as described in “Cabling and Switches” on page 4-3.

3.Check to see that you are running a supported Linux distribution/kernel. See Table 2-2 or Table 5-1.

4.Verify that the BIOS for your system is configured for use with the QLogic adapter. See “Configuring the BIOS” on page 4-4.

5.Following the safety instructions in “Safety with Electricity” on page 4-5. Unpack the adapter (“Unpacking Information” on page 4-5) and verify the package contents.

6.Install the adapter by following the instructions in “Hardware Installation” on page 4-9.

7.Cable the adapter to the switch, as described in “Cabling the Adapter to the InfiniBand Switch” on page 4-17. Check that all InfiniBand switches are configured.

8.Follow the steps in “Completing the Installation” on page 4-18 to finish the installation.

IB0056101-00 G

3-1

3 – Step-by-Step Installation Checklist

Software Installation

S

Software Installation

The following steps summarize the basic InfiniPath and OpenFabrics software installation and startup. These steps must be performed on each node in the cluster:

1.Make sure that the HCA hardware installation has been completed according to the instructions in “Hardware Installation” on page 4-1.

2.Verify that the Linux kernel software is installed on each node in the cluster. The required kernels and supported Linux distributions for both InfiniPath and OpenFabrics are defined in Table 5-1.

3.Make sure that your environment has been set up as described in “Setting Up Your Environment” on page 5-2.

4.Download your version of the InfiniPath/OpenFabrics software from the QLogic web site to a local server directory. See “Choosing the Appropriate Download Files” on page 5-3.

5.Unpack the tar file and check for any missing files or RPMs. See “Unpacking the InfiniPath tar File” on page 5-4 and “Check for Missing Files or RPMs” on page 5-5.

6.Install the appropriate packages on each cluster node as described in “Installing the InfiniPath and OpenFabrics RPMs” on page 5-6. Also see Table 5-4 for a list of RPMs to install.

NOTE:

Rocks may be used as a cluster install method. See “Managing and Installing Software Using Rocks” on page 5-36.

7.Under the following circumstances, the system needs to be rebooted:

If this is the first InfiniPath installation OR

If you have installed VNIC with the OpenFabrics RPM set

The system can be rebooted after all the software has been installed. If you are only upgrading from a prior InfiniPath software installation, drivers can be restarted manually. See “Starting the InfiniPath Service” on page 5-10.

8.If you want to use the optional InfiniPath (ipath_ether) and OpenFabrics drivers (ipoib) and services (opensm, srp), configure them as described in “Configuring the InfiniPath Drivers” on page 5-11 and “OpenFabrics Drivers and Services Configuration and Startup” on page 5-16.

9.Check the system state by observing the LEDs. See “LED Link and Data Indicators” on page 5-29.

3-2	IB0056101-00 G

A

3 – Step-by-Step Installation Checklist

Software Installation

10.Optimize your adapter for the best performance. See “Adapter Settings” on page 5-30. Also see the Performance and Management Tips section in the

QLogic HCA and InfiniPath Software User Guide.

11.Perform the recommended health checks. See “Customer Acceptance Utility” on page 5-31.

12.After installing the InfiniPath and OpenFabrics software, refer to the QLogic HCA and InfiniPath Software User Guide for more information about using InfiniPath, QLogic MPI, and OpenFabrics products.

IB0056101-00 G

3-3

3 – Step-by-Step Installation Checklist

Software Installation

Notes

S

3-4	IB0056101-00 G

4 Hardware Installation

This section lists the requirements and provides instructions for installing the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic InfiniPath PCIe adapter and PCIe riser card, QLE7140; and the QHT7040 or QHT7140 adapter hardware and HTX riser card. These components are collectively referred to as the adapter and the riser card in the remainder of this document.

The adapter is a low-latency, high-bandwidth, high message rate cluster interconnect for InfiniBand. The QLogic interconnect is InfiniBand 4X, with a raw data rate of 20Gbps (data rate of 16Gbps) for the QLE7240 and QLE7280; and 10Gbps (data rate of 8Gbps) for the QLE7140, QHT7040, and QHT7140.

OpenFabrics is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases.

Hardware Installation Requirements

This section lists hardware and software environment requirements for installing the QLogic QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140.

Hardware

QLogic interconnect adapters are for use with UL listed computers. The following statement is true for all the adapters:

This device complies with part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) This device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operations.

Different adapter cards work on different platforms. Table 4-1 shows the relationship between the adapter model and different types of motherboards.

Table 4-1. Adapter Models and Related Platforms

QLogic	Platform	Plugs Into
Model
Number
QLE7240	PCI Express systems	Standard PCI Express x8 or x16 slot
QLE7280	PCI Express systems	Standard PCI Express x16 slot

IB0056101-00 G

4-1

4 – Hardware Installation

Hardware Installation Requirements

S

Table 4-1. Adapter Models and Related Platforms (Continued)

QLogic	Platform	Plugs Into
Model
Number
QLE7140	PCI Express systems	Standard PCI Express x8 or x16 slot
QHT7040	Motherboards with HTX connectors	HyperTransport HTX slot
QHT7140	Motherboards with HTX connectors	HyperTransport HTX slot

Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of a PCI Express (PCIe) slot in a typical motherboard. See Figure 4-7 for an illustration of an HTX slot for a typical Opteron motherboard.

The motherboard vendor is the optimal source for information on the layout and use of HyperTransport and PCI Express-enabled expansion slots on supported motherboards.

Form Factors

The QLE7240, QLE7280, and QLE7140 are the model numbers for the adapters that ship in the standard PCI Express half-height, short-form factor. These adapters can be used with either full-height or low-profile face plates.

The QHT7040 is the model number for the adapter that shipped in the HTX full-height factor. The HTX low-profile form factor is referred to as the QHT7140. It is the same as the QHT7040, except for its more compact size. In either case, the adapter is backward and forward compatible for the motherboards in which it is supported. The QHT7040 and QHT7140 HTX adapters are collectively referred to as the QHT7140 unless otherwise stated.

When the QHT7040 or QHT7140 adapter is installed with the riser card, it may prevent some or all of the other PCI expansion slots from being used, depending on the form factor of the adapter and motherboard.

Run ipath_control -i to see information on which form adapter is installed.

The file /sys/bus/pci/drivers/ib_ipath/00/boardversion contains the same information. For more information, see the Troubleshooting appendix in the

QLogic HCA and InfiniPath Software User Guide.

4-2	IB0056101-00 G

A

4 – Hardware Installation

Hardware Installation Requirements

Cabling and Switches

The cable installation uses a standard InfiniBand (IB) 4X cable. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cable that QLogic has currently qualified is 20 meters. For DDR-capable adapters and switches, the DDR-capable passive copper cables cannot be longer than 10 meters. Active cables can eliminate some of the cable length restrictions.

InfiniBand switches are available through QLogic.

NOTE:

If you are using the QLE7240 or QLE7280 and want to use DDR mode, then DDR-capable switches must be used.

The copper cables listed in Table 4-2 are available from QLogic:

Table 4-2. QLogic InfiniBand Cables

Product Number	Description
7104-1M-Cable	4x-4x cable—1 meter
7104-2M-Cable	4x-4x cable—2 meters
7104-3M-Cable	4x-4x cable—3 meters
7104-4M-Cable	4x-4x cable—4 meters
7104-5M-Cable	4x-4x cable—5 meters
7104-6M-Cable	4x-4x cable—6 meters
7104-7M-Cable	4x-4x cable—7 meters
7104-8M-Cable	4x-4x cable—8 meters
7104-9M-Cable	4x-4x Cable—9 meters
7104-10M-Cable	4x-4x cable—10 meters
7104-12M-Cable	4x-4x cable—12 meters (SDR only)
7104-14M-Cable	4x-4x cable—14 meters (SDR only)
7104-16M-Cable	4x-4x cable—16 meters (SDR only)
7104-18M-Cable	4x-4x cable—18 meters (SDR only)

For cabling instructions, see “Cabling the Adapter to the InfiniBand Switch” on page 4-17.

IB0056101-00 G

4-3

4 – Hardware Installation

Hardware Installation Requirements

S

Optical Fibre Option

The QLogic adapter also supports connection to the switch by means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these types of convertors. For more information on the EMCORE convertor, see www.emcore.com.

Intel® and Zarlink™ also offer optical cable solutions. See www.intel.com and www.zarlink.com for more information.

Configuring the BIOS

To achieve the best performance with QLogic adapters, you need to configure your BIOS with specific settings. The BIOS settings, which are stored in non-volatile memory, contain certain parameters characterizing the system. These parameters may include date and time, configuration settings, and information about the installed hardware.

There are two issues concerning BIOS settings of which you need to be aware:

Advanced Configuration and Power Interface (ACPI) needs to be enabled.

Memory Type Range Registers (MTRR) mapping needs to be set to “Discrete”.

MTRR is used by the InfiniPath driver to enable write combining to the on-chip transmit buffers. This option improves write bandwidth to the QLogic chip by writing multiple words in a single bus transaction (typically 64 bytes). This option applies only to x86_64 systems.

However, some BIOS’ do not have the MTRR mapping option. It may have a different name, depending on the chipset, vendor, BIOS, or other factors. For example, it is sometimes referred to as 32 bit memory hole, which must be enabled.

You can check and adjust these BIOS settings using the BIOS Setup utility. For specific instructions, follow the hardware documentation that came with your system.

QLogic also provides a script, ipath_mtrr, which sets the MTRR registers. These registers enable maximum performance from the InfiniPath driver. This script can be run after the InfiniPath software has been installed. It needs to be run after each system reboot.

For more details, see “BIOS Settings” on page A-1.

4-4	IB0056101-00 G

A

4 – Hardware Installation

Safety with Electricity

Safety with Electricity

Observe these guidelines and safety precautions when working around computer hardware and electrical equipment:

Locate the power source shutoff for the computer room or lab where you are working. This is where you will turn OFF the power in the event of an emergency or accident. Never assume that power has been disconnected for a circuit; always check first.

Do not wear loose clothing. Fasten your tie or scarf, remove jewelry, and roll up your sleeves. Wear safety glasses when working under any conditions that might be hazardous to your eyes.

Shut down and disconnect the system’s power supply from AC service before you begin work, to insure that standby power is not active. Power off all attached devices such as monitors, printers, and external components. Note that many motherboards and power supplies maintain standby power at all times. Inserting or removing components while standby is active can damage them.

Use normal precautions to prevent electrostatic discharge, which can damage integrated circuits.

Unpacking Information

This section provides instructions for safely unpacking and handling the QLogic adapter. To avoid damaging the adapter, always take normal precautions to avoid electrostatic discharge.

Verify the Package Contents

The QLogic adapter system should arrive in good condition. Before unpacking, check for any obvious damage to the packaging. If you find any obvious damage to the packaging or to the contents, please notify your reseller immediately.

List of the Package Contents

The package contents for the QLE7240 adapter are:

QLogic QLE7240

Additional short bracket

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or motherboard vendor.

IB0056101-00 G

4-5

4 – Hardware Installation

Unpacking Information

S

The package contents for the QLE7280 adapter are:

QLogic QLE7280

Additional short bracket

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or motherboard vendor.

The package contents for the QLE7140 adapter are:

QLogic QLE7140

Quick Start Guide

Standard PCIe risers can be used, typically supplied by your system or motherboard vendor. The contents are illustrated in Figure 4-2.

The package contents for the QHT7140 adapter are:

QLogic QHT7140

HTX riser card for use in 1U or 2U chassis

Quick Start Guide

The contents are illustrated in Figure 4-3.

The IBA6120, IBA6110, and IBA7220 are the QLogic ASICs, which are the central components of the interconnect. The location of the IBA7220 ASIC on the adapter is shown in Figure 4-1. The location of the IBA6120 ASIC on the adapter is shown in Figure 4-2. The location of the IBA6110 ASIC on the adapter is shown in Figure 4-3.

4-6	IB0056101-00 G

A

4 – Hardware Installation

Unpacking Information

PCI Express

Edge Connectors

InfiniBand

Connector

Face Plate

IBA7220 ASIC

Figure 4-1. QLogic QLE7280 with IBA7220 ASIC

PCI Express Riser Card. Not supplied; shown for reference.

PCI Express		InfiniBand
Edge Connectors
		Connector
	IBA6120 ASIC	Face Plate

Figure 4-2. QLogic QLE7140 Card with Riser, Top View

IB0056101-00 G

4-7

4 – Hardware Installation

Unpacking Information

S

HTX Edge

Connectors

HTX Riser Card

InfiniBand

Connector

IBA6110 ASIC	QHT7140 Low
	Profile Card	Face Plate

PathScale

InfiniBand

Connector

Face Plate

PathScale

QHT7040 Full Height

Short Card

Figure 4-3. QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View

Unpacking the QLogic Adapter

Follow these steps when unpacking the QLogic adapter:

1.When unpacking, ground yourself before removing the QLogic adapter from the anti-static bag.

2.Grasping the QLogic adapter by its face plate, pull the adapter out of the anti-static bag. Handle the adapter only by its edges or the face plate. Do not allow the adapter or any of its components to touch any metal parts.

3.After checking for visual damage, store the adapter and the riser card in their anti-static bags until you are ready to install them.

4-8	IB0056101-00 G

A

4 – Hardware Installation

Hardware Installation

Hardware Installation

This section contains hardware installation instructions for the QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140.

Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser

Installation for the QLE7240, QLE7280, and QLE7140 is similar. The following instructions are for the QLE7140, but can be used for any of these three adapters.

Most installations will be in 1U and 2U chassis, using a PCIe right angle riser card. This results in an installation of the adapter that is parallel to the motherboard. This type of installation is described first. Installation in a 3U chassis is described in “Hardware Installation for the QHT7140 Without an HTX Riser” on page 4-16.

Installing the QLogic QLE7140 in a 1U or 2U chassis requires a PCIe right angle riser card.

A taller riser card can be used if necessary. The QLE7140 can connect to any of the standard compatible PCI Express riser cards.

Dual Adapter Installation

If you have a motherboard with dual PCIe slots, dual adapters can be installed. The adapters must match. For example, on a motherboard with two x16 slots, dual QLE7280 adapters can be installed, but not a QLE7240 adapter and a QLE7280 adapter. Check the design of your motherboard to see how riser cards can be used.

Follow the instructions in “Installation Steps” on page 4-9.

See the Using MPI section in the QLogic HCA and InfiniPath Software User Guide for information on using the IPATH_UNIT environment variable to control which HCA to use.

Installation Steps

To install the QLogic adapter with a PCIe riser card:

1.The BIOS should already be configured properly by the motherboard manufacturer. However, if any additional BIOS configuration is required, it will usually need to be done before installing the QLogic adapter. See “Configuring the BIOS” on page 4-4.

2.Shut down the power supply to the system into which you will install the QLogic adapter.

3.Take precautions to avoid electrostatic damage (ESD) to the cards by properly grounding yourself or touching the metal chassis to discharge static electricity before handling the cards.

IB0056101-00 G

4-9

4 – Hardware Installation

Hardware Installation

S

4.Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system.

5.Locate the PCIe slot on your motherboard. Note that the PCIe slot has two separate sections, with the smaller slot opening located towards the front (see Figure 4-4). These two sections correspond to the shorter and longer connector edges of the adapter and riser.

PCIe slot in typical motherboard

Figure 4-4. PCIe Slot in a Typical Motherboard

6.Determine if a blanking panel is installed in your chassis. If it is, remove it so that the InfiniBand connector will be accessible. Refer to your system vendor instructions for how to remove the blanking panel.

7.Remove the QLogic adapter from the anti-static bag.

8.Locate the face plate on the connector edge of the card.

4-10	IB0056101-00 G

A

4 – Hardware Installation

Hardware Installation

9.Connect the QLogic adapter and PCIe riser card together, forming the assembly that you will insert into your motherboard. First, visually line up the adapter slot connector edge with the edge connector of the PCIe riser card (see Figure 4-5).

.

PCIe Riser Card

QLogic Adapter	Face Plate

LEDs

InfiniBand connector

Figure 4-5. QLogic PCIe HCA Assembly with Riser Card

10.Holding the QLogic adapter by its edges, carefully insert the card slot connector into the PCIe riser card edge connector, as show in Figure 4-5. The result is a combined L-shaped assembly of the PCIe riser card and QLogic adapter. This assembly is what you will insert into the PCIe slot on the motherboard in the next step.

11.Turn the assembly so that the riser card connector edge is facing the PCIe slot on the motherboard, and the face plate is toward the front of the chassis.

12.Holding this assembly above the motherboard at about a 45 degree angle, slowly lower it so that the connector on the face plate clears the blanking panel opening of the chassis from the inside. Slowly align the connector edge of the riser card with the motherboard’s PCIe slot. The short section of the connector must align with the short section of the slot.

IB0056101-00 G

4-11