INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR
OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS
OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING
TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE,
MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death.
SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND
ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL
CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT
LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS
SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics
of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever
for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design
with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published
specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-5484725, or go to: http://www.intel.com/design/literature.htm
Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
This chapter describes the contents, intended audience, and organization of the Intel®
Adapter Hardware Installation Guide.
The Intel
adapters. The following adapters are covered in this guide:
• QLE7340 PCI Express
• QLE7342 PCI Express
1.1Intended Audience
®
Adapter Hardware Installation Guide contains instructions for installing the
This installation guide is intended for administrators responsible for installing the Intel®
QLE7340 or QLE7342 adapter in their Linux
and instructions for administering an Intel
Host Software User Guide.
The Intel
®
Adapter Hardware Installation Guide assumes that you are familiar with the
specific hardware that you plan to use. Before installing the adapter, you should have
basic knowledge of your host and target operating systems.
This document does not contain all the information you need to use basic Linux
commands or to perform all system administration tasks. For this information, see the
software documentation you received with your system.
1.2How this Guide is Organized
The Intel® Adapter Hardware Installation Guide is organized into these sections:
• Chapter 1.0, “Introduction”, contains an overview of the host channel adapters and
software, describes interoperability with other products, lists all related
documentation, and provides Intel contact information.
• Chapter 2.0, “Hardware Installation Checklist”, provides a high-level overview of
the hardware installation procedures.
• Chapter 3.0, “Hardware Installation”, includes instructions for installing the Intel
QLE7340, and QLE7342 adapters.
1.3Overview
The material in this documentation pertains to an OFED+ cluster. A cluster is defined as
a collection of nodes, each attached to an InfiniBand
interconnect. The nodes are generally Linux
processors/cores.
The Intel
QLE7342 adapters have a raw data rate of 40Gbps.
The Intel
interconnect is designed to work with all InfiniBand
®
adapters are InfiniBand* 4X. The Quad Data Rate (QDR) QLE7340 and
®
adapters utilize Intel® True Scale Fabric switches and cabling. The Intel
*
cluster. Additional detailed information
®
cluster can be found in the Intel® OFED+
*
*
*
-based computers having multiple
-based fabric through the Intel®
*
-compliant switches.
®
Note:If you are using the QLE7340 or QLE7342, and want to use QDR mode, then QDR-
capable switches must be used.
®
Intel
OFED+ Host Software is interoperable with other vendors’ host channel adapters
running compatible OFED+ releases. There are several options for subnet management
in your cluster:
July 2015Installation Guide
Order Number: G91929004US5
True Scale Fabric Adapter Hardware
Adapter Hardware
• Use the embedded Subnet Manager (SM) in one or more managed switches
supplied by your InfiniBand
• Use a host-based Subnet Manager such as the Intel
Manager, as a part of the Intel
*
switch vendor.
®
True Scale Fabric Suite download.
• Use the Open source Subnet Manager (OpenSM) component of OFED+.
True Scale Fabric Suite FastFabric Command Line Interface Reference Guide
®
OFED+ Host Software Release Notes
®
True Scale Fabric Suite Software Release Notes
1.5Documentation Conventions
This guide uses the following documentation conventions:
• Note: provides additional information.
• Caution: indicates the presence of a hazard that has the potential of causing
damage to data or equipment.
• Warning: indicates the presence of a hazard that has the potential of causing
personal injury.
• Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this
guide, and links to Web sites are also shown in blue. For example:
— Table 2 lists problems related to the user interface and remote agent.
— See “Installation Checklist” on page 6.
— For more information, visit www.intel.com.
• Text in bold font indicates user interface elements such as a menu items, buttons,
check boxes, or column headings. For example:
— Click the Start button, point to Programs, point to Accessories, and then
click Command Prompt.
— Under Notification Options, select the Warning Alarms check box.
• Text in
For example:
• Key names and key strokes are indicated with UPPERCASE:
• Text in italics indicates terms, emphasis, variables, or document titles. For
example:
Courier font indicates a file name, directory path, or command line text.
— To return to the root directory from anywhere in the file structure:
— For a complete listing of license agreements, refer to the Intel Software End
User License Agreement.
— What are shortcut keys?
— To enter the date type mm/dd/yyyy (where mm is the month, dd is the day,
and yyyy is the year).
• Topic titles between quotation marks identify related topics either within this
manual or in the online help throughout this document.
1.6License Agreements
Refer to the Intel Software End User License Agreement for a complete listing of all
license agreements affecting this product.
1.7Technical Support
Intel True Scale Technical Support for products under warranty is available during local
standard working hours excluding Intel Observed Holidays. For customers with
extended service, consult your plan for available hours. For Support information, see
the Support link at
www.intel.com/truescale.
§ §
July 2015Installation Guide
Order Number: G91929004US7
This section provides an overview of the hardware installation procedures. Detailed
steps are found in Chapter 3.0, “Hardware Installation”.
The following steps summarize the basic hardware installation procedure:
1. Check that the adapter hardware is appropriate for your platform. See Table 1.
2. Check to see that you have the appropriate cables and switches, as described in
“Cabling and Switches” on page 11.
3. Following the safety instructions in “Safety with Electricity” on page 12. Unpack the
adapter (“Unpacking Information” on page 12) and verify the package contents.
4. Install the adapter by following the instructions in “Hardware Installation” on
page 13.
5. Cable the adapter to the switch, as described in “Cabling the Adapter to the Switch”
on page 17. Check that all switches are configured.
6. Follow the steps in “Completing the Installation” on page 17 to finish the
installation.
§ §
July 2015Installation Guide
Order Number: G91929004US9
This section lists the requirements and provides instructions for installing the Intel®
adapters. Instructions are included for the Intel
®
QDR PCI Express adapters, the
QLE7340 and QLE7342. These components are collectively referred to as the adapter
and the riser card in the remainder of this document.
The adapter is a low-latency, high-bandwidth, high message rate cluster interconnect
for InfiniBand
*
. The Intel interconnect is InfiniBand* 4X, with a raw data rate of 40Gbps
for the QLE7340 and QLE7342.
®
Intel
True Scale Fabric OFED+ Host Software is interoperable with other vendors’ host
This section lists hardware and software environment requirements for installing the
®
Intel
QLE7340, QLE7342.
3.1.1Hardware
Intel® adapters are for use with UL listed computers. The following statement is true
for all the adapters:
This device complies with part 15 of the FCC Rules. Operation is subject to the
following two conditions: (1) This device may not cause harmful interference, and
(2) this device must accept any interference received, including interference that
may cause undesired operations.
Different adapters work on different platforms. Table 1 shows the relationship between
the adapter model and different types of motherboards.
Installation of the QLE7340, QLE7342 in a 1U or 2U chassis requires the use of a riser
card. See Figure 5 for an illustration of a PCI Express (PCIe) slot in a typical
motherboard.
The motherboard vendor is the optimal source for information about the layout and use
of PCI Express-enabled expansion slots on supported motherboards.
3.1.1.1Form Factors
The QLE7340 and QLE7342 are the model numbers for the adapters that ship in the
standard PCI Express half-height, short-form factor. These adapters can be used with
either full-height or low-profile face plates.
3.1.2Cabling and Switches
The adapters use standard InfiniBand* cables. These cables can be passive copper,
active copper, fiber optic or Quad Small Form-factor Pluggable (QSFP) active optical.
Contact Intel Sales or your authorized Reseller for a list of qualified InfiniBand
*
cables.
July 2015Installation Guide
Order Number: G91929004US11
True Scale Fabric Adapter Hardware
Adapter Hardware
Note:If you want to use the QLE7340 or QLE7342 in QDR mode, a QDR-capable InfiniBand*
switch must be used.
For cabling instructions, see “Cabling the Adapter to the Switch” on page 17.
3.2Safety with Electricity
Observe these guidelines and safety precautions when working around computer
hardware and electrical equipment:
• Locate the power source shutoff for the computer room or lab where you are
working. This is where you will turn OFF the power in the event of an emergency or
accident. Never assume that power has been disconnected for a circuit; always
check first.
• Do not wear loose clothing. Fasten your tie or scarf, remove jewelry, and roll up
your sleeves. Wear safety glasses when working under any conditions that might
be hazardous to your eyes.
• Shut down and disconnect the system’s power supply from AC service before you
begin work, to insure that standby power is not active. Power off all attached
devices such as monitors, printers, and external components. Note that many
motherboards and power supplies maintain standby power at all times. Inserting or
removing components while standby is active can damage them.
• Use normal precautions to prevent electrostatic discharge, which can damage
integrated circuits.
3.3Unpacking Information
This section provides instructions for safely unpacking and handling the Intel® adapter.
To avoid damaging the adapter, always take normal precautions to avoid electrostatic
discharge.
3.3.1Verify the Package Contents
The Intel® adapter system should arrive in good condition. Before unpacking, check for
any obvious damage to the packaging. If you find any obvious damage to the
packaging or to the contents, please notify your reseller immediately.
3.3.2List of the Package Contents
The package contents for the QLE7340 adapter are:
®
• Intel
• Additional short bracket
• Readme First
Standard PCIe risers can be used, typically supplied by your system or motherboard
vendor.
Standard PCIe risers can be used, typically supplied by your system or motherboard
vendor.
The IBA7322 is the Intel
®
ASIC that is the central component of the interconnect. The
location of the IBA7322 ASIC on the adapter is shown in Figure 1.
Note:The QLE7342 also contains the IBA7322 ASIC. The only difference is the QLE7342 has
two external InfiniBand
Figure 1.Intel
®
QLE7340 with IBA7322 ASIC
*
ports.
PCI Express
Edge Connectors
InfiniBand*
Connector
Face Plate
IBA7322 ASIC
3.3.3Unpacking the Intel Adapter
Follow these steps when unpacking the Intel® adapter:
®
1. When unpacking, ground yourself before removing the Intel
anti-static bag.
®
2. Grasping the Intel
adapter by its face plate, pull the adapter out of the anti-static
bag. Handle the adapter only by its edges or the face plate. Do not allow the
adapter or any of its components to touch any metal parts.
3. After checking for visual damage, store the adapter and the riser card in their antistatic bags until you are ready to install them.
adapter from the
3.4Hardware Installation
This section contains hardware installation instructions for the QLE7340 and QLE7342.
3.4.1Hardware Installation for QLE734x with PCI Express Riser
Most installations will be in 1U and 2U chassis, using a PCIe right angle riser card. This
results in an installation of the adapter that is parallel to the motherboard. .
Installing the adapters in a 1U or 2U chassis requires a PCIe right angle riser card.
A taller riser card can be used if necessary. The adapter can connect to any of the
standard compatible PCI Express riser cards.
July 2015Installation Guide
Order Number: G91929004US13
True Scale Fabric Adapter Hardware
3.4.1.1Dual Adapter Installation
If you have a motherboard with dual PCIe slots, dual adapters can be installed. The
adapters must match. For example, on a motherboard with two x16 slots, dual
QLE7340 adapters can be installed, but not a QLE7340 adapter and a QLE7342 adapter.
Check the design of your motherboard to see how riser cards can be used.
Follow the instructions in “Installation Steps” on page 14.
See the Using MPI section in the Intel
information on using the
IPATH_UNIT environment variable to control which host
channel adapter to use.
3.4.1.2Installation Steps
To install the Intel® adapter with a PCIe riser card:
1. Shut down the power supply to the system into which you will install the Intel
adapter.
2. Take precautions to avoid electrostatic damage (ESD) to the cards by properly
grounding yourself or touching the metal chassis to discharge static electricity
before handling the cards.
3. Remove the cover screws and cover plate to expose the system’s motherboard. For
specific instructions on how to do this, follow the hardware documentation that
came with your system.
4. Locate the PCIe slot on your motherboard. Note that the PCIe slot has two separate
sections, with the smaller slot opening located towards the front (see Figure 5).
These two sections correspond to the shorter and longer connector edges of the
adapter and riser.
Adapter Hardware
®
OFED+ Host Software User Guide for
®
Figure 2.PCIe Slot in a Typical Motherboard
PCIe slot in
typical motherboard
5. Determine if a blanking panel is installed in your chassis. If it is, remove it so that
the InfiniBand
instructions for how to remove the blanking panel.
6. Remove the Intel
7. Locate the face plate on the connector edge of the card.
*
connector will be accessible. Refer to your system vendor
8. Connect the Intel® adapter and PCIe riser card together, forming the assembly that
you will insert into your motherboard. First, visually line up the adapter slot
connector edge with the edge connector of the PCIe riser card (see Figure 9).
Figure 3.Intel PCIe Host Channel Adapter Assembly with Riser Card
PCIe Riser Card
Intel Adapter
Face Plate
LEDs
InfiniBand* connector
9. Holding the Intel
the PCIe riser card edge connector, as show in Figure 9. The result is a combined Lshaped assembly of the PCIe riser card and Intel
you will insert into the PCIe slot on the motherboard in the next step.
10. Turn the assembly so that the riser card connector edge is facing the PCIe slot on
the motherboard, and the face plate is toward the front of the chassis.
11. Holding this assembly above the motherboard at about a 45 degree angle, slowly
lower it so that the connector on the face plate clears the blanking panel opening of
the chassis from the inside. Slowly align the connector edge of the riser card with
the motherboard’s PCIe slot. The short section of the connector must align with the
short section of the slot.
12. Insert the riser assembly into the motherboard’s PCIe slot, ensuring good contact.
The Intel
®
above it (see Figure 4).
®
adapter by its edges, carefully insert the card slot connector into
®
adapter. This assembly is what
adapter should now be parallel to the motherboard and about one inch
July 2015Installation Guide
Order Number: G91929004US15
True Scale Fabric Adapter Hardware
Figure 4.Assembled PCIe Host Channel Adapter with Riser
13. Secure the face plate to the chassis. The Intel® adapter has a screw hole on the
side of the face plate that can be attached to the chassis with a retention screw.
The securing method may vary depending on the chassis manufacturer. Refer to
the system documentation for information about mounting details such as
mounting holes, screws to secure the card, or other brackets.
The Intel
®
PCIe host channel adapter with PCIe riser card is now installed. Next, install
the cables as described in “Cabling the Adapter to the Switch” on page 17. Then test
your installation by powering up and verifying link status (see “Completing the
Installation” on page 17).
Adapter Hardware
3.4.2Hardware Installation for QLE734x Without a PCI Express Riser
Installing the Intel® QLE734x without a PCI Express riser card requires a 3U or larger
chassis.
To install the Intel
1. Shut down the power supply to the system into which you will install the Intel
adapter.
2. Take precautions to avoid electrostatic discharge (ESD) damage to the cards by
properly grounding yourself or touching the metal chassis to discharge static
electricity before handling the cards.
3. If you are installing the Intel
screws and cover plate to expose the system’s motherboard. For specific
instructions on how to do this, follow the hardware documentation that came with
your system.
4. Locate the PCIe slot on your motherboard.
5. Remove the Intel
horizontal section of the bracket, and the top rear corner of the card. Be careful not
to touch any of the components on the printed circuit card.
6. Without fully inserting, gently align and rest the adapter card’s gold fingers on top
of the motherboard’s PCIe slot.
7. Insert the card by pressing firmly and evenly on the top of the horizontal bracket
and the top rear corner of the card simultaneously. The card should insert evenly
®
adapter without a riser card:
®
adapter into a covered system, remove the cover
®
adapter from the anti-static bag. Hold the card by the top
into the slot. Be careful not to push, grab, or put pressure on any other part of the
card, and avoid touching any of the components.
®
8. Secure the face plate to the chassis. The Intel
adapter has a screw hole on the
side of the face plate that can be attached to the chassis with a retention screw.
The securing method may vary depending on the chassis manufacturer. Refer to
the system documentation for information about mounting details such as
mounting holes, and screws to secure the card or other brackets.
Next, install the cables, as described in “Cabling the Adapter to the Switch” on page 17.
Then test your installation by powering up the system (see “Completing the
Installation” on page 17).
3.5Switch Configuration and Monitoring
The Intel interconnect is designed to work with all InfiniBand*-compliant switches, such
as the the Intel
®
12000 series. Follow the applicable switch documentation for
installing and configuring your switches.
3.6Cabling the Adapter to the Switch
Follow the recommendations of your cable vendor for cable management and proper
bend radius.
The QLE7340 and QLE7342 are cabled the same way.
To install the cables:
1. Check that you have removed the protector plugs from the cable connector ends.
2. Different vendor cables might have different latch mechanisms. Determine if your
cable has a spring-loaded latch mechanism.
• If your cable is spring-loaded, grasp the metal shell and pull on the plastic latch to
release the cable. To insert, push and the cable snaps into place. You will hear a
short “click” sound from the cable connector when it snaps in.
• If your cable latch mechanism is not spring-loaded, push on the metal case, then
push the plastic latch to lock the cable in place.
*
3. The InfiniBand
Connect the cable to the connector on the Intel
the side latches of the cable when connecting. (On some cables, this latch is
located at the top of the cable connector.) Make sure the lanyard handle on the
cable connector is slid forward toward the card connector until fully engaged.
4. Connect the other end of the cable to the switch.
cables are symmetric; either end can be plugged into the switch.
®
QLE7340 and QLE7342. Depress
3.7Completing the Installation
To complete the hardware installation:
1. Complete any other installation steps for other components.
2. Replace the cover plate and back panel.
3. Verify that the power cable is properly connected.
4. Turn on the power supply and boot the system normally.
5. Watch the LED indicators. The LEDs will flash only once, briefly, at power-up. The
LEDs are functional only after the True Scale software has been installed, the driver
has been loaded, and the system is connected to an InfiniBand
LEDs to check the state of the adapter, see “LED Link and Data Indicators”.
True Scale Fabric Adapter Hardware
July 2015Installation Guide
Order Number: G91929004US17
*
switch. To use the
3.8LED Link and Data Indicators
The LEDs function as link and data indicators once the True Scale software has been
installed, the driver has been loaded, and the fabric is being actively managed by a
subnet manager.
Table 2 describes the LED states. The green LED indicates the physical link signal; the
amber LED indicates the link. The green LED normally illuminates first. The normal
state is Green On, Amber On. The QLE7342 and QLE7240 have an additional state, as
shown in Table 2.
Table 2.LED Link and Data Indicators
LED StatesIndication
Green OFF
Amber OFF
Green ON
Amber OFF
Green ON
Amber ON
Green BLINKING (quickly)
Amber ON
1
Green BLINKING
Amber BLINKING
1. This feature is available only on the QLE7340 and QLE7342 adapters
The switch is not powered up. The software is neither installed nor
started. Loss of signal.
Verify that the software is installed and configured with ipath_control
-i. If correct, check both cable connectors.
Signal detected and the physical link is up. Ready to talk to SM to bring
the link fully up.
If this state persists, the SM may be missing or the link may not be
configured.
Use ipath_control -i to verify the software state. If all host channel
adapters are in this state, then the SM is not running. Check the SM
configuration, or install and run opensmd.
The link is configured, properly connected, and ready. Signal detected.
Ready to talk to an SM to bring the link fully up.
The link is configured. Properly connected and ready to receive data
and link packets.
Indicates traffic
Locates the adapter
This feature is controlled by ipath_control -b [On | Off]