HPE InfiniBand Options for HPE BladeSystems c-Class
Overview
Page 1
HPE InfiniBand Options for HPE BladeSystems c-Class
Hewlett Packard Enterprise supports 56 Gbps Fourteen Data Rate (FDR) and 40Gbps Quad Data Rate (QDR) InfiniBand (IB)
products that include mezzanine Host Channel Adapters (HCA) for server blades, dual
adaptors, switch blades for c
focuses
on mezzanine adaptors for server blades, and InfiniBand switch blades for c
switches, standup PCI Express HCAs and FlexLOM Adaptors for
The
enclosure. It is based on the Mellanox Switch X technology. The FDR IB switch blade has 16 downlink ports to connect up to
server blades in the c7000 enclosure, and 18 QSFP uplink ports for inter
capable of supporting 56Gbps (FDR) bandwidth. A subnet manager has to be provided; see the paragraph on subnet manage
more details.
The
Platinum enclosure. It is based on the Mellanox Switch X technology. The FDR IB switch blade has 16 downlink ports to
to 16 server blades in the c7000 enclosure, and 18 QSFP uplink ports for inter
ports are capable of supporting 56Gbps (FDR) bandwidth. The switch contains an integrated management module that is capable of
managing a cluster of up to 648 nodes.
links. The most commonly deployed fabric topology is a fat tree or its variations. A subnet manager is require
control an InfiniBand fabric. The subnet manager functionality can be provided by either the
rack-mount InfiniBand switch with an embedded fabric manager (aka internally managed switch) or host-based subnet manager
mode InfiniBand or Ethernet mezzanine
-Class enclosures, and rack switches and cables for building scale-out solutions. This QuickSpecs
-Class enclosures; for details on the InfiniBand rack
HPE ProLiant and Apollo
HPE InfiniBand for HPE ProLiant and Apollo servers QuickSpecs at:
• HPE BLc FDR IB Switch for HPE BladeSystem c-Class
for HPE BladeSystem c-Class are based on the Mellanox ConnectX-3 Pro technology. The 544+M
the PCI Express 3.0 x8 interface. The 544+M Mezzanine adapters are dual function adapters that are
dual InfiniBand ports or dual Ethernet ports when connected to supported Ethernet switches in c7000 enclosure
d port and one Ethernet port is not supported on 544+M Mezzanine adapters)
HPE BladeSystem c-Class Gen 9 blade servers.
HPE BladeSystem c-Class from HPE:
the PCI Express 3.0 x16 interface. Two ports provide more than 100Gb/s of throughput in dual port FDR InfiniBand mode
n improvement in message rate compared with previous generations of QDR InfiniBand cards.
The 544+M and 545M Mezzanine Adapters are only supported in the HPE BLc7000 Platinum Enclosures.
tack software (driver) is required to run on servers connected to the InfiniBand network. For mezzanine HCAs based
Hewlett Packard Enterprise supports Mellanox OFED Driver on Linux 64-bit operating systems, and
icrosoft Windows.
HPE BLc FDR IB Switch for HPE BladeSystem c-Class is a double wide switch for the HPE BladeSystem c7000 Platinum
HPE BLc FDR Managed IB Switch for HPE BladeSystem c-Class is a double wide switch for the HPE BladeSystem c7000
HPE BladeSystem c-Class is based on the Mellanox Connect-IB technology. The 545M adapter
-driven server and storage clustering applications. Additional application acceleration is achieved
u_section=34.
16
-switch links or to connect to external servers. All ports are
connect up
-switch links or to connect to external servers. All
An InfiniBand fabric consists of one or more InfiniBand switches connected via inter-switch
d to manage and
HPE BLc FDR Managed IB Switch or a
rs for
QuickSpecs
HPE InfiniBand Options for HPE BladeSystems c-
Class
Overview
Page 2
software on a server connected to the fabric. OpenSM is a host-based subnet manager that runs on a server connected to the
InfiniBand fabric. Mellanox OFED software stack includes OpenSM for Linux, and Mellanox WinOF includes OpenSM for Windows. For
comprehensive management and monitoring capabilities, Mellanox Unified Fabric Manager
managing the InfiniBand fabric based on Mellanox/ InfiniBand switch products and Mellanox based mezzanine HCA.
An embedded fabric
QDR switches.
Please refer to:
mount
HP
from 1M to 7M for HCA to switch, or inter
for HCA to switch, or inter
switch, or inter
Please refer to:
InfiniBand cables.
What's New
•
− PCI Express 3.0 x8
™ Advanced (UFM) is recommended for
manager is available on Mellanox internally managed 36-port FDR switch, and modular FDR and
https://www.hpe.com/h20195/v2/GetHTML.aspx?docname=c04154440 for information about InfiniBand rack
switches.
E supports InfiniBand copper and fiber optic cables with QSFP to QSFP connectors. The QSFP to QSFP QDR copper cables range
-switch links at QDR speed. The QSFP to QSFP FDR copper cables range from 0.5M to 3M
-switch links at FDR speed. The QSFP to QSFP FDR optical cables range from 3M to 30M for HCA to
-switch links at FDR speed.
https://www.hpe.com/h20195/v2/GetHTML.aspx?docname=c04154440 for more details on supported
Mellanox Connect-IB technology
Dual port FDR InfiniBand
PCI Express 3.0 x16 interface for increased throughput
Improved message rate
Designed for use with HPE BLc7000 Platinum enclosures
− Support Mellanox OFED Linux driver stack r
For firmware release notes, please visit the support website
system.
− HPE BLc FDR IB Switch for HPE BladeSystem c-Class
Double wide switch blade for the Platinum c7000 enclosure with 16 downlink ports
to connect server blades via the midplane, and 18 QSFP unlink ports for the interswitch links or to connect to external servers. All ports are capable of supporting
56Gbps (FDR) ban
version of the switch (648312-B21)
to be provided on the fabric (see the subnet manager discussion above). The other
version of the switch (648311-B21) contains a management module and can run
the subnet manager.
then select your operating
Reliability
• All ports on FDR HCA cards and switch are capable of supporting 56Gbps signaling rate, with a peak
data rate of 52 Gbps in each direction.
• All ports on QDR HCA card and switch are capable of supporting 40Gbps signaling rate, with a peak
• The 545M Mezzanine Adapter: PCI Express revision 3.0x16 compliant
• IBTA version 1.2 compatible
Page
QuickSpecs
HPE InfiniBand Options for HPE BladeSystems c-
Class
Standard Features
Page 5
Operating systems
The 544+M series devices configured as IB only mode, or Ethernet only mode are supported on MLNX_OFED
WinOF 5.10 on
0) are
Interconnect Bays
HPE IB switch blades are designed to fit into the double wide switch bays on the c-Class enclosures. Depending
InfiniBand Fabric
Auto-negotiation/self-discovery:
support
3.1-1.0.3 on the following Linux operating systems:
• RHEL: 6.5; 6.6; 6.7; 7.0; 7.1
• CentOS: 6.5; 6.6; 6.7; 7.0; 7.1
• SLES: 11 SP1; SP2; SP3; SP4; 12
• OEL: 6.3; 6.4; 6.5; 6.6; 6.7; 7.0; 7.1
• Ubuntu: 12.04; 14.04; 14.10; 15.04
The 544+M series devices configured as IB only mode or Ethernet only mode are supported on
the following Windows operating systems:
• Windows Server 2012 R2 (x64 only)
• Windows Server 2012 (x64 only)
• Windows Server 2008 R2 (x64 only)
The 544+M series devices configured as Ethernet only mode (with VMware 4.15.2.0 or VMware 1.9.10also supported in the following virtualized environments:
• ESXI 6.x
• ESX 5.5
• ESX 5.1
The 545M adapter is supported on MLNX_OFED 3.1-1.0.3 on the following Linux operating systems:
• RHEL: 6.5; 6.6; 6.7; 7.0; 7.1
• CentOS: 6.5; 6.6; 6.7; 7.0; 7.1
• SLES: 11 SP1; SP2; SP3; SP4; 12
• OEL: 6.3; 6.4; 6.5; 6.6; 6.7; 7.0; 7.1
• Ubuntu: 12.04; 14.04; 14.10; 15.04
on c-Class
enclosure
on the mezzanine connectors used for IB HCA mezzanine cards, the IB switch blade has to be inserted into
corresponding interconnect bays.
• The FDR switch may be installed in interconnect bays 3 and 4, 5 and 6, 7 and 8 in the HPE BLc7000
InfiniBand FDR or QDR ports support auto-negotiation down to lower frequencies. For example, external uplink
ports of a FDR switch blade will negotiate down to QDR or DDR speed if they are connected to a device which
can only operate at those speeds.
Self-discovery is supported by subnet manager. Check the documents of the specific subnet manager for more
http://www.docs.hp.com.
details
Management Support:
Mellanox OFED includes drivers and utilities to configure and manage the HCA in a Linux environment and
Mellanox WinOF includes drivers and utilities to configure and manage the HCA in the Microsoft Windows
environment.
QuickSpecs
HPE InfiniBand Options for HPE BladeSystems c-
Class
Standard Features
Page 6
InfiniBand fabric has to be set up with a subnet manager: see the subnet manager discussion above.
Warranty
Hewlett Packard Enterprise branded hardware options qualified for BladeSystem c-Class and p-Class servers
agreements.
are covered by a global limited warranty and supported by HPE Services and a worldwide network of Hewlett
Packard Enterprise Authorized Channel Partners. The Hewlett Packard Enterprise branded hardware option
diagnostic support and repair is available for one year from date of purchase, or the length of the server they
are attached to, whichever is greater. Support for software and initial setup is available for 90 days from date of
purchase. Additional support may be covered under the warranty or available for an additional fee.
Enhancements to warranty services are available through HPE Care Pack services or customized service
QuickSpecs
HPE InfiniBand Options for HPE BladeSystems c-
Class
Service and Support
7
Service and
HPE Technology Services
Recommended
Recommended HPE Care Pack Services for optimal satisfaction with your HPE product
HPE Technology Services offers you consultants and support experts to solve your most complex
infrastructure problems. We help keep your business running, boost availability and avoid downtime.
Protect your business beyond warranty with HPE Care Pack Services
When you buy HPE Options, it's also a good time to think about what level of service you may need. HPE Care
Pack services provide total care and support expertise with committed response choices designed to meet your
IT and business need.
HPE Foundation Care services deliver scalable support-packages for Hewlett Packard Enterprise industrystandard servers and software. You can choose the type and level of service that is most suitable for your
business needs. If you are running business critical environments, Hewlett Packard Enterprise offers Proactive
Care. These services help you deliver high levels of application availability through proactive service
management and advanced technical response.
Here is the support service recommendation from the Foundation Care and Proactive Care portfolio. For
customized support service solution, Hewlett Packard Enterprise can work with you to tailor a service solution
for your unique support requirements using broader services portfolio of Foundation Care and Proactive Care.
Hewlett Packard Enterprise recommends Proactive Care for your BladeSystem servers, enclosures and
components to prevent problems with proactive advice and assistance, resolve issues and rapidly recover by
connecting to The Hewlett Packard Enterprise enhanced call experience and advanced technical expertise
HPE ProLiant Server Hardware Installation
Provides for the basic hardware installation of Hewlett Packard Enterprise branded
networking options to assist you in bringing your new hardware into operation in a timely and professional
manner.
Provides a flexible way to purchase HPE best-in-class consultancy and technical services. You can buy
Proactive Select Service Credits when you purchase your hardware and then use the credits over the next 12
months.
Page
QuickSpecs
Class
Relat
8
Cabling Options
FDR Cables
HP 0.5M FDR Quad Small Form Factor Pluggable InfiniBand Copper Cable
670759-B21
HP 1M FDR Quad Small Form Factor Pluggable InfiniBand Copper Cable
670759-B22
HP 1.5M FDR Quad Small Form Factor Pluggable InfiniBand Copper Cable
670759-B23
HP 2M FDR Quad Small Form Factor Pluggable InfiniBand Copper Cable
670759-B24
HP 3M FDR Quad Small Form Factor Pluggable InfiniBand Copper Cable
670759-B25
HP 3 Meter InfiniBand FDR QSFP V-series Optical Cable
808722-B21
HP 5 Meter InfiniBand FDR QSFP V-series Optical Cable
808722-B22
HP 7 Meter InfiniBand FDR QSFP V-series Optical Cable
808722-B23
HP 10 Meter InfiniBand FDR QSFP V-series Optical Cable
808722-B24
HP 12 Meter InfiniBand FDR QSFP V-series Optical Cable
808722-B25
HP 15 Meter InfiniBand FDR QSFP V-series Optical Cable
808722-B26
HP 20 Meter InfiniBand FDR QSFP V-series Optical Cable
808722-B27
HP 30 Meter InfiniBand FDR QSFP V-series Optical Cable
808722-B28
HPE Ethernet
Modules
HP 10GbE Ethernet Pass-Thru Module for c-Class BladeSystem
538113-B21
Mellanox SX1018HP Ethernet Switch for c-Class BladeSystem
689638-B21
HPE InfiniBand Options for HPE BladeSystems c-
ed Options
Interconnect
Page
QuickSpecs
HPE InfiniBand Options for HPE BladeSystems c-
Class
Technical Specifications
9
Compliance
Mezzanine:
HPE 544+M, 545M, PCI Express 3.0, ROHS-R6
Switches:
IBTA version 1.2 compatible
ROHS-R6
General
Communications
Mezzanine:
The 544+M: Mellanox ConnectX-3 Pro
HPE The 545M: Mellanox Connect-IB