Mellanox ConnectX-6 Dx operation manual

Exported onOct/22/2020 06:45 PM

https://docs.mellanox.com/x/esV-AQ

NVIDIA Mellanox ConnectX-6 Dx PCIe HHHL
Ethernet Adapter Cards User Manual

2

Table of Contents

Table of Contents .......................................................................................................2

Introduction................................................................................................................9

Product Overview .................................................................................................................... 9

Features and Benefits........................................................................................................... 11

Operating Systems/Distributions ......................................................................................... 13

Connectivity ........................................................................................................................... 13

Manageability ........................................................................................................................ 14

Interfaces .................................................................................................................15

ConnectX-6 Dx IC Interface.................................................................................................. 16

Encryption......................................................................................................................... 16

PCI Express Interface ........................................................................................................... 16

Networking Ports LEDs Interface ........................................................................................ 16

Scheme 1: One Bi-Color LED .......................................................................................... 17

Scheme 3: One Bi-Color LED and one Single Color LED............................................... 17

Voltage Regulators................................................................................................................ 18

Heatsink................................................................................................................................. 18

Hardware Installation ..............................................................................................19

Safety Warnings..................................................................................................................... 19

Installation Procedure Overview........................................................................................... 19

System Requirements........................................................................................................... 20

Hardware Requirements ................................................................................................. 20

Airflow Requirements .................................................................................................20

Software Requirements ...................................................................................................20

Safety Precautions ................................................................................................................ 20

Unpacking the Package ........................................................................................................ 21

Pre-Installation Checklist..................................................................................................... 21

Bracket Replacement Instructions ...................................................................................... 22

Installation Instructions........................................................................................................ 22

Uninstalling the Card ............................................................................................................ 24

Cables and Modules.............................................................................................................. 25

Identifying the Card in Your System ..................................................................................... 25

Uninstalling the Card ............................................................................................................ 26

3

Driver Installation ....................................................................................................28

Linux Driver Installation........................................................................................................ 28

Prerequisites ....................................................................................................................28

Downloading Mellanox OFED .......................................................................................... 28

Installing Mellanox OFED ................................................................................................ 29

Installation Script ........................................................................................................29

Installation Procedure ................................................................................................30

Installation Results .....................................................................................................32

Installation Logs..........................................................................................................33

openibd Script..............................................................................................................33

Driver Load Upon System Boot ..................................................................................33

mlnxofedinstall Return Codes....................................................................................34

Uninstalling MLNX_OFED........................................................................................... 34

Installing MLNX_OFED Using YUM ................................................................................. 34

Setting up MLNX_OFED YUM Repository................................................................... 34

Installing MLNX_OFED Using the YUM Tool ..............................................................35

Uninstalling MLNX_OFED Using the YUM Tool .........................................................36

Installing MLNX_OFED Using apt-get Tool..................................................................... 36

Setting up MLNX_OFED apt-get Repository ..............................................................37

Installing MLNX_OFED Using the apt-get Tool..........................................................37

Uninstalling MLNX_OFED Using the apt-get Tool..................................................... 38

Updating Firmware After Installation.............................................................................. 38

Updating the Device Online......................................................................................... 38

Updating the Device Manually ....................................................................................38

Updating the Device Firmware Automatically upon System Boot ............................39

UEFI Secure Boot.............................................................................................................39

Enrolling Mellanox's x.509 Public Key on Your Systems...........................................39

Removing Signature from kernel Modules ................................................................40

Performance Tuning ........................................................................................................41

Windows Driver Installation.................................................................................................. 41

Software Requirements .................................................................................................. 41

Downloading Mellanox WinOF-2 Driver ..........................................................................41

Installing Mellanox WinOF-2 Driver ............................................................................... 42

Attended Installation................................................................................................... 42

4

Unattended Installation...............................................................................................47

Installation Results .....................................................................................................48

Uninstalling Mellanox WinOF-2 Driver............................................................................ 48

Attended Uninstallation ..............................................................................................48

Unattended Uninstallation.......................................................................................... 48

Extracting Files Without Running Installation ................................................................ 49

Firmware Upgrade ...........................................................................................................51

VMware Driver Installation ................................................................................................... 51

Hardware and Software Requirements........................................................................... 51

Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 52

Removing Earlier Mellanox Drivers................................................................................. 52

Firmware Programming ..................................................................................................53

Updating Adapter Firmware ....................................................................................54

Troubleshooting .......................................................................................................55

GeneralTroubleshooting ...................................................................................................... 55

LinuxTroubleshooting .......................................................................................................... 55

WindowsTroubleshooting..................................................................................................... 56

Specifications ...........................................................................................................57

MCX621102A[C/E/N]-ADAT Specifications........................................................................... 57

MCX623102A[C/N]-ADATSpecifications .............................................................................. 58

MCX623102A[C/N/S/E]-GDAT Specifications ...................................................................... 59

MCX623106A[C/N/S/E]-CDATSpecifications ....................................................................... 60

MCX623106P[C/N/E]-CDATSpecifications........................................................................... 61

MCX623105A[N/E]-CDATSpecifications .............................................................................. 62

MCX623105A[C/N/S/E]-VDATSpecifications........................................................................ 63

Dual-Port SFP28/SFP56 x8 Adapter Cards Mechanical Drawing................................. 64

Dual-PortSFP28/SFP56 x16 Adapter Cards Mechanical Drawing............................... 64

Single-Port QSFP56 x16 Adapter Cards Mechanical Drawing...................................... 65

Dual-Port QSFP56 x16 Adapter Cards Mechanical Drawing ........................................ 65

Bracket Mechanical Drawing ............................................................................................... 65

Dual-Port SFP28/SFP56 Adapter Card ...........................................................................66

Short Bracket ..............................................................................................................66

Tall Bracket .................................................................................................................66

Single-Port QSFP56 Adapter Card .................................................................................. 66

5

Short Bracket ..............................................................................................................66

Tall Bracket .................................................................................................................66

Dual-Port QSFP56 Adapter Card..................................................................................... 67

Short Bracket ..............................................................................................................67

Tall Bracket .................................................................................................................67

Finding the MAC on the Adapter Card......................................................................68

Document Revision History ......................................................................................71

6

About This Manual

This User Manual describes NVIDIA® Mellanox®ConnectX®-6 Dx Ethernet adapter cards. It provides
details as to the interfaces of the board, specifications, required software and firmware for operating
the board, and relevant documentation.

Ordering Part Numbers

The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet
adapter cards.

Configuration OPN Marketing Description

25GbE PCIe
OPNs

50GbE OPNs MCX623102AC-

100GbE OPNs MCX623105AN-

MCX621102AC­ADAT

MCX621102AN­ADAT

MCX621102AE­ADAT

MCX623102AC­ADAT

MCX623102AN­ADAT

GDAT

MCX623102AN­GDAT

MCX623102AS­GDAT

MCX623102AE­GDAT

CDAT

MCX623105AE­CDAT

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,
Crypto and Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,
No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,
Crypto, No Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0
x16, Crypto and Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0
x16, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0
x16, Crypto and Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 50GbE , Dual-port SFP56, PCIe 4.0
x16, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0
x16, Secure Boot, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0
x16, Crypto, No Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 100GbE, Single-port QSFP56, PCIe

4.0 x16, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 100GbE, Single-port QSFP56, PCIe

4.0 x16, Crypto, No Secure Boot, Tall Bracket

100GbE with PPS
In/Out OPNs

MCX623106AC­CDAT

MCX623106AN­CDAT

MCX623106AS­CDAT

MCX623106AE­CDAT

MCX623106PC­CDAT

MCX623106PN­CDAT

MCX623106PE­CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0
x16, Crypto and Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0
x16, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0
x16, Secure Boot, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0
x16, Crypto, No Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS
In/Out, PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS
In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS
In/Out, PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket

7

Configuration OPN Marketing Description

200GbE OPNs MCX623105AC-

VDAT

MCX623105AN­VDAT

MCX623105AS­VDAT

MCX623105AE­VDAT

ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe

4.0 x16, Crypto and Secure Boot, Tall Bracket

ConnectX®-6 Dx EN adapter card, 200GbE , Single-port QSFP56, PCIe

4.0 x16, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe

4.0 x16, Secure Boot, No Crypto, Tall Bracket

ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe

4.0 x16, Crypto, No Secure Boot, Tall Bracket

Intended Audience

This manual is intended for the installer and user of these cards.The manual assumes basic familiarity

with Ethernet network and architecture specifications.

Technical Support

Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough

the following methods:

•

URL:http://www.mellanox.com> Support

•

E-mail:[e-mail]

•

Tel: [phone]

Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails

regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first

seekassistance through their reseller.

Related Documentation

Mellanox OFED for
Linux User Manual
and Release Notes

WinOF-2 for

WindowsUser Manual

and Release Notes

Mellanox VMware for

Ethernet User Manual

and Release Notes

Mellanox Firmware
Utility (mlxup) User

Manual and Release

Notes

Mellanox Firmware

Tools (MFT) User

Manual

User Manual describing OFED features, performance, band diagnostic, tools
content and configuration. SeeMellanox OFED for Linux Documentation.

User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content and configuration. SeeWinOF-2 for Windows Documentation.

User Manual describing the various components of the Mellanox ConnectX®
NATIVE ESXi stack. See http://www.mellanox.comProducts > Software > Ethernet
Drivers > VMware Driver

Mellanox firmware update and query utility used to update the firmware. Seehttp:/

/www.mellanox.com> Products > Software > Firmware Tools >mlxup Firmware

Utility

User Manual describing the set of MFT firmware management tools for a single
node. SeeMFT User Manual.

8

IEEE Std 802.3

Specification

IEEE Ethernet specification at http://standards.ieee.org/

PCI Express

Specifications

Mellanox LinkX

Interconnect

Solutions

Industry Standard PCI Express Base and Card Electromechanical Specifications at

https://pcisig.com/specifications

Mellanox LinkX Ethernet cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high­bandwidth, low-latency connections between compute nodes and switch nodes.
Mellanox offers one of industry’s broadest portfolio 40Gb/s, 56Gb/s, 100Gb/s and
200Gb/s cables, including Direct Attach Copper cables (DACs), copper splitter
cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths
from 0.5m to 10km.Mellanox tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.Read more at https://

www.mellanox.com/products/interconnect/infiniband-overview.php

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.

Revision History

A list of the changes made to this document are provided inDocument Revision History.

9

Introduction

Product Overview

This is the

User Manual

for Mellanox Technologies Ethernet adapter cards based on the ConnectX®-6

Dx integrated circuit device.

As the world's most advanced cloud SmartNIC, ConnectX-6 Dx provides up to two ports of 25, 50 or
100Gb/s or a single-port of 200Gb/s Ethernet connectivity, powered by 50Gb/s PAM4 SerDes technology

and PCIe Gen 4.0 host connectivity.ConnectX-6 Dx continues among Mellanox's innovation path in

scalable cloud fabrics, delivering unparalleled performance and efficiency at every scale. ConnectX-6
Dx's innovative hardware offload engines, including IPsec and TLS inline data-in-motion encryption, are

ideal for enabling secure network connectivity in modern data-center environments. Please refer toFe

ature and Benefits for more details.

Mellanox ConnectX-6 Dx 25GbE Adapter Cards

Part Number MCX621102AC-

ADAT

Form Factor/
Dimensions

Data Transmission
Rate

Network Connector
Type

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x8 PCIe Gen 4.0 SERDES @ 16.0GT/s

5.59in. x 2.71in (142.00mm x 68.90mm)

Ethernet: 1/10/25 Gb/s

Dual-port
SFP28

MCX621102AE-

ADAT

Dual-port
SFP28

MCX621102AN-

ADAT

Dual-port SFP28 Dual-port

MCX623102AC­ADAT

SFP28

x16

MCX623102AN-

ADAT

Dual-port SFP28

Crypto √ √ - √ -

Secure Boot √ - - √ -

Adapter IC Part
Number

RoHS RoHS Compliant

MT28928A0­CCCF-CE

MT28928A0­NCCF-CE

MT28928A0­NCCF-CE

MT28928A0­CCCF-CE

MT28928A0­NCCF-CE

Mellanox ConnectX-6 Dx 50GbE Adapter Cards

Part Number MCX623102AC-GDAT MCX623102AN-GDAT MCX623102AS-GDAT

Form Factor/Dimensions 5.59in. x 2.71in (142.00mm x 68.90mm)

Data Transmission Rate Ethernet: 1/10/25/40/50 Gb/s

Network Connector Type Dual-port SFP56

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16

10

Crypto √ √ - -

Secure Boot √ - - √

Adapter IC Part Number MT28928A0-CCCF-CE MT28928A0-

CCCF-CE

RoHS RoHS Compliant

MT28928A0-NCCF-CE MT28928A0-NCCF-CE

Mellanox ConnectX-6 Dx 100GbE Adapter Cards

Part Number MCX623105A

N-CDAT

Form Factor/
Dimensions

Data
Transmission
Rate

Network
Connector Type

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16

Crypto - √ √ √ - -

Secure Boot - - - √ - √

5.59in. x 2.71in (142.00mm x 68.90mm)

Ethernet: 1/10/25/40/50/100 Gb/s

Single-port
QSFP56

MCX623105A
E-CDAT

Single-port
QSFP56

MCX623106A
E-CDAT

Dual-port
QSFP56

MCX623106A

C-CDAT

Dual-port
QSFP56

MCX623106A

N-CDAT

Dual-port
QSFP56

MCX623106A
S-CDAT

Dual-port
QSFP56

Adapter IC Part
Number

RoHS RoHS Compliant

MT28928A0­XCCF-CE

MT28928A0­XCCF-CE

MT28928A0­CCCF-CE

MT28928A0­CCCF-CE

MT28928A0­NCCF-CE

Mellanox ConnectX-6 Dx 100GbE Adapter Cards withPPS In/Out SMAs

Part Number MCX623106PN-CDAT MCX623106PE-CDAT MCX623106PC-CDAT

Form Factor/Dimensions 5.59in. x 2.71in (142.00mm x 68.90mm)

Data Transmission Rate Ethernet: 1/10/25/40/50/100 Gb/s

Network Connector Type Dual-port QSFP56 Dual-port QSFP56 Dual-port QSFP56

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16

Crypto - √ √

Secure Boot - - √

PPS In/Out SMAs √ √ √

Adapter IC Part Number MT28928A0-CCCF-CE MT28928A0-NCCF-CE MT28928A0-NCCF-CE

MT28928A0­NCCF-CE

RoHS RoHS Compliant

11

Mellanox ConnectX-6 Dx 200GbE Adapter Cards

Part Number MCX623105AC-

VDAT

Form Factor/
Dimensions

Data Transmission
Rate

Network Connector
Type

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16

Crypto √ √ - -

Secure Boot √ - - √

Adapter IC Part
Number

RoHS RoHS Compliant

5.59in. x 2.71in (142.00mm x 68.90mm)

Ethernet: 1/10/25/40/50/100/200 Gb/s

Single-port QSFP56

MT28928A0-CCCF-CEMT28928A0-CCCF-CEMT28928A0-NCCF-CEMT28928A0-NCCF-

MCX623105AE­VDAT

MCX623105AN­VDAT

MCX623105AS­VDAT

For more detailed information seeSpecifications.

CE

Features and Benefits

This section describes hardware features and capabilities. Please refer to the relevant driver



and/or firmware release notes for feature availability.

Feature Description

PCI
Express
(PCIe)

PCIe Gen 4.0 SERDES@ 8.0GT/s / 16.0GT/s through x8/x16 Edge Connector

12

Feature Description

Mellanox adapters comply with the following IEEE 802.3 standards:

200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE

– IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
– IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
– IEEE 802.3ba 40 Gigabit Ethernet
– IEEE 802.3by 25 Gigabit Ethernet
– IEEE 802.3ae 10 Gigabit Ethernet
– IEEE 802.3ap based auto-negotiation and KR startup
– IEEE 802.3ad, 802.1AX Link Aggregation
– IEEE 802.1Q, 802.1P VLAN tags and priority
– IEEE 802.1Qau (QCN)
– Congestion Notification
– IEEE 802.1Qaz (ETS)
– IEEE 802.1Qbb (PFC)
– IEEE 802.1Qbg
– IEEE 1588v2
– Jumbo frame support (9.6KB)

Memory PCI Express - stores and accesses Ethernet fabric connection information and packet data.

Overlay
Networks

RDMA
overConve
rged
Ethernet
(RoCE)

Mellanox
PeerDirect

®

CPU
Offload

•

•

SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by
Macronix)

•

FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity
is 128Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus(Available
only in QSFP cards).

In order to better scale their networks, data center operators often create overlay networks that
carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as
NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the
hardware offloading engines, placing higher loads on the host CPU. Mellanox ConnectX-6 Dx
effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines
that encapsulate and de-capsulate the overlay protocol.

Mellanox ConnectX-6 Dx, utilizing RoCE (RDMA over Converged Ethernet) technology, delivers
low-latency and high-performance over Band and Ethernet networks. Leveraging data center
bridging (DCB) capabilities as well as Mellanox ConnectX-6 Dx advanced congestion control
hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and
Layer 3 networks.

Mellanox PeerDirect® communication provides high-efficiency RDMA access by eliminating
unnecessary internal data copies between components on the PCIe bus (for example, from GPU to
CPU), and therefore significantly reduces application run time. Mellanox ConnectX-6 Dx advanced
acceleration technology enables higher cluster efficiency and scalability to tens of thousands of
nodes.

Adapter functionality enabling reduced CPU overhead allowing more available CPU for
computation tasks.

Open vSwitch (OVS) offload using ASAP2- Accelerated Switch and Packet

• Flexible match-action flow tables

• Tunnelingencapsulation / decapsulation

Quality of
Service
(QoS)

Hardware­based I/
OVirtualiza
tion

Support for port-based Quality of Service enabling various application requirements for latency
and SLA.

Mellanox ConnectX-6 Dx provides dedicated adapter resources and guaranteed isolation and
protection for virtual machines within the server.

13

Feature Description

Storage
Acceleratio
n

SR-IOV Mellanox ConnectX-6 Dx SR-IOV technology provides dedicated adapter resources and guaranteed

High­Performan
ceAccelera
tions

PPS In/Out
SMAs

A consolidated compute and storage network achieves significant cost-performance advantages
over multi-fabric networks. Standard block and file access protocols can leverage

• RDMA for high-performance storage access

• NVMe over Fabric offloads for target machine

• Erasure Coding

• T10-DIF Signature Handover

isolation and protection for virtual machines (VM) within the server.

• Tag Matching and Rendezvous Offloads

• Adaptive Routing on Reliable Transport

• Burst Buffer Offloads for Background Checkpointing

Mellanox offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related
features called “5T”. Mellanox PTP and 5T software solution are designed to meet the most
demanding PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that
allows ConnectX-6 Dx to achieve sub 20u Sec accuracy and also offers many timing related
functions such as time-triggered scheduling or time-based SND accelerations (time-based
ASAP²) . Furthermore, the 5T technology enables the software application to transmit fronthaul
(ORAN) compatible in high bandwidth. The PTP part supports slave clock, master clock, and
boundary clock.

ConnectX-6 Dx PTP solution allows you to run any PTP stack on your host.

With respect to testing and measurements, selected Mellanox adapters allow you to use the PPS­out signal from onboard SMA connecter, ConnectX-6 Dx also allows to measure PTP in scale, with
PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal,
and the timestamp will be sent to the SW.

Enabled in MCX623106PN-CDAT, MCX623106PE-CDAT and MCX623106PC-CDAT.

Operating Systems/Distributions

•

RHEL/CentOS

•

Windows

•

FreeBSD

•

VMware

•

OpenFabrics Enterprise Distribution (OFED)

•

OpenFabrics Windows Distribution (WinOF-2)

Connectivity

•

Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches

•

Passive copper cable with ESD protection

•

Powered connectors for optical and active cable support

14

Manageability

Mellanox ConnectX-6 Dx technology maintains support for manageability through a BMC. Mellanox
ConnectX-6 Dx PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP
over PCIe protocols as if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter
for the specific manageability solution in use by the server, please contact Mellanox Support.

15

Interfaces

The below figures show the component side of the Mellanox ConnectX-6 Dx adapter card. Each
numbered interface that is referenced in the figures is described in the following table with a link to
detailed information.

The below figures are for illustration purposes only and might not reflect the current revision



of the adapter card.

Callout Item Description

1 “ConnectX-6 Dx IC” Mellanox ConnectX-6 Dx IC on the board.

2 “PCI Express Interface" PCIe Gen 3.0/4.0 through an x8/x16 edge connector.

3 “Ethernet SFP28/SFP56/QSFP56

Interface”

4 “Networking Ports LEDs Interface” There are two I/O LEDs per port to indicate speed

“SMBus Interface” Allows BMC connectivity using MCTP over SMBus or

“Voltage Regulators” Voltage supply pins that feed onboard regulators.

Ethernet traffic is transmitted through the adapter's
SFP28/SFP56/QSFP56 connectors. The networking
connectors allow for the use of modules, optical and
passive cable interconnect solutions.

and link status.

MCTP over PCIe protocols.

16

ConnectX-6 Dx IC Interface

The ConnectX®-6 Dx EN family of adapter IC devicesdelivers two ports of 10/25/40/50/100Gb/s or a

single-port of 200Gb/s Ethernet connectivity paired with best-in-class hardware capabilities that
accelerate and secure cloud and data-center workloads.

Encryption

Applies to Crypto OPNs only.



ConnectX-6 Dx brings security to every end-point, including:

•

Purpose-built inline acceleration engines that offload IPsec and TLS data-in-motion and XTS­AES data-at-rest cryptographic operations.

•

Stateful firewall solution acceleration, powered by Open vSwitch connection tracking and

Mellanox’s ASAP2 technology.

•

Embedded hardware root-of-trust and support for RSA-based secure firmware update and
secure boot, providing guaranteed integrity of the network adapter.

Ethernet SFP28 / SFP56 / QSFP56 Interfaces

The network ports of the ConnectX-6 Dx adapter card are compliant with the IEEE 802.3 Ethernet
standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28 / SFP56 /

QSFP56connector on the adaptercard.

The adapter card includes special circuits to protect from ESD shocks to the card/server when



plugging copper cables.

PCI Express Interface

ConnectX-6 Dx adapter cards support PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible) through x8/16
edge connectors. The device can be either a master initiating the PCI Express bus operations, or a
slave responding to PCI bus operations.
The following lists PCIe interface features:

•

PCIe Gen 4.0 and 3.0 compliant, 2.0 and 1.1 compatible

•

2.5, 5.0, 8.0, or 16.0 GT/s link rate x8 or x16 lanes

•

Auto-negotiates to x16, x8, x4, x2, or x1

•

Support for MSI/MSI-X mechanisms

Networking Ports LEDs Interface

For the networking ports LEDs description, follow the below table depending on the OPN you have

purchased.

17

OPN LEDs Scheme

MCX621102AC-ADAT, MCX621102AN-ADAT,
MCX621102AE-ADAT

MCX623102A[C/N]-ADAT, MCX623102A[C/N]­GDAT,MCX623102A[S/E/N/C]-GDAT
MCX623105A[N/E]-CDAT,MCX623106A[C/N/S/E]­CDAT,MCX623106P[C/N/E]-CDAT,
MCX623105A[C/N/S/E]-VDAT

Scheme 1: One Bi-Color LED

Scheme 3: One Bi-Color LED and one Single Color LED

Scheme 1: One Bi-Color LED

There is one bicolor(Yellow and Green) I/O LED per port to indicate speed and link status.

Physical and Logical Link Indications

State Bi-Color LED (Yellow/Green)

Physical link speed

Beacon command for locating the adapter card 1Hz blinking Yellow

Error 4Hz blinking Yellow indicates a problem with the link

Physical Activity • A constant Green indicates a physical link with the

maximum networking speed.

• A constant Yellow indicates a physical link with less than
the maximum networking speed.

Link Up • A constant Green indicates a physical link with

themaximum networking speed.

• A constant Yellow indicates a physical link with less than
the maximum networking speed.

Scheme 3: One Bi-Color LED and one Single Color LED

There are two I/O LEDs per port to indicate speed and link status. LED1 is a bicolor LED (Yellow and
green) and LED2 is a single color LED (green).

Physical and Logical Link Indications (Ethernet Mode)

State Bi-Color LED (Yellow/Green)

Physical link speed

Beacon command for locating the
adapter card

Error 4Hz blinking Yellow indicates a problem

1Hz blinking Yellow OFF

with the link

Single Color LED (Green)

Logical link/activity

ON

18

State Bi-Color LED (Yellow/Green)

Physical link speed

Single Color LED (Green)

Logical link/activity

Physical Activity • A constant Green indicates a physical

link with the maximum networking speed.

• A constant Yellow indicates a physical
link with less than the maximum
networking speed.

Link Up • A constant Green indicates a physical

link with themaximum networking speed.

• A constant Yellow indicates a physical
link with less than the maximum
networking speed.

Blinking

ON

SMBus Interface

ConnectX-6 Dx technology maintains support for manageability through a BMC. ConnectX-6 Dx PCIe
stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as
if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific
manageability solution in use by the server, please contact Mellanox Support.

Voltage Regulators

The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These
voltage supply pins feed on-board regulators that provide the necessary power to the various
components on the card.

Heatsink

The heatsink is attached to the ConnectX-6 Dx IC in order to dissipate the heat. It is attached either by
using four spring-loaded push pins that insert into four mounting holes, or by screws. ConnectX-6 Dx
IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX-6 Dx card in
cases of high-temperature event, improper thermal coupling or heatsink removal.

19

Hardware Installation

Installation and initialization of ConnectX-6 Dx adapter cards require attention to the mechanical
attributes, power specification, and precautions for electronic equipment.

Safety Warnings

Safety warnings are provided here in the English language. For safety warnings in other



languages, refer to the Adapter Installation Safety Instructions document available on

mellanox.com.

Please observe all safety warnings to avoidinjury and prevent damage to system components. Note
that not all warnings are relevant to all models.

Unable to render include or excerpt-include. Could not retrieve page.

Installation Procedure Overview

The installation procedure of ConnectX-6 Dx adapter cards involves the following steps:

Step Procedure Direct Link

1 Check the system’s hardware and software

requirements.

2 Pay attention to the airflow consideration within the

host system

3 Follow the safety precautions Refer to Safety Precautions

4 Unpack the package Refer to Unpack the package

5 Follow the pre-installation checklist Refer to Pre-Installation Checklist

6 (Optional) Replace the full-height mounting bracket

with the supplied short bracket

7 Install the ConnectX-6 Dx and PCIe Auxiliary

connection cards in the system

8 Connect cables or modules to the card Refer to Cables and Modules

9 Identify ConnectX-6 Dx in the system Refer to Identifying Your Card

Refer to System Requirements

Refer to Airflow Requirements

Refer to Bracket Replacement

Instructions

Refer to Installation Instructions

20

System Requirements

Hardware Requirements

Unless otherwise specified, Mellanox products are designed to work in an environmentally



controlled data center with low levels of gaseous and dust (particulate) contamination.

The operation environment should meet severity level G1 as per ISA 71.04 for gaseous
contamination and ISO 14644-1 class 8 for cleanliness level.

A system with a PCI Express slot of corresponding bus width is required for installing the card.

For proper operation and performance, please make sure to use a PCIe slot with a



corresponding bus width and that can supply sufficient power to your card. Refer to the

Specifications section of the manual for more power requirements.

Airflow Requirements

ConnectX-6 Dx adapter cards are offered with two airflow patterns: from the heatsink to the network
ports.

Please refer to the Specificationssection for airflow numbers for each specific card model.

All cards in the system should be planned with the same airflow direction.



Software Requirements

•

See Operating Systems/Distributionssection under the Introduction section.

•

Software Stacks - Mellanox OpenFabric software package MLNX_OFED for Linux, WinOF-2 for
Windows, and VMware. See the Driver Installationsection.

Safety Precautions

The adapter is being installed in a system that operates with voltages that can be lethal. Before
opening the case of the system, observe the following precautions to avoid injury and prevent damage
to system components.

•

Remove any metallic objects from your hands and wrists.

•

Make sure to use only insulated tools.

•

Verify that the system is powered off and is unplugged.

•

It is strongly recommended to use an ESD strap or other antistatic devices.

21

Unpacking the Package

Check against the package contents list that all the parts have been sent.Check the parts for visible

damage that may have occurred during shipping. Please note that the cards must be placed on an

antistatic surface.

Category Qty. Item

Cards 1 ConnectX-6 Dx adapter card

Accessories 1 Adapter card short bracket

1 Adapter card tall bracket (shipped assembled on

the card)

Please note that if the card is removed hastily from the antistatic bag, the plastic ziplock may



harm the EMI fingers on the networking connector. Carefully remove the card from the
antistatic bag to avoid damaging the EMI fingers.

Pre-Installation Checklist

1.

Verify that your system meets the hardware and software requirements stated above.

2.

Shut down your system if active.

Turn off the power to the system, and disconnect the power cord. Refer to the system
documentation for instructions. Before you install the ConnectX-6 Dx card, make sure that the
system is disconnected from power.

3.

(Optional) Check the mounting bracket on the card.

If required for your system, replace the full-height mounting bracket that is shipped mounted on
the card with the supplied low-profile bracket. Refer toBracket Replacement Instructions.

22

Bracket Replacement Instructions

The ConnectX-6 Dx card is usually shipped with an assembled high-profile bracket. If this form factor

is suitable for your requirements, you can skip the remainder of this section and move toInstallation

Instructions. If you need to replace the high-profile bracket with the short bracket that is included in

the shipping box, please follow the instructions in this section.

Due to risk of damaging the EMI gasket, it is not recommended to replace the bracket more



than three times.

To replace the bracket you will need the following parts:

•

The new brackets of the proper height

•

The 2 screws saved from the removal of the bracket

Removing the Existing Bracket

1.

Using a torque driver, remove the two screws holding the bracket in place.

2.

Separate the bracket from the ConnectX-6 Dx card.

Be careful not to put stress on the LEDs on the adapter card.



3.

Save the two screws.

Installing the New Bracket

1.

Place the bracket onto the card until the screw holes line up.

Do not force the bracket onto the adapter card.



2.

Screw on the bracket using the screws saved from the bracket removal procedure above.

Use a torque driver to apply up to 2 lbs-in torque on the screws.



Installation Instructions

This section provides detailed instructions on how to install your adapter card in a system.

Please note that the following figures are for illustration purposes only.



Connect the adapter Card in an available PCI Express x16 slot in the chassis.

Step 1:Locate an available PCI Express x16 slot and insert the adapter card to the chassis.