NVIDIA Ubuntu 20.04 Linux Inbox Driver User Manual
Ubuntu 20.04 Linux Inbox Driver User
Manual
20.04
Ubuntu 20.04 Linux Inbox Driver User Manual | ii
Document History
Version
Date
Description of Change
Ubuntu 20.04
November 2020
Initial release of this document
Ubuntu 20.04 Linux Inbox Driver User Manual | iii
Table of Contents
1 Firmware Burning .............................................................................................. 4
2 Port Type Management ....................................................................................... 5
2.1 Port Type Management/VPI Cards Configuration ........................................................... 5
3 Modules Loading and Unloading ........................................................................ 9
4 Important Packages and Their Installation ...................................................... 10
5 SR-IOV Configuration ........................................................................................ 10
5.1 Setting up SR-IOV in ConnectX Adapters ...................................................................... 10
Default RoCE Mode Setting for RDMA_CM Application ........................................... 12
Ubuntu 20.04 Linux Inbox Driver User Manual | 4
1 Firmware Burning
1.
lspci | grep Mellanox
Example:
04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family
[ConnectX-4]
04:00.1 InfiniBand controller: Mellanox Technologies MT27700 Family
[ConnectX-4]
07:00.0 Ethernet controller: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx]
07:00.1 Ethernet controller: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx]
0a:00.0 Network controller: Mellanox Technologies MT27520 Family
[ConnectX-3 Pro]
21:00.0 InfiniBand controller: Mellanox Technologies MT27600 [Connect-IB]
24:00.0 Ethernet controller: Mellanox Technologies MT28800 Family
[ConnectX-5 Ex]
24:00.1 InfiniBand controller: Mellanox Technologies MT28800 Family
[ConnectX-5 Ex]
2.
# mstflint -d 81:00.0 q
Image type: FS4
FW Version: 16.26.4012
FW Release Date: 10.12.2019
Product Version: 16.26.4012
Rom Info: type=UEFI version=14.19.17 cpu=AMD64
type=PXE version=3.5.805 cpu=AMD64
Description: UID GuidsNumber
Base GUID: ec0d9a0300d42de4 8
Base MAC: ec0d9ad42de4 8
Image VSD: N/A
Device VSD: N/A
PSID: MT_0000000009
Security Attributes: N/A
3. Download the firmware BIN file from the Mellanox website that matches your card's PSID:
www.mellanox.com → Support →Support → Firmware Download
4. Burn the firmware.
# mstflint -d <lspci-device-id> -i <image-file> b
5. Reboot your machine after the firmware burning is completed.
6. Validate new firmware burned successfully:
# ethtool -i ens3
driver: mlx5_core
Ubuntu 20.04 Linux Inbox Driver User Manual | 5
version: 5.0-0
firmware-version: 16.26.4012 (MT_0000000009)
expansion-rom-version:
bus-info: 0000:24:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
2 Port Type Management
2.1 Port Type Management/VPI Cards
Configuration
ConnectX®-3/ConnectX®-3 Pro/ConnectX®-4 ports can be individually configured to work as
InfiniBand or Ethernet ports. By default, both ConnectX®-5 VPI ports are initialized as
InfiniBand ports. If you wish to change the port type use the mstconfig after the driver is
loaded.
1. Install mstflint tools.
apt install mstflint
2. Check the PCI address.
lspci | grep Mellanox
Example:
24:00.0 Ethernet controller: Mellanox Technologies MT28800 Family
[ConnectX-5 Ex]
3. Use mstconfig to change the link type as desired IB -- for InfiniBand, ETH -- for
Ethernet.
mstconfig –d <device pci> s LINK_TYPE_P1/2=<ETH|IB|VPI>
Example:
# mstconfig -d 00:06.0 s LINK_TYPE_P1=ETH
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-EDA_Ax
Description: ConnectX-5 Ex VPI adapter card; EDR IB (100Gb/s)
and 100GbE; dual-port QSFP28; PCIe4.0 x16; tall bracket; ROHS R6
Loading...