Cavium QL41112HFCU-DE, QL41164HMRJ-DE, QL41162HMRJ-DE, QL41164HMCU-DE, QL41262HFCU-DE User Manual
...
User’s Guide
Converged Network Adapters
41xxx Series
AH0054602-00 J
Third party information brought to
you courtesy of Dell EMC.
User’s Guide—Converged Network Adapters
41xxx Series
Document Revision History
Revision A, April 28, 2017
Revision B, August 24, 2017
Revision C, October 1, 2017
Revision D, January 24, 2018
Revision E, March 15, 2018
Revision F, April 19, 2018
Revision G, May 22, 2018
Revision H, August 23, 2018
Rev
ision J, January 24, 2019
Changes Sections Affected
In Table 2-2, updated the OS requirements:
Removed support for Windows Nano. Added
support for 2008 SP2 R2 SP2 and Windows
2019.
Removed support for RHEL 6.8, 7.2, 7.3, 7.4.
Added support for RHEL 7.5 and 7.6.
Removed support for SLES 11 SP4 and
SLES 12 SP2. Added support for SLES 15.
Removed support for ESXi 6.0 U3 and later for
25G adapters. Added support for vSphere ESXi
6.5 U2 and vSphere ESXi 6.7 U1.
Following Step 4, added a note: “The maximum
number of supported qedentv Ethernet interfaces
on an ESXi host is 32, because the vmkernel
allows only 32 interfaces to register for management callback.”
Added a note describing the driver limitation as it
applies to the iSCSI vmnic.
Moved iSCSI boot from SAN information from the
iSCSI Configuration chapter to a new chapter.
Added and corrected iSCSI BFS information.
“System Requirements” on page 4
“Installing VMware Drivers” on page 29
“iSCSI Support” on page 34
Chapter 6 Boot from SAN Configuration
Moved FCoE boot from SAN information from the
FCoE Configuration chapter to a new chapter.
Added and corrected FCoE BFS information.
Chapter 6 Boot from SAN Configuration
ii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Updated Ta bl e 7 -1 to reflect the current OS
support for RoCE v1, RoCE v2, iWARP, iSER, and
OFED.
In Table 7-2, replaced the RDMA Mode property
with the NetworkDirect Technology property.
In Step 1, updated the supported kernels for
RoCE v2 for Linux.
In Step 5, updated the guest OS for VM for
PVRDMA.
Updated the following iWARP procedures:
In Step 1 c of the To enable iWARP on the
Windows host and verify RDMA procedure,
changed the parameter to NetworkDirect
Tec hnology (was RDMA Mode).
In Step 4 b of the To verify iWARP traffic
procedure, changed the command to netstat
-xan (was net -xan).
In the To configure the NFS server procedure:
Added a new Step 1 that describes how to
create an nfs-server directory and grant
permission.
In Step 2, added rw, to the command.
Replaced Step 4 for loading the service.
“Supported Operating Systems and OFED” on
page 131
“Configuring RoCE on the Adapter for Windows
Server” on page 135
“RoCE v2 Configuration for Linux” on page 148
“Configuring a Paravirtual RDMA Device
(PVRDMA)” on page 157
“Configuring iWARP on Windows” on page 167
“Configuring NFS-RDMA” on page 175
In the To configure the NFS client procedure:
Added a new Step 1 that describes how to
create an nfs-client directory and grant
permission.
In Step 3
e
V
Updated the list of OSs for configuring iSER in
inbox OFED.
Following Step 12, added a fourth bullet to the
note: “Switch dependent teaming (IEEE 802.3ad
LACP and Generic/Static Link Aggregation (Trunking) cannot use a switch independent partitioned
virtual adapter. IEEE standards require Switch
Dependent Teaming (IEEE 802.3ad LACP and
Generic/Static Link Aggregation (Trunking)) mode
to work per the entire port instead of just the MAC
address (fraction of a port) granularity.”
, updated the command used for NFS
rsion 4.
“Configuring NFS-RDMA” on page 175
“Before You Begin” on page 179
“Configuring Microsoft Initiator to Use Cavium’s
iSCSI Offload” on page 192
iii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Added support for Windows Server 2019 to the
section.
In the To create a Hyper-V virtual switch with an
RDMA NIC procedure:
Changed the section title to “...RDMA NIC” (was
“...RDMA Virtual NIC”.
Changed procedure title (was To create a
Hyper-V virtual switch with an RDMA virtual
NIC),
Added a new Step 1: “On all physical inter-
faces, set the value of the NetworkDirect
Functionality parameter to Enabled.”
In the To add host virtual NICs procedure:
In Step 2, corrected the last digit of the com-
mand to “S” (was “5”).
Changed the note’s first bullet, second sen-
tence to: “The same VLAN ID must be assigned
to ports on the switch” (was: “The same VLAN
ID must be assigned to all the interfaces, and
on the switch”).
Deleted the obsolete section, Default and
Maximum VMMQ Virtual NIC.
“Windows Server 2012 R2, 2016, and 2019 iSCSI
Boot Installation” on page 199
“Creating a Hyper-V Virtual Switch with an RDMA
NIC” on page 239
“Adding a vLAN ID to Host Virtual NIC” on
page 240
“Configuring VMMQ” on page 254
In Step 4, deleted the note and example. “Creating a VM and Enabling VMMQ on VMNet-
workadapters in the VM” on page 257
In the first bullet, deleted the statement, “The
Management OS VNIC has four VMMQs.”
Added a note referencing the new parameter
added for Windows Server 2016 and Windows
Server 2019 to configure the maximum quantity of
queue pairs on a virtual port.
Deleted the obsolete “Deploying and Managing a
Nano Server” section.
Added a new chapter describing the features of
Windows Server 2019.
In Table B-1, added support for the Dell 10/25G
dual rate transceiver.
Added terms and definitions. Glossary
“Enabling and Disabling VMMQ on a Management
NIC” on page 258
“Monitoring Traffic Statistics” on page 258
Chapter 14 Windows Server 2016
Chapter 15 Windows Server 2019
“Tested Cables and Optical Modules” on page 280
iv AH0054602-00 J
Table of Contents
Preface
Supported Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
What Is in This Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
Laser Safety—FDA Notice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
Agency Certification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
EMI and EMC Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
KCC: Class A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
VCCI: Class A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
Product Safety Compliance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
1 Product Overview
Functional Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Adapter Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Physical Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Standards Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Hardware Installation
System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Preinstallation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Installing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3 Driver Installation
Installing Linux Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Installing the Linux Drivers Without RDMA . . . . . . . . . . . . . . . . . . . . . 10
Removing the Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Installing Linux Drivers Using the src RPM Package . . . . . . . . . 12
Installing Linux Drivers Using the kmp/kmod RPM Package . . . 13
Installing Linux Drivers Using the TAR File. . . . . . . . . . . . . . . . . 13
v AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Installing the Linux Drivers with RDMA . . . . . . . . . . . . . . . . . . . . . . . . 14
Linux Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Linux Driver Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Importing a Public Key for Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . 16
Installing Windows Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Installing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Running the DUP in the GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
DUP Installation Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
DUP Installation Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Removing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Managing Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Setting Power Management Options. . . . . . . . . . . . . . . . . . . . . . . . . . 27
Installing VMware Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
VMware Drivers and Driver Packages. . . . . . . . . . . . . . . . . . . . . . . . . 28
Installing VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . 31
VMware Driver Parameter Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Removing the VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
FCoE Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
iSCSI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4 Upgrading the Firmware
Running the DUP by Double-Clicking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Running the DUP from a Command Line. . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Running the DUP Using the .bin File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5 Adapter Preboot Configuration
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Displaying Firmware Image Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Configuring Device-level Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Configuring NIC Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Configuring Data Center Bridging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Configuring FCoE Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Configuring Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Partitioning for VMware ESXi 6.0 and ESXi 6.5 . . . . . . . . . . . . . . . . . 62
vi AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
6 Boot from SAN Configuration
iSCSI Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
iSCSI Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 65
iSCSI Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Setting the BIOS Boot Mode to UEFI . . . . . . . . . . . . . . . . . . . . . 66
Enabling NPAR and the iSCSI HBA . . . . . . . . . . . . . . . . . . . . . . 68
Selecting the iSCSI UEFI Boot Protocol. . . . . . . . . . . . . . . . . . . 69
Configuring the Storage Target. . . . . . . . . . . . . . . . . . . . . . . . . . 70
Configuring iSCSI Boot Parameters . . . . . . . . . . . . . . . . . . . . . . 70
Configuring BIOS Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Configuring iSCSI Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . 73
Configuring the DHCP Server to Support iSCSI Boot . . . . . . . . 85
Configuring iSCSI Boot from SAN on Windows . . . . . . . . . . . . . . . . . 89
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Selecting the Preferred iSCSI Boot Mode . . . . . . . . . . . . . . . . . 90
Configuring iSCSI General Parameters . . . . . . . . . . . . . . . . . . . 90
Configuring the iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Configuring the iSCSI Targets . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Detecting the iSCSI LUN and Injecting the QLogic Drivers . . . . 92
Configuring iSCSI Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 94
Configuring iSCSI Boot from SAN for RHEL 7.5 and Later . . . . 95
Configuring iSCSI Boot from SAN for SLES 12 SP3
and Later . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Configuring iSCSI Boot from SAN for Other Linux
Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Configuring iSCSI Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 109
Setting the UEFI Main Configuration . . . . . . . . . . . . . . . . . . . . . 109
Configuring the System BIOS for iSCSI Boot (L2) . . . . . . . . . . . 111
Mapping the CD or DVD for OS Installation . . . . . . . . . . . . . . . . 113
FCoE Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
FCoE Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 115
FCoE Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Specifying the BIOS Boot Protocol. . . . . . . . . . . . . . . . . . . . . . . 116
Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 116
Configuring FCoE Boot from SAN on Windows . . . . . . . . . . . . . . . . . 121
Windows Server 2012 R2 and 2016 FCoE Boot Installation . . . 121
Configuring FCoE on Windows . . . . . . . . . . . . . . . . . . . . . . . . .
FCoE Crash Dump
Injecting (Slipstreaming) Adapter Drivers into Windows
Image Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
on Windows
122
. . . . . . . . . . . . . . . . . . . . . . . . . 122
vii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Configuring FCoE Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 124
Prerequisites for Linux FCoE Boot from SAN. . . . . . . . . . . . . . . 124
Configuring Linux FCoE Boot from SAN . . . . . . . . . . . . . . . . . . 125
Configuring FCoE Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 127
Injecting (Slipstreaming) ESXi Adapter Drivers into Image Files
Installing the Customized ESXi ISO . . . . . . . . . . . . . . . . . . . . . . 128
7 RoCE Configuration
Supported Operating Systems and OFED . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Planning for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Preparing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Preparing the Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Configuring the Cisco Nexus 6000 Ethernet Switch . . . . . . . . . . . . . . 134
Configuring the Dell Z9100 Ethernet Switch . . . . . . . . . . . . . . . . . . . . 135
Configuring RoCE on the Adapter for Windows Server . . . . . . . . . . . . . . . . 135
Viewing RDMA Counters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Configuring RoCE on the Adapter for Linux . . . . . . . . . . . . . . . . . . . . . . . . . 143
RoCE Configuration for RHEL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
RoCE Configuration for SLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Verifying the RoCE Configuration on Linux . . . . . . . . . . . . . . . . . . . . . 145
vLAN Interfaces and GID Index Values. . . . . . . . . . . . . . . . . . . . . . . . 147
RoCE v2 Configuration for Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Identifying the RoCE v2 GID Index or Address . . . . . . . . . . . . . 148
Verifying the RoCE v1 or RoCE v2 GID Index and Address
from sys and class Parameters . . . . . . . . . . . . . . . . . . . . . . . . 149
Verifying the RoCE v1 or RoCE v2 Function Through
perftest Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Configuring RoCE on the Adapter for VMware ESX . . . . . . . . . . . . . . . . . . 154
Configuring RDMA Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Configuring MTU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
RoCE Mode and Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Configuring a Paravirtual RDMA Device (PVRDMA). . . . . . . . . . . . . . 157
Configuring DCQCN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
DCQCN Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
DCQCN Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
DCB-related Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Global Settings on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Setting vLAN Priority on RDMA Traffic . . . . . . . . . . . . . . . . . . . . 162
Setting ECN on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Setting DSCP on RDMA Traffic . . . . . . . . . . . . . . . . . . . . . . . . . 162
. 127
viii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Configuring DCQCN (continued)
Configuring DSCP-PFC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Enabling DCQCN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Configuring CNP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
MAC Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Script Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
8 iWARP Configuration
Preparing the Adapter for iWARP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Configuring iWARP on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Configuring iWARP on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Installing the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Configuring iWARP and RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Detecting the Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Supported iWARP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Running Perftest for iWARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Configuring NFS-RDMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
iWARP RDMA-Core Support on SLES 12 SP3 and OFED 4.8x . . . . . 176
9 iSER Configuration
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Configuring iSER for RHEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Configuring iSER for SLES 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Using iSER with iWARP on RHEL and SLES. . . . . . . . . . . . . . . . . . . . . . . . 184
Optimizing Linux Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Configuring CPUs to Maximum Performance Mode . . . . . . . . . . . . . . 186
Configuring Kernel sysctl Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Configuring IRQ Affinity Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring Block Device Staging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring iSER on ESXi 6.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Before You Begin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring iSER for ESXi 6.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
10 iSCSI Configuration
iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
iSCSI Offload in Windows Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Installing Cavium QLogic Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Installing the Microsoft iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . 192
Configuring Microsoft Initiator to Use Cavium’s iSCSI Offload . . . . . . 192
ix AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
iSCSI Offload in Windows Server (continued)
iSCSI Offload FAQs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Windows Server 2012 R2, 2016, and 2019 iSCSI Boot Installation . . 199
iSCSI Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
iSCSI Offload in Linux Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Differences from bnx2i. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Configuring qedi.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Verifying iSCSI Interfaces in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
11 FCoE Configuration
Configuring Linux FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Differences Between qedf and bnx2fc. . . . . . . . . . . . . . . . . . . . . . . . . 205
Configuring qedf.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Verifying FCoE Devices in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
12 SR-IOV Configuration
Configuring SR-IOV on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Configuring SR-IOV on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Enabling IOMMU for SR-IOV in UEFI-based Linux OS Installations
Configuring SR-IOV on VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
. . . 220
13 NVMe-oF Configuration with RDMA
Installing Device Drivers on Both Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Configuring the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Configuring the Initiator Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Preconditioning the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Testing the NVMe-oF Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Optimizing Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
.IRQ Affinity (multi_rss-affin.sh). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
CPU Frequency (cpufreq.sh). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
14 Windows Server 2016
Configuring RoCE Interfaces with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . 238
Creating a Hyper-V Virtual Switch with an RDMA NIC . . . . . . . . . . . . 239
Adding a vLAN ID to Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . 240
Verifying If RoCE is Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Adding Host Virtual NICs (Virtual Ports) . . . . . . . . . . . . . . . . . . . . . . . 242
Mapping the SMB Drive and Running RoCE Traffic . . . . . . . . . . . . . . 242
RoCE over Switch Embedded Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Creating a Hyper-V Virtual Switch with SET and RDMA
Virtual NICs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
x AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
RoCE over Switch Embedded Teaming (continued)
Enabling RDMA on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Assigning a vLAN ID on SET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Running RDMA Traffic on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Configuring QoS for RoCE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Configuring QoS by Disabling DCBX on the Adapter . . . . . . . . . . . . . 246
Configuring QoS by Enabling DCBX on the Adapter. . . . . . . . . . . . . . 250
Configuring VMMQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
Enabling VMMQ on the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
Creating a Virtual Machine Switch with or Without SR-IOV . . . . . . . . 255
Enabling VMMQ on the Virtual Machine Switch . . . . . . . . . . . . . . . . . 256
Getting the Virtual Machine Switch Capability. . . . . . . . . . . . . . . . . . . 257
Creating a VM and Enabling VMMQ on VMNetworkadapters
in the VM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Enabling and Disabling VMMQ on a Management NIC . . . . . . . . . . . 258
Monitoring Traffic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Configuring VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Enabling VXLAN Offload on the Adapter. . . . . . . . . . . . . . . . . . . . . . . 259
Deploying a Software Defined Network. . . . . . . . . . . . . . . . . . . . . . . . 259
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Configuring the Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Deploying a Hyper-Converged System . . . . . . . . . . . . . . . . . . . . . . . . 261
Deploying the Operating System . . . . . . . . . . . . . . . . . . . . . . . . 261
Configuring the Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . 263
15 Windows Server 2019
RSSv2 for Hyper-V. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
RSSv2 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Known Event Log Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Windows Server 2019 Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
VMMQ Is Enabled by Default . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Inbox Driver Network Direct (RDMA) Is Disabled by Default. . . . . . . . 268
New Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Max Queue Pairs (L2) Per VPort. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Network Direct Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Virtualization Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
VMQ and VMMQ Default Accelerations . . . . . . . . . . . . . . . . . . . . . . . 271
Single VPort Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
xi AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
16 Troubleshooting
Troubleshooting Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Verifying that Current Drivers Are Loaded . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Verifying Drivers in Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Verifying Drivers in Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Verifying Drivers in VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Testing Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Testing Network Connectivity for Windows . . . . . . . . . . . . . . . . . . . . . 275
Testing Network Connectivity for Linux . . . . . . . . . . . . . . . . . . . . . . . . 276
Microsoft Virtualization with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Linux-specific Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Miscellaneous Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Collecting Debug Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
AAdapter LEDS
B Cables and Optical Modules
Supported Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
Tested Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
C Dell Z9100 Switch Configuration
D Feature Constraints
Glossary
xii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
List of Figures
Figure Page
3-1 Dell Update Package Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3-2 QLogic InstallShield Wizard: Welcome Window . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3-3 QLogic InstallShield Wizard: License Agreement Window. . . . . . . . . . . . . . . . . . . . 20
3-4 InstallShield Wizard: Setup Type Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3-5 InstallShield Wizard: Custom Setup Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3-6 InstallShield Wizard: Ready to Install the Program Window . . . . . . . . . . . . . . . . . . 22
3-7 InstallShield Wizard: Completed Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3-8 Dell Update Package Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3-9 Setting Advanced Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3-10 Power Management Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4-1 Dell Update Package: Splash Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4-2 Dell Update Package: Loading New Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4-3 Dell Update Package: Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4-4 Dell Update Package: Finish Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4-5 DUP Command Line Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5-1 System Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5-2 System Setup: Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5-3 Main Configuration Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5-4 Main Configuration Page, Setting Partitioning Mode to NPAR. . . . . . . . . . . . . . . . . 42
5-5 Firmware Image Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5-6 Device Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5-7 NIC Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5-8 System Setup: Data Center Bridging (DCB) Settings . . . . . . . . . . . . . . . . . . . . . . . 50
5-9 FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5-10 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5-11 iSCSI General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5-12 iSCSI Initiator Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5-13 iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5-14 iSCSI Second Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5-15 NIC Partitioning Configuration, Global Bandwidth Allocation. . . . . . . . . . . . . . . . . . 57
5-16 Global Bandwidth Allocation Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5-17 Partition 1 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5-18 Partition 2 Configuration: FCoE Offload. . . . . . . . . . .
. . . . .
5-19 Partition 3 Configuration: iSCSI Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5-20 Partition 4 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6-1 System Setup: Boot Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6-2 System Setup: Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6-3 System Setup: NIC Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6-4 System Setup: Boot Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6-5 Selecting NIC Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6-6 System Setup: NIC Configuration, Boot Protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6-7 System Setup: iSCSI Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6-8 System Setup: Selecting General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
. . . . . . . . . . . . . . . . . . 60
xiii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
6-9 System Setup: iSCSI General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6-10 System Setup: Selecting iSCSI Initiator Parameters . . . . . . . . . . . . . . . . . . . . . . . . 77
6-11 System Setup: iSCSI Initiator Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6-12 System Setup: Selecting iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . 79
6-13 System Setup: iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6-14 System Setup: iSCSI Second Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6-15 System Setup: Saving iSCSI Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6-16 System Setup: iSCSI General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6-17 System Setup: iSCSI General Parameters, VLAN ID . . . . . . . . . . . . . . . . . . . . . . . 89
6-18 Detecting the iSCSI LUN Using UEFI Shell (Version 2). . . . . . . . . . . . . . . . . . . . . . 93
6-19 Windows Setup: Selecting Installation Destination . . . . . . . . . . . . . . . . . . . . . . . . . 93
6-20 Windows Setup: Selecting Driver to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6-21 Integrated NIC: Device Level Configuration for VMware . . . . . . . . . . . . . . . . . . . . . 109
6-22 Integrated NIC: Partition 2 Configuration for VMware . . . . . . . . . . . . . . . . . . . . . . . 110
6-23 Integrated NIC: System BIOS, Boot Settings for VMware . . . . . . . . . . . . . . . . . . . . 111
6-24 Integrated NIC: System BIOS, Connection 1 Settings for VMware . . . . . . . . . . . . . 112
6-25 Integrated NIC: System BIOS, Connection 1 Settings (Target) for VMware . . . . . . 112
6-26 VMware iSCSI BFS: Selecting a Disk to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6-27 VMware iSCSI Boot from SAN Successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6-28 System Setup: Selecting Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6-29 System Setup: Device Settings, Port Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6-30 System Setup: NIC Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
6-31 System Setup: FCoE Mode Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6-32 System Setup: FCoE General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6-33 System Setup: FCoE General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6-34 Selecting the Driver Disk Image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6-35 Selecting the Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6-36 ESXi-Customizer Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6-37 Select a VMware Disk to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6-38 VMware Generic USB Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7-1 Configuring RoCE Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7-2 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . .
. . . . .
. . . . . . . . . . . . . . . . . . 138
7-3 Performance Monitor: Cavium FastLinQ Counters . . . . . . . . . . . . . . . . . . . . . . . . . 140
7-4 Switch Settings, Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7-5 Switch Settings, Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7-6 Configuring RDMA_CM Applications: Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7-7 Configuring RDMA_CM Applications: Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7-8 Configuring a New Distributed Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
7-9 Assigning a vmknic for PVRDMA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7-10 Setting the Firewall Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8-1 Windows PowerShell Command: Get-NetAdapterRdma . . . . . . . . . . . . . . . . . . . . . 168
8-2 Windows PowerShell Command: Get-NetOffloadGlobalSetting . . . . . . . . . . . . . . . 168
8-3 Perfmon: Add Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
8-4 Perfmon: Verifying iWARP Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
9-1 RDMA Ping Successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
xiv AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
9-2 iSER Portal Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9-3 Iface Transport Confirmed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9-4 Checking for New iSCSI Device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
9-5 LIO Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
10-1 iSCSI Initiator Properties, Configuration Page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
10-2 iSCSI Initiator Node Name Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
10-3 iSCSI Initiator—Discover Target Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
10-4 Target Portal IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
10-5 Selecting the Initiator IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
10-6 Connecting to the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10-7 Connect To Target Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
12-1 System Setup for SR-IOV: Integrated Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
12-2 System Setup for SR-IOV: Device Level Configuration . . . . . . . . . . . . . . . . . . . . . . 209
12-3 Adapter Properties, Advanced: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . 210
12-4 Virtual Switch Manager: Enabling SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
12-5 Settings for VM: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
12-6 Device Manager: VM with QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
12-7 Windows PowerShell Command: Get-NetadapterSriovVf . . . . . . . . . . . . . . . . . . . . 214
12-8 System Setup: Processor Settings for SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
12-9 System Setup for SR-IOV: Integrated Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
12-10 Editing the grub.conf File for SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
12-11 Command Output for sriov_numvfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
12-12 Command Output for ip link show Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
12-13 RHEL68 Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
12-14 Add New Virtual Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
12-15 VMware Host Edit Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
13-1 NVMe-oF Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
13-2 Subsystem NQN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
13-3 Confirm NVMe-oF Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
13-4 FIO Utility Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
14-1 Enabling RDMA in Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
14-2 Hyper-V Virtual Ethernet Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
14-3 Windows PowerShell Command: Get-VMNetworkAdapter . . . . . . . . . . . . . . . . . . . 241
14-4 Windows PowerShell Command: Get-NetAdap
terRdma
. . . . . . . . . . . . . . . . . . . . . 241
14-5 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
14-6 Performance Monitor Shows RoCE Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
14-7 Windows PowerShell Command: New-VMSwitch . . . . . . . . . . . . . . . . . . . . . . . . . . 244
14-8 Windows PowerShell Command: Get-NetAdapter. . . . . . . . . . . . . . . . . . . . . . . . . . 245
14-9 Advanced Properties: Enable QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
14-10 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
14-11 Advanced Properties: Enabling QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
14-12 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
14-13 Advanced Properties: Enabling Virtual Switch RSS. . . . . . . . . . . . . . . . . . . . . . . . . 255
14-14 Virtual Switch Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
14-15 Windows PowerShell Command: Get-VMSwitch. . . . . . . . . . . . . . . . . . . . . . . . . . . 257
xv AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
14-16 Advanced Properties: Enabling VXLAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
14-17 Example Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
15-1 RSSv2 Event Log Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
xvi AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
List of Tables
Table Page
2-1 Host Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2-2 Minimum Host Operating System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3-1 Cavium QLogic 41xxx Series Adapters Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . 9
3-2 qede Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3-3 Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3-4 VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3-5 ESXi Driver Packages by Release. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3-6 VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3-7 VMware Driver Parameter Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5-1 Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6-1 iSCSI Out-of-Box and Inbox Boot from SAN Support . . . . . . . . . . . . . . . . . . . . . . . 65
6-2 iSCSI General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6-3 DHCP Option 17 Parameter Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6-4 DHCP Option 43 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6-5 DHCP Option 17 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6-6 FCoE Out-of-Box and Inbox Boot from SAN Support . . . . . . . . . . . . . . . . . . . . . . . 115
7-1 OS Support for RoCE v1, RoCE v2, iWARP, iSER, and OFED . . . . . . . . . . . . . . . . 131
7-2 Advanced Properties for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7-3 Cavium FastLinQ RDMA Error Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7-4 DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
13-1 Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
15-1 Windows 2019 Virtualization Resources for Dell 41xxx Series Adapters. . . . . . . . . 270
15-2 Windows 2019 VMQ and VMMQ Accelerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
16-1 Collecting Debug Data Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
A-1 Adapter Port Link and Activity LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
B-1 Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
B-2 Switches Tested for Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
xvii AH0054602-00 J
Preface
This preface lists the supported products, specifies the intended audience,
explains the typographic conventions used in this guide, and describes legal
notices.
Supported Products
This user’s guide describes the following Cavium® products:
QL41112HFCU-DE 10Gb Converged Network Adapter, full-height bracket
QL41112HLCU-DE 10Gb Converged Network Adapter, low-profile bracket
QL41162HFRJ-DE 10Gb Converged Network Adapter, full-height bracket
QL41162HLRJ-DE 10Gb Converged Network Adapter, low-profile bracket
QL41162HMRJ-DE 10Gb Converged Network Adapter
QL41164HMCU-DE 10Gb Converged Network Adapter
QL41164HMRJ-DE 10Gb Converged Network Adapter
QL41262HFCU-DE 10/25Gb Converged Network Adapter, full-height
bracket
QL41262HLCU-DE 10/25Gb Converged Network Adapter, low-profile
bracket
QL41262HMCU-DE 10/25Gb Converged Network
QL41264HMCU-DE 10/25Gb Converged Network Adapter
Intended Audience
This guide is intended for system administrators and other technical staff
members responsible for configuring and managing adapters installed on Dell
®
PowerEdge
servers in Windows®, Linux®, or VMware® environments.
®
xviii AH0054602-00 J
Preface
What Is in This Guide
What Is in This Guide
Following this preface, the remainder of this guide is organized into the following
chapters and appendices:
Chapter 1 Product Overview provides a product functional description, a list
of features, and the adapter specifications.
Chapter 2 Hardware Installation describes how to install the adapter,
including the list of system requirements and a preinstallation checklist.
Chapter 3 Driver Installation describes the installation of the adapter drivers
on Windows, Linux, and VMware.
Chapter 4 Upgrading the Firmware describes how to use the Dell Update
Package (DUP) to upgrade the adapter firmware.
Chapter 5 Adapter Preboot Configuration describes the preboot adapter
configuration tasks using the Human Infrastructure Interface (HII)
application.
Chapter 6 Boot from SAN Configuration covers boot from SAN configuration
for both iSCSI and FCoE.
Chapter 7 RoCE Configuration describes how to configure the adapter, the
Ethernet switch, and the host to use RDMA over converged Ethernet
(RoCE).
Chapter 8 iWARP Configuration provides procedures for configuring Internet
wide area RDMA protocol (iWARP) on Windows, Linux, and VMware
ESXi 6.7 systems.
Chapter 9 iSER Configuration describes how to configure iSCSI Extensions
for RDMA (iSER) for Linux RHEL, SLES, Ubuntu, and ESXi 6.7.
Chapter 10 iSCSI Configuration describes iSCSI boot and iSCSI offload for
Windows and Linux.
Chapter 11 FCoE Configuration covers configuring Linux FCoE offload.
Chapter 12 SR-IOV Configuration provides procedures for configuring single
root input/output virtualization (SR-IOV) on Windows, Linux, and VMware
systems.
Chapter 13 NVMe-oF Configuration with RDMA demonstrates how to
configure NVMe-oF on a simple network for 41xxx Series Adapters.
Chapter 14 Windows Server 2016 describes the Windows Server 2016
features.
xix AH0054602-00 J
Preface
NOTE
CAUTION
CAUTION
!
!
WARNING
Documentation Conventions
Chapter 15 Windows Server 2019 describes the Windows Server 2019
features.
Chapter 16 Troubleshooting describes a variety of troubleshooting methods
and resources.
Appendix A Adapter LEDS lists the adapter LEDs and their significance.
Appendix B Cables and Optical Modules lists the cables, optical modules,
and switches that the 41xxx Series Adapters support.
Appendix C Dell Z9100 Switch Configuration describes how to configure the
Dell Z9100 switch port for 25Gbps.
Appendix D Feature Constraints provides information about feature
constraints implemented in the current release.
At the end of this guide is a glossary of terms.
Documentation Conventions
This guide uses the following documentation conventions:
provides additional information.
without an alert symbol indicates the presence of a hazard
that could cause damage to equipment or loss of data.
with an alert symbol indicates the presence of a hazard that
could cause minor or moderate injury.
indicates the presence of a hazard that could cause serious
injury or death.
Text in blue font indicates a hyperlink (jump) to a figure, table, or section in
this guide, and links to Web sites are shown in underlined blue
example:
Table 9-2 lists problems related to the user interface and remote agent.
See “Installation Checklist” on page 6.
For more information, visit www.marvell.com
Text in bold font indicates user interface elements such as a menu items,
buttons, check boxes, or column headings. For example:
.
. For
Click the Start button, point to Programs, point to Accessories, and
then click Command Prompt.
Under Notification Options, select the Warning Alarms check box.
xx AH0054602-00 J
Preface
NOTE
Documentation Conventions
Te x t in Courier font indicates a file name, directory path, or command line
text. For example:
To return to the root directory from anywhere in the file structure:
cd/ root and press ENTER.
Type
Issue the following command:
sh ./install.bin.
Key names and key strokes are indicated with UPPERCASE:
Press CTRL+P.
Press the UP ARROW key.
Text in italics indicates terms, emphasis, variables, or document titles. For
example:
What are shortcut keys?
To enter the date type mm/dd/yyyy (where mm is the month, dd is the
day, and yyyy is the year).
Topic titles between quotation marks identify related topics either within this
manual or in the online help, which is also referred to as the help system
throughout this document.
Command line interface (CLI) command syntax conventions include the
following:
Plain text indicates items that you must type as shown. For example:
qaucli -pr nic -ei
<> (angle brackets) indicate a variable whose value you must
specify. For example:
<serial_number>
For CLI commands only, variable names are always indicated
using angle brackets instead of italics.
[] (square brackets) indicate an optional parameter. For example:
[<file_name>] means specify a file name, or omit it to select
the default file name.
xxi AH0054602-00 J
Preface
CLASS I LASER
Legal Notices
| (vertical bar) indicates mutually exclusive options; select one option
only. For example:
on|off
1|2|3|4
... (ellipsis) indicates that the preceding item may be repeated. For
example:
x... means one or more instances of x.
[x...] means zero or more instances of x.
⋮ (vertical ellipsis) within command example output indicate where
portions of repetitious output data have been intentionally omitted.
() (parentheses) and { } (braces) are used to avoid logical
ambiguity. For example:
a|b c is ambiguous
{(a|b) c} means a or b, followed by c
{a|(b c)}
means either a, or b c
Legal Notices
Legal notices covered in this section include laser safety (FDA notice), agency
certification, and product safety compliance.
Laser Safety—FDA Notice
This product complies with DHHS Rules 21CFR Chapter I, Subchapter J. This
product has been designed and manufactured according to IEC60825-1 on the
safety label of laser product.
Class 1
Laser Product
Appareil laser
de classe 1
Produkt der
Laser Klasse 1
Luokan 1 Laserlaite Varoitus—Luokan 1 lasersäteilyä, kun laite on auki
Caution—Class 1 laser radiation when open
Do not view directly with optical instruments
Attention—Radiation laser de classe 1
Ne pas regarder directement avec des instruments optiques
Vorsicht—Laserstrahlung der Klasse 1 bei geöffneter Abdeckung
Direktes Ansehen mit optischen Instrumenten vermeiden
Älä katso suoraan laitteeseen käyttämällä optisia instrumenttej
xxii AH0054602-00 J
Preface
Legal Notices
Agency Certification
The following sections summarize the EMC and EMI test specifications performed
on the 41xxx Series Adapters to comply with emission, immunity, and product
safety standards.
EMI and EMC Requirements
FCC Part 15 compliance: Class A
FCC compliance information statement: This device complies with Part 15 of
the FCC Rules. Operation is subject to the following two conditions: (1) this device
may not cause harmful interference, and (2) this device must accept any
interference received, including interference that may cause undesired operation.
ICES-003 Compliance: Class A
This Class A digital apparatus complies with Canadian ICES-003. Cet appareil
numériqué de la classe A est conformé à la norme NMB-003 du Canada.
CE Mark 2014/30/EU, 2014/35/EU EMC Directive Compliance:
EN55032:2012/ CISPR 32:2015 Class A
EN55024: 2010
EN61000-3-2: Harmonic Current Emission
EN61000-3-3: Voltage Fluctuation and Flicker
Immunity Standards
EN61000-4-2: ESD
EN61000-4-3: RF Electro Magnetic Field
EN61000-4-4: Fast Transient/Burst
EN61000-4-5: Fast Surge Common/ Differential
EN61000-4-6: RF Conducted Susceptibility
EN61000-4-8: Power Frequency Magnetic Field
EN61000-4-11: Voltage Dips and Interrupt
VCCI: 2015-04; Class A
AS/NZS; CISPR 32: 2015 Class A
CNS 13438: 2006 Class A
xxiii AH0054602-00 J
Preface
Legal Notices
KCC: Class A
Korea RRA Class A Certified
Product Name/Model: Converged Network Adapters and
Intelligent Ethernet Adapters
Certification holder: QLogic Corporation
Manufactured date: Refer to date code listed on product
Manufacturer/Country of origin: QLogic Corporation/USA
VCCI: Class A
A class equipment
(Business purpose
info/telecommunications
equipment)
As this equipment has undergone EMC registration for business purpose, the seller and/or the buyer is asked to beware
of this point and in case a wrongful sale or purchase has
been made, it is asked that a change to household use be
made.
Korean Language Format—Class A
This is a Class A product based on the standard of the Voluntary Control Council
for Interference (VCCI). If this equipment is used in a domestic environment, radio
interference may occur, in which case the user may be required to take corrective
actions.
Product Safety Compliance
UL, cUL product safety:
UL 60950-1 (2nd Edition) A1 + A2 2014-10-14
CSA C22.2 No.60950-1-07 (2nd Edition) A1 +A2 2014-10
Use only with listed ITE or equivalent.
Complies with 21 CFR 1040.10 and 1040.11, 2014/30/EU, 2014/35/EU.
xxiv AH0054602-00 J
Preface
Legal Notices
2006/95/EC low voltage directive:
TUV EN60950-1:2006+A11+A1+A12+A2 2nd Edition
TUV IEC 60950-1: 2005 2nd Edition Am1: 2009 + Am2: 2013 CB
CB Certified to IEC 60950-1 2nd Edition
xxv AH0054602-00 J
1 Product Overview
This chapter provides the following information for the 41xxx Series Adapters:
Functional Description
Features
“Adapter Specifications” on page 3
Functional Description
The Cavium FastLinQ 41000 Series Adapters include 10 and 25Gb Converged
Network Adapters and Intelligent Ethernet Adapters that are designed to perform
accelerated data networking for server systems. The 41000 Series Adapter
includes a 10/25Gb Ethernet MAC with full-duplex capability.
Using the operating system’s teaming feature, you can split your network into
virtual LANs (vLANs), as well as group multiple network adapters together into
teams to provide network load balancing and fault tolerance. For more information
about teaming, see your operating system documentation.
Features
The 41xxx Series Adapters provide the following features. Some features may not
be available on all adapters:
NIC partitioning (NPAR)
Single-chip solution:
10/25Gb MAC
SerDes interface for direct attach copper (DAC) transceiver connection
PCI Express
Zero copy capable hardware
®
(PCIe®) 3.0 x8
1 AH0054602-00 J
1–Product Overview
Features
Performance features:
TCP, IP, UDP checksum offloads
TCP segmentation offload (TSO)
Large segment offload (LSO)
Generic segment offload (GSO)
Large receive offload (LRO)
Receive segment coalescing (RSC)
Microsoft
®
dynamic virtual machine queue (VMQ), and Linux
Multiqueue
Adaptive interrupts:
Transmit/receive side scaling (TSS/RSS)
Stateless offloads for Network Virtualization using Generic Routing
Encapsulation (NVGRE) and virtual LAN (VXLAN) L2/L3 GRE
1
tunneled traffic
Manageability:
System management bus (SMB) controller
Advanced Configuration and Power Interface (ACPI) 1.1a compliant
(multiple power modes)
Network controller-sideband interface (NC-SI) support
Advanced network features:
Jumbo frames (up to 9,600 bytes). The OS and the link partner must
support jumbo frames.
Virtual LANs (vLANs)
Flow control (IEEE Std 802.3x)
Logical link control (IEEE Std 802.2)
High-speed on-chip reduced instruction set computer (RISC) processor
Integrated 96KB frame buffer memory (not applicable to all models)
1,024 classification filters (not applicable to all models)
Support for multicast addresses through 128-bit hashing hardware function
Serial flash NVRAM memory
PCI Power Management Interface (v1.1)
64-bit base address register (BAR) support
1
This feature requires OS or Hypervisor support to use the offloads.
2 AH0054602-00 J
1–Product Overview
Adapter Specifications
EM64T processor support
iSCSI and FCoE boot support
Adapter Specifications
The 41xxx Series Adapter specifications include the adapter’s physical
characteristics and standards-compliance references.
Physical Characteristics
The 41xxx Series Adapters are standard PCIe cards and ship with either a
full-height or a low-profile bracket for use in a standard PCIe slot.
Standards Specifications
Supported standards specifications include:
PCI Express Base Specification, rev. 3.1
PCI Express Card Electromechanical Specification, rev. 3.0
PCI Bus Power Management Interface Specification, rev. 1.2
IEEE Specifications:
802.1ad (QinQ)
802.1AX (Link Aggregation)
802.1p (Priority Encoding)
802.1q (VLAN)
802.3-2015 IEEE Standard for Ethernet (flow control)
802.3-2015 Clause 78 Energy Efficient Ethernet (EEE)
1588-2002 PTPv1 (Precision Time Protocol)
1588-2008 PTPv2
2
IPv4 (RFQ 791)
IPv6 (RFC 2460)
2
Hardware support limit of SR-IOV VFs varies. The limit may be lower in some OS environments;
refer to the appropriate section for your OS.
3 AH0054602-00 J
2 Hardware Installation
This chapter provides the following hardware installation information:
System Requirements
“Safety Precautions” on page 5
“Preinstallation Checklist” on page 6
“Installing the Adapter” on page 6
System Requirements
Before you install a Cavium 41xxx Series Adapter, verify that your system meets
the hardware and operating system requirements shown in Table 2-1 and
Table 2-2. For a complete list of supported operating systems, visit the Marvell
Web site.
Table 2-1. Host Hardware Requirements
Hardware Requirement
Architecture IA-32 or EMT64 that meets operating system requirements
PCIe Gen 2 x8 (2x10G NIC)
PCIe
Memory 8GB RAM (minimum)
Cables and Optical
Modules
PCIe Gen 3 x8 (2x25G NIC)
Full dual-port 25Gb bandwidth is supported on PCIe Gen 3 x8 or
faster slots.
The 41xxx Series Adapters have been tested for interoperability
with a variety of 1G, 10G, and 25G cables and optical modules.
See “Tested Cables and Optical Modules” on page 280.
4 AH0054602-00 J
2–Hardware Installation
NOTE
!
WARNING
Safety Precautions
Table 2-2. Minimum Host Operating System Requirements
Operating
System
Windows Server 2008 R2 SP1, 2008 SP2, 2012, 2012 R2, 2016, 2019
Linux
VMware
XenServer 7.0, 7.1
Table 2-2 denotes minimum host OS requirements. For a complete list of
supported operating systems, visit the Marvell Web site.
Safety Precautions
The adapter is being installed in a system that operates with voltages that
can be lethal. Before you open the case of your system, observe the
following precautions to protect yourself and to prevent damage to the
system components.
Remove any metallic objects or jewelry from your hands and wrists.
Make sure to use only insulated or nonconducting tools.
Requirement
®
RHEL
SLES
vSphere® ESXi 6.5 U2 and vSphere ESXi 6.7 U1
6.9, 6.10, 7.5, and 7.6
®
12 SP3 and SLES 15
Verify that the system is powered OFF and is unplugged before you
touch internal components.
Install or remove adapters in a static-free environment. The use of a
properly grounded wrist strap or other personal antistatic devices and an
antistatic mat is strongly recommended.
5 AH0054602-00 J
Loading...