How it Works
Cavium
Loading...
QL41112HLCU-CK
User Manual
299 pgs6.34 Mb0
Table of contents
Loading...

Cavium QL41112HLRJ-BK, QL41162HLRJ-BK, QL41162HLRJ-CK, QL41162HLRJ-SP, QL41212HLCU-CK User Manual

...
User’s Guide
Converged Network Adapters and
Intelligent Ethernet Adapters
FastLinQ 41000 Series
AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
Revision 01, February 7, 2017 Revision A, June 26, 2017 Revision B, November 21, 2017
Changes Sections Affected
Updated EMI/EMC requirements:
CE Mark EMC Directive ComplianceReplaced EN55022 with EN55032.VCCIAS/NZSAdded CNS 13438:2006 Class A
Updated product safety compliance:
Updated UL and UL CSA compliance for 2014Deleted the 2006/95/EC low voltage directiveUpdated TUV EN60950-1 and TUV IEC
60950-1
Updated Table 3-6 to include a column for ESXi driver versions.
Updated Table 6-1 for clarification. “Supported Operating Systems and OFED” on
Updated Step 2 to add the Dynamic option. “Preparing the Adapter” on page 59 Updated Table 6-2 to add the Quality of Service
property
“EMI and EMC Requirements” on page xxii
“Product Safety Compliance” on page xxiv
“VMware Drivers and Driver Packages” on page 25
page 57
“Configuring RoCE on the Adapter for Windows Server” on page 64
Added Configuring RoCE on the Adapter for ESX to chapter 6.
Updated the procedure introduction to re ad : To
configure iWARP through HII in Default mode.
Updated Step 4b, to remove the IWARP+RoCE option.
Updated Step 8, to correct DCB to iWARP. Deleted former Step 2: Unzip rdma-core-mas-
ter.zip, and then issue following command s . . .
Moved the former Chapter 11, iSER Configuration to following Chapter 7, iWARP Configuration.
Added Chapter 12, NVMe-oF Configuration with RDMA.
“Configuring RoCE on the Adapter for ESX” on page 80
“Preparing the Adapter for iWARP” on page 86
“iWARP RDMA-Core Support on SLES 12 SP3, RHEL 7.4, and OFED 4.8x” on page 95
Chapter 8 iSER Configuration
Chapter 12 NVMe-oF Configuration with RDMA
ii AH0054601-00 B

Table of Contents

Preface
Supported Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
What Is in This Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Related Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
License Agreements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Technical Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Downloading Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Contact Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
Knowledge Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
Warranty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
Laser Safety—FDA Notice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
Agency Certification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
EMI and EMC Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
KCC: Class A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
Product Safety Compliance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
1 Product Overview
Functional Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Adapter Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
QLogic Control Suite CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
QLogic QConvergeConsole GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
QLogic QConvergeConsole vCenter Plug-in. . . . . . . . . . . . . . . . . . . . 4
QConvergeConsole PowerKit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
FastLinQ ESXCLI VMware Plug-in . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Adapter Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Physical Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Standards Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
iii AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
2 Hardware Installation
System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Safety Precautions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Preinstallation Checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Installing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3 Driver Installation
Installing Linux Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Installing the Linux Drivers Without RDMA . . . . . . . . . . . . . . . . . . . . . 13
Removing the Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Installing Linux Drivers Using the src RPM Package . . . . . . . . . 15
Installing Linux Drivers Using the kmp/kmod RPM Package . . . 16
Installing Ubuntu Linux Drivers. . . . . . . . . . . . . . . . . . . . . . . . . . 16
Installing Linux Drivers Using the TAR File. . . . . . . . . . . . . . . . . 16
Installing the Linux Drivers with RDMA . . . . . . . . . . . . . . . . . . . . . . . . 17
Linux Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Linux Driver Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Installing Windows Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Installing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Removing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Managing Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Setting Power Management Options. . . . . . . . . . . . . . . . . . . . . . . . . . 22
Installing Drivers for Windows Nano Server . . . . . . . . . . . . . . . . . . . . 22
Creating a Nano ISO Image, Injecting Drivers, and Updating the
Multiboot/Flash Image on a Nano Server . . . . . . . . . . . . . . . . . . . . . 23
Installing VMware Driver Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
VMware Drivers and Driver Packages. . . . . . . . . . . . . . . . . . . . . . . . . 25
Installing VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
VMware Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . 28
VMware Driver Parameter Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Removing the VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
FCoE Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
iSCSI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4 Firmware Upgrade Utility
Image Verification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Upgrading Adapter Firmware on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Upgrading Adapter Firmware on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . 33
iv AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
Upgrading Adapter Firmware on Windows Nano. . . . . . . . . . . . . . . . . . . . . 34
5 Adapter Preboot Configuration
Getting Started. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Displaying Firmware Image Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Configuring Device-level Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Configuring Port-level Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Configuring FCoE Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Configuring Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6 RoCE Configuration
Supported Operating Systems and OFED. . . . . . . . . . . . . . . . . . . . . . . . . . 57
Planning for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Preparing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Preparing the Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Configuring the Cisco Nexus 6000 Ethernet Switch . . . . . . . . . . . . . . 59
Configuring the Dell Z9100 Ethernet Switch . . . . . . . . . . . . . . . . . . . . 61
Configuring the Arista 7060X Ethernet Switch. . . . . . . . . . . . . . . . . . . 63
Configuring RoCE on the Adapter for Windows Server . . . . . . . . . . . . . . . . 64
Configuring RoCE on the Adapter for Linux. . . . . . . . . . . . . . . . . . . . . . . . . 68
RoCE Configuration for RHEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
RoCE Configuration for SLES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
RoCE Configuration for Ubuntu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Verifying the RoCE Configuration on Linux. . . . . . . . . . . . . . . . . . . . . 72
VLAN Interfaces and GID Index Values . . . . . . . . . . . . . . . . . . . . . . . 74
RoCE v2 Configuration for Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Identifying RoCE v2 GID Index or Address . . . . . . . . . . . . . . . . 75
Verifying RoCE v1 or RoCE v2 GID Index and Address from
sys and class Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Verifying RoCE v1 or RoCE v2 Functionality Through perftest
Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Configuring RoCE on the Adapter for ESX. . . . . . . . . . . . . . . . . . . . . . . . . . 80
Configuring RDMA Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Configuring MTU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
RoCE Mode and Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Configuring a Paravirtual RDMA Device (PVRDMA). . . . . . . . . . . . . . 83
7 iW A RP Configuration
Preparing the Adapter for iWARP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Configuring iWARP on the Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
v AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
Configuring iWARP on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Installing the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Configuring iWARP and RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Detecting the Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Supported iWARP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Running Perftest for iWARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Configuring NFS-RDMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
iWARP RDMA-Core Support on SLES 12 SP3, RHEL 7.4, and
OFED 4.8x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
8 iSER Configuration
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Configuring iSER for RHEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Configuring iSER for SLES 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Using iSER with iWARP on RHEL and SLES. . . . . . . . . . . . . . . . . . . . . . . . 103
Configuring iSER for Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Configuring LIO as Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Configuring the Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Optimizing Linux Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Configuring CPUs to Maximum Performance Mode . . . . . . . . . . . . . . 111
Configuring Kernel sysctl Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Configuring IRQ Affinity Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Configuring Block Device Staging. . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
9 iSCSI Configuration
iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
iSCSI Boot Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Selecting the Preferred iSCSI Boot Mode . . . . . . . . . . . . . . . . . 114
Configuring the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Configuring iSCSI Boot Parameters. . . . . . . . . . . . . . . . . . . . . . 115
Adapter UEFI Boot Mode Configuration . . . . . . . . . . . . . . . . . . . . . . . 117
Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Static iSCSI Boot Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Dynamic iSCSI Boot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Enabling CHAP Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Configuring the DHCP Server to Support iSCSI Boot . . . . . . . . . . . . . . . . . 129
DHCP iSCSI Boot Configurations for IPv4 . . . . . . . . . . . . . . . . . . . . . 129
DHCP Option 17, Root Path. . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
DHCP Option 43, Vendor-specific Information. . . . . . . . . . . . . . 130
Configuring the DHCP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
vi AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
Configuring DHCP iSCSI Boot for IPv6. . . . . . . . . . . . . . . . . . . . . . . . 131
DHCPv6 Option 16, Vendor Class Option . . . . . . . . . . . . . . . . . 131
DHCPv6 Option 17, Vendor-Specific Information. . . . . . . . . . . . 131
Configuring VLANs for iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Configuring iSCSI Boot from SAN for SLES 12 . . . . . . . . . . . . . . . . . . . . . . 133
Configuring iSCSI Boot from SAN for RHEL 7.4 . . . . . . . . . . . . . . . . . . . . . 136
iSCSI Offload in Windows Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Installing QLogic Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Installing the Microsoft iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . 139
Configuring Microsoft Initiator to Use QLogic’s iSCSI Offload. . . . . . . 140
iSCSI Offload FAQs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
Windows Server 2012 R2 and 2016 iSCSI Boot Installation. . . . . . . . 147
iSCSI Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
iSCSI Offload in Linux Environments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Differences from bnx2i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Configuring qedi.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Verifying iSCSI Interfaces in Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Open-iSCSI and Boot from SAN Considerations. . . . . . . . . . . . . . . . . . . . . 152
10 FCoE Configuration
FCoE Boot from SAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Preparing System BIOS for FCoE Build and Boot. . . . . . . . . . . . . . . . 158
Specifying the BIOS Boot Protocol. . . . . . . . . . . . . . . . . . . . . . . 158
Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 158
Windows FCoE Boot from SAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Windows Server 2012 R2 and 2016 FCoE Boot Installation . . . 163
Configuring FCoE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
FCoE Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Injecting (Slipstreaming) Adapter Drivers into Windows Image Files. . . . . . 165
Configuring Linux FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Differences Between qedf and bnx2fc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Configuring qedf.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Verifying FCoE Devices in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Boot from SAN Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
11 SR-IOV Configuration
Configuring SR-IOV on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Configuring SR-IOV on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Configuring SR-IOV on VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
vii AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
12 NVMe-oF Configuration with RDMA
Installing Device Drivers on Both Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Configuring the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Configuring the Initiator Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
Preconditioning the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Testing the NVMe-oF Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Optimizing Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
.IRQ Affinity (multi_rss-affin.sh). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
CPU Frequency (cpufreq.sh). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
13 Windows Server 2016
Configuring RoCE Interfaces with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . 198
Creating a Hyper-V Virtual Switch with an RDMA Virtual NIC. . . . . . . 199
Adding a VLAN ID to Host Virtual NIC. . . . . . . . . . . . . . . . . . . . . . . . . 200
Verifying If RoCE is Enabled. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Adding Host Virtual NICs (Virtual Ports) . . . . . . . . . . . . . . . . . . . . . . . 201
Mapping the SMB Drive and Running RoCE Traffic . . . . . . . . . . . . . . 202
RoCE over Switch Embedded Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Creating a Hyper-V Virtual Switch with SET and RDMA Virtual NICs . 204
Enabling RDMA on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Assigning a VLAN ID on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Running RDMA Traffic on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Configuring QoS for RoCE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Configuring QoS by Disabling DCBX on the Adapter . . . . . . . . . . . . . 205
Configuring QoS by Enabling DCBX on the Adapter . . . . . . . . . . . . . . 210
Configuring VMMQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Enabling VMMQ on the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Setting the VMMQ Max QPs Default and Non-Default VPort . . . . . . . 214
Creating a Virtual Machine Switch with or Without SR-IOV . . . . . . . . 215
Enabling VMMQ on the Virtual Machine Switch . . . . . . . . . . . . . . . . . 217
Getting the Virtual Machine Switch Capability. . . . . . . . . . . . . . . . . . . 217
Creating a VM and Enabling VMMQ on VMNetworkadapters
in the VM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Default and Maximum VMMQ Virtual NIC. . . . . . . . . . . . . . . . . . . . . . 219
Enabling and Disabling VMMQ on a Management NIC . . . . . . . . . . . 219
Monitoring Traffic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Configuring VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Enabling VXLAN Offload on the Adapter. . . . . . . . . . . . . . . . . . . . . . . 220
Deploying a Software Defined Network. . . . . . . . . . . . . . . . . . . . . . . . 221
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
viii AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
Configuring the Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Deploying a Hyper-Converged System. . . . . . . . . . . . . . . . . . . . . . . . 222
Deploying the Operating System . . . . . . . . . . . . . . . . . . . . . . . . 223
Configuring the Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . 226
Deploying and Managing a Nano Server . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Roles and Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Deploying a Nano Server on a Physical Server . . . . . . . . . . . . . . . . . 230
Deploying a Nano Server in a Virtual Machine . . . . . . . . . . . . . . . . . . 233
Managing a Nano Server Remotely . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Managing a Nano Server with Windows PowerShell Remoting. 235
Adding the Nano Server to a List of Trusted Hosts . . . . . . . . . . 235
Starting the Remote Windows PowerShell Session. . . . . . . . . . 235
Managing QLogic Adapters on a Windows Nano Server . . . . . . . . . . 236
RoCE Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
14 Troubleshooting
Troubleshooting Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Verifying that Current Drivers Are Loaded . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Verifying Drivers in Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Verifying Drivers in Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Verifying Drivers in VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Testing Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Testing Network Connectivity for Windows . . . . . . . . . . . . . . . . . . . . . 241
Testing Network Connectivity for Linux . . . . . . . . . . . . . . . . . . . . . . . . 242
Microsoft Virtualization with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Linux-specific Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Miscellaneous Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Troubleshooting Windows FCoE and iSCSI Boot from SAN . . . . . . . . . . . . 243
Collecting Debug Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
A Adapter LEDS B Cables and Optical Modules
Supported Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Tested Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
C Feature Constraints Glossary
ix AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
Index
x AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series

List of Figures

Figure Page
3-1 Setting Advanced Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3-2 Power Management Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5-1 System Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5-2 System Setup: Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5-3 Main Configuration Page, Setting Default Partitioning Mode. . . . . . . . . . . . . . . . . . 37
5-4 Main Configuration Page, Setting NPAR Partitioning Mode. . . . . . . . . . . . . . . . . . . 38
5-5 Firmware Information Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5-6 Device Level Configuration Page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5-7 Port Level Configuration Page: Setting Link Speed. . . . . . . . . . . . . . . . . . . . . . . . . 43
5-8 Port Level Configuration Page: Setting Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . 44
5-9 FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5-10 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5-11 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5-12 iSCSI Initiator Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5-13 iSCSI First Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5-14 iSCSI Second Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5-15 Partitions Configuration Page (No FCoE Offload or iSCSI Offload). . . . . . . . . . . . . 51
5-16 Partitions Configuration Page (with FCoE Offload and iSCSI Offload) . . . . . . . . . . 52
5-17 Global Bandwidth Allocation Page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5-18 Partition 1 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5-19 Partition 2 Configuration: FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5-20 Partition 3 Configuration: iSCSI Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5-21 Partition 4 Configuration: Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6-1 Configuring RoCE Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6-2 Switch Settings, Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6-3 Switch Settings, Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6-4 Configuring RDMA_CM Applications: Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6-5 Configuring RDMA_CM Applications: Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6-6 Configuring a New Distributed Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6-7 Assigning a vmknic for PVRDMA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6-8 Setting the Firewall Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7-1 Windows PowerShell Command: Get-NetAdapterRdma. . . . . . . . . . . . . . . . . . . . . 87
7-2 Windows PowerShell Command: Get-NetOffloadGlobalSetting . . . . . . . . . . . . . . . 88
7-3 Perfmon: Add Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7-4 Perfmon: Verifying iWARP Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
8-1 RDMA Ping Successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
8-2 iSER Portal Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
8-3 Iface Transport Confirmed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
8-4 Checking for New iSCSI Device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
8-5 LIO Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
9-1 Systems Utilities at Boot Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
9-2 Configuration Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
9-3 Selecting Port Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
xi AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
9-4 Port Level Configuration, Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
9-5 Selecting iSCSI Boot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
9-6 Selecting General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
9-7 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
9-8 Selecting iSCSI Initiator Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
9-9 iSCSI Initiator Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
9-10 iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
9-11 iSCSI First Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
9-12 iSCSI Second Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
9-13 Saving iSCSI Changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
9-14 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
9-15 iSCSI Initiator Configuration, VLAN ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
9-16 System Configuration: Setting iSCSI Initiator Parameters. . . . . . . . . . . . . . . . . . . . 134
9-17 Prompt for Out-of-Box Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
9-18 Red Hat Enterprise Linux 7.4 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
9-19 iSCSI Initiator Properties, Configuration Page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
9-20 iSCSI Initiator Node Name Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
9-21 iSCSI Initiator—Discover Target Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
9-22 Target Portal IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
9-23 Selecting the Initiator IP Address. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
9-24 Connecting to the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
9-25 Connect To Target Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
10-1 System Utilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
10-2 System Configuration, Port Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
10-3 Port Level Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
10-4 Boot Mode in Port Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
10-5 FCoE Offload Enabled. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10-6 Selecting General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10-7 FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
10-8 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
11-1 Device Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
11-2 Adapter Properties, Advanced: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . 172
11-3 Virtual Switch Manager: Enabling SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
11-4 Settings for VM: Enabling SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
11-5 Device Manager: VM with QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
11-6 Windows PowerShell Command: Get-NetadapterSriovVf . . . . . . . . . . . . . . . . . . . . 177
11-7 Integrated Devices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
11-8 Editing the grub.conf File for SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
11-9 Command Output for sriov_numvfs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
11-10 Command Output for ip link show Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
11-11 RHEL68 Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
11-12 Add New Virtual Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
11-13 VMware Host Edit Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
12-1 NVMe-oF Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
12-2 Subsystem NQN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
xii AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series
12-3 Confirm NVMe-oF Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
12-4 FIO Utility Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
13-1 Enabling RDMA in Host Virtual NIC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
13-2 Hyper-V Virtual Ethernet Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
13-3 Windows PowerShell Command: Get-VMNetworkAdapter . . . . . . . . . . . . . . . . . . . 200
13-4 Windows PowerShell Command: Get-NetAdapterRdma. . . . . . . . . . . . . . . . . . . . . 201
13-5 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
13-6 Performance Monitor Shows RoCE Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
13-7 Windows PowerShell Command: New-VMSwitch . . . . . . . . . . . . . . . . . . . . . . . . . . 204
13-8 Windows PowerShell Command: Get-NetAdapter. . . . . . . . . . . . . . . . . . . . . . . . . . 204
13-9 Advanced Properties: Enable QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
13-10 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
13-11 Advanced Properties: Enabling QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
13-12 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
13-13 Advanced Properties: Enabling Virtual Switch RSS. . . . . . . . . . . . . . . . . . . . . . . . . 214
13-14 Advanced Properties: Setting VMMQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
13-15 Virtual Switch Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
13-16 Windows PowerShell Command: Get-VMSwitch. . . . . . . . . . . . . . . . . . . . . . . . . . . 217
13-17 Advanced Properties: Enabling VXLAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
13-18 Example Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
13-19 Windows PowerShell Command: Get-NetAdapter. . . . . . . . . . . . . . . . . . . . . . . . . . 237
13-20 Windows PowerShell Command: Get-NetAdapterRdma. . . . . . . . . . . . . . . . . . . . . 237
13-21 Windows PowerShell Command: New-Item . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
13-22 Windows PowerShell Command: New-SMBShare . . . . . . . . . . . . . . . . . . . . . . . . . 238
13-23 Windows PowerShell Command: Get-NetAdapterStatistics . . . . . . . . . . . . . . . . . . 238
14-1 Windows Setup Error Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
xiii AH0054601-00 B
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series

List of Tables

Table Page
2-1 Host Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2-2 Minimum Host Operating System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3-1 QLogic 41000 Series Adapters Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3-2 qede Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3-3 Linux Driver Operation Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3-4 Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3-5 VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3-6 ESXi Driver Packages by Release. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3-7 VMware Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3-8 VMware Driver Parameter Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3-9 QLogic 41000 Series Adapter VMware FCoE Driver. . . . . . . . . . . . . . . . . . . . . . . . 30
3-10 QLogic 41000 Series Adapter iSCSI Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5-1 Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
6-1 OS Support for RoCE, RoCEv2, iWARP, iSER, and OFED . . . . . . . . . . . . . . . . . . 57
6-2 Advanced Properties for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
9-1 Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
9-2 DHCP Option 17 Parameter Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
9-3 DHCP Option 43 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
9-4 DHCP Option 17 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
12-1 Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
13-1 Roles and Features of Nano Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
14-1 Collecting Debug Data Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
A-1 Adapter Port Link and Activity LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
B-1 Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
B-2 Switches Tested for Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
xiv AH0054601-00 B

Preface

This preface lists the supported products, specifies the intended audience, explains the typographic conventions used in this guide, lists related documents, provides technical support and contact information, and describes legal notices.

Supported Products

This user’s guide describes the following Cavium FastLinQ® products:
10Gb Intelligent Ethernet Adapter: QL41112HLCU-CK/SP/BK 10Gb Intelligent Ethernet Adapter: QL41112HLRJ-CK/SP/BK 10Gb Converged Network Adapter: QL41162HLRJ-CK/SP/BK 10/25Gb Intelligent Ethernet Adapter: QL41212HLCU-CK/SP/BK 10/25Gb Converged Network Adapter: QL41262HLCU-CK/SP/BK

Intended Audience

This guide is intended for system administrators and other technical staff members responsible for configuring and managing adapters installed on servers
in Windows, Linux
®
, or VMware® environments.

What Is in This Guide

Following this preface, the remainder of this guide is organized into the following chapters and appendices:
Chapter 1 Product Overview provides a product functional description, a list
of features, adapter management tool descriptions, and the adapter specifications.
Chapter 2 Hardware Installation describes how to install the adapter,
including the list of system requirements and a preinstallation checklist.
Chapter 3 Driver Installation describes the installation of the adapter drivers
on Windows, Linux, and VMware.
Chapter 4 Firmware Upgrade Utility describes the use of the utility to
upgrade adapter firmware and boot code.
xv AH0054601-00 B
Preface What Is in This Guide
Chapter 5 Adapter Preboot Configuration describes the preboot adapter
Chapter 6 RoCE Configuration describes how to configure the adapter, the
Chapter 7 iWARP Configuration provides procedures for configuring Internet
Chapter 8 iSER Configuration describes how to configure iSCSI Extensions
Chapter 9 iSCSI Configuration describes iSCSI boot, iSCSI crash dump,
Chapter 10 FCoE Configuration describes Fibre Channel over Ethernet
Chapter 11 SR-IOV Configuration provides procedures for configuring single
configuration tasks using the Human Infrastructure Interface (HII) application.
Ethernet switch, and the host to use RDMA over converged Ethernet (RoCE).
wide area RDMA protocol (iWARP) on Windows and Linux systems.
for RDMA (iSER) for Linux RHEL and SLES.
and iSCSI offload for Windows and Linux.
(FCoE) boot from SAN and booting from SAN after installation.
root input/output virtualization (SR-IOV) on Windows, Linux, and VMware systems.
Chapter 12 NVMe-oF Configuration with RDMA demonstrates how to
configure NVMe-oF on a simple network.
Chapter 13 Windows Server 2016 describes the Windows Server 2016
features.
Chapter 14 Troubleshooting describes a variety of troubleshooting methods
and resources.
Appendix A Adapter LEDS lists the adapter LEDs and their significance. Appendix B Cables and Optical Modules lists the cables and optical
modules that the 41000 Series Adapters support.
Appendix C Feature Constraints provides information about feature
constraints implemented in the current release.
At the end of this guide is a glossary of terms and an index to help you quickly locate the information you need.
xvi AH0054601-00 B
Preface
NOTE
CAUTION
CAUTION
!
!
WARNING

Related Materials

Related Materials
For additional information, refer to the following documents that are available on the Downloads and Documentation page of the QLogic Web site:
http://driverdownloads.qlogic.com
Installation Guide—QConvergeConsole GUI (part number SN0051105-00)
contains detailed information on how to install and use the QConvergeConsole GUI management tool.
Help System—QConvergeConsole GUI help topics, available while using
the QCC GUI.
User's Guide—QLogic Control Suite CLI (part number BC0054511-00)
contains detailed information on how to install, start, and u se QLogic Control Suite CLI.
User's Guide—PowerShell (part number BC0054518-00) contains detailed
information on how to install QConvergeConsole FastLinQ PowerKit to manage the QLogic FastLinQ adapters on the system.
User’s Guide—QConvergeConsole Plug-ins for vSphere (part number
SN0054677-00) provides details for using the two plug-ins to extend the capabilities of VMware vCenter Server and the vSphere Web Client.
User’s Guide—FastLinQ ESXCLI VMware Plug-in (part number
BC0151101-00) describes the plug-in that extends the capabilities of the
®
ESX
CLI to manage QLogic 3400, 8400, and 45000 Series Adapters
installed in VMware ESX/ESXi hosts.

Documentation Conventions

This guide uses the following documentation conventions:
provides additional information.
without an alert symbol indicates the presence of a hazard
that could cause damage to equipment or loss of data.
with an alert symbol indicates the presence of a hazard that
could cause minor or moderate injury.
indicates the presence of a hazard that could cause serious
injury or death.
xvii AH0054601-00 B
Preface Documentation Conventions
Text in blue font indicates a hyperlink (jump) to a figure, table, or section in
this guide, and links to Web sites are shown in underlined blue example:
Table 9-2 lists problems related to the user interface and remote agent. See “Installation Checklist” on page 6.
. For
For more information, visit www.cavium.com
.
Text in bold font indicates user interface elements such as a menu items,
buttons, check boxes, or column headings. For example: Click the Start button, point to Programs, point to Accessories, and
then click Command Prompt.
Under Notification Options, select the Warning Alarms check box.
Text in
Courier font indicates a file name, directory path, or command line
text. For example: To return to the root directory from anywhere in the file structure:
Type
cd /root and press ENTER.
Issue the following command:
sh ./install.bin
Key names and key strokes are indicated with UPPERCASE:
Press CTRL+P. Press the UP ARROW key.
Text in italics indicates terms, emphasis, variables, or document titles. For
example:
For a complete listing of license agreements, refer to the Software End
User License Agreement.
What are shortcut keys? To enter the date, type mm/dd/yyyy (where mm is the month, dd is the
day, and yyyy is the year).
Topic titles between quotation marks identify related topics either within this
manual or in the online help, which is also referred to as the help system throughout this document.
Command line interface (CLI) command syntax conventions include the
following:
Plain text indicates items that you must type as shown. For example:
qaucli -pr nic -ei
xviii AH0054601-00 B
Preface
NOTE

License Agreements

<> (angle brackets) indicate a variable whose value you must
specify. For example:
<serial_number>
For CLI commands only, variable names are always indicated using angle brackets instead of italics.
[] (square brackets) indicate an optional parameter. For example:
[<file_name>] means specify a file name, or omit it to select
the default file name.
| (vertical bar) indicates mutually exclusive options; select one option
only. For example:
on|off
1|2|3|4
... (ellipsis) indicates that the preceding item may be repeated. For
example:
x... means one or more instances of x.
[x...] means zero or more instances of x.
Vertical ellipses within command example output indicate where
portions of repetitious output data have been intentionally omitted.
() (parentheses) and { } (braces) are used to avoid logical
ambiguity. For example:
a|b c is ambiguous
{(a|b) c} means a or b, followed by c {a|(b c)}
License Agreements
Refer to the QLogic Software End User License Agreement for a complete listing of all license agreements affecting this product.
means either a, or b c
xix AH0054601-00 B
Preface Technical Support

Technical Support

Customers should contact their authorized maintenance provider for technical support of their QLogic products. QLogic-direct customers may contact Technical Support; others will be redirected to their authorized maintenance provider. Visit the Support Web site listed in Contact Information for the latest firmware and software updates.

Downloading Updates

The QLogic Web site provides periodic updates to product firmware, software, and documentation.
To download firmware, software, and documentation:
1. Go to the Downloads and Documentation page:
driverdownloads.qlogic.com
2. Type the QLogic model name in the search box.

Training

3. In the search results list, locate and select the firmware, software, or documentation for your product.
4. View the product details Web page to ensure that you have the correct firmware, software, or documentation. For additional information, click Read Me and Release Notes under Support Files.
5. Click Download Now.
6. Save the file to your computer.
7. If you have downloaded firmware, software, drivers, or boot code, follow the installation instructions in the Read Me file.
Instead of typing a model name in the search box, you can perform a guided search as follows:
1. Click the product type tab: Adapters, Switches, Routers, or ASICs.
2. Click the corresponding button to search by model or operating system.
3. Click an item in each selection column to define the search, and then click Go.
4. Locate the firmware, software, or document you need, and then click the item’s name or icon to download or open the item.
Global Training maintains a Web site at www.qlogictraining.com offering online and instructor-led training for all QLogic products. In addition, sales and technical professionals may obtain Associate and Specialist-level certifications to qualify for additional benefits.
xx AH0054601-00 B
Preface Legal Notices

Contact Information

Technical Support for products under warranty is available during local standard working hours excluding Cavium Observed Holidays. For Support phone numbers, see the Contact Support link:
support.qlogic.com
Support Headquarters
QLogic Web Site Technical Support Web Site Technical Support E-mail Technical Training E-mail

Knowledge Database

The QLogic knowledge database is an extensive collection of QLogic product information that you can search for specific solutions. QLogic is constantly adding to the collection of information in the database to provide answers to your most urgent questions. Access the database from the QLogic Support Center:
support.qlogic.com

Legal Notices

Legal notices covered in this section include warranty, laser safety (FDA notice), agency certification, and product safety compliance.
Cavium, Inc. 12900 Whitewater Drive Suite 140 Minnetonka, MN 55343 USA
www.qlogic.com support.qlogic.com globalsupport@cavium.com training@qlogic.com

Warranty

For warranty details, please check the QLogic Web site:
www.qlogic.com/Support/Pages/Warranty.aspx
xxi AH0054601-00 B
Preface
CLASS I LASER
Legal Notices

Laser Safety—FDA Notice

This product complies with DHHS Rules 21CFR Chapter I, Subchapter J. This product has been designed and manufactured according to IEC60825-1 on the safety label of laser product.
Class 1 Laser Product
Appareil laser de classe 1
Produkt der Laser Klasse 1
Luokan 1 Laserlaite Varoitus—Luokan 1 lasersäteilyä, kun laite on auki

Agency Certification

The following sections summarize the EMC and EMI test specifications performed on the 41000 Series Adapters.
EMI and EMC Requirements
FCC Part 15 compliance: Class A
FCC compliance information statement: This device complies with Part 15 of
the FCC Rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation.
ICES-003 Compliance: Class A
This Class A digital apparatus complies with Canadian ICES-003. Cet appareil numériqué de la classe A est conformé à la norme NMB-003 du Canada.
Caution—Class 1 laser radiation when open Do not view directly with optical instruments
Attention—Radiation laser de classe 1 Ne pas regarder directement avec des instruments optiques
Vorsicht—Laserstrahlung der Klasse 1 bei geöffneter Abdeckung Direktes Ansehe n mit optischen Instrumenten vermeiden
Älä katso suoraan laitteeseen käyttämällä optisia instrumenttej
CE Mark 2014/30/EU, 2014/35/EU EMC Directive Compliance:
EN55032:2012/ CISPR 32:2015 Class A EN55024:2010
EN61000-3-2 : Harmonic Current Emission EN61000-3-3 : Voltage Fluctuation and Flicker
xxii AH0054601-00 B
Preface Legal Notices
KCC: Class A
Immunity Standards EN61000-4-2 : ESD EN61000-4-3 : RF Electro Magnetic Field EN61000-4-4 : Fast Transient/Burst EN61000-4-5 : Fast Surge Common/ Differential EN61000-4-6 : RF Conducted Susceptibility EN61000-4-8 : Power Frequency Magnetic Field EN61000-4-11: Voltage Dips and Interrupt
VCCI: 2015-04; Class A AS/NZS; CISPR 32: 2015 Class A CNS 13438: 2006 Class A
Korea RRA Class A Certified
Product Name/Model: Converged Network Adapters and Intelligent Ethernet Adapters Certification holder: QLogic Corporation Manufactured date: Refer to date code listed on product Manufacturer/Country of origin: QLogic Corporation/USA
A class equipm ent (Business purpose
info/telecommunications equipment)
As this equipment has undergone EMC registration for busi­ness purpose, the seller and/or the buyer is asked to beware of this point and in case a wrongful sale or purchase has been made, it is asked that a change to household use be made.
Korean Language Format—Class A
xxiii AH0054601-00 B
Preface Legal Notices

Product Safety Compliance

UL, cUL product safety:
UL 60950-1 (2nd Edition) A1 + A2 2014-10-14 CSA C22.2 No.60950-1-07 (2nd Edition) A1 +A2 2014-10
Use only with listed ITE or equivalent. Complies with 21 CFR 1040.10 and 1040.11, 2014/30/EU, 2014/35/EU. 2006/95/EC low voltage directive: TUV EN60950-1:2006+A11+A1+A12+A2 2nd Edition
TUV IEC 60950-1: 2005 2nd Edition Am1: 2009 + Am2: 2013 CB CB Certified to IEC 60950-1 2nd Edition
xxiv AH0054601-00 B

1 Product Overview

This chapter provides the following information for the 41000 Series Adapters:
Functional Description Features Adapter Management Adapter Specifications

Functional Description

The QLogic FastLinQ 41000 Series Adapters include 10 and 25Gb Converged Network Adapters and Intelligent Ethernet Adapters that are designed to perform accelerated data networking for server systems. The 41000 Series Adapter includes a 10/25Gb Ethernet MAC with full-duplex capability.
Using the operating system’s teaming feature, you can split your network into virtual LANs (VLANs), as well as group multiple network adapters together into teams to provide network load balancing and fault tolerance. For more information about teaming, see your operating system documentation.

Features

The 41000 Series Adapters provide the following features. Some features may not be available on all adapters:
NIC partitioning (NPAR) FCoE offload iSCSI offload Universal RDMA: Remote direct memory access over converged Ethernet,
versions 1 and 2 (RoCE v1/v2) and Internet Wide Area Protocol (iWARP):
iSCSI Extensions for RDMA (iSER) NVMe over Fabrics (NVMe-oF) (For details, download the Deployment
Guide: Non-volatile Memory Express over Fabrics with Universal Remote Direct Memory Access, part number BC0054519-00)
NFS over RDMA (NFSoRDMA)
1 AH0054601-00 B
1–Product Overview Features
Data center bridging (DCB):
Single-chip solution:
Performance features:
Enhanced transmission selection (ETS; IEEE 802.1Qaz) Priority-based flow control (PFC; IEEE 802.1Qbb) Data center bridging eXchange protocol (DCBX; CEE version 1.01,
IEEE)
10/25Gb MAC SerDes interface for direct attach copper (DAC) transceiver connection RJ-45 10GBASE-T interface for CAT cable PCI Express 3.0 x8 Zero copy capable hardware
TCP, IP, UDP checksum offloads TCP segmentation offload (TSO)
Large segment offload (LSO) Generic segment offload (GSO) HW based Generic Receive Offload (HW-GRO) Large receive offload (LRO) Receive segment coalescing (RSC)
Microsoft
®
dynamic virtual machine queue (VMQ), and Linux
multiqueue
Adaptive interrupts:
Transmit/receive side scaling (TSS/RSS) Stateless offloads for Network Virtualization using Generic Routing
Encapsulation (NVGRE) and virtual LAN (VXLAN) L2/L3 GRE tunneled traffic
1
Manageability:
System management bus (SMB) controller Advanced Configuration and Power Interface (ACPI) 1.1a compliant
(multiple power modes)
Network controller-sideband interface (NC-SI) support The management applications listed in Adapter Management
1
This feature requires OS or Hypervisor support to use the offloads.
2 AH0054601-00 B
1–Product Overview
NOTE

Adapter Management

Advanced network features:
Logical link control (IEEE Std 802.2) High-speed on-chip reduced instruction set computer (RISC) processor Integrated 96KB frame buffer memory (not applicable to all models) 1,024 classification filters (not applicable to all models) Support for multicast addresses through 128-bit hashing hardware function Serial flash NVRAM memory PCI Power Management Interface (v1.1) 64-bit base address register (BAR) support
Jumbo frames (up to 9,600 bytes). The OS and the link partner must
support jumbo frames.
Virtual LANs (VLAN) Flow control (IEEE Std 802.3x)
EM64T processor support iSCSI and FCoE boot support
Adapter Management
The following applications are available to manage 41000 Series Adapters:
QLogic Control Suite CLI QLogic QConvergeConsole GUI QLogic QConvergeConsole vCenter Plug-in QConvergeConsole PowerKit FastLinQ ESXCLI VMware Plug-in
QCS CLI, QCC GUI, and QCC PowerKit also require installation of their specific RPC agent.
For document part numbers and download instructions, see “Related Materials”
on page xvii.
2
2
Hardware support limit of SR-IOV VFs varies. The limit may be lower on some OS environments;
refer to the appropriate section for your OS.
3 AH0054601-00 B
1–Product Overview
NOTE
Adapter Management

QLogic Control Suite CLI

QLogic Control Suite (QCS) CLI is a console application that you can run from a Windows command prompt or a Linux terminal console. Use QCS CLI to manage QLogic FastLinQ 3400/8400/41000/45000 Series Adapters and any QLogic adapter based on 57xx/57xxx controllers on both local and remote computer systems. For information about installing and using QCS CLI, see the User’s Guide—QLogic Control Suite CLI.
Although QLogic Control Suite CLI does not run on a Windows Nano Server shell, you can install the QLNXRemote agent on a Nano Server system and manage it remotely through QCS CLI using the
removehost
remote QLNXRemote Nano agent, you must disable or correctly configure the firewall on the Nano system. For instructions on installing the QLNXRemote agent on a Nano Server, refer to the QConvergeConsole Windows Agent Installers Read me located inside the QCS CLI for Windows download available on the QLogic Downloads and Documentation page:
www.qlogic.com
(Remove Host) commands. For QCS CLI to connect to the
addhost (Add Host) and

QLogic QConvergeConsole GUI

The QConvergeConsole (QCC) GUI is a Web-based management tool for configuring and managing QLogic Fibre Channel Adapters, Converged Network Adapters, and Intelligent Ethernet Adapters. You can use QCC GUI on Windows and Linux platforms to manage QLogic adapters on both local and remote computer systems. QCC GUI is dependent upon additional software (a management agent) for the adapter. The management agent is installed separately from QCC GUI. QCC GUI cannot communicate with the hardware until the management agent has been installed.
For information about installing QCC GUI, see the Installation Guide— QConvergeConsole GUI. For information about using the QCC GUI, see the online help.

QLogic QConvergeConsole vCenter Plug-in

The QConvergeConsole vCenter® Plug-in is a Web-based management tool that is integrated into the VMware vCenter Server for configuring and managing QLogic Fibre Channel adapters, Converged Network Adapters, and Intelligent Ethernet Adapters in a virtual environment. You can use the vCenter Plug-in installed in the VMware vSphere clients to manage QLogic adapters. For information about installing and using the vCenter Plug-in, see the User’s Guide— QConvergeConsole Plug-ins for vSphere.
4 AH0054601-00 B
1–Product Overview Adapter Specifications

QConvergeConsole PowerKit

The QConvergeConsole PowerKit lets you manage QLogic FastLinQ 3400/8400/41000/45000 Series Adapters on the system using QLogic cmdlets in
the Windows PowerShell Microsoft-developed scriptable language for performing task automation and configuration management both locally and remotely. Windows PowerShell is based on the .NET framework and includes a command-line shell and a GUI-integrated scripting environment (ISE) that allows you to create scripts without having to type all the commands in the command line. This feature allows you to streamline and automate repetitive and monotonous Windows and Linux server jobs through scripts by linking multiple instructions together. In addition to being a powerful scripting tool, the QLogic PowerKit comes with a selection of preconfigured cmdlets (scripts that perform a single function) to monitor and manage your QLogic FastLinQ adapters. For information about installing and using the QConvergeConsole PowerKit, refer to the User's Guide—PowerShell.
®
application. Windows PowerShell is a

FastLinQ ESXCLI VMware Plug-in

The FastLinQ ESXCLI plug-in extends the capabilities of the ESX® CLI to manage QLogic FastLinQ 3400/8400/41000/45000 Series Adapters installed in VMware ESX/ESXi hosts. For information about installing and using the ESXCLI plug-in, see the User’s Guide—FastLinQ ESXCLI VMware Plug-in.

Adapter Specifications

The 41000 Series Adapter specifications include the adapter’s physical characteristics and standards-compliance references.

Physical Characteristics

The 41000 Series Adapte rs are standard PCI Express® cards and ship with either a full-height or a low-profile bracket for use in a standard PCIe
®
slot.
5 AH0054601-00 B
1–Product Overview Adapter Specifications

Standards Specifications

Supported standards specifications include:
PCI Express Base Specification, rev. 3.0 PCI Express Card Electromechanical Specification, rev. 3.0 PCI Bus Power Management Interface Specification, rev. 1.2 IEEE Specifications:
802.3-2012 IEEE Standard for Ethernet (flow control) 802.1q (VLAN) 802.1AX (Link Aggregation) 802.1ad (QinQ) 802.1p (Priority Encoding) 1588-2002 PTPv1 (Precision Time Protocol) 1588-2008 PTPv2 IEEE 802.3az Energy Efficient Ethernet (EEE)
IPv4 (RFQ 791) IPv6 (RFC 2460)
6 AH0054601-00 B
Loading...
+ 269 hidden pages
Buy Points How it Works FAQ Contact Us Questions and Suggestions Privacy Policy Cookie Policy Terms of Use