Chelsio Communications Chelsio T5, Chelsio T4 Installation And User Manual

Chelsio T5/T4 Unified Wire for Linux ii
This document and related products are distributed under licenses restricting their use, copying, distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission by Chelsio Communications.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to sales@chelsio.com
Support
For all support related questions please send email to support@chelsio.com Copyright © 2016.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications. All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters) 209 North Fair Oaks Avenue, Sunnyvale, CA 94085 U.S.A
www.chelsio.com
Tel: 408.962.3600 Fax: 408.962.3661
Chelsio (India) Private Limited Subramanya Arcade, Floor 3, Tower B No. 12, Bannerghatta Road, Bangalore-560029 Karnataka, India
Tel: +91-80-4039-6800 Fax: +91-80-4039-6807
Chelsio KK (Japan) Regus Place Canada Akasaka 7-3-37 Minato-ku, Tokyo, Japan 107-0052
Tel: 03-6234-4353
Chelsio T5/T4 Unified Wire for Linux iii
Document History
Version
Revision Date
1.0.0
12/08/2011
1.0.1
01/09/2013
1.0.2
01/27/2013
1.0.3
03/26/2013
1.0.4
04/12/2013
1.0.5
06/20/2013
1.0.6
08/17/2013
1.0.7
10/22/2013
1.0.8
03/08/2013
1.0.9
05/15/2013
1.1.0
07/26/2013
1.1.1
08/14/2013
1.1.2
12/06/2013
1.1.3
12/19/2013
1.1.4
03/13/2014
1.1.5
05/02/2014
1.1.6
06/30/2014
1.1.7
10/22/2014
1.1.8
11/04/2014
1.1.9
02/05/2015
1.2.0
03/04/2015
1.2.1
03/25/2015
1.2.2
06/03/2015
1.2.3
08/05/2015
1.2.4
02/29/2016
Chelsio T5/T4 Unified Wire for Linux iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 12
1. Introduction 13
1.1. Features 13
1.2. Hardware Requirements 14
1.3. Software Requirements 14
1.4. Package Contents 15
2. Hardware Installation 18
3. Software/Driver Installation 21
3.1. Pre-requisites 22
3.2. Installing Chelsio Unified Wire from source 22
3.3. Installing Chelsio Unified Wire from RPM 33
3.4. Firmware update 35
4. Software/Driver Uninstallation 37
4.1. Uninstalling Chelsio Unified Wire from source 37
4.2. Uninstalling Chelsio Unified Wire from RPM 41
5. Configuring Chelsio Network Interfaces 43
5.1. Configuring 40G adapters 43
5.2. Configuring network-scripts 45
5.3. Creating network-scripts 45
5.4. Checking Link 46
6. Performance Tuning 47
7. Software/Driver Update 48
II. NETWORK (NIC/TOE) 49
1. Introduction 50
1.1. Hardware Requirements 50
1.2. Software Requirements 51
2. Software/Driver Loading 52
2.1. Loading in NIC mode (without full offload support) 52
2.2. Loading in TOE mode (with full offload support) 52
3. Software/Driver Unloading 53
3.1. Unloading the NIC driver 53
3.2. Unloading the TOE driver 53
4. Software/Driver Configuration and Fine-tuning 55
4.1. Instantiate Virtual Functions (SR-IOV) 55
4.2. Enabling Busy waiting 55
4.3. Performance Tuning 56
III. VIRTUAL FUNCTION NETWORK (VNIC) 62
Chelsio T5/T4 Unified Wire for Linux v
1. Introduction 63
1.1. Hardware Requirements 63
1.2. Software Requirements 64
2. Software/Driver Loading 65
2.1. Instantiate Virtual Functions 65
2.2. Loading the driver 65
3. Software/Driver Unloading 66
3.1. Unloading the driver 66
IV. IWARP (RDMA) 67
1. Introduction 68
1.1. Hardware Requirements 68
1.2. Software Requirements 68
2. Software/Driver Loading 70
2.1. Loading iWARP driver 70
3. Software/Driver Unloading 71
4. Software/Driver Configuration and Fine-tuning 72
4.1. Testing connectivity with ping and rping 72
4.2. Enabling various MPIs 73
4.3. Setting up NFS-RDMA 80
4.4. Performance Tuning 82
V. RDMA BLOCK DEVICE DRIVER (RBD) 83
1. Introduction 84
1.1. Hardware Requirements 84
1.2. Software Requirements 84
2. Software/Driver Loading 85
3. Software/Driver Unloading 86
4. Software/Driver Configuration and Fine-tuning 87
VI. WD-UDP 88
1. Introduction 89
1.1. Hardware Requirements 89
1.2. Software Requirements 89
2. Software/Driver Loading 91
3. Software/Driver Unloading 92
4. Software/Driver Configuration and Fine-tuning 93
4.1. Accelerating UDP Socket communications 93
VII. ISCSI PDU OFFLOAD TARGET 99
1. Introduction 100
Chelsio T5/T4 Unified Wire for Linux vi
1.1. Features 100
1.2. Hardware Requirements 101
1.3. Software Requirements 102
2. Software/Driver Loading 104
2.1. Latest iSCSI Software Stack Driver Software 104
3. Software/Driver Unloading 106
4. Software/Driver Configuration and Fine-tuning 107
4.1. Command Line Tools 107
4.2. iSCSI Configuration File 107
4.3. A Quick Start Guide for Target 108
4.4. The iSCSI Configuration File 110
4.5. Challenge-Handshake Authenticate Protocol (CHAP) 121
4.6. Target Access Control List (ACL) Configuration 123
4.7. Target Storage Device Configuration 125
4.8. Target Redirection Support 127
4.9. The command line interface tools “iscsictl” & “chisns” 128
4.10. Rules of Target Reload (i.e. “on the fly” changes) 133
4.11. System Wide Parameters 135
4.12. Performance Tuning 136
VIII. ISCSI PDU OFFLOAD INITIATOR 137
1. Introduction 138
1.1. Hardware Requirements 138
1.2. Software Requirements 139
2. Software/Driver Loading 140
3. Software/Driver Unloading 141
4. Software/Driver Configuration and Fine-tuning 142
4.1. Accelerating open-iSCSI Initiator 142
4.2. Auto login from cxgb4i initiator at OS bootup 144
IX. DATA CENTER BRIDGING (DCB) 146
1. Introduction 147
1.1. Hardware Requirements 147
1.2. Software Requirements 147
2. Software/Driver Loading 149
3. Software/Driver Unloading 150
4. Software/Driver Configuration and Fine-tuning 151
4.1. Configuring Cisco Nexus 5010 switch 151
4.2. Configuring the Brocade 8000 switch 154
5. Running NIC & iSCSI Traffic together with DCBx 156
Chelsio T5/T4 Unified Wire for Linux vii
X. FCOE PDU OFFLOAD TARGET 157
1. Introduction 158
1.1. Hardware Requirements 158
1.2. Software Requirements 158
2. Software/Driver Loading 159
3. Software/Driver Unloading 160
4. Software/Driver Configuration and Fine-tuning 161
4.1. Configuring Cisco Nexus 5010 switch 161
4.2. Collecting port information 164
4.3. Configuring LUNs on Target 167
4.4. Configuring Persistent Target 169
4.5. Verifying initiators connected to the target 170
4.6. Removing LUNs 171
4.7. Performance Tuning 171
XI. FCOE FULL OFFLOAD INITIATOR 174
1. Introduction 175
1.1. Hardware Requirements 175
1.2. Software Requirements 175
2. Software/Driver Loading 177
3. Software/Driver Unloading 178
4. Software/Driver Configuration and Fine-tuning 179
4.1. Configuring Cisco Nexus 5010 and Brocade switch 179
4.2. FCoE fabric discovery verification 179
4.3. Formatting the LUNs and Mounting the Filesystem 183
4.4. Creating Filesystem 184
4.5. Mounting the formatted LUN 185
XII. OFFLOAD BONDING DRIVER 186
1. Introduction 187
1.1. Hardware Requirements 187
1.2. Software Requirements 187
2. Software/Driver Loading 189
3. Software/Driver Unloading 190
4. Software/Driver Configuration and Fine-tuning 191
4.1. Offloading TCP traffic over a bonded interface 191
XIII. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) 192
1. Introduction 193
1.1. Hardware Requirements 193
1.2. Software Requirements 194
Chelsio T5/T4 Unified Wire for Linux viii
2. Software/Driver Loading 195
3. Software/Driver Unloading 196
4. Software/Driver Configuration and Fine-tuning 197
4.1. Offloading TCP traffic over a bonded interface 197
XIV. UDP SEGMENTATION OFFLOAD AND PACING 198
1. Introduction 199
1.1. Hardware Requirements 199
1.2. Software Requirements 200
2. Software/Driver Loading 201
3. Software/Driver Unloading 202
4. Software/Driver Configuration and Fine-tuning 203
4.1. Modifying the application 203
4.2. Configuring UDP Pacing 205
XV. OFFLOAD IPV6 DRIVER 207
1. Introduction 208
1.1. Hardware Requirements 208
1.2. Software Requirements 208
2. Software/Driver Loading 210
3. Software/Driver Unloading 211
3.1. Unloading the NIC driver 211
3.2. Unloading the TOE driver 211
XVI. BYPASS DRIVER 212
1. Introduction 213
1.1. Features 213
1.2. Hardware Requirements 214
1.3. Software Requirements 215
2. Software/Driver Loading 216
3. Software/Driver Unloading 217
4. Software/Driver Configuration and Fine-tuning 218
4.1. Starting ba server 218
4.2. Bypass API (CLI) 218
XVII. WD SNIFFING AND TRACING 224
1. Theory of Operation 225
1.1. Hardware Requirements 226
1.2. Software Requirements 227
2. Installation and Usage 228
Chelsio T5/T4 Unified Wire for Linux ix
2.1. Installing basic support 228
2.2. Using Sniffer (wd_sniffer) 228
2.3. Using Tracer (wd_tcpdump_trace) 228
XVIII.CLASSIFICATION AND FILTERING 230
1. Introduction 231
1.1. Hardware Requirements 231
1.2. Software Requirements 232
2. Usage 233
2.1. Configuration 233
2.2. Creating Filter Rules 234
2.3. Listing Filter Rules 235
2.4. Removing Filter Rules 235
2.5. Layer 3 example 236
2.6. Layer 2 example 239
3. Hash/DDR Filters 243
3.1. Creating Filter Rules 243
3.2. Listing Filter Rules 244
3.3. Removing Filter Rules 245
3.4. Swap MAC feature 245
3.5. Hit Counters 246
XIX. TRAFFIC MANAGEMENT 248
1. Introduction 249
1.1. Hardware Requirements 249
1.2. Software Requirements 250
2. Software/Driver Loading 251
3. Software/Driver Unloading 252
4. Software/Driver Configuration and Fine-tuning 253
4.1. Traffic Management Rules 253
4.2. Configuring Traffic Management 255
5. Usage 258
5.1. Non-Offloaded Connections 258
5.2. Offloaded Connections 258
5.3. Offloaded Connections with Modified Application 259
XX. UNIFIED WIRE MANAGER (UM) 260
1. Introduction 261
1.1. Features 261
1.2. Reference Architecture 262
1.3. Unified Wire Manager Components 262
Chelsio T5/T4 Unified Wire for Linux x
1.4. Authentication and encryption 263
2. Hardware and Software 264
2.1. Supported Adapters 264
2.2. Platform/Component Matrix 265
2.3. Platform/Driver Matrix 265
3. Installing Unified Wire Manager 266
4. Verifying UM components status 267
4.1. Verifying Management Agent 267
4.2. Verifying Management Client 268
4.3. Verifying Management Station 268
5. Management Agent 269
5.1. Communication 269
5.2. Configuration 269
5.3. Service configuration 269
5.4. Firewall 270
6. CLI client 271
6.1. CLI Help system 271
6.2. Client conflict resolution 271
7. Web GUI client 272
7.1. Management Station 272
7.2. Accessing Web Management Interface 273
7.3. Layout and Navigation 276
7.4. Home page 277
7.5. System page 290
7.6. Network page 302
7.7. Storage Page 331
7.8. Hardware Features 364
8. Uninstalling Unified Wire Manager 380
8.1. Uninstalling Management Agent 380
8.2. Uninstalling Management Client 380
8.3. Uninstalling Management Station 381
XXI. UNIFIED BOOT 382
1. Introduction 383
1.1. Hardware Requirements 383
1.2. Software Requirements 385
2. Flashing firmware and option ROM 386
2.1. Preparing USB flash drive 386
2.2. Legacy 387
2.3. uEFI 390
3. Configuring PXE Server 395
Chelsio T5/T4 Unified Wire for Linux xi
4. PXE boot process 396
4.1. Legacy PXE boot 396
4.2. uEFI PXE Boot 401
5. FCoE boot process 407
5.1. Legacy FCoE boot 407
5.2. uEFI FCoE Boot 416
6. iSCSI boot process 426
6.1. Legacy iSCSI boot 426
6.2. uEFI iSCSI Boot 439
7. Creating Driver Update Disk (DUD) 451
7.1. Creating DUD for RedHat Enterprise Linux 451
7.2. Creating DUD for Suse Enterprise Linux 451
8. OS Installation 453
8.1. Installation using Chelsio NIC DUD (PXE only) 453
8.2. Installation on FCoE LUN 464
8.3. Installation on iSCSI LUN 484
XXII. LUSTRE FILE SYSTEM 501
1. Introduction 502
1.1. Hardware Requirements 502
1.2. Software Requirements 503
2. Creating/Configuring Lustre File System 504
XXIII. APPENDIX A 506
1. Troubleshooting 507
2. Chelsio End-User License Agreement (EULA) 509
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 12
I. Chelsio Unified Wire
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 13
1. Introduction
Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip, single firmware cards provide enterprises and data centers with high performance solutions for various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized 40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, iSCSI, FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to thousands of connections, and allows simultaneous low latency and high bandwidth operation thanks to multiple physical channels through the ASIC.
The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and Filtering.
Ideal for all data, storage and high performance clustering applications, the T5/T4 Adapters enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets, Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high performance computing environments, Chelsio T5/T4 adapters bring a new level of performance metrics and functional capabilities to the computer networking industry.
Chelsio Unified Wire software comes in two formats: Source code and RPM package forms. Installing from source requires compiling the package to generate the necessary binaries. You can choose this method when you are using a custom-built kernel. You can also install the package using the interactive GUI installer. In other cases, download the RPM package specific to your operating system and follow the steps mentioned to install the package. Please note that the OFED software required to install Chelsio iWARP driver comes bundled in both source as well as RPM packages.
This document describes the installation, use and maintenance of the software and its various components.
1.1. Features
The Chelsio Unified Wire Package uses a single command to install various drivers and utilities. It consists of the following software:
Network (NIC/TOE) Virtual Function Network (vNIC) iWARP (RDMA)
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 14
RDMA Block Device driver (RBD) WD-UDP * iSCSI PDU Offload Target iSCSI PDU Offload Initiator Data Center Bridiging (DCB) * FCoE PDU Offload Target * FCoE full offload Initiator * Offload Bonding driver Offload Multi-Adapter Failover(MAFO) UDP Segmentation Offload and Pacing Offload IPv6 driver Bypass driver Classification and Filtering feature Traffic Management feature (TM) Unified Wire Manager (UM) Unified Boot Software^ Lustre File System Utility Tools (cop, cxgbtool, t4_perftune, benchmark tools, sniffer & tracer) libs (iWARP and WD-UDP libraries)
For detailed instructions on loading, unloading and configuring the drivers/tools please refer to their respective sections.
* This product is not supported with the current release. Support will be added in upcoming
releases. ^ FCoE and iSCSI Boot are not supported with the current release. Support will be added
in upcoming releases.
1.2. Hardware Requirements
The Chelsio Unified Wire software supports Chelsio T5 and T4 Series of Unified Wire Adapters. To know more about the list of adapters supported by each driver, please refer to their respective sections.
1.3. Software Requirements
The Chelsio Unified Wire software has been developed to run on 64-bit Linux based platforms and therefore it is a base requirement for running the driver. To know more about the complete list of operating systems supported by each driver, please refer to their respective sections.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 15
1.4. Package Contents
1.4.1. Source Package
The Chelsio Unified Wire source package consists of the following files/directories:
debrules: This directory contains packaging specification files required for building Debian
packages.
docs: This directory contains support documents - README, Release Notes and User’s
Guide (this document) for the software.
libs: This directory is for libraries required to install the WD-UDP and iWARP drivers. The
libibverbs library has implementation of RDMA verbs which will be used by iWARP
applications for data transfers. The librdmacm library works as an RDMA connection
manager. The libcxgb4 library works as an interface between the above mentioned generic
libraries and Chelsio iWARP driver. The libcxgb4_sock library is a LD_PRELOAD-able
library that accelerates UDP Socket communications transparently and without
recompilation of the user application.
lustre: This directory contains patch needed to be applied to the lustre kernel. OFED: This directory contains supported OFED packages. RPM-Manager: This directory contains support scripts used for cluster deployment. scripts: Support scripts used by the Unified Wire Installer. specs: The packaging specification files required for building RPM packages. src: Source code for different drivers. support: This directory contains source files for the dialog utility. tools:
autoconf-2.63: This directory contains the source for autoconf v2.63 needed for WD-
UDP and iWARP libraries.
ba_server: Management and configuration tools for bypass adapters.  benchmarks: This directory contains various benchmarking tools to measure
throughput and latency of various networks.
chelsio_adapter_config: This directory contains scripts and binaries needed to
configure Chelsio 40G Adapters.
cop: The cop tool compiles offload policies into a simple program form that can be
loaded into the kernel and interpreted. These offload policies are used to determine the settings to be used for various connections. The connections to which the settings are applied are based on matching filter specifications. Please find more details on this tool in its manual page (run man cop command).
cudbg: Chelsio Unified Debug tool which facilitates collection and viewing of various
debug entities like register dump, Devlog, CIM LA, etc.
cxgbtool: The cxgbtool queries or sets various aspects of Chelsio network interface
cards. It complements standard tools used to configure network settings and provides functionality not available through such tools. Please find more details on this tool in its manual page (run man cxgbtool command).
To use cxbtool for FCoE Initiator driver, use [root@host~]# cxgbtool stor -h
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 16
rdma_tools: This directory contains iWARP benchmarking tools.  t4_sniffer: This directory contains sniffer tracing and filtering libraries. See WD
Sniffing and Tracing chapter for more information.
um: This directory contains Unified Wire Manager RPMs for different distributions
and Management Station configuration files.
90-rdma.rules: This file contains udev rules needed for running RDMA applications
as a non-root user.
chdebug: This script collects operating system environment details and debug
information which can be sent to the support team, to troubleshoot Chelsio hardware/software related issues.
chiscsi_set_affinity.sh: This shell script is used for mapping iSCSI Worker threads
to different CPUs.
chsetup: The chsetup tool loads NIC, TOE and iWARP drivers, and creates WD-
UDP configuration file.
chstatus: This utility provides status information on any Chelsio NIC in the system.  t4_latencytune.sh: Script used for latency tuning of Chelsio Adapters. t4_perftune.sh: This shell script is to tune the system for higher performance. It
achieves it through modifying the IRQ-CPU binding. This script can also be used to change Tx coalescing settings.
t4-forward.sh: RFC2544 Forward test tuning script.  uname_r: This file is used by chstatus script to verify if the Linux platform is
supported or not.
wdload: UDP acceleration tool.  wdunload: Used to unload all the loaded Chelsio drivers.
install.py, dialog.py: Python scripts needed for the GUI installer. EULA: Chelsio’s End User License Agreement install.log: File containing installation summary. Makefile: The Makefile for building and installing from the source. sample_machinefile: Sample file used during iWARP installation on cluster nodes. Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD.
The OptionROM directory contains Unified Boot Option ROM image (cubt4.bin), uEFI driver
(ChelsioUD.efi), default boot configuration file (bootcfg) and a legacy flash utility (cfut4.exe),
which can be used to flash the option ROM onto Chelsio's T5 and T4 based Converged
Network Adapters (CNAs).
The LinuxDUD directory contains image (.img) files required to update drivers for Linux
distributions.
1.4.2. RPM package
The Chelsio Unified Wire RPM package consists of the following:
config: This directory contains T5/T4 firmware configuration files.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 17
docs: This directory contains support documents i.e. README, Release Notes and User’s
Guide (this document) for the software.
DRIVER-RPMS: RPM packages of Chelsio drivers. OFED-RPMS: OFED RPM packages required to install iWARP driver. scripts: Support scripts used by the Unified Wire Installer. EULA: Chelsio’s End User License Agreement. install.py: Python script that installs the RPM package. See Software/Driver Installation
section for more information.
uninstall.py: Python script that uninstalls the RPM package. See Software/Driver
Uninstallation section for more information.
Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD.
The OptionROM directory contains Unified Boot Option ROM image (cubt4.bin), uEFI driver
(ChelsioUD.efi), default boot configuration file (bootcfg) and a legacy flash utility (cfut4.exe),
which can be used to flash the option ROM onto Chelsio's T5 and T4 based Converged
Network Adapters (CNAs).
The LinuxDUD directory contains image (.img) files required to update drivers for Linux
distributions.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 18
2. Hardware Installation
Follow these steps to install Chelsio Adapter in your system:
1. Shutdown/power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIe
x8/x16 slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 40/10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Run update-pciids command to download the current version of PCI ID list
[root@host~]# update-pciids
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 198k 100 198k 0 0 491k 0 --:--:-- --:--:-- --:--:-- 626k
Done.
13. Verify if the adapter was installed successfully by using the lspci command
a. For T5 adapters :
[root@host~]# lspci |grep -i Chelsio
07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.3 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified Wire Ethernet Controller 07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 19
Wire Ethernet Controller 07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR Unified Wire Storage Controller 07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire Storage Controller
b. And for T4 adapters :
[root@host~]# lspci | grep –i Chelsio 03:00.0 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller 03:00.1 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller 03:00.2 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller 03:00.3 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller 03:00.4 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller 03:00.5 SCSI storage controller: Chelsio Communications Inc T420-CR Unified Wire Storage Controller 03:00.6 Fibre Channel: Chelsio Communications Inc T420-CR Unified Wire Storage Controller 03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000
For Chelsio T5/T4 adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter Physical function 4: for all NIC functions of the adapter Physical function 5: for iSCSI Physical function 6: for FCoE Physical function 7: Currently not assigned
Once Unified Wire package is installed and loaded, examine the output of dmesg to see if the card is discovered.
For T5 adapters:
eth2: Chelsio T520-LL rev 1 1000/10GBASE-SFP RNIC MSI-X, Offload capable 0000:07:00.4: S/N: RE12130097, P/N: 11011675004
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 20
And, for T4 adapters:
eth0: Chelsio T420-CR rev 2 1000/10GBASE-SFP RNIC MSI-X, Offload capable 0000:04:00.4: S/N: PT18111226, P/N: 110112140D0
The above outputs indicate the hardware configuration of the adapters as well as the Serial numbers.
Network device names for Chelsio’s physical ports are assigned using the following convention: the port farthest from the motherboard will appear as the first network interface. However, for T5 40G and T420-BT adapters, the association of physical Ethernet ports and their corresponding network device names is opposite. For these adapters, the port nearest to the motherboard will appear as the first network interface.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 21
3. Software/Driver Installation
There are two main methods to install the Chelsio Unified Wire package: from source and RPM. If you decide to use source, you can install the package using CLI or GUI mode. If you decide to use RPM, you can install the package using Menu or CLI mode.
Irrespective of the method chosen for installation, the machine needs to be rebooted for changes to take effect.
The following table describes the various configuration tuning options available during installation and drivers/software installed with each option by default:
T5/T4
Configuration
Tuning Option
Description
Driver/Software installed
Unified Wire
Configures adapters to run multiple protocols
like NIC/TOE, iWARP, iSCSI and FCoE
Initiator simultaneously.
NIC/TOE, vNIC, iWARP, RBD, WD-UDP, iSCSI Target, iSCSI Initiator, DCB, FCoE
Target, FCoE Initiator, Bonding, MAFO,
IPv6, Sniffer & Tracer, UM (Agent, Client,
WebGUI), Filtering, TM, Lustre
Low latency
Networking
Configures adapters to run NIC/TOE and
iWARP traffic with low latency specially
needed for financial applications.
NIC/TOE, iWARP, RBD, WD-UDP, IPv6,
Sniffer & Tracer, Bonding, MAFO, UM (Agent, Client, WebGUI), Filtering, TM
High capacity
RDMA
Configures adapters to establish a large
number of RDMA connections.
NIC/TOE, iWARP, RBD, WD-UDP, Bonding,
MAFO, IPv6, Sniffer & Tracer, UM (Agent,
Client, WebGUI), Filtering, TM
RDMA
Performance
Improves RDMA performance on T5/T4
adapters.
NIC/TOE, iWARP, RBD
UM (Agent, Client, WebGUI)
High capacity
TOE
Configures adapters to establish a large
number of TOE connections.
NIC/TOE, Bonding, MAFO, IPv6, UM (Agent,
Client, WebGUI), Filtering,
TM
iSCSI
Performance*
Improves iSCSI performance on T5
adapters.
NIC/TOE, iSCSI Target, iSCSI Initiator,
Bonding, DCB,
UM (Agent, Client, WebGUI)
UDP Seg.Offload
& Pacing
Configures adapters to establish a large
number of UDP Segmentation Offload
connections.
NIC/TOE, IPv6, USO, Bonding, UM (Agent,
Client, WebGUI), Filtering, TM
T5 Wire Direct
Latency*
Configures T5 adapters to provide low Wire
Direct latency.
NIC/TOE, iWARP, RBD, WD-UDP, UM
(Agent, Client, WebGUI)
High Capacity WD
Configures adapters to establish a large
number of WD-UDP connections.
NIC/TOE, WD-UDP, UM (Agent, Client,
WebGUI)
T5 Hash Filter*
Configures T5 adapters to create more
filters.
NIC, Filtering, UM (Agent, Client, WebGUI)
* Supported only on T5 adapters.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 22
3.1. Pre-requisites
Depending on the component you choose to install, please ensure that the following requirements are met, before proceeding with the installation.
If you want to install OFED with NFS-RDMA support, please refer “Setting up NFS-RDMA”
in iWARP (RDMA) (Click here). If you’re planning to install iSCSI PDU Offload Initiator, please install openssl-devel
package. IPv6 should be enabled in the machine to use the RPM Packages.
3.2. Installing Chelsio Unified Wire from source
3.2.1. GUI mode (with Dialog utility)
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
ii. Untar the tarball using the following command:
[root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the GUI installer:
[root@host~]# ./install.py
iv. If Dialog utility is present, you can skip to step (v). If not, press ‘y’ to install it when the
installer prompts for input.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 23
v. Select “install” under “Choose an action”
vi. Select Enable IPv6-Offload to install drivers with IPv6 Offload support or Disable IPv6-
offload to continue installation without IPv6 offload support.
vii. Select the required T5/T4 configuration tuning option:
The tuning options may vary depending on the Linux distribution.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 24
viii. Under “Choose install components”, select “all” to install all the related components for the
option chosen in step (vii) or select “custom” to install specific components.
ix. Select the required performance tuning option.
a. Enable Binding IRQs to CPUs: Bind MSI-X interrupts to different CPUs and disable
IRQ balance daemon. b. Retain IRQ balance daemon: Do not disable IRQ balance daemon. c. TX-Coalasce: Write tx_coal=2 to modprobe.d/conf.
For more information on the Performance tuning options, please refer to
Performance Tuning section of the Network (NIC/TOE) chapter.
Note
To install benchmark tools, please select “custom option”. To install Bypass or FCoE PDU Offload Target drivers, please select
Unified Wire in step (vii). Then select “custom” option.
Important
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 25
x. If you already have the required version of OFED software installed, you can skip this step
by selecting Skip-OFED. To install OFED-3.18-1 choose the Install-OFED option. To install OFED-3.12-1, select Choose-OFED-Version and then OFED-3.12-1.
xi. The selected components will now be installed:
This step will be prompted only for OFED supported platforms.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 26
xii. After successful installation, summary of installed components will be displayed.
xiii. Select “View log” to view the installation log or “Exit” to continue.
xiv. Select “Yes” to exit the installer or “No” to go back.
xv. Reboot your machine for changes to take effect.
Press Esc or Ctrl+C to exit the installer at any point of time.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 27
3.2.1.1. Installation on updated kernels
If the kernel version on your Linux distribution is updated, follow the steps mentioned below to install the Unified Wire package:
i. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the GUI installer:
[root@host~]# ./install.py
ii. Select “Yes” to continue with the installation on the updated kernel or “No” to exit.
iii. Select the nearest supported kernel version from the list and select “OK”.
iv. Follow steps (v) to (xv) mentioned in the previous section.
3.2.2. CLI mode (without Dialog utility)
If your system does not have Dialog or you choose not to install it, follow the steps mentioned below to install the Unified Wire package:
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 28
ii. Untar the tarball using the following command:
[root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the installer:
[root@host~]# ./install.py
iv. When the installer prompts you for your input, press ‘n’ to continue installation without the
Dialog utility.
v. Enter the number corresponding to the Configuration tuning option in the Input field and
press Enter.
vi. If you already have the required version of OFED software installed, you can skip this step.
To install OFED-3.18-1 choose the Install-OFED option. To skip this step, select Skip- OFED.
vii. The selected components will now be installed.
After successful installation you can press 1 to view the installation log. Press any other key to exit from the installer.
viii. Reboot your machine for changes to take effect.
3.2.2.1. iWARP driver installation on Cluster nodes
Chelsio’s Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a
single command. Follow the procedure mentioned below: i. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in
the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed.
To install Bypass driver, run
[root@host~]# ./install.py -c <target> and follow steps (iv) - (vi)
mentioned above.
To customize the installation, view the help by typing
[root@host~]#./install.py –h
Important
Please make sure that you have enabled password less authentication with ssh on the peer nodes for this feature to work.
Important
This step will be prompted only for OFED supported platforms.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 29
ii. Now, execute the following command:
[root@host~]# ./install.py -C -m <machinefilename>
iii. Select the required T5/T4 configuration tuning option. The tuning options may vary
depending on the Linux distribution.
iv. Select the required Cluster Configuration.
v. If you already have the required version of OFED software installed, you can skip this step.
To install OFED-3.18-1 choose the Install-OFED option.To skip this step, select Skip-OFED.
vi. The selected components will now be installed. The above commands will install iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes
listed in the machinefilename file.
3.2.3. CLI mode
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
ii. Untar the tarball using the following command:
[root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and build
the source using :
[root@host~]# make
iv. Install the drivers, tools and libraries using the following command:
[root@host~]# make install
v. The default configuration tuning option is Unified Wire. The configuration tuning can be
selected using the following commands:
[root@host~]# make CONF=<T5/T4 configuration>
[root@host~]# make CONF=<T5/T4 configuration> install
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 30
vi. Reboot your machine for changes to take effect.
3.2.3.1. Installation on updated kernels
If the kernel version on your Linux distribution is updated, please execute the following command to install the Unified Wire package:
[root@host~]# make UNAME_R=<kernel_version>
Where kernel_version is the nearest supported kernel version. For example, if you want to install the package on a RHEL 6 distribution updated to 2.6.32-
431.20.3. el6 kernel, run the following commands:
[root@host~]# make UNAME_R=2.6.32-431.el6 [root@host~]# make UNAME_R=2.6.32-431.el6 install
To view the list of the supported kernel versions, run the following command:
[root@host~]# make list_kernels
Reboot your machine for changes to take effect.
3.2.4. CLI mode (individual drivers)
You can also choose to install drivers individually. Provided here are steps to build and install NIC, TOE, iWARP, RDM Block Device, Bypass, WD-UDP, UDP Segmentation Offload, FCoE PDU Offload target, DCB drivers and benchmarking tools. To know about other drivers, view help by running make help.
To view the different configuration tuning options, view help by typing
[root@host~]#make help
Note
Steps (iv) and (v) mentioned above will NOT install Bypass, FCoE PDU offload target, DCB drivers and benchmark tools.They will have to be installed manually.
Please refer to section CLI mode (individual drivers) for instructions on installing them.
Important
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 31
To build and install NIC driver without offload support :
[root@host~]# make nic [root@host~]# make nic_install
To build and install NIC driver with offload support and Offload drivers:
[root@host~]# make toe [root@host~]# make toe_install
To build and install Offload drivers without IPv6 support:
[root@host~]# make toe_ipv4 [root@host~]# make toe_ipv4_install
To build and install iWARP driver against outbox OFED:
[root@host~]# make iwarp [root@host~]# make iwarp_install
To build and install RDMA Block Device driver:
[root@host~]# make rdma_block_device [root@host~]# make rdma_block_device_install
To build and install all drivers without IPv6 support:
[root@host~]# make ipv6_disable=1 [root@host~]# make ipv6_disable=1 install
The above step will not install Bypass driver. Run the following commands to install it:
[root@host~]# make bypass [root@host~]# make bypass_install
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 32
To build and install all drivers with DCB support:
[root@host~]# make dcbx=1 [root@host~]# make dcbx=1 install
The offload drivers support UDP Segmentation Offload with limited number of connections
(1024 connections).To build and install UDP Offload drivers which support large number of offload connections (approx 10K):
[root@host~]# make udp_offload [root@host~]# make udp_offload_install
To build and install FCoE Target drivers:
[root@host~]# make fcoe_pdu_offload_target [root@host~]# make fcoe_pdu_offload_target_install
The default T5/T4 configuration tuning option is Unified Wire. The configuration tuning can
be selected using the following commands:
[root@host~]# make CONF=<T5/T4 configuration> <Build Target>
[root@host~]# make CONF=<T5/T4 configuration> <Install Target>
To build and install drivers along with benchmarks:
[root@host~]# make BENCHMARKS=1 [root@host~]# make BENCHMARKS=1 install
Unified Wire Manager will be installed by default. To skip the installation:
[root@host~]# make INSTALL_UM=0 install
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 33
The drivers will be installed as RPMs or Debian packages (for ubuntu). To skip this and
install drivers:
[root@host~]# make SKIP_RPM=1 install
3.3. Installing Chelsio Unified Wire from RPM
3.3.1. Menu Mode
i. Download the tarball specific to your operating system and architecture from Chelsio
Download Center, http://service.chelsio.com/
ii. Untar the tarball:
E.g. for RHEL 6.6, untar using the following command:
[root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x-RHEL6.6_x86_64.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and run
the following command:
[root@host~]# ./install.py
iv. Select the Installation type as described below. Enter the corresponding number in the Input
field and press Enter.
1. Unified Wire: Install all the drivers in the Unified Wire software package. This option will not install OFED and drivers built against OFED.
2. T5 Wire Direct Latency: Install Wire Direct Latency drivers needed for Low latency applications.
3. Custom: Customize the installation. Use this option to install drivers/software and related components (like OFED-3.18-1) according to the tuning option selected.
To view the different configuration tuning options, view the help by typing
[root@host~]#make help
Note
If IPv6 is administratively disabled in the machine, the drivers will be built and installed without IPv6 Offload support by default.
Note
Drivers installed from RPM Packages do not have DCB support.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 34
4. EXIT: Exit the installer.
v. The selected components will now be installed. vi. Reboot your machine for changes to take effect.
3.3.2. CLI mode
i. Download the tarball specific to your operating system and architecture from Chelsio
Download Center, http://service.chelsio.com/
ii. Untar the tarball:
E.g. For RHEL 6.6, untar using the following command:
[root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x-RHEL6.6_x86_64.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and install
Unified Wire using:
[root@host~]# ./install.py –i <nic_toe/all/bypass/udpso/wd>
nic_toe : NIC and TOE drivers only all : all Chelsio drivers built against inbox OFED bypass : bypass drivers and tools udpso : UDP segmentation offload capable NIC and TOE drivers only wd : Wire Direct drivers and libraries only
If the installation aborts with the message "Resolve the errors/dependencies manually and restart the installation", please go through the install.log to resolve errors/dependencies and then start the installation again.
Note
The Installation options may vary depending on the Configuration tuning option selected.
Note
The Installation options may vary depending on the Linux distribution.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 35
iv. The default configuration tuning option is Unified Wire. The configuration tuning can be
selected using the following command:
[root@host~]# ./install.py –i <Installation mode> -c <Configuration>
v. To install OFED and Chelsio drivers built against OFED, run the above command with -o
option.
[root@host~]# ./install.py –i <Installation mode> -c <Configuration> -o
vi. Reboot your machine for changes to take effect.
3.3.2.1. iWARP driver installation on cluster nodes
i. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in
the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed.
ii. Navigate to ChelsioUwire directory and execute the following command:
[root@host~]# ./install.py -C -m <machinefilename> -i
<nic_toe/all/bypass/udpso/wd> -c <T5/T4 configuration> -o
Here, -o parameter will install OFED and Chelsio drivers built against OFED. The above command will install iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes
listed in the <machinefilename> file. iii. Reboot your machine for changes to take effect.
3.4. Firmware update
The T5 and T4 firmwares are installed on the system, typically in /lib/firmware/cxgb4, and the driver will auto-load the firmwares if an update is required. The kernel must be configured to enable userspace firmware loading support:
Device Drivers -> Generic Driver Options -> Userspace firmware loading support
Note
To view the different configuration tuning options, view the help by typing
[root@host~]# ./install.py –h
Please make sure that you have enabled password less authentication with ssh on the peer nodes for this feature to work.
Important
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 36
The firmware version can be verified using ethtool:
[root@host~]# ethtool -i <iface>
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 37
4. Software/Driver Uninstallation
Similar to installation, the Chelsio Unified Wire package can be uninstalled using two main methods: from the source and RPM, based on the method used for installation. If you decide to use source, you can uninstall the package using CLI or GUI mode.
4.1. Uninstalling Chelsio Unified Wire from source
4.1.1. GUI mode (with Dialog utility)
i. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the GUI installer:
[root@host~]# ./install.py
ii. Select “uninstall” , Under “Choose an action”
iii. Select “all” to uninstall all the installed drivers, libraries and tools or select “custom” to
remove specific components.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 38
iv. The selected components will now be uninstalled.
v. After successful uninstalltion, summary of the uninstalled components will be displayed.
vi. Select “View log” to view uninstallation log or “Exit” to continue.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 39
vii. Select “Yes” to exit the installer or “No” to go back.
4.1.2. CLI mode (without Dialog utility)
Run the following script with –u option to uninstall the Unified Wire Package:
[root@host~]# ./install.py –u <target>
4.1.3. CLI mode
Change your current working directory to Chelsio Unified Wire package directory and uninstall using the following command:
[root@host~]# make uninstall
4.1.3.1. iWARP driver uninstallation on Cluster nodes
To uninstal iWARP drivers on multiple Cluster nodes with a single command, run the following command:
[root@host~]# ./install.py -C -m <machinefilename> -u all
Press Esc or Ctrl+C to exit the installer at any point of time.
Note
View help by typing [root@host~]# ./install.py –h for more information
Note
Uninstalling Unified Wire package will not uninstall Unified Wire Manager. Refer to the next section, CLI mode (individual drivers) to remove the software manually.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 40
The above command will remove Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file.
4.1.4. CLI mode (individual drivers/software)
You can also choose to uninstall drivers/software individually. Provided here are steps to uninstall NIC, TOE, iWARP, RDMA Block Device, Bypass, UDP Segmentation Offload, FCoE PDU Offload target drivers and Unified Wire Manager (UM). To know about other drivers, access help by running make help
To uninstall NIC driver :
[root@host~]# make nic_uninstall
To uninstall offload driver:
[root@host~]# make toe_uninstall
To uninstall iWARP driver:
[root@host~]# make iwarp_uninstall
To uninstall RDMA Block Device driver:
[root@host~]# make rdma_block_device_uninstall
To uninstall Bypass driver:
[root@host~]# make bypass_uninstall
To uninstall UDP Segmentation Offload driver:
[root@host~]# make udp_offload_uninstall
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 41
To uninstall FCoE Target driver:
[root@host~]# make fcoe_pdu_offload_target_uninstall
To uninstall Unified Wire Manager (UM):
[root@host~]# make uninstall UM_UNINST=1
OR
[root@host~]# make tools_uninstall UM_UNINST=1
4.2. Uninstalling Chelsio Unified Wire from RPM
Change your current working directory to Chelsio Unified Wire package directory and and run the following command:
[root@host~]# ./uninstall.py <inbox/ofed>
inbox : for removing all Chelsio drivers. ofed : for removing OFED and Chelsio drivers.
Uninstalling Unified Wire package will not uninstall Unified Wire Manager. Refer to the Unified Wire Manager (UM) chapter to remove the software manually (Click here).
Note
The uninstallation options may vary depending on Linux distribution. View help by typing [root@host~]# ./uninstall.py –h for more information.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 42
4.2.1.1. iWARP driver uninstallation on Cluster nodes
To uninstal iWARP drivers on multiple Cluster nodes with a single command, run the following:
[root@host~]# ./install.py -C -m <machinefilename> -u
The above command will remove Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 43
5. Configuring Chelsio Network Interfaces
In order to test Chelsio adapters’ features it is required to use two machines both with Chelsio’s (T5, T4 or both) network adapters installed. These two machines can be connected directly without a switch (back-to-back), or both connected to a switch. The interfaces have to be declared and configured. The configuration files for network interfaces on Red Hat Enterprise Linux (RHEL) distributions are kept under /etc/sysconfig/network-scripts.
5.1. Configuring 40G adapters
Chelsio T5 40G adapters can be configured in the following three modes: i. 2X40Gbps: This is the default mode of operation where each port functions as 40Gbps link.
The port nearest to the motherboard will appear as the first network interface (Port 0). ii. 4X10Gbps: In this mode, port 0 functions as 4 10Gbps links and port 1 is disabled. iii. QSA: This mode adds support for QSA (QSFP to SFP+) modules, enabling smooth, cost-
effective, connections between 40 Gigabit Ethernet adapters and 1 or 10 Gigabit Ethernet
networks using existing SFP+ based cabling. The port farthest from the motherboard will
appear as the first network interface (Port 0).
To configure/change the mode of operation, use the following procedure: i. Unload all Chelsio drivers using the rmmod command:
[root@host~]# rmmod <chelsio_driver>
Some operating systems may attempt to auto-configure the detected hardware and some may not detect all ports on a multi-port adapter. If this happens, please refer to the operating system documentation for manually configuring the network device.
Note
This is an alpha release of QSA mode.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 44
ii. Run the chelsio_adapter_config command to detect all T5 40G adapter(s) present in the
system.
[root@host~]# chelsio_adapter_config Chelsio T580 card detected Chelsio T580 PCI devices :
|------------------------------|
| 1 T580-LP-CR 01:00.0 |
| 2 T580-CR 03:00.0 |
| 3 T580-LP-SO-CR 04:00.0 |
|------------------------------|
iii. Select the adapter to configure by specifying the adapter index.
iv. Select the required mode:
Possible T580 adapter modes: |------------------------------|
| 1: 2x40G |
| 2: 4x10G |
| 3: QSA |
|------------------------------|
Select mode for adapter (1,2,3):
v. Reload the network driver for changes to take effect.
[root@host~]# rmmod cxgb4 [root@host~]# modprobe cxgb4
In case of T580-SO-CR adapters, reboot the machine for changes to take effect.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 45
5.2. Configuring network-scripts
A typical interface network-script (e.g. eth0) on RHEL 6.X looks like the following:
# file: /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
HWADDR=00:30:48:32:6A:AA
ONBOOT="yes"
NM_CONTROLLED="no"
BOOTPROTO="static"
IPADDR=10.192.167.111
NETMASK=255.255.240.0
In the case of DHCP addressing the last two lines should be removed and
BOOTPROTO="static" should be changed to BOOTPROTO="dhcp"
The ifcfg-ethX files have to be created manually. They are required for bringing the interfaces up and down and attribute the desired IP addresses.
5.3. Creating network-scripts
To spot the new interfaces, make sure the driver is unloaded first. To that point ifconfig -a |
grep HWaddr should display all non-chelsio interfaces whose drivers are loaded, whether the
interfaces are up or not.
[root@host~]# ifconfig -a | grep HWaddr eth0 Link encap:Ethernet HWaddr 00:30:48:32:6A:AA
Then load the driver using the modprobe cxgb4 command (for the moment it does not make any difference whether we are using NIC-only or the TOE-enabling driver). The output of ifconfig should display the T5/T4 interfaces as:
[root@host~]# ifconfig -a | grep HWaddr eth0 Link encap:Ethernet HWaddr 00:30:48:32:6A:AA eth1 Link encap:Ethernet HWaddr 00:07:43:04:6B:E9
eth2 Link encap:Ethernet HWaddr 00:07:43:04:6B:F1
On earlier versions of RHEL the NETMASK attribute is named IPMASK. Make sure you are using the right attribute name.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 46
For each interface you can write a configuration file in /etc/sysconfig/network-scripts. The ifcfg-eth1 could look like:
# file: /etc/sysconfig/network-scripts/ifcfg-eth1 DEVICE="eth1" HWADDR=00:07:43:04:6B:E9 ONBOOT="no" NM_CONTROLLED="no" BOOTPROTO="static" IPADDR=10.192.167.112 NETMASK=255.255.240.0
From now on, the eth1 interface of the adapter can be brought up and down through the ifup
eth1 and ifdown eth1 commands respectively. Note that it is of course not compulsory to
create a configuration file for every interface if you are not planning to use them all.
5.4. Checking Link
Once the network-scripts are created for the interfaces you should check the link i.e. make sure it is actually connected to the network. First, bring up the interface you want to test using
ifup eth1.
You should now be able to ping any other machine from your network provided it has ping response enabled.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 47
6. Performance Tuning
In order to auto tune the system for best performance, Chelsio recommends: Disabling virtualization, c-state technology, VT-d, Intel I/O AT and SR-IOV in the BIOS
settings
Installing the adapter into a PCIe Gen3 x8/x16 slot. Installing the tools which will copy t4_perftune.sh script to /sbin directory. Run the script
to map the adapter queues to different CPUs:
[root@host~]# t4_perftune.sh
Also, follow the steps mentioned below to lower your latency:
i. Disable SELinux ii. Run the following script to disable few services.
[root@host~]# t4_latencytune.sh <interface>
iii. Set sysctl param net.ipv4.tcp_low_latency to 1
[root@host~]# sysctl -w net.ipv4.tcp_low_latency=1
To optimize your system for different protocols, please refer to their respective chapters.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 48
7. Software/Driver Update
For any distribution specific problems, please check README and Release Notes included in the release for possible workaround.
Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various software/drivers. You can also subscribe to our newsletter for the latest software updates.
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 49
II. Network (NIC/TOE)
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 50
1. Introduction
Chelsio’s T5 and T4 series of Unified Wire Adapters provide extensive support for NIC operation, including all stateless offload mechanisms for both IPv4 and IPv6 (IP, TCP and UDP checksum offload, LSO - Large Send Offload aka TSO - TCP Segmentation Offload, and assist mechanisms for accelerating LRO - Large Receive Offload).
A high performance fully offloaded and fully featured TCP/IP stack meets or exceeds software implementations in RFC compliance. Chelsio’s T5/T4 engine provides unparalleled performance through a specialized data flow processor implementation and a host of features designed for high throughput and low latency in demanding conditions and networking environments.
TCP offload is fully implemented in the hardware, thus freeing the CPU from TCP/IP overhead. The freed CPU can be used for any computing needs. The TCP offload in turn removes network bottlenecks and enables applications to take full advantage of the networking capabilities.
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the currently shipping Chelsio Adapters that are compatible with Chelsio Network driver:
T580-OCP-SO* T520-OCP-SO* T520-BT T580-CR T580-SO-CR* T580-LP-CR T520-LL-CR T520-SO-CR* T520-CR T540-CR T420-CR T440-CR T422-CR T420-SO-CR T404-BT T420-BCH T440-LP-CR T420-BT T420-LL-CR T420-CX
*Only NIC driver supported
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 51
1.2. Software Requirements
1.2.1. Linux Requirements
Currently the Network driver is available for the following versions:
RHEL 7.2, 3.10.0-327.el7 * RHEL 7.1, 3.10.0-229.el7 RHEL 7.1 BE, 3.10.0-229.el7.ppc64 (POWER7) RHEL 7.1 LE, 3.10.0-229.ael7b.ppc64le (POWER8) RHEL 6.7, 2.6.32-573.el6* RHEL 6.6, 2.6.32-504.el6* RHEL 6.5, 2.6.32-431.el6* SLES 12 SP1, 3.12.49-11-default * SLES 12, 3.12.28-4-default* SLES 11 SP4, 3.0.101-63-default * SLES 11 SP3, 3.0.76-0.11-default Ubuntu 14.04.3, 3.19.0-25-generic * Ubuntu 14.04.2, 3.16.0-30-generic * Kernel.org linux-4.1 * Kernel.org linux-3.18 * Kernel.org linux-3.17
Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. * Limited QA performed.
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 52
2. Software/Driver Loading
The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.
2.1. Loading in NIC mode (without full offload support)
To load the Network driver without full offload support, run the following command:
[root@host~]# modprobe cxgb4
2.2. Loading in TOE mode (with full offload support)
To enable full offload support, run the following command:
[root@host~]# modprobe t4_tom
In VMDirect Path environment, it is recommended to load the offload driver using the following command:
[root@host~]# modprobe t4_tom vmdirectio=1
Offload support needs to be enabled upon each reboot of the system. This can be done manually as shown above.
Note
Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers.
Important
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 53
3. Software/Driver Unloading
3.1. Unloading the NIC driver
To unload the NIC driver, run the following command:
[root@host~]# rmmod cxgb4
3.2. Unloading the TOE driver
A reboot is required to unload the TOE driver. To avoid rebooting, follow the steps mentioned below:
i. Load t4_tom driver with unsupported_allow_unload parameter.
[root@host~]# modprobe t4_tom unsupported_allow_unload=1
ii. Stop all the offloaded traffic, servers and connections. Check for the reference count.
[root@host~]# cat /sys/module/t4_tom/refcnt
If the reference count is 0, the driver can be directly unloaded. Skip to step (iii) If the count is non-zero, load a COP policy which disables offload using the following procedure:
a. Create a policy file which will disable offload
[root@host~]# cat policy_file all => !offload
b. Compile and apply the output policy file
[root@host~]# cop –o no-offload.cop policy_file [root@host~]# cxgbtool ethX policy no-offload.cop
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 54
iii. Unload the driver:
[root@host~]# rmmod t4_tom [root@host~]# rmmod toecore [root@host~]# rmmod cxgb4
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 55
4. Software/Driver Configuration and Fine-tuning
4.1. Instantiate Virtual Functions (SR-IOV)
To instantiate the Virtual functions, load the cxgb4 driver with num_vf parameter with a non-zero value. For example:
[root@host~]# modprobe cxgb4 num_vf=1,0,0,0
The number(s) provided for num_vf parameter specifies the number of Virtual Functions to be instantiated per Physical Function. The Virtual Functions can be assigned to Virtual Machines (Guests). A maximum of 64 Virtual Functions can be instantiated with 16 Virtual Functions per Physical Function. Loading the cxgb4 driver with num_vf parameter loads the cxgb4vf module (the driver for Virtual Functions) in the host by default. Hence unload the cxgb4vf module (on the host) before assigning Virtual Functions to the Virtual Machines (Guests), using the following command:
[root@host~]# rmmod cxgb4vf
4.2. Enabling Busy waiting
Busy waiting/polling is a technique where a process repeatedly checks to see if an event has occurred, by spinning in a tight loop. By making use of similar technique, Linux kernel provides the ability for the socket layer code to poll directly on an Ethernet device's Rx queue. This eliminates the cost of interrupts and context switching, and with proper tuning allows to achieve latency performance similar to that of hardware.
Chelsio's NIC and TOE drivers support this feature and can be enabled on Chelsio supported devices to attain improved latency.
To make use of BUSY_POLL feature, follow the steps mentioned below: i. Enable BUSY_POLL support in kernel config file by setting CONFIG_NET_RX_BUSY_POLL=y
ii. Enable BUSY_POLL globally in the system by setting the values of following sysctl
parameters depending on the number of connections:
To get familiar with physical and virtual function terminologies, please refer the PCI Express specification.
Note
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 56
sysctl -w net.core.busy_read=<value>
sysctl -w net.core.busy_poll=<value>
Set the values of the above parameters to 50 for 100 or less connections; and 100 for more than 100 connections.
4.3. Performance Tuning
Receiver Side Scaling (RSS) Receiver Side Scaling enables the receiving network traffic to scale with the available number of
processors on a modern networked computer. RSS enables parallel receive processing and dynamically balances the load among multiple processors. Chelsio’s T5/T4 network controller fully supports Receiver Side Scaling for IPv4 and IPv6.
This script first determines the number of CPUs on the system and then each receiving queue is bound to an entry in the system interrupt table and assigned to a specific CPU. Thus, each receiving queue interrupts a specific CPU through a specific interrupt now. For example, on a 4­core system, t4_perftune.sh gives the following output:
[root@host~]# t4_perftune.sh
Discovering Chelsio T4/T5 devices ... Configuring Chelsio T4/T5 devices ... Tuning eth7 IRQ table length 4 Writing 1 in /proc/irq/62/smp_affinity Writing 2 in /proc/irq/63/smp_affinity Writing 4 in /proc/irq/64/smp_affinity Writing 8 in /proc/irq/65/smp_affinity
eth7 now up and tuned ...
Because there are 4 CPUs on the system, 4 entries of interrupts are assigned. For other T5/T4 network interfaces, you should see similar output message.
Now the receiving traffic is dynamically assigned to one of the system’s CPUs through a T5/T4 queue. This achieves a balanced usage among all the processors. This can be verified, for example, by using the iperf tool. First set up a server on the receiver host:
BUSY_POLL can also be enabled on a per-connection basis by making use of
SO_BUSY_POLL option in the socket application code. Refer socket man-page for more details.
Note
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 57
[root@receiver_host~]# iperf –s
Then on the sender host, send data to the server using the iperf client mode. To emulate a moderate traffic workload, use -P option to request 20 TCP streams from the server:
[root@sender_host~]# iperf -c receiver_host_name_or_IP -P 20
Then on the receiver host, look at interrupt rate at /proc/interrupts:
[root@receiver_host~]# cat /proc/interrupts | grep eth6
Id CPU0 CPU1 CPU2 CPU3 type interface
36: 115229 0 0 1 PCI-MSI-edge eth6 (queue 0)
37: 0 121083 1 0 PCI-MSI-edge eth6 (queue 1)
38: 0 0 105423 1 PCI-MSI-edge eth6 (queue 2)
39: 0 0 0 115724 PCI-MSI-edge eth6 (queue 3)
Now interrupts from eth6 are evenly distributed among the 4 CPUs.
Without T5/T4’s RSS support, the interrupts caused by network traffic may be distributed
unevenly over CPUs. For your information, the traffic produced by the same iperf commands gives the following output in /proc/interrupts.
[root@receiver_host~]# cat /proc/interrupts | grep eth6
Id CPU0 CPU1 CPU2 CPU3 type interface
36: 0 9 0 17418 PCI-MSI-edge eth6 (queue 0)
37: 0 0 21718 2063 PCI-MSI-edge eth6 (queue 1)
38: 0 7 391519 222 PCI-MSI-edge eth6 (queue 2)
39: 1 0 33 17798 PCI-MSI-edge eth6 (queue 3)
Here there are 4 receiving queues from the eth6 interface, but they are not bound to a specific CPU or interrupt entry. Queue 2 has caused a very large number of interrupts on CPU2 while CPU0 and CPU1 are barely used by any of the four queues. Enabling RSS is thus essential for best performance.
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 58
Interrupt Coalescing The idea behind Interrupt Coalescing (IC) is to avoid flooding the host CPUs with too many
interrupts. Instead of throwing one interrupt per incoming packet, IC waits for ‘n’ packets to be
available in the Rx queues and placed into the host memory through DMA operations before an interrupt is thrown, reducing the CPU load and thus improving latency. It can be changed using the following command:
[root@host~]# ethtool –C ethX rx-frames n
Configuring sysctl, adaptive interrupts, select_queue (NIC)
i. Turn off irqbalance
[root@host~]# /etc/init.d/irqbalance stop
ii. Add the following sysctl parameters to /etc/sysctl.conf
sysctl -w net.ipv4.tcp_timestamps=0 sysctl -w net.ipv4.tcp_low_latency=1 sysctl -w net.core.netdev_max_backlog=250000 sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216 sysctl -w net.core.rmem_default=16777216 sysctl -w net.core.wmem_default=16777216 sysctl -w net.core.optmem_max=16777216 sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216' sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'
Linux’s irqbalance may take charge of distributing interrupts among CPUs on a multiprocessor platform. However, irqbalance distributes interrupt requests from all hardware devices across processors. For a server with T5/T4 network card constantly receiving large volume of data at 40/10Gbps, the network interrupt demands are significantly high. Under such circumstances, it is necessary to enable RSS to balance the network load across multiple processors and achieve the best performance.
Note
For more information, run the following command:
[root@host~]# ethtool -h
Note
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 59
iii. Bring up the network interfaces and run the following command:
[root@host~]# ethtool -C ethXX adaptive-rx on
Read back the ethtool settings with the following command:
[root@host~]# ethtool -c ethXX
Output should show adaptive-rx as on. iv. Change select_queue parameter’s value to 1:
[root@host~]# cat /sys/module/cxgb4/parameters/select_queue 0 [root@host~]# echo 1 > /sys/module/cxgb4/parameters/select_queue [root@host~]# cat /sys/module/cxgb4/parameters/select_queue 1
For TOE performance, follow the first two steps mentioned above and then set the following sysctl parameter:
[root@host~]# sysctl -w toe.toe0_tom.delayed_ack=3
Large Receive Offload / Generic Receive Offload Large Receive Offload or Generic Receive Offload is a performance improvement feature at the
receiving side. LRO/GRO aggregates the received packets that belong to same stream, and combines them to form a larger packet before pushing them to the receive host network stack. By doing this, rather than processing every small packet, the receiver CPU works on fewer packet headers but with same amount of data. This helps reduce the receive host CPU load and improve throughput in a 40/10Gb network environment where CPU can be the bottleneck.
LRO and GRO are different names to refer to the same receiver packets aggregating feature. LRO and GRO actually differ in their implementation of the feature in the Linux kernel. The feature was first added into the Linux kernel in version 2.6.24 and named Large Receive Offload (LRO). However LRO only works for TCP and IPv4. As from kernel 2.6.29, a new protocol­independent implementation removing the limitation is added to Linux, and it is named Generic Receive Offload (GRO). The old LRO code is still available in the kernel sources but whenever both GRO and LRO are presented GRO is always the preferred one to use.
Please note that if your Linux system has IP forwarding enabled, i.e. acting as a bridge or router, the LRO needs to be disabled. This is due to a known kernel issue.
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 60
Chelsio’s T5/T4 card supports both hardware assisted GRO/LRO and Linux-based GRO/LRO.
t4_tom is the kernel module that enables the hardware assisted GRO/LRO. If it is not already in
the kernel module list, use the following command to insert it:
[root@host~]# lsmod | grep t4_tom [root@host~]# modprobe t4_tom
[root@host~]# lsmod | grep t4_tom t4_tom 88378 0 [permanent] toecore 21618 1 t4_tom cxgb4 225342 1 t4_tom
Then T5/T4’s hardware GRO/LRO implementation is enabled. If you would like to use the Linux GRO/LRO for any reason, first the t4_tom kernel module
needs to be removed from kernel module list. Please note you might need to reboot your system.
After removing the t4_tom module, you can use ethtool to check the status of current GRO/LRO settings, for example:
[root@host~]# ethtool -k eth6 Offload parameters for eth6: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off
Now the generic-receive-offload option is on. This means GRO is enabled. Please note that there are two offload options here: generic-receive-offload and large-receive-
offload. This is because on this Linux system (RHEL6.0), the kernel supports both GRO and
LRO. As mentioned earlier, GRO is always the preferred option when both of them are present. On other systems LRO might be the only available option. Then ethtool could be used to switch LRO on and off as well.
Chapter II. Network (NIC/TOE)
Chelsio T5/T4 Unified Wire for Linux Page 61
When Linux’s GRO is enabled, Chelsio’s T5/T4 driver provides two GRO-related statistics. They are displayed using the following command:
[root@host~]# ethtool -S eth6 ...
GROPackets : 0 GROMerged : 897723 ...
GROPackets is the number of held packets. Those are candidate packets held by the kernel to
be processed individually or to be merged to larger packets. This number is usually zero.
GROMerged is the number of packets that merged to larger packets. Usually this number
increases if there is any continuous traffic stream present.
ethtool can also be used to switch off the GRO/LRO options when necessary:
[root@host~]# ethtool -K eth6 gro off [root@host~]# ethtool -k eth6 Offload parameters for eth6: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off
The output above shows a disabled GRO.
Chapter III. Virtual Function Network (vNIC)
Chelsio T5/T4 Unified Wire for Linux Page 62
III. Virtual Function Network (vNIC)
Chapter III. Virtual Function Network (vNIC)
Chelsio T5/T4 Unified Wire for Linux Page 63
1. Introduction
The ever increasing network infrastructure of IT enterprises has lead to a phenomenal increase in maintenance and operational costs. IT managers are forced to acquire more physical servers and other data center resources to satisfy storage and network demands. To solve the Network and I/O overhead, users are opting for server virtualization which consolidates I/O workloads onto lesser physical servers thus resulting in efficient, dynamic and economical data center environments. Other benefits of Virtualization include improved disaster recovery, server portability, cloud computing, Virtual Desktop Infrastructure (VDI), etc.
Chelsio’s T5 and T4 Unified Wire family of Adapters deliver increased bandwidth, lower latency and lower power with virtualization features to maximize cloud scaling and utilization. The adapters also provide full support for PCI-SIG SR-IOV to improve I/O performance on a virtualized system. User can configure up to 64 Virtual and 8 Physical functions (with 4 PFs as SR-IOV capable) along with 336 virtual MAC addresses.
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the currently shipping Chelsio Adapters that are compatible with the Chelsio vNIC driver:
T520-BT
T580-CR
T520-LL-CR
T520-CR
T580-LP-CR
T540-CR
T420-CR
T440-CR
T422-CR
T420-SO-CR
T404-BT
T440-LP-CR
T420-BT
T420-LL-CR
T420-CX
Chapter III. Virtual Function Network (vNIC)
Chelsio T5/T4 Unified Wire for Linux Page 64
1.2. Software Requirements
1.2.1. Linux Requirements
Currently the vNIC driver is available for the following versions:
RHEL 7.2, 3.10.0-327.el7 *
RHEL 7.1, 3.10.0-229.el7
RHEL 6.7, 2.6.32-573.el6*
RHEL 6.6, 2.6.32-504.el6*
RHEL 6.5, 2.6.32-431.el6*
SLES 12 SP1, 3.12.49-11-default *
SLES 12, 3.12.28-4-default*
SLES 11 SP4, 3.0.101-63-default *
SLES 11 SP3, 3.0.76-0.11-default
Ubuntu 14.04.3, 3.19.0-25-generic *
Ubuntu 14.04.2, 3.16.0-30-generic *
Kernel.org linux-4.1 *
Kernel.org linux-3.18 *
Kernel.org linux-3.17
Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work.
* Limited QA performed.
Chapter III. Virtual Function Network (vNIC)
Chelsio T5/T4 Unified Wire for Linux Page 65
2. Software/Driver Loading
2.1. Instantiate Virtual Functions
To instantiate Chelsio Virtual Functions, please refer the Network (NIC/TOE) section (click
here)
2.2. Loading the driver
The vNIC driver must be loaded on the Guest OS by the root user. Any attempt to load the driver as a regular user will fail.
To load the driver, run the following command:
[root@host~]# modprobe cxgb4vf
Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers.
Important
Chapter III. Virtual Function Network (vNIC)
Chelsio T5/T4 Unified Wire for Linux Page 66
3. Software/Driver Unloading
3.1. Unloading the driver
The vNIC driver must be unloaded on the Guest OS by the root user. Any attempt to unload the driver as a regular user will fail.
To unload the driver, execute the following command:
[root@host~]# rmmod cxgb4vf
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 67
IV. iWARP (RDMA)
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 68
1. Introduction
Chelsio’s T5/T4 engine implements a feature rich RDMA implementation which adheres to the IETF standards with optional markers and MPA CRC-32C.
The iWARP RDMA operation benefits from the virtualization, traffic management and QoS mechanisms provided by T5/T4 engine. It is possible to ACL process iWARP RDMA packets. It is also possible to rate control the iWARP traffic on a per-connection or per-class basis, and to give higher priority to QPs that implement distributed locking mechanisms. The iWARP operation also benefits from the high performance and low latency TCP implementation in the offload engine.
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the currently shipping Chelsio Adapters that are compatible with Chelsio iWARP driver:
T520-BT
T580-CR
T520-LL-CR
T520-CR
T580-LP-CR
T540-CR
T420-CR
T440-CR
T422-CR
T404-BT
T440-LP-CR
T420-LL-CR
T420-CX
1.2. Software Requirements
1.2.1. Linux Requirements
Currently the iWARP driver is available for the following versions:
RHEL 7.2, 3.10.0-327.el7 *
RHEL 7.1, 3.10.0-229.el7
RHEL 7.1 BE, 3.10.0-229.el7.ppc64 (POWER7)
RHEL 7.1 LE, 3.10.0-229.ael7b.ppc64le (POWER8)
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 69
RHEL 6.7, 2.6.32-573.el6*
RHEL 6.6, 2.6.32-504.el6*
RHEL 6.5, 2.6.32-431.el6*
SLES 12 SP1, 3.12.49-11-default *
SLES 12, 3.12.28-4-default*
SLES 11 SP4, 3.0.101-63-default *
SLES 11 SP3, 3.0.76-0.11-default
Ubuntu 14.04.3, 3.19.0-25-generic*
Ubuntu 14.04.2, 3.16.0-30-generic *
Kernel.org linux-4.1 *
Kernel.org linux-3.18 *
Kernel.org linux-3.17
Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work *Limited QA performed
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 70
2. Software/Driver Loading
2.1. Loading iWARP driver
The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.
To load the iWARP driver we need to load the NIC driver and core RDMA drivers first. Run the following commands:
[root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4
[root@host~]# modprobe rdma_ucm
Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers.
Important
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 71
3. Software/Driver Unloading
To unload the iWARP driver, run the following command:
[root@host~]# rmmod iw_cxgb4
openmpi-1.4.3 can cause IMB benchmark stalls due to a shared memory BTL issue. This issue is fixed in openmpi-1.4.5 and later releases. Hence, it is recommended that you download and install the latest stable release from Open MPI's official website, http://www.open-mpi.org
Important
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 72
4. Software/Driver Configuration and Fine-tuning
4.1. Testing connectivity with ping and rping
Load the NIC, iWARP & core RDMA modules as mentioned in Software/Driver Loading section. After which, you will see two or four ethernet interfaces for the T5/T4 device. Configure them with an appropriate ip address, netmask, etc. You can use the Linux ping command to test basic connectivity via the T5/T4 interface. To test RDMA, use the rping command that is included in the librdmacm-utils RPM:
Run the following command on the server machine:
[root@host~]# rping -s -a server_ip_addr -p 9999
Run the following command on the client machine:
[root@host~]# rping -c –Vv -C10 -a server_ip_addr -p 9999
You should see ping data like this on the client:
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA client DISCONNECT EVENT... #
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 73
4.2. Enabling various MPIs
4.2.1. Setting shell for Remote Login
User needs to set up authentication on the user account on all systems in the cluster to allow user to remotely logon or executing commands without password.
Quick steps to set up user authentication: i. Change to user home directory
[root@host~]# cd
ii. Generate authentication key
[root@host~]# ssh-keygen -t rsa
iii. Hit [Enter] upon prompting to accept default setup and empty password phrase iv. Create authorization file
[root@host~]# cd .ssh [root@host~]# cat *.pub > authorized_keys [root@host~]# chmod 600 authorized_keys
v. Copy directory .ssh to all systems in the cluster
[root@host~]# cd [root@host~]# scp -r /root/.ssh remotehostname-or-ipaddress:
4.2.2. Configuration of various MPIs (Installation and Setup)
Intel-MPI
i. Download latest Intel MPI from the Intel website ii. Copy the license file (.lic file) into l_mpi_p_x.y.z directory iii. Create machines.LINUX (list of node names) in l_mpi_p_x.y.z iv. Select advanced options during installation and register the MPI. v. Install software on every node.
[root@host~]# ./install.py
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 74
vi. Set IntelMPI with mpi-selector (do this on all nodes).
[root@host~]# mpi-selector --register intelmpi --source-dir /opt/intel/impi/3.1/bin/ [root@host~]# mpi-selector --set intelmpi
vii. Edit .bashrc and add these lines:
export RSH=ssh export DAPL_MAX_INLINE=64 export I_MPI_DEVICE=rdssm:chelsio export MPIEXEC_TIMEOUT=180 export MPI_BIT_MODE=64
viii. Logout & log back in. ix. Populate mpd.hosts with node names.
x. Contact Intel for obtaining their MPI with DAPL support. xi. To run Intel MPI over RDMA interface, DAPL 2.0 should be set up as follows:
Enable the Chelsio device by adding an entry at the beginning of the /etc/dat.conf file for the Chelsio interface. For instance, if your Chelsio interface name is eth2, then the following line adds a DAT version 2.0 device named chelsio2" for that interface:
chelsio2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
Open MPI (Installation and Setup)
Open MPI iWARP support is only available in Open MPI version 1.3 or greater. Open MPI will work without any specific configuration via the openib btl. Users wishing to
performance tune the configurable options may wish to inspect the receive queue values. Those can be found in the "Chelsio T4" section of mca-btl-openib-device-params.ini. Follow the steps mentioned below to install and configure Open MPI.
i. If not alreay done, install mpi-selector tool.
The hosts in this file should be Chelsio interface IP addresses. I_MPI_DEVICE=rdssm:chelsio assumes you have an entry in
/etc/dat.conf named chelsio.
MPIEXEC_TIMEOUT value might be required to increase if heavy traffic is
going across the systems.
Note
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 75
ii. Download the latest stable/feature version of openMPI from OpenMPI website
,http://www.open-mpi.org iii. Untar and change your current working directory to openMPI package directory. iv. Configure and install as:
[root@host~]#./configure --with-openib=/usr CC=gcc CXX=g++ F77=gfortran FC=gfortran --enable-mpirun-prefix-by-default --prefix=/usr/mpi/gcc/openmpi­x.y.z/ --with-openib-libdir=/usr/lib64/ --libdir=/usr/mpi/gcc/openmpi­x.y.z/lib64/ --with-contrib-vt-flags=--disable-iotrace [root@host~]# make [root@host~]# make install
The above step will install openMPI in /usr/mpi/gcc/openmpi-x.y.z/
v. Next, create a shell script , mpivars.csh, with the following entry:
# path if ("" == "`echo $path | grep /usr/mpi/gcc/openmpi-x.y.z/bin`") then set path=(/usr/mpi/gcc/openmpi-x.y.z/bin $path) endif
# LD_LIBRARY_PATH if ("1" == "$?LD_LIBRARY_PATH") then if ("$LD_LIBRARY_PATH" !~ */usr/mpi/gcc/openmpi-x.y.z/lib64*) then setenv LD_LIBRARY_PATH /usr/mpi/gcc/openmpi­x.y.z/lib64:${LD_LIBRARY_PATH} endif else setenv LD_LIBRARY_PATH /usr/mpi/gcc/openmpi-x.y.z/lib64 endif
# MPI_ROOT setenv MPI_ROOT /usr/mpi/gcc/openmpi-x.y.z
To enable multithreading, add “--enable-mpi-thread-multiple and --with-threads=posix” parameters to the above configure command.
Note
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 76
vi. Simlarly, create another shell script, mpivars.sh, with the following entry:
# PATH if test -z "`echo $PATH | grep /usr/mpi/gcc/openmpi-x.y.z/bin`"; then PATH=/usr/mpi/gcc/openmpi-x.y.z/bin:${PATH} export PATH fi
# LD_LIBRARY_PATH if test -z "`echo $LD_LIBRARY_PATH | grep /usr/mpi/gcc/openmpi- x.y.z/lib64`"; then LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi­ x.y.z/lib64${LD_LIBRARY_PATH:+:}${LD_LIBRARY_PATH} export LD_LIBRARY_PATH
fi
# MPI_ROOT MPI_ROOT=/usr/mpi/gcc/openmpi-x.y.z export MPI_ROOT
vii. Next, copy the two files created in steps (v) and (vi) to /usr/mpi/gcc/openmpi-x.y.z/bin and
/usr/mpi/gcc/openmpi-x.y.z/etc
viii. Register OpenMPI with MPI-selector:
[root@host~]# mpi-selector --register openmpi --source-dir /usr/mpi/gcc/openmpi-x.y.z/bin
ix. Verify if it is listed in mpi-selector:
[root@host~]# mpi-selector --l
x. Set OpenMPI:
[root@host~]# mpi-selector --set openmpi –yes
xi. Logut and log back in.
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 77
MVAPICH2 (Installation and Setup)
i. Download the latest MVAPICH2 software package from http://mvapich.cse.ohio-state.edu/
ii. Untar and change your current working directory to MVAPICH2 package directory. iii. Configure and install as:
[root@host~]# ./configure --prefix=/usr/mpi/gcc/mvapich2-x.y/ --with­device=ch3:mrail --with-rdma=gen2 --enable-shared --with-ib­libpath=/usr/lib64/ -enable-rdma-cm --libdir=/usr/mpi/gcc/mvapich2-x.y/lib64 [root@host~]# make [root@host~]# make install
The above step will install MVAPICH2 in /usr/mpi/gcc/mvapich2-x.y/ iv. Next, create a shell script , mpivars.csh, with the following entry:
# path
if ("" == "`echo $path | grep /usr/mpi/gcc/mvapich2-x.y/bin`") then
set path=(/usr/mpi/gcc/mvapich2-x.y/bin $path)
endif
# LD_LIBRARY_PATH
if ("1" == "$?LD_LIBRARY_PATH") then
if ("$LD_LIBRARY_PATH" !~ */usr/mpi/gcc/mvapich2-x.y/lib64*) then
setenv LD_LIBRARY_PATH /usr/mpi/gcc/mvapich2-
x.y/lib64:${LD_LIBRARY_PATH}
endif
else
setenv LD_LIBRARY_PATH /usr/mpi/gcc/mvapich2-x.y/lib64
endif
# MPI_ROOT
setenv MPI_ROOT /usr/mpi/gcc/mvapich2-x.y
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 78
v. Simlarly, create another shell script, mpivars.sh, with the following entry:
# PATH
if test -z "`echo $PATH | grep /usr/mpi/gcc/ mvapich2-x.y/bin`"; then
PATH=/usr/mpi/gcc/mvapich2-x.y/bin:${PATH}
export PATH
fi
# LD_LIBRARY_PATH
if test -z "`echo $LD_LIBRARY_PATH | grep /usr/mpi/gcc/mvapich2-
x.y/lib64`"; then
LD_LIBRARY_PATH=/usr/mpi/gcc/mvapich2-
x.y/lib64${LD_LIBRARY_PATH:+:}${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH
fi
# MPI_ROOT
MPI_ROOT=/usr/mpi/gcc/mvapich2-x.y
export MPI_ROOT
vi. Next, copy the two files created in steps (iv) and (v) to /usr/mpi/gcc/mvapich2-x.y/bin and
/usr/mpi/gcc/mvapich2-x.y/etc
vii. Add the following entries in .bashrc file:
export MVAPICH2_HOME=/usr/mpi/gcc/mvapich2-x.y/ export MV2_USE_IWARP_MODE=1 export MV2_USE_RDMA_CM=1
viii. Register MPI:
[root@host~]# mpi-selector --register mvapich2 --source-dir /usr/mpi/gcc/mvapich2-x.y/bin/
ix. Verify if it is listed in mpi-selector:
[root@host~]# mpi-selector --l
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 79
x. Set MVAPICH2:
[root@host~]# mpi-selector --set mvapich2 –yes
xi. Logut and log back in. xii. Populate mpd.hosts with node names. xiii. On each node, create /etc/mv2.conf with a single line containing the IP address of the local
T4/T5 interface. This is how MVAPICH2 picks which interface to use for RDMA traffic.
4.2.3. Building MPI tests
i. Download Intel’s MPI Benchmarks from http://software.intel.com/en-us/articles/intel-mpi-
benchmarks
ii. Untar and change your current working directory to src directory. iii. Edit make_mpich file and set MPI_HOME variable to the MPI which you want to build the
benchmarks tool against. For example, in case of openMPI-1.6.4 set the variable as:
MPI_HOME=/usr/mpi/gcc/openmpi-1.6.4/
iv. Next, build and install the benchmarks using:
[root@host~]# gmake -f make_mpich
The above step will install IMB-MPI1, IMB-IO and IMB-EXT benchmarks in the current working directory (i.e. src).
v. Change your working directory to the MPI installation directory. In case of OpenMPI, it will
be /usr/mpi/gcc/openmpi-x.y.z/ vi. Create a directory called tests and then another directory called imb under tests. vii. Copy the benchmarks built and installed in step (iv) to the imb directory. viii. Follow steps (v), (vi) and (vii) for all the nodes.
4.2.4. Running MPI applications
Run Intel MPI applications as:
mpdboot -n <no_of_nodes_in_cluster> -r ssh mpdtrace mpiexec -ppn -n 2 /opt/intel/impi/3.1/tests/IMB-3.1/IMB-MPI1
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 80
The performance is best with NIC MTU set to 9000 bytes.
Run Open MPI application as:
mpirun --host node1,node2 -mca btl openib,sm,self /usr/mpi/gcc/openmpi­x.y.z/tests/imb/IMB-MPI1
The RDMA CM returned an event error while attempting to make a connection. This type of error usually indicates a network configuration error.
Local host: core96n3.asicdesigners.com Local device: Unknown Error name: RDMA_CM_EVENT_ADDR_ERROR Peer: core96n8
Workaround: Increase the OpenMPI rdma route resolution timeout. The default is 1000, or 1000ms. Increase it to 30000 with this parameter:
--mca btl_openib_connect_rdmacm_resolve_timeout 30000
Run MVAPICH2 application as :
mpirun_rsh -ssh -np 8 -hostfile mpd.hosts $MVAPICH2_HOME/tests/imb/IMB-MPI1
4.3. Setting up NFS-RDMA
4.3.1. Starting NFS-RDMA
Server-side settings
Follow the steps mentioned below to set up an NFS-RDMA server.
For OpenMPI/RDMA clusters with node counts greater than or equal to 8 nodes, and process counts greater than or equal to 64, you may experience the following RDMA address resolution error when running MPI jobs with the default OpenMPI settings:
Note
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 81
i. Make entry in /etc/exports file for the directories you need to export using NFS-RDMA on
server as:
/share/rdma *(fsid=0,async,insecure,no_root_squash) /share/rdma1 *(fsid=1,async,insecure,no_root_squash)
Note that for each directory you export, you should have DIFFERENT fsid’s. ii. Load the iwarp modules and make sure peer2peer is set to 1.
iii. Load xprtrdma and svcrdma modules as:
[root@host~]# modprobe xprtrdma [root@host~]# modprobe svcrdma
iv. Start the nfs service as:
[root@host~]# service nfs start
All services in NFS should start without errors. v. Now we need to edit the file portlist in the path /proc/fs/nfsd/
Include the rdma port 2050 into this file as:
[root@host~]# echo rdma 2050 > /proc/fs/nfsd/portlist
vi. Run exportfs to make local directories available for Network File System (NFS) clients to
mount.
[root@host~]# exportfs
Now the NFS-RDMA server is ready.
Client-side settings Follow the steps mentioned below at the client side. i. Load the iwarp modules and make sure peer2peer is set to 1. Make sure you are able to
ping and ssh to the server Chelsio interface through which directories will be exported.
Chapter IV. iWARP (RDMA)
Chelsio T5/T4 Unified Wire for Linux Page 82
ii. Load the xprtrdma module.
[root@host~]# modprobe xprtrdma
iii. Run the showmount command to show all directories from server as:
[root@host~]# showmount –e <server-chelsio-ip>
iv. Once the exported directories are listed, mount them as:
[root@host~]# mount.nfs <serverip>:<directory> <mountpoint-on-client> -o vers=3,rdma,port=2050,wsize=65536,rsize=65536
4.4. Performance Tuning
See performance tuning section in the Unified Wire chapter for generic performance settings.
Chapter V. RDMA Block Device Driver (RBD)
Chelsio T5/T4 Unified Wire for Linux Page 83
V. RDMA Block Device Driver (RBD)
Chapter V. RDMA Block Device Driver (RBD)
Chelsio T5/T4 Unified Wire for Linux Page 84
1. Introduction
RDMA Block Device Driver implements RDMA initiator and target for arbitrary block devices. The initiator registers as a blkdev driver locally. The target opens the backend block device submitting BIO operations on behalf of the initiator. Data flows via RDMA into/out of fast registered memory regions with zero copying added.
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the currently shipping Chelsio adapters that are compatible with Chelsio RDMA block device driver:
T520-BT
T580-CR
T520-LL-CR
T520-CR
T580-LP-CR
T540-CR
1.2. Software Requirements
1.2.1. Linux Requirements
Currently the RDMA block device driver is available for the following version(s):
RHEL 7.2, 3.10.0-327.el7 *
RHEL 7.1, 3.10.0-229.el7 *
SLES 12 SP1, 3.12.49-11-default *
SLES 12, 3.12.28-4-default *
Kernel.org linux-4.1 *
Kernel.org linux-3.18 *
Kernel.org linux-3.17 Other kernel versions have not been tested and are not guaranteed to work.
* Limited QA performed
Chapter V. RDMA Block Device Driver (RBD)
Chelsio T5/T4 Unified Wire for Linux Page 85
2. Software/Driver Loading
The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.
On Target
Run the following commands to load the RDMA block device driver on the target machine:
[root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm [root@host~]# modprobe rbdt
On Initiator
Run the following commands to load the RDMA block device driver on the initiator machine:
[root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm [root@host~]# modprobe rbdi
Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers.
Important
Chapter V. RDMA Block Device Driver (RBD)
Chelsio T5/T4 Unified Wire for Linux Page 86
3. Software/Driver Unloading
On Target
Run the following commands to unload the RDMA block device driver on the target machine:
[root@host~]# rmmod rbdt [root@host~]# rmmod rdma_ucm [root@host~]# rmmod iw_cxgb4
On Initiator
Run the following commands to unload the RDMA block device driver on the initiator machine:
[root@host~]# rmmod rbdi [root@host~]# rmmod rdma_ucm [root@host~]# rmmod iw_cxgb4
Chapter V. RDMA Block Device Driver (RBD)
Chelsio T5/T4 Unified Wire for Linux Page 87
4. Software/Driver Configuration and Fine-tuning
Adding a Target
On the initiator machine, run the following command to add a target:
[root@host~]# rbdctl -n -a <target_ip> -d <target_block_device> -p <target_port_number>
E.g.:
[root@host~]# rbdctl -n -a 102.1.1.106 -d /dev/ram0 -p 65000
Removing a Target
Run the following command to remove a target from the initiator machine:
[root@host~]# rbdctl -r -d <initiator_device>
E.g.:
[root@host~]# rbdctl -r -d /dev/rbdi0
Listing Targets
Run the following command on the initiator, to list all the targets available:
[root@host~]# rbdctl -l
E.g.:
[root@host~]# rbdctl -l
local /dev/rbdi0 102.1.1.105:49093 remote /dev/ram0 102.1.1.106:65000 state CONNECTED local /dev/rbdi1 102.1.1.105:49685 remote /dev/ram1 102.1.1.106:65000 state CONNECTED
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 88
VI. WD-UDP
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 89
1. Introduction
Chelsio WD-UDP (Wire Direct-User Datagram Protocol) with Multicast is a user-space UDP stack with Multicast address reception and socket acceleration that enables users to run their existing UDP socket applications unmodified.
It features software modules that enable direct wire access from user space to the Chelsio T5/T4 network adapter with complete bypass of the kernel, which results in an ultra-low latency 40/10Gb Ethernet solution for high frequency trading and other delay-sensitive applications.
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the currently shipping Chelsio Adapters that are compatible with Chelsio WD­UDP driver:
T520-BT
T580-CR
T520-LL-CR
T520-CR
T580-LP-CR
T540-CR
T420-CR
T440-CR
T422-CR
T404-BT
T440-LP-CR
T420-LL-CR
T420-CX
1.2. Software Requirements
1.2.1. Linux Requirements
Currently the WD-UDP driver is available for the following versions:
RHEL 7.2, 3.10.0-327.el7 *
RHEL 7.1, 3.10.0-229.el7
RHEL 6.7, 2.6.32-573.el6*
RHEL 6.6, 2.6.32-504.el6*
RHEL 6.5, 2.6.32-431.el6*
Note
This product is not supported with the current release. Support will be added in upcoming releases.
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 90
SLES 12 SP1, 3.12.49-11-default *
SLES12, 3.12.28-4-default*
SLES 11 SP4, 3.0.101-63-default *
SLES 11 SP3, 3.0.76-0.11-default
Ubuntu 14.04.3, 3.19.0-25-generic *
Ubuntu 14.04.2, 3.16.0-30-generic *
Kernel.org linux-4.1 *
Kernel.org linux-3.18 *
Kernel.org linux-3.17
Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work.
*Limited QA performed
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 91
2. Software/Driver Loading
The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.
RDMA core modules from the OFED package should be loaded before proceeding. To load the WD-UDP driver, use the following commands which will automatically load RDMA core modules:
[root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4
[root@host~]# modprobe rdma_ucm
Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers.
Important
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 92
3. Software/Driver Unloading
To unload the WD-UDP driver, run the following command:
[root@host~]# rmmod iw_cxgb4
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 93
4. Software/Driver Configuration and Fine-tuning
4.1. Accelerating UDP Socket communications
The libcxgb4_sock library is a LD_PRELOAD-able library that accelerates UDP Socket communications transparently and without recompilation of the user application. This section describes how to use libcxgb4_sock.
By preloading libcxgb4_sock, all sockets created by the application are intercepted and possibly
accelerated based on the user’s configuration. Once accelerated, data for the UDP endpoint are
transmitted or received via HW queues allocated specifically for the accelerated endpoint, bypassing the kernel, the host networking stack and sockets framework, and enabling ultra-low latency and high bandwidth utilization.
Due to HW resource limitations, only a small number of queues can be allocated for UDP acceleration. Therefore only performance critical UDP applications should use libcxgb4_sock.
Only 64 IPv4 UDP / 28 IPv6 UDP sockets can be accelerated per Chelsio T5/T4 device, with Unified Wire Configuration tuning option. If you want more sockets to be accelerated, please use Low Latency or High Capacity WD tuning option.
4.1.1. Application Requirements
Certain application behavior is not supported by libcxb4_sock in this release. If your application does any of the following, it will not work with libcxgb4_sock:
Calling fork() after creating UDP sockets and using the UDP socket in the child process.
Using multiple threads on a single UDP socket without serialization. For instance, having
one thread sending concurrently with another thread receiving. If your application does this, you need to serialize these paths with a spin or mutex lock.
Only 1 UDP endpoint is allowed to bind to a given port per host. So if you have multiple
processes on the same host binding to the same UDP port number, you cannot use
libcxgb4_sock. Applications must have root privileges to use libcxgb4_sock. Applications requiring bonded T5/T4 interfaces are not currently supported.
The performance benefit observed with libcxgb4_sock will vary based on your application’s behavior. While all UDP IO is handled properly, only certain datagrams are accelerated. Non accelerated IO is handled by libcxgb4_sock via the host networking stack seamlessly. Both Unicast and Multicast datagrams can be accelerated, but the datagrams must meet the following criteria:
Non fragmented. In other words, they fit in a single IP datagram that is <= the T5/T4 device
MTU.
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 94
Routed through the T5/T4 acceleration device. If the ingress datagram arrives via a device
other than the T5/T4 acceleration device, then it will not utilize the acceleration path. On
egress, if the destination IP address will not route out via the T5/T4 device, then it too will
not be accelerated.
4.1.2. Using libcxgb4_sock
The libcxgb4_sock library utilizes the Linux RDMA Verbs subsystem, and thus requires the RDMA modules be loaded. Ensure that your systems load the iw_cxgb4 and rdma_ucm modules:
[root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm
Now, preload libcxgb4_sock, using one of the methods mentioned below when starting your application:
Preloading using wdload script:
[root@host~]# PROT=UDP wdload <pathto>/your_application
The above command will generate an end point file, libcxgb4_sock.conf at /etc/. Parameters like interface name and port number can be changed in this file.
Preloading manually
Create a configuration file that defines which UDP endpoints should be accelerated, their vlan and priority if any, as well as which T5/T4 interface/port should be used. The file
/etc/libcxgb4_sock.conf contains these endpoint entries. Create this file on all systems
using libcxgb4_sock. Here is the syntax:
# # Syntax: # # endpoint {attributes} ... # where attributes include: # interface = interface-name # port = udp-port-number # vlan = vlan-id # priority = vlan-priority
If you encounter error while using wdload on kernels built on RHEL 5.x distribution, then run the above command as :
[root@host~]# NUMA=0 PROT=UDP wdload <pathto>/your_application
Note
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 95
# # e.g. # endpoint { # interface=eth2.5 # port = 8000 vlan = 5 priority=1 # } # endpoint { interface=eth2 port=9999} # # endpoints that bind to port 0 (requesting the host allocate a port)
# can be accelerated with port=0: # # endpoint {interface=eth1 port=0} #
Assume your T5/T4 interface is eth2. To accelerate all applications that preload libcxgb4_sock using eth2, you only need one entry in /etc/libcxgb4_sock.conf:
endpoint {interface=eth2 port=0}
If you have eth2 and eth3 configured for example, you can define certain endpoints to eth2 and others to eth3:
endpoint {interface=eth2 port=9999} endpoint {interface=eth3 port=8888}
For VLAN support, create your VLANs using the normal OS service (like vconfig, for example), then add entries to define the VLAN and priority for each endpoint to be accelerated:
endpoint {interface = eth2.5 port=10000} endpoint {interface = eth2.7 priority=3 port=9000}
Now, preload libcxgb4_sock:
[root@host~]# CXGB4_SOCK_CFG=<path to config file> LD_PRELOAD=libcxgb4_sock.so <pathto>/your_application
In order to offload IPv6 UDP sockets, please select “low latency networking” as T5/T4 configuration tuning option during installation.
Note
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 96
4.1.3. Running WD-UDP in debug mode
To use libcxgb4_sock’s debug capabilities, use the libcxgb4_sock_debug library provided in the package. Follow the steps mentioned below:
i. Make the following entry in the /etc/syslog.conf file:
*.debug /var/log/cxgb4.log
ii. Restart the service:
[root@host~]# /etc/init.d/syslog restart
iii. Finally, preload libcxgb4_sock_debug using the command mentioned below when starting
your application:
[root@host~]# LD_PRELOAD=libcxgb4_sock_debug.so CXGB4_SOCK_DEBUG=-1 <pathto>/your_application
4.1.4. Running WD-UDP with larger I/O size
If the I/O size is > 3988, execute the commands mentioned below:
[root@host~]# echo 1024 > /proc/sys/vm/nr_hugepages [root@host~]# CXGB4_SOCK_HUGE_PAGES=1 PROT=UDP wdload <pathto>/your_application
4.1.5. Example with hpcbench/udp
The udp benchmark from the hpcbench suite can be used to show the benefits of libcxgb4_sock. The hpcbench suite can be found at:
Source: http://hpcbench.sourceforge.net/index.html Sample: http://hpcbench.sourceforge.net/udp.html The nodes in this example, r9 and r10, have T5/T4 eth1 configured and the ports are connected
point-to-point.
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 97
[root@r9 ~]# ifconfig eth1|grep inet inet addr:192.168.2.111 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::7:4300:104:465a/64 Scope:Link
[root@r9 ~]# [root@r10 ~]# ifconfig eth1|grep inet inet addr:192.168.2.112 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::7:4300:104:456a/64 Scope:Link [root@r10 ~]#
For this benchmark, we need a simple “accelerate all” configuration on both nodes:
[root@r9 ~]# cat /etc/libcxgb4_sock.conf endpoint {interface=eth1 port=0} [root@r9 ~]#
[root@r10 ~]# cat /etc/libcxgb4_sock.conf endpoint {interface=eth1 port=0} [root@r10 ~]#
On R10, we run udpserver on port 9000 without libcxgb4_sock preloaded, and on port 90001 with preload:
[root@r10 ~]# /usr/local/src/hpcbench/udp/udpserver -p 9000 & [1] 11453 [root@r10 ~]# TCP socket listening on port [9000]
[root@r10 ~]# LD_PRELOAD=libcxgb4_sock.so /usr/local/src/hpcbench/udp/udpserver -p 9001 & [2] 11454 [root@r10 ~]# TCP socket listening on port [9001] [root@r10 ~]#
Then on r9, we run udptest to port 9000 to see the host stack UDP latency:
[root@r9 ~]# /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.112 -p 9000
Running the same test with libcxgb4_sock:
[root@r9 ~]# LD_PRELOAD=libcxgb4_sock.so /usr/local/src/hpcbench/udp/udptest
-r 5 -a -h 192.168.1.112 -p 9001
Chapter VI. WD-UDP
Chelsio T5/T4 Unified Wire for Linux Page 98
4.1.6. Determining if the application is being offloaded
To see if the application is being offloaded, open a window on one of the machines, and run
tcpdump against the Chelsio interface. If you see minimal UDP output on the interface, then the
UDP traffic is being properly offloaded.
Chapter VII. iSCSI PDU Offload Target
Chelsio T5/T4 Unified Wire for Linux Page 99
VII. iSCSI PDU Offload Target
Chapter VII. iSCSI PDU Offload Target
Chelsio T5/T4 Unified Wire for Linux Page 100
1. Introduction
This section describes how to install and configure iSCSI PDU Offload Target software for use as a key element in your iSCSI SAN. The software runs on Linux-based systems that use Chelsio or non-Chelsio based Ethernet adapters. However, to guarantee highest performance, Chelsio recommends using Chelsio adapters. Chelsio’s adapters include offerings that range from stateless offload adapters (regular NIC) to the full line of TCP/IP Offload Engine (TOE) adapters.
The software implements RFC 3720, the iSCSI standard of the IETF. The software has been fully tested for compliance to that RFC and others and it has been exhaustively tested for interoperability with the major iSCSI vendors.
The software implements most of the iSCSI protocol in software running in kernel mode on the host with the remaining portion, which consists of the entire fast data path, in hardware when used with Chelsio’s TOE adapters. When standard NIC Adapters are used the entire iSCSI protocol is executed in software.
The performance of this iSCSI stack is outstanding and when used with Chelsio’s hardware it is enhanced further. Because of the tight integration with Chelsio’s TOE adapters, this software
has a distinct performance advantage over the regular NIC. The entire solution, which includes this software, Chelsio TOE hardware, an appropriate base computer system – including a high end disk subsystem, has industry leading performance. This can be seen when the entire solution is compared to others based on other technologies currently available on the market in terms of throughput and IOPS.
1.1. Features
Chelsio’s iSCSI driver stack supports the iSCSI protocol in the Target mode. From henceforth “iSCSI Software Entity” term refers to the iSCSI target.
The Chelsio iSCSI PDU Offload Target software provides the following high level features:
Expanded NIC Support
Chelsio TCP Offload Engine (TOE) Support
T5/T4 Based HBAs (T5/T4xx Series cards)
Non-Chelsio
Runs on regular NICs
Chelsio Terminator ASIC Support
Offloads iSCSI Fast Data Path with Direct Data Placement (DDP) Offloads iSCSI Header and Data Digest Calculations Offload Speeds at 1 Gb, 10 Gb and 40Gb Offloads TCP/IP for NAS simultaneously with iSCSI
Target Specific features
Loading...