Chelsio Communications Chelsio T5, Chelsio T4 Installation And User Manual
Chelsio T5/T4 Unified Wire for Linux ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to sales@chelsio.com
Support
For all support related questions please send email to support@chelsio.com
Copyright © 2016.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: 408.962.3600
Fax: 408.962.3661
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: +91-80-4039-6800
Fax: +91-80-4039-6807
Chelsio KK (Japan)
Regus Place Canada
Akasaka 7-3-37
Minato-ku, Tokyo,
Japan 107-0052
Tel: 03-6234-4353
Chelsio T5/T4 Unified Wire for Linux iii
Document History
Version
Revision Date
1.0.0
12/08/2011
1.0.1
01/09/2013
1.0.2
01/27/2013
1.0.3
03/26/2013
1.0.4
04/12/2013
1.0.5
06/20/2013
1.0.6
08/17/2013
1.0.7
10/22/2013
1.0.8
03/08/2013
1.0.9
05/15/2013
1.1.0
07/26/2013
1.1.1
08/14/2013
1.1.2
12/06/2013
1.1.3
12/19/2013
1.1.4
03/13/2014
1.1.5
05/02/2014
1.1.6
06/30/2014
1.1.7
10/22/2014
1.1.8
11/04/2014
1.1.9
02/05/2015
1.2.0
03/04/2015
1.2.1
03/25/2015
1.2.2
06/03/2015
1.2.3
08/05/2015
1.2.4
02/29/2016
Chelsio T5/T4 Unified Wire for Linux iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 12
1. Introduction 13
1.1. Features 13
1.2. Hardware Requirements 14
1.3. Software Requirements 14
1.4. Package Contents 15
2. Hardware Installation 18
3. Software/Driver Installation 21
3.1. Pre-requisites 22
3.2. Installing Chelsio Unified Wire from source 22
3.3. Installing Chelsio Unified Wire from RPM 33
3.4. Firmware update 35
4. Software/Driver Uninstallation 37
4.1. Uninstalling Chelsio Unified Wire from source 37
4.2. Uninstalling Chelsio Unified Wire from RPM 41
5. Configuring Chelsio Network Interfaces 43
5.1. Configuring 40G adapters 43
5.2. Configuring network-scripts 45
5.3. Creating network-scripts 45
5.4. Checking Link 46
6. Performance Tuning 47
7. Software/Driver Update 48
II. NETWORK (NIC/TOE) 49
1. Introduction 50
1.1. Hardware Requirements 50
1.2. Software Requirements 51
2. Software/Driver Loading 52
2.1. Loading in NIC mode (without full offload support) 52
2.2. Loading in TOE mode (with full offload support) 52
3. Software/Driver Unloading 53
3.1. Unloading the NIC driver 53
3.2. Unloading the TOE driver 53
4. Software/Driver Configuration and Fine-tuning 55
4.1. Instantiate Virtual Functions (SR-IOV) 55
4.2. Enabling Busy waiting 55
4.3. Performance Tuning 56
III. VIRTUAL FUNCTION NETWORK (VNIC) 62
Chelsio T5/T4 Unified Wire for Linux v
1. Introduction 63
1.1. Hardware Requirements 63
1.2. Software Requirements 64
2. Software/Driver Loading 65
2.1. Instantiate Virtual Functions 65
2.2. Loading the driver 65
3. Software/Driver Unloading 66
3.1. Unloading the driver 66
IV. IWARP (RDMA) 67
1. Introduction 68
1.1. Hardware Requirements 68
1.2. Software Requirements 68
2. Software/Driver Loading 70
2.1. Loading iWARP driver 70
3. Software/Driver Unloading 71
4. Software/Driver Configuration and Fine-tuning 72
4.1. Testing connectivity with ping and rping 72
4.2. Enabling various MPIs 73
4.3. Setting up NFS-RDMA 80
4.4. Performance Tuning 82
V. RDMA BLOCK DEVICE DRIVER (RBD) 83
1. Introduction 84
1.1. Hardware Requirements 84
1.2. Software Requirements 84
2. Software/Driver Loading 85
3. Software/Driver Unloading 86
4. Software/Driver Configuration and Fine-tuning 87
VI. WD-UDP 88
1. Introduction 89
1.1. Hardware Requirements 89
1.2. Software Requirements 89
2. Software/Driver Loading 91
3. Software/Driver Unloading 92
4. Software/Driver Configuration and Fine-tuning 93
4.1. Accelerating UDP Socket communications 93
VII. ISCSI PDU OFFLOAD TARGET 99
1. Introduction 100
Chelsio T5/T4 Unified Wire for Linux vi
1.1. Features 100
1.2. Hardware Requirements 101
1.3. Software Requirements 102
2. Software/Driver Loading 104
2.1. Latest iSCSI Software Stack Driver Software 104
3. Software/Driver Unloading 106
4. Software/Driver Configuration and Fine-tuning 107
4.1. Command Line Tools 107
4.2. iSCSI Configuration File 107
4.3. A Quick Start Guide for Target 108
4.4. The iSCSI Configuration File 110
4.5. Challenge-Handshake Authenticate Protocol (CHAP) 121
4.6. Target Access Control List (ACL) Configuration 123
4.7. Target Storage Device Configuration 125
4.8. Target Redirection Support 127
4.9. The command line interface tools “iscsictl” & “chisns” 128
4.10. Rules of Target Reload (i.e. “on the fly” changes) 133
4.11. System Wide Parameters 135
4.12. Performance Tuning 136
VIII. ISCSI PDU OFFLOAD INITIATOR 137
1. Introduction 138
1.1. Hardware Requirements 138
1.2. Software Requirements 139
2. Software/Driver Loading 140
3. Software/Driver Unloading 141
4. Software/Driver Configuration and Fine-tuning 142
4.1. Accelerating open-iSCSI Initiator 142
4.2. Auto login from cxgb4i initiator at OS bootup 144
IX. DATA CENTER BRIDGING (DCB) 146
1. Introduction 147
1.1. Hardware Requirements 147
1.2. Software Requirements 147
2. Software/Driver Loading 149
3. Software/Driver Unloading 150
4. Software/Driver Configuration and Fine-tuning 151
4.1. Configuring Cisco Nexus 5010 switch 151
4.2. Configuring the Brocade 8000 switch 154
5. Running NIC & iSCSI Traffic together with DCBx 156
Chelsio T5/T4 Unified Wire for Linux vii
X. FCOE PDU OFFLOAD TARGET 157
1. Introduction 158
1.1. Hardware Requirements 158
1.2. Software Requirements 158
2. Software/Driver Loading 159
3. Software/Driver Unloading 160
4. Software/Driver Configuration and Fine-tuning 161
4.1. Configuring Cisco Nexus 5010 switch 161
4.2. Collecting port information 164
4.3. Configuring LUNs on Target 167
4.4. Configuring Persistent Target 169
4.5. Verifying initiators connected to the target 170
4.6. Removing LUNs 171
4.7. Performance Tuning 171
XI. FCOE FULL OFFLOAD INITIATOR 174
1. Introduction 175
1.1. Hardware Requirements 175
1.2. Software Requirements 175
2. Software/Driver Loading 177
3. Software/Driver Unloading 178
4. Software/Driver Configuration and Fine-tuning 179
4.1. Configuring Cisco Nexus 5010 and Brocade switch 179
4.2. FCoE fabric discovery verification 179
4.3. Formatting the LUNs and Mounting the Filesystem 183
4.4. Creating Filesystem 184
4.5. Mounting the formatted LUN 185
XII. OFFLOAD BONDING DRIVER 186
1. Introduction 187
1.1. Hardware Requirements 187
1.2. Software Requirements 187
2. Software/Driver Loading 189
3. Software/Driver Unloading 190
4. Software/Driver Configuration and Fine-tuning 191
4.1. Offloading TCP traffic over a bonded interface 191
XIII. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) 192
1. Introduction 193
1.1. Hardware Requirements 193
1.2. Software Requirements 194
Chelsio T5/T4 Unified Wire for Linux viii
2. Software/Driver Loading 195
3. Software/Driver Unloading 196
4. Software/Driver Configuration and Fine-tuning 197
4.1. Offloading TCP traffic over a bonded interface 197
XIV. UDP SEGMENTATION OFFLOAD AND PACING 198
1. Introduction 199
1.1. Hardware Requirements 199
1.2. Software Requirements 200
2. Software/Driver Loading 201
3. Software/Driver Unloading 202
4. Software/Driver Configuration and Fine-tuning 203
4.1. Modifying the application 203
4.2. Configuring UDP Pacing 205
XV. OFFLOAD IPV6 DRIVER 207
1. Introduction 208
1.1. Hardware Requirements 208
1.2. Software Requirements 208
2. Software/Driver Loading 210
3. Software/Driver Unloading 211
3.1. Unloading the NIC driver 211
3.2. Unloading the TOE driver 211
XVI. BYPASS DRIVER 212
1. Introduction 213
1.1. Features 213
1.2. Hardware Requirements 214
1.3. Software Requirements 215
2. Software/Driver Loading 216
3. Software/Driver Unloading 217
4. Software/Driver Configuration and Fine-tuning 218
4.1. Starting ba server 218
4.2. Bypass API (CLI) 218
XVII. WD SNIFFING AND TRACING 224
1. Theory of Operation 225
1.1. Hardware Requirements 226
1.2. Software Requirements 227
2. Installation and Usage 228
Chelsio T5/T4 Unified Wire for Linux ix
2.1. Installing basic support 228
2.2. Using Sniffer (wd_sniffer) 228
2.3. Using Tracer (wd_tcpdump_trace) 228
XVIII.CLASSIFICATION AND FILTERING 230
1. Introduction 231
1.1. Hardware Requirements 231
1.2. Software Requirements 232
2. Usage 233
2.1. Configuration 233
2.2. Creating Filter Rules 234
2.3. Listing Filter Rules 235
2.4. Removing Filter Rules 235
2.5. Layer 3 example 236
2.6. Layer 2 example 239
3. Hash/DDR Filters 243
3.1. Creating Filter Rules 243
3.2. Listing Filter Rules 244
3.3. Removing Filter Rules 245
3.4. Swap MAC feature 245
3.5. Hit Counters 246
XIX. TRAFFIC MANAGEMENT 248
1. Introduction 249
1.1. Hardware Requirements 249
1.2. Software Requirements 250
2. Software/Driver Loading 251
3. Software/Driver Unloading 252
4. Software/Driver Configuration and Fine-tuning 253
4.1. Traffic Management Rules 253
4.2. Configuring Traffic Management 255
5. Usage 258
5.1. Non-Offloaded Connections 258
5.2. Offloaded Connections 258
5.3. Offloaded Connections with Modified Application 259
XX. UNIFIED WIRE MANAGER (UM) 260
1. Introduction 261
1.1. Features 261
1.2. Reference Architecture 262
1.3. Unified Wire Manager Components 262
Chelsio T5/T4 Unified Wire for Linux x
1.4. Authentication and encryption 263
2. Hardware and Software 264
2.1. Supported Adapters 264
2.2. Platform/Component Matrix 265
2.3. Platform/Driver Matrix 265
3. Installing Unified Wire Manager 266
4. Verifying UM components status 267
4.1. Verifying Management Agent 267
4.2. Verifying Management Client 268
4.3. Verifying Management Station 268
5. Management Agent 269
5.1. Communication 269
5.2. Configuration 269
5.3. Service configuration 269
5.4. Firewall 270
6. CLI client 271
6.1. CLI Help system 271
6.2. Client conflict resolution 271
7. Web GUI client 272
7.1. Management Station 272
7.2. Accessing Web Management Interface 273
7.3. Layout and Navigation 276
7.4. Home page 277
7.5. System page 290
7.6. Network page 302
7.7. Storage Page 331
7.8. Hardware Features 364
8. Uninstalling Unified Wire Manager 380
8.1. Uninstalling Management Agent 380
8.2. Uninstalling Management Client 380
8.3. Uninstalling Management Station 381
XXI. UNIFIED BOOT 382
1. Introduction 383
1.1. Hardware Requirements 383
1.2. Software Requirements 385
2. Flashing firmware and option ROM 386
2.1. Preparing USB flash drive 386
2.2. Legacy 387
2.3. uEFI 390
3. Configuring PXE Server 395
Chelsio T5/T4 Unified Wire for Linux xi
4. PXE boot process 396
4.1. Legacy PXE boot 396
4.2. uEFI PXE Boot 401
5. FCoE boot process 407
5.1. Legacy FCoE boot 407
5.2. uEFI FCoE Boot 416
6. iSCSI boot process 426
6.1. Legacy iSCSI boot 426
6.2. uEFI iSCSI Boot 439
7. Creating Driver Update Disk (DUD) 451
7.1. Creating DUD for RedHat Enterprise Linux 451
7.2. Creating DUD for Suse Enterprise Linux 451
8. OS Installation 453
8.1. Installation using Chelsio NIC DUD (PXE only) 453
8.2. Installation on FCoE LUN 464
8.3. Installation on iSCSI LUN 484
XXII. LUSTRE FILE SYSTEM 501
1. Introduction 502
1.1. Hardware Requirements 502
1.2. Software Requirements 503
2. Creating/Configuring Lustre File System 504
XXIII. APPENDIX A 506
1. Troubleshooting 507
2. Chelsio End-User License Agreement (EULA) 509
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 12
I. Chelsio Unified Wire
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 13
1. Introduction
Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, iSCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified
Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and
Filtering.
Ideal for all data, storage and high performance clustering applications, the T5/T4 Adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5/T4 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
Chelsio Unified Wire software comes in two formats: Source code and RPM package forms.
Installing from source requires compiling the package to generate the necessary binaries. You
can choose this method when you are using a custom-built kernel. You can also install the
package using the interactive GUI installer. In other cases, download the RPM package specific
to your operating system and follow the steps mentioned to install the package. Please note that
the OFED software required to install Chelsio iWARP driver comes bundled in both source as
well as RPM packages.
This document describes the installation, use and maintenance of the software and its
various components.
1.1. Features
The Chelsio Unified Wire Package uses a single command to install various drivers and utilities.
It consists of the following software:
Network (NIC/TOE)
Virtual Function Network (vNIC)
iWARP (RDMA)
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 14
RDMA Block Device driver (RBD)
WD-UDP *
iSCSI PDU Offload Target
iSCSI PDU Offload Initiator
Data Center Bridiging (DCB) *
FCoE PDU Offload Target *
FCoE full offload Initiator *
Offload Bonding driver
Offload Multi-Adapter Failover(MAFO)
UDP Segmentation Offload and Pacing
Offload IPv6 driver
Bypass driver
Classification and Filtering feature
Traffic Management feature (TM)
Unified Wire Manager (UM)
Unified Boot Software^
Lustre File System
Utility Tools (cop, cxgbtool, t4_perftune, benchmark tools, sniffer & tracer)
libs (iWARP and WD-UDP libraries)
For detailed instructions on loading, unloading and configuring the drivers/tools please refer to
their respective sections.
* This product is not supported with the current release. Support will be added in upcoming
releases.
^ FCoE and iSCSI Boot are not supported with the current release. Support will be added
in upcoming releases.
1.2. Hardware Requirements
The Chelsio Unified Wire software supports Chelsio T5 and T4 Series of Unified Wire Adapters.
To know more about the list of adapters supported by each driver, please refer to their
respective sections.
1.3. Software Requirements
The Chelsio Unified Wire software has been developed to run on 64-bit Linux based platforms
and therefore it is a base requirement for running the driver. To know more about the complete
list of operating systems supported by each driver, please refer to their respective sections.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 15
1.4. Package Contents
1.4.1. Source Package
The Chelsio Unified Wire source package consists of the following files/directories:
debrules: This directory contains packaging specification files required for building Debian
packages.
docs: This directory contains support documents - README, Release Notes and User’s
Guide (this document) for the software.
libs: This directory is for libraries required to install the WD-UDP and iWARP drivers. The
libibverbs library has implementation of RDMA verbs which will be used by iWARP
applications for data transfers. The librdmacm library works as an RDMA connection
manager. The libcxgb4 library works as an interface between the above mentioned generic
libraries and Chelsio iWARP driver. The libcxgb4_sock library is a LD_PRELOAD-able
library that accelerates UDP Socket communications transparently and without
recompilation of the user application.
lustre: This directory contains patch needed to be applied to the lustre kernel.
OFED: This directory contains supported OFED packages.
RPM-Manager: This directory contains support scripts used for cluster deployment.
scripts: Support scripts used by the Unified Wire Installer.
specs: The packaging specification files required for building RPM packages.
src: Source code for different drivers.
support: This directory contains source files for the dialog utility.
tools:
autoconf-2.63: This directory contains the source for autoconf v2.63 needed for WD-
UDP and iWARP libraries.
ba_server: Management and configuration tools for bypass adapters.
benchmarks: This directory contains various benchmarking tools to measure
throughput and latency of various networks.
chelsio_adapter_config: This directory contains scripts and binaries needed to
configure Chelsio 40G Adapters.
cop: The cop tool compiles offload policies into a simple program form that can be
loaded into the kernel and interpreted. These offload policies are used to determine
the settings to be used for various connections. The connections to which the
settings are applied are based on matching filter specifications. Please find more
details on this tool in its manual page (run man cop command).
cudbg: Chelsio Unified Debug tool which facilitates collection and viewing of various
debug entities like register dump, Devlog, CIM LA, etc.
cxgbtool: The cxgbtool queries or sets various aspects of Chelsio network interface
cards. It complements standard tools used to configure network settings and
provides functionality not available through such tools. Please find more details on
this tool in its manual page (run man cxgbtool command).
To use cxbtool for FCoE Initiator driver, use [root@host~]# cxgbtool stor -h
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 16
rdma_tools: This directory contains iWARP benchmarking tools.
t4_sniffer: This directory contains sniffer tracing and filtering libraries. See WD
Sniffing and Tracing chapter for more information.
um: This directory contains Unified Wire Manager RPMs for different distributions
and Management Station configuration files.
90-rdma.rules: This file contains udev rules needed for running RDMA applications
as a non-root user.
chdebug: This script collects operating system environment details and debug
information which can be sent to the support team, to troubleshoot Chelsio
hardware/software related issues.
chiscsi_set_affinity.sh: This shell script is used for mapping iSCSI Worker threads
to different CPUs.
chsetup: The chsetup tool loads NIC, TOE and iWARP drivers, and creates WD-
UDP configuration file.
chstatus: This utility provides status information on any Chelsio NIC in the system.
t4_latencytune.sh: Script used for latency tuning of Chelsio Adapters.
t4_perftune.sh: This shell script is to tune the system for higher performance. It
achieves it through modifying the IRQ-CPU binding. This script can also be used to
change Tx coalescing settings.
t4-forward.sh: RFC2544 Forward test tuning script.
uname_r: This file is used by chstatus script to verify if the Linux platform is
supported or not.
wdload: UDP acceleration tool.
wdunload: Used to unload all the loaded Chelsio drivers.
install.py, dialog.py: Python scripts needed for the GUI installer.
EULA: Chelsio’s End User License Agreement
install.log: File containing installation summary.
Makefile: The Makefile for building and installing from the source.
sample_machinefile: Sample file used during iWARP installation on cluster nodes.
Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD.
The OptionROM directory contains Unified Boot Option ROM image (cubt4.bin), uEFI driver
(ChelsioUD.efi), default boot configuration file (bootcfg) and a legacy flash utility (cfut4.exe),
which can be used to flash the option ROM onto Chelsio's T5 and T4 based Converged
Network Adapters (CNAs).
The LinuxDUD directory contains image (.img) files required to update drivers for Linux
distributions.
1.4.2. RPM package
The Chelsio Unified Wire RPM package consists of the following:
config: This directory contains T5/T4 firmware configuration files.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 17
docs: This directory contains support documents i.e. README, Release Notes and User’s
Guide (this document) for the software.
DRIVER-RPMS: RPM packages of Chelsio drivers.
OFED-RPMS: OFED RPM packages required to install iWARP driver.
scripts: Support scripts used by the Unified Wire Installer.
EULA: Chelsio’s End User License Agreement.
install.py: Python script that installs the RPM package. See Software/Driver Installation
section for more information.
uninstall.py: Python script that uninstalls the RPM package. See Software/Driver
Uninstallation section for more information.
Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD.
The OptionROM directory contains Unified Boot Option ROM image (cubt4.bin), uEFI driver
(ChelsioUD.efi), default boot configuration file (bootcfg) and a legacy flash utility (cfut4.exe),
which can be used to flash the option ROM onto Chelsio's T5 and T4 based Converged
Network Adapters (CNAs).
The LinuxDUD directory contains image (.img) files required to update drivers for Linux
distributions.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 18
2. Hardware Installation
Follow these steps to install Chelsio Adapter in your system:
1. Shutdown/power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIe
x8/x16 slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 40/10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Run update-pciids command to download the current version of PCI ID list
[root@host~]# update-pciids
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 198k 100 198k 0 0 491k 0 --:--:-- --:--:-- --:--:-- 626k
Done.
13. Verify if the adapter was installed successfully by using the lspci command
a. For T5 adapters :
[root@host~]# lspci |grep -i Chelsio
07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.3 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 19
Wire Ethernet Controller
07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR
Unified Wire Storage Controller
07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire
Storage Controller
b. And for T4 adapters :
[root@host~]# lspci | grep –i Chelsio
03:00.0 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.1 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.2 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.3 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.4 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.5 SCSI storage controller: Chelsio Communications Inc T420-CR Unified
Wire Storage Controller
03:00.6 Fibre Channel: Chelsio Communications Inc T420-CR Unified Wire
Storage Controller
03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000
For Chelsio T5/T4 adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
Once Unified Wire package is installed and loaded, examine the output of dmesg to see if the
card is discovered.
For T5 adapters:
eth2: Chelsio T520-LL rev 1 1000/10GBASE-SFP RNIC MSI-X, Offload capable
0000:07:00.4: S/N: RE12130097, P/N: 11011675004
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 20
And, for T4 adapters:
eth0: Chelsio T420-CR rev 2 1000/10GBASE-SFP RNIC MSI-X, Offload capable
0000:04:00.4: S/N: PT18111226, P/N: 110112140D0
The above outputs indicate the hardware configuration of the adapters as well as the Serial
numbers.
Network device names for Chelsio’s physical ports are assigned using the following
convention: the port farthest from the motherboard will appear as the first network
interface. However, for T5 40G and T420-BT adapters, the association of physical
Ethernet ports and their corresponding network device names is opposite. For
these adapters, the port nearest to the motherboard will appear as the first network
interface.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 21
3. Software/Driver Installation
There are two main methods to install the Chelsio Unified Wire package: from source and RPM.
If you decide to use source, you can install the package using CLI or GUI mode.
If you decide to use RPM, you can install the package using Menu or CLI mode.
Irrespective of the method chosen for installation, the machine needs to be rebooted for
changes to take effect.
The following table describes the various configuration tuning options available during
installation and drivers/software installed with each option by default:
T5/T4
Configuration
Tuning Option
Description
Driver/Software installed
Unified Wire
Configures adapters to run multiple protocols
like NIC/TOE, iWARP, iSCSI and FCoE
Initiator simultaneously.
NIC/TOE, vNIC, iWARP, RBD, WD-UDP,
iSCSI Target, iSCSI Initiator, DCB, FCoE
Target, FCoE Initiator, Bonding, MAFO,
IPv6, Sniffer & Tracer, UM (Agent, Client,
WebGUI), Filtering, TM, Lustre
Low latency
Networking
Configures adapters to run NIC/TOE and
iWARP traffic with low latency specially
needed for financial applications.
NIC/TOE, iWARP, RBD, WD-UDP, IPv6,
Sniffer & Tracer, Bonding, MAFO, UM
(Agent, Client, WebGUI), Filtering, TM
High capacity
RDMA
Configures adapters to establish a large
number of RDMA connections.
NIC/TOE, iWARP, RBD, WD-UDP, Bonding,
MAFO, IPv6, Sniffer & Tracer, UM (Agent,
Client, WebGUI), Filtering, TM
RDMA
Performance
Improves RDMA performance on T5/T4
adapters.
NIC/TOE, iWARP, RBD
UM (Agent, Client, WebGUI)
High capacity
TOE
Configures adapters to establish a large
number of TOE connections.
NIC/TOE, Bonding, MAFO, IPv6, UM (Agent,
Client, WebGUI), Filtering,
TM
iSCSI
Performance*
Improves iSCSI performance on T5
adapters.
NIC/TOE, iSCSI Target, iSCSI Initiator,
Bonding, DCB,
UM (Agent, Client, WebGUI)
UDP Seg.Offload
& Pacing
Configures adapters to establish a large
number of UDP Segmentation Offload
connections.
NIC/TOE, IPv6, USO, Bonding, UM (Agent,
Client, WebGUI), Filtering, TM
T5 Wire Direct
Latency*
Configures T5 adapters to provide low Wire
Direct latency.
NIC/TOE, iWARP, RBD, WD-UDP, UM
(Agent, Client, WebGUI)
High Capacity WD
Configures adapters to establish a large
number of WD-UDP connections.
NIC/TOE, WD-UDP, UM (Agent, Client,
WebGUI)
T5 Hash Filter*
Configures T5 adapters to create more
filters.
NIC, Filtering, UM (Agent, Client, WebGUI)
* Supported only on T5 adapters.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 22
3.1. Pre-requisites
Depending on the component you choose to install, please ensure that the following
requirements are met, before proceeding with the installation.
If you want to install OFED with NFS-RDMA support, please refer “Setting up NFS-RDMA”
in iWARP (RDMA) (Click here).
If you’re planning to install iSCSI PDU Offload Initiator, please install openssl-devel
package.
IPv6 should be enabled in the machine to use the RPM Packages.
3.2. Installing Chelsio Unified Wire from source
3.2.1. GUI mode (with Dialog utility)
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
ii. Untar the tarball using the following command:
[root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the GUI installer:
[root@host~]# ./install.py
iv. If Dialog utility is present, you can skip to step (v). If not, press ‘y’ to install it when the
installer prompts for input.
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 23
v. Select “install” under “Choose an action”
vi. Select Enable IPv6-Offload to install drivers with IPv6 Offload support or Disable IPv6-
offload to continue installation without IPv6 offload support.
vii. Select the required T5/T4 configuration tuning option:
The tuning options may vary depending on the Linux distribution.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 24
viii. Under “Choose install components”, select “all” to install all the related components for the
option chosen in step (vii) or select “custom” to install specific components.
ix. Select the required performance tuning option.
a. Enable Binding IRQs to CPUs: Bind MSI-X interrupts to different CPUs and disable
IRQ balance daemon.
b. Retain IRQ balance daemon: Do not disable IRQ balance daemon.
c. TX-Coalasce: Write tx_coal=2 to modprobe.d/conf.
For more information on the Performance tuning options, please refer to
Performance Tuning section of the Network (NIC/TOE) chapter.
Note
To install benchmark tools, please select “custom option”.
To install Bypass or FCoE PDU Offload Target drivers, please select
Unified Wire in step (vii). Then select “custom” option.
Important
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 25
x. If you already have the required version of OFED software installed, you can skip this step
by selecting Skip-OFED.
To install OFED-3.18-1 choose the Install-OFED option. To install OFED-3.12-1, select
Choose-OFED-Version and then OFED-3.12-1.
xi. The selected components will now be installed:
This step will be prompted only for OFED supported platforms.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 26
xii. After successful installation, summary of installed components will be displayed.
xiii. Select “View log” to view the installation log or “Exit” to continue.
xiv. Select “Yes” to exit the installer or “No” to go back.
xv. Reboot your machine for changes to take effect.
Press Esc or Ctrl+C to exit the installer at any point of time.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 27
3.2.1.1. Installation on updated kernels
If the kernel version on your Linux distribution is updated, follow the steps mentioned below to
install the Unified Wire package:
i. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the GUI installer:
[root@host~]# ./install.py
ii. Select “Yes” to continue with the installation on the updated kernel or “No” to exit.
iii. Select the nearest supported kernel version from the list and select “OK”.
iv. Follow steps (v) to (xv) mentioned in the previous section.
3.2.2. CLI mode (without Dialog utility)
If your system does not have Dialog or you choose not to install it, follow the steps mentioned
below to install the Unified Wire package:
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 28
ii. Untar the tarball using the following command:
[root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the installer:
[root@host~]# ./install.py
iv. When the installer prompts you for your input, press ‘n’ to continue installation without the
Dialog utility.
v. Enter the number corresponding to the Configuration tuning option in the Input field and
press Enter.
vi. If you already have the required version of OFED software installed, you can skip this step.
To install OFED-3.18-1 choose the Install-OFED option. To skip this step, select Skip-
OFED.
vii. The selected components will now be installed.
After successful installation you can press 1 to view the installation log. Press any other key
to exit from the installer.
viii. Reboot your machine for changes to take effect.
3.2.2.1. iWARP driver installation on Cluster nodes
Chelsio’s Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a
single command. Follow the procedure mentioned below:
i. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in
the cluster. You can view the sample file, sample_machinefile, provided in the package to
view the format in which the nodes have to be listed.
To install Bypass driver, run
[root@host~]# ./install.py -c <target> and follow steps (iv) - (vi)
mentioned above.
To customize the installation, view the help by typing
[root@host~]#./install.py –h
Important
Please make sure that you have enabled password less authentication with ssh
on the peer nodes for this feature to work.
Important
This step will be prompted only for OFED supported platforms.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 29
ii. Now, execute the following command:
[root@host~]# ./install.py -C -m <machinefilename>
iii. Select the required T5/T4 configuration tuning option. The tuning options may vary
depending on the Linux distribution.
iv. Select the required Cluster Configuration.
v. If you already have the required version of OFED software installed, you can skip this step.
To install OFED-3.18-1 choose the Install-OFED option.To skip this step, select Skip-OFED.
vi. The selected components will now be installed.
The above commands will install iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes
listed in the machinefilename file.
3.2.3. CLI mode
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
ii. Untar the tarball using the following command:
[root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and build
the source using :
[root@host~]# make
iv. Install the drivers, tools and libraries using the following command:
[root@host~]# make install
v. The default configuration tuning option is Unified Wire. The configuration tuning can be
selected using the following commands:
[root@host~]# make CONF=<T5/T4 configuration>
[root@host~]# make CONF=<T5/T4 configuration> install
Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Linux Page 30
vi. Reboot your machine for changes to take effect.
3.2.3.1. Installation on updated kernels
If the kernel version on your Linux distribution is updated, please execute the following
command to install the Unified Wire package:
[root@host~]# make UNAME_R=<kernel_version>
Where kernel_version is the nearest supported kernel version.
For example, if you want to install the package on a RHEL 6 distribution updated to 2.6.32-
431.20.3. el6 kernel, run the following commands:
[root@host~]# make UNAME_R=2.6.32-431.el6
[root@host~]# make UNAME_R=2.6.32-431.el6 install
To view the list of the supported kernel versions, run the following command:
[root@host~]# make list_kernels
Reboot your machine for changes to take effect.
3.2.4. CLI mode (individual drivers)
You can also choose to install drivers individually. Provided here are steps to build and install
NIC, TOE, iWARP, RDM Block Device, Bypass, WD-UDP, UDP Segmentation Offload, FCoE
PDU Offload target, DCB drivers and benchmarking tools. To know about other drivers, view
help by running make help.
To view the different configuration tuning options, view help by typing
[root@host~]#make help
Note
Steps (iv) and (v) mentioned above will NOT install Bypass, FCoE PDU offload
target, DCB drivers and benchmark tools.They will have to be installed manually.
Please refer to section CLI mode (individual drivers) for instructions on
installing them.
Important
Loading...