Chelsio Communications T520-CR, T580-CR, T520-LL-CR, T522-CR, T580-LP-CR User Manual
...
Chelsio Communications (Headquarters)
370 San Aleso Ave.
Suite 100
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: 408.962.3600
Fax: 408.962.3661
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: +91-80-4039-6800
Fax: +91-80-4039-6807
Chelsio KK (Japan)
SHIMA Akasaka Bldg.
Minato-ku, Tokyo
Japan 107-0052
Tel: 03-6234-4353
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to sales@chelsio.com
Support
For all support related questions please send email to support@chelsio.com
Copyright © 2014.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio T5/T4 Unified Wire For Linux ii
Version
Revision Date
1.0.0
12/08/2011
1.0.1
01/09/2013
1.0.2
01/27/2013
1.0.3
03/26/2013
1.0.4
04/12/2013
1.0.5
06/20/2013
1.0.6
08/17/2013
1.0.7
10/22/2013
1.0.8
03/08/2013
1.0.9
05/15/2013
1.1.0
07/26/2013
1.1.1
08/14/2013
1.1.2
12/06/2013
1.1.3
12/19/2013
1.1.4
03/13/2014
1.1.5
05/02/2014
1.1.6
06/30/2014
Document History
Chelsio T5/T4 Unified Wire For Linux iii
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 12
1. Introduction 13
1.1. Features 13
1.2. Hardware Requirements 14
1.3. Software Requirements 14
1.4. Package Contents 14
2. Hardware Installation 17
3. Software/Driver Installation 20
3.1. Pre-requisites 21
3.2. Installing Chelsio Unified Wire from source 21
3.3. Installing Chelsio Unified Wire from RPM 32
3.4. Firmware update 35
4. Software/Driver Uninstallation 36
4.1. Uninstalling Chelsio Unified Wire from source 36
4.2. Uninstalling Chelsio Unified Wire from RPM 40
5. Configuring Chelsio Network Interfaces 42
5.1. Configuring 40G adapters 42
5.2. Configuring network-scripts 43
5.3. Creating network-scripts 44
5.4. Checking Link 45
6. Software/Driver Update 46
II. NETWORK (NIC/TOE) 47
1. Introduction 48
1.1. Hardware Requirements 48
1.2. Software Requirements 49
2. Software/Driver Loading 50
2.1. Loading in NIC mode (without full offload support) 50
2.2. Loading in TOE mode (with full offload support) 50
3. Software/Driver Unloading 51
3.1. Unloading the NIC driver 51
3.2. Unloading the TOE driver 51
4. Software/Driver Configuration and Fine-tuning 52
4.1. Instantiate Virtual Functions (SR-IOV) 52
4.2. Performance tuning 52
III. VIRTUAL FUNCTION NETWORK (VNIC) 58
1. Introduction 59
1.1. Hardware Requirements 59
Chelsio T5/T4 Unified Wire For Linux iv
1.2. Software Requirements 60
2. Software/Driver Loading 61
2.1. Loading the driver 61
3. Software/Driver Unloading 62
3.1. Unloading the driver 62
4. Software/Driver Configuration and Fine-tuning 63
4.1. Instantiate Virtual Functions 63
IV. IWARP (RDMA) 64
1. Introduction 65
1.1. Hardware Requirements 65
1.2. Software Requirements 65
2. Software/Driver Loading 67
2.1. Compiling and Loading iWARP driver 67
3. Software/Driver Unloading 68
4. Software/Driver Configuration and Fine-tuning 69
4.1. Testing connectivity with ping and rping 69
4.2. Enabling various MPIs 70
4.3. Setting up NFS-RDMA 78
V. WD-UDP 80
1. Introduction 81
1.1. Hardware Requirements 81
1.2. Software Requirements 81
2. Software/Driver Loading 83
3. Software/Driver Unloading 84
4. Software/Driver Configuration and Fine-tuning 85
4.1. Accelerating UDP Socket communications 85
VI. WD-TOE 91
1. Introduction 92
1.1. Hardware Requirements 92
1.2. Software Requirements 92
2. Software/Driver Loading 94
3. Software/Driver Unloading 95
4. Software/Driver Configuration and Fine-tuning 96
4.1. Running the application 96
VII. ISCSI PDU OFFLOAD TARGET 97
1. Introduction 98
Chelsio T5/T4 Unified Wire For Linux v
1.1. Features 98
1.2. Hardware Requirements 99
1.3. Software Requirements 101
2. Software/Driver Loading 103
2.1. Latest iSCSI Software Stack Driver Software 103
2.2. Generating single RPM for T3 and T4 adapters 105
2.3. Obtaining the iSCSI Software License 106
3. Software/Driver Unloading 108
4. Software/Driver Configuration and Fine-tuning 109
4.1. Command Line Tools 109
4.2. iSCSI Configuration File 109
4.3. A Quick Start Guide for Target 110
4.4. The iSCSI Configuration File 113
4.5. Challenge-Handshake Authenticate Protocol (CHAP) 122
4.6. Target Access Control List (ACL) Configuration 124
4.7. Target Storage Device Configuration 125
4.8. Target Redirection Support 128
4.9. The command line interface tools “iscsictl” & “chisns” 129
4.10. Rules of Target Reload (i.e. “on the fly” changes) 134
4.11. System Wide Parameters 135
VIII. ISCSI PDU OFFLOAD INITIATOR 137
1. Introduction 138
1.1. Hardware Requirements 138
1.2. Software Requirements 139
2. Software/Driver Loading 140
3. Software/Driver Unloading 142
4. Software/Driver Configuration and Fine-tuning 143
4.1. Accelerating open-iSCSI Initiator 143
4.2. Auto login from cxgb4i initiator at OS bootup 146
IX. DATA CENTER BRIDGING (DCB) 147
1. Introduction 148
1.1. Hardware Requirements 148
1.2. Software Requirements 148
2. Software/Driver Loading 149
3. Software/Driver Unloading 150
4. Software/Driver Configuration and Fine-tuning 151
4.1. Configuring Cisco Nexus 5010 switch 151
4.2. Configuring the Brocade 8000 switch 154
Chelsio T5/T4 Unified Wire For Linux vi
X. FCOE PDU OFFLOAD TARGET 157
1. Introduction 158
1.1. Hardware Requirements 158
1.2. Software Requirements 158
2. Software/Driver Loading 161
3. Software/Driver Configuration and Fine-tuning 162
3.1. Configuring Cisco Nexus 5010 switch 162
3.2. Collecting port information 165
3.3. Configuring LUNs on Target 169
3.4. Configuring Persistent Target 171
3.5. Verifying initiators connected to the target 172
3.6. Removing LUNs 173
3.7. Performance tuning 173
4. Software/Driver Unloading 176
XI. FCOE FULL OFFLOAD INITIATOR 177
1. Introduction 178
1.1. Hardware Requirements 178
1.2. Software Requirements 178
2. Software/Driver Loading 179
3. Software/Driver Unloading 180
4. Software/Driver Configuration and Fine-tuning 181
4.1. Configuring Cisco Nexus 5010 and Brocade switch 181
4.2. FCoE fabric discovery verification 181
4.3. Formatting the LUNs and Mounting the Filesystem 185
4.4. Creating Filesystem 187
4.5. Mounting the formatted LUN 188
XII. OFFLOAD BONDING DRIVER 189
1. Introduction 190
1.1. Hardware Requirements 190
1.2. Software Requirements 190
2. Software/Driver Loading 192
3. Software/Driver Unloading 193
4. Software/Driver Configuration and Fine-tuning 194
4.1. Offloading TCP traffic over a bonded interface 194
XIII. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) 195
1. Introduction 196
1.1. Hardware Requirements 196
1.2. Software Requirements 197
Chelsio T5/T4 Unified Wire For Linux vii
2. Software/Driver Loading 198
3. Software/Driver Unloading 199
4. Software/Driver Configuration and Fine-tuning 200
4.1. Offloading TCP traffic over a bonded interface 200
4.2. Network Device Configuration 201
XIV. UDP SEGMENTATION OFFLOAD AND PACING 202
1. Introduction 203
1.1. Hardware Requirements 204
1.2. Software Requirements 204
2. Software/Driver Loading 206
3. Software/Driver Unloading 207
4. Software/Driver Configuration and Fine-tuning 208
4.1. Modifying the application 208
4.2. Configuring UDP Pacing 210
XV. OFFLOAD IPV6 DRIVER 211
1. Introduction 212
1.1. Hardware Requirements 212
1.2. Software Requirements 212
2. Software/Driver Loading 214
3. Software/Driver Unloading 215
3.1. Unloading the NIC driver 215
3.2. Unloading the TOE driver 215
XVI. BYPASS DRIVER 216
1. Introduction 217
1.1. Features 217
1.2. Hardware Requirements 218
1.3. Software Requirements 219
2. Software/Driver Loading 220
3. Software/Driver Unloading 221
4. Software/Driver Configuration and Fine-tuning 222
4.1. Starting ba server 222
4.2. Bypass API (CLI) 222
XVII. WD SNIFFING AND TRACING 228
1. Theory of Operation 229
1.1. Hardware Requirements 230
1.2. Software Requirements 231
2. Installation and Usage 232
Chelsio T5/T4 Unified Wire For Linux viii
2.1. Installing basic support 232
2.2. Using Sniffer (wd_sniffer) 232
2.3. Using Tracer (wd_tcpdump_trace) 232
XVIII.CLASSIFICATION AND FILTERING 234
1. Introduction 235
1.1. Hardware Requirements 235
1.2. Software Requirements 236
2. Usage 237
2.1. Configuration 237
2.2. Creating Filter Rules 237
2.3. Listing Filter Rules 239
2.4. Removing Filter Rules 239
2.5. Layer 3 example 240
2.6. Layer 2 example 243
3. Hash/DDR Filters 247
3.1. Creating Filter Rules 247
3.2. Listing Filter Rules 248
3.3. Removing Filter Rules 249
3.4. Swap MAC feature 249
3.5. Hit Counters 250
XIX. TRAFFIC MANAGEMENT 252
1. Introduction 253
1.1. Hardware Requirements 253
1.2. Software Requirements 254
2. Software/Driver Loading 255
3. Software/Driver Unloading 256
4. Software/Driver Configuration and Fine-tuning 257
4.1. Traffic Management Rules 257
4.2. Configuring Traffic Management 259
5. Usage 262
5.1. Non-Offloaded Connections 262
5.2. Offloaded Connections 262
5.3. Offloaded Connections with Modified Application 263
XX. UNIFIED WIRE MANAGER (UM) 264
1. Introduction 265
1.1. Features 265
1.2. Reference Architecture 266
1.3. Unified Wire Manager Components 266
Chelsio T5/T4 Unified Wire For Linux ix
1.4. Authentication and encryption 267
2. Hardware and Software 268
2.1. Supported Adapters 268
2.2. Platform/Component Matrix 269
2.3. Platform/Driver Matrix 269
3. Installing Unified Wire Manager 270
4. Verifying UM components status 271
4.1. Verifying Management Agent 271
4.2. Verifying Management Client 272
4.3. Verifying Management Station 272
5. Management Agent 273
5.1. Communication 273
5.2. Configuration 273
5.3. Service configuration 273
5.4. Firewall 274
6. CLI client 275
6.1. CLI Help system 275
6.2. Client conflict resolution 275
7. Web GUI client 276
7.1. Management Station 276
7.2. Accessing Web Management Interface 277
7.3. Layout and Navigation 280
7.4. Home page 281
7.5. System page 294
7.6. Network page 306
7.7. Storage Page 335
7.8. Hardware Features 369
8. Uninstalling Unified Wire Manager 385
8.1. Uninstalling Management Agent 385
8.2. Uninstalling Management Client 385
8.3. Uninstalling Management Station 386
XXI. UNIFIED BOOT 387
1. Introduction 388
1.1. Hardware Requirements 388
1.2. Software Requirements 389
2. Flashing firmware and option ROM 390
2.1. Using Flash Utility 390
2.2. Using cxgbtool 394
3. Configuring PXE Server 395
4. PXE boot process 396
Chelsio T5/T4 Unified Wire For Linux x
4.1. Legacy PXE boot 396
5. FCoE boot process 401
5.1. Legacy FCoE boot 401
6. iSCSI boot process 410
6.1. Legacy iSCSI boot 410
7. Creating Driver Update Disk (DUD) 423
7.1. Creating DUD for RedHat Enterprise Linux 423
7.2. Creating DUD for Suse Enterprise Linux 423
8. OS Installation 425
8.1. Installation using Chelsio NIC DUD (PXE only) 425
8.2. Installation on FCoE LUN 435
8.3. Installation on iSCSI LUN 440
XXII. LUSTRE FILE SYSTEM 450
1. Introduction 451
1.1. Hardware Requirements 451
1.2. Software Requirements 452
2. Creating/Configuring Lustre File System 453
XXIII. APPENDIX A 455
1. Troubleshooting 456
2. Chelsio End-User License Agreement (EULA) 458
Chelsio T5/T4 Unified Wire For Linux xi
Chapter I. Chelsio Unified Wire
I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For Linux Page 12
Chapter I. Chelsio Unified Wire
1. Introduction
Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, ISCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified
Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and
Filtering.
Ideal for all data, storage and high performance clustering applications, the T5/T4 Adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5/T4 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
Chelsio Unified Wire software comes in two formats: Source code and RPM package forms.
Installing from source requires compiling the package to generate the necessary binaries. You
can choose this method when you are using a custom-built kernel. You can also install the
package using the interactive GUI installer. In other cases, download the RPM package specific
to your operating system and follow the steps mentioned to install the package. Please note that
the OFED software required to install Chelsio iWARP driver comes bundled in both source as
well as RPM packages.
This document describes the installation, use and maintenance of the software and its
various components.
1.1. Features
The Chelsio Unified Wire Package uses a single command to install various drivers and utilities.
It consists of the following software:
Network (NIC/TOE)
Virtual Function Network (vNIC)
iWARP (RDMA)
Chelsio T5/T4 Unified Wire For Linux Page 13
Chapter I. Chelsio Unified Wire
WD-UDP
WD-TOE
iSCSI PDU Offload Target
iSCSI PDU Offload Initiator
Data Center Bridiging (DCB)
FCoE PDU Offload Target
FCoE full offload Initiator
Offload Bonding driver
Offload Multi-Adapter Failover(MAFO)
UDP Segmentation Offload and Pacing
Offload IPv6 driver
Bypass driver
Classification and Filtering feature
Traffic Management feature (TM)
Unified Wire Manager (UM)
Unified Boot Software
Lustre File System
Utility Tools(cop,cxgbtool,t4_perftune,benchmark tools, sniffer & tracer)
libs (iWARP and WD-UDP libraries)
For detailed instructions on loading, unloading and configuring the drivers/tools please refer to
their respective sections.
1.2. Hardware Requirements
The Chelsio Unified Wire software supports Chelsio T5 and T4 Series of Unified Wire Adapters.
To know more about the list of adapters supported by each driver, please refer to their
respective sections.
1.3. Software Requirements
The Chelsio Unified Wire software has been developed to run on 64-bit Linux based platforms
and therefore it is a base requirement for running the driver. To know more about the complete
list of operating systems supported by each driver, please refer to their respective sections.
1.4. Package Contents
1.4.1. Source Package
The Chelsio Unified Wire source package consists of the following files/directories:
Chelsio T5/T4 Unified Wire For Linux Page 14
Chapter I. Chelsio Unified Wire
To use cxbtool for FCoE Initiator driver, use [root@host]# cxgbtool stor –h
Note
install.py,dialog.py: Python scripts needed for the GUI installer.
EULA: Chelsio’s End User License Agreement
install.log: File containing installation summary.
docs: The docs directory contains support documents - README, Release Notes and
User’s Guide (this document) for the software.
libs: This directory is for libraries required to install the WD-TOE, WD-UDP and iWARP
drivers. The libibverbs library has implementation of RDMA verbs which will be used by
iWARP applications for data transfers. The librdmacm library works as an RDMA connection
manager. The libcxgb4 library works as an interface between the above mentioned generic
libraries and Chelsio iWARP driver. The libcxgb4_sock library is a LD_PRELOAD-able
library that accelerates UDP Socket communications transparently and without
recompilation of the user application.
Makefile: The Makefile for building and installing from the source.
sample_machinefile: Sample file used during iWARP installation on cluster nodes.
scripts: Support scripts used by the Unified Wire Installer.
specs: The packaging specification files required for building RPM packages.
src: Source code for different drivers.
support: This directory contains source files for the dialog utility.
tools:
ba_tools: Management and configuration tools for bypass adapters.
benchmarks: This directory contains various benchmarking tools to measure
throughput and latency of various networks.
cop: The cop tool compiles offload policies into a simple program form that can be
loaded into the kernel and interpreted. These offload policies are used to determine
the settings to be used for various connections. The connections to which the
settings are applied are based on matching filter specifications. Please find more
details on this tool in its manual page (run man cop command).
cxgbtool: The cxgbtool queries or sets various aspects of Chelsio network
interface cards. It complements standard tools used to configure network settings
and provides functionality not available through such tools. Please find more details
on this tool in its manual page (run man cxgbtool command).
rdma_tools: This directory contains iWARP benchmarking tools.
sniffer: This directory contains sniffer tracing and filtering libraries. See WD Sniffing
and Tracing chapter for more information.
um: This directory contains Unified Wire Manager RPMs for different distributions
and Management Station configuration files.
chsetup: The chsetup tool loads NIC,TOE and iWARP drivers, and creates WD-
UDP configuration file.
chstatus: This utility provides status information on any Chelsio NIC in the system.
Chelsio T5/T4 Unified Wire For Linux Page 15
Chapter I. Chelsio Unified Wire
t4_perftune.sh: This shell script is to tune the system for higher performance. It
achieves it through modifying the IRQ-CPU binding. This script can also be used to
change Tx coalescing settings.
t4-forward.sh: RFC2544 Forward test tuning script.
uname_r: This file is used by chstatus script to verify if the Linux platform is
supported or not.
wdload: UDP acceleration tool.
wdunload: Used to unload all the loaded Chelsio drivers.
bootcfg: This utility generates a default boot configuration file in binary format.
chiscsi_set_affinity.sh: This shell script is used for mapping iSCSI Worker threads
to different CPUs.
chelsio_adapter_config: This directory contains scripts and binaries needed to
configure Chelsio 40G Adapters.
Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD.
The OptionROM directory contains Unified Boot Option ROM image (cuwlbt4.bin) and a
Legacy Flash Utility (cfut4.exe), which can be used to flash Unified Boot Option ROM onto
Chelsio's T5 and T4 Converged Network Adapters (CNAs).
The LinuxDUD directory contains image (.img) files required to update drivers for Linux
(RHEL and SLES) distributions.
1.4.2. RPM package
The Chelsio Unified Wire RPM package consists of the following:
docs: The docs directory contains support documents i.e. README, Release Notes and
User’s Guide (this document) for the software.
scripts: Support scripts used by the Unified Wire Installer.
DRIVER-RPMS: RPM packages of Chelsio drivers.
OFED-RPMS: OFED RPM packages required to install iWARP driver.
install.py: Python script that installs the RPM package. See Software/Driver Installation
section for more information.
uninstall.py: Python script that uninstalls the RPM package. See Software/Driver
Uninstallation section for more information.
EULA: Chelsio’s End User License Agreement.
Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD.
The OptionROM directory contains Unified Boot Option ROM image (cuwlbt4.bin) and a
Legacy Flash Utility (cfut4.exe), which can be used to flash Unified Boot Option ROM onto
Chelsio's T5 and T4 Converged Network Adapters (CNAs).
The LinuxDUD directory contains image (.img) files required to update drivers for Linux
(RHEL and SLES) distributions.
Chelsio T5/T4 Unified Wire For Linux Page 16
Chapter I. Chelsio Unified Wire
2. Hardware Installation
Follow these steps to install Chelsio Adapter in your system:
1. Shutdown/power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIE x8
slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 40/10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Verify if the adapter was installed successfully by using the lspci command
a. For T5 adapters :
[root@host]# lspci |grep -i Chelsio
07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.3 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR
Unified Wire Storage Controller
07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire
Storage Controller
Chelsio T5/T4 Unified Wire For Linux Page 17
Chapter I. Chelsio Unified Wire
b. And for T4 adapters :
[root@host]# lspci | grep –i Chelsio
03:00.0 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.1 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.2 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.3 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.4 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.5 SCSI storage controller: Chelsio Communications Inc T420-CR Unified
Wire Storage Controller
03:00.6 Fibre Channel: Chelsio Communications Inc T420-CR Unified Wire
Storage Controller
03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000
For Chelsio T5/T4 adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
Once Unified Wire package is installed and loaded, examine the output of dmesg to see if the
card is discovered.
For T5 adapters:
eth2: Chelsio T520-LL rev 0 10GBASE-SFP RNIC PCIe 8 GT/s x8 MSI-X, Offload
capable
0000:07:00.4: S/N: RE12130097, P/N: 11011675004
And, for T4 adapters:
eth0: Chelsio T420-CR rev 2 10GBASE-SFP RNIC PCIe x8 MSI-X
0000:04:00.4: S/N: PT18111226, P/N: 110112140D0
Chelsio T5/T4 Unified Wire For Linux Page 18
Chapter I. Chelsio Unified Wire
Network device names for Chelsio’s physical ports are assigned using the
following convention: the port farthest from the motherboard will appear as the
first network interface. However, for T5 40G and T420-BT adapters, the
association of physical Ethernet ports and their corresponding network device
names is opposite. For these adapters, the port nearest to the motherboard will
appear as the first network interface.
Note
The above outputs indicate the hardware configuration of the adapters as well as the Serial
numbers. As observed by the x8, the card is properly installed in an x8 slot on the machine and
using MSI-X interrupts.
Chelsio T5/T4 Unified Wire For Linux Page 19
Chapter I. Chelsio Unified Wire
T5/T4
Configuration
Tuning Option
Description
Driver/Software installed
Unified Wire
Configures adapters to run multiple protocols
like NIC/TOE, iWARP, iSCSI and FCoE
Initiator simultaneously.
NIC/TOE, vNIC, iWARP, WD-UDP,
iSCSITarget, iSCSI Initiator, Bonding,
MAFO,IPv6, Sniffer & Tracer, DCB, FCoE
Initiator, UM(Agent,Client,WebGUI),
Filtering, TM, Lustre
Low latency
Networking
Configures adapters to run NIC/TOE and
iWARP traffic with low latency specially
needed for financial applications.
NIC/TOE, vNIC, iWARP, WD-UDP, WD-
TOE, IPv6, Sniffer & Tracer,
UM(Agent,Client,WebGUI), Filtering, TM
High capacity
RDMA
Configures adapters to establish a large
number of RDMA connections.
NIC/TOE, vNIC, iWARP, WD-UDP, Bonding,
MAFO,IPv6, Sniffer & Tracer,
UM(Agent,Client,WebGUI), Filtering, TM
RDMA
Performance
Improves RDMA performance on T5/T4
adapters.
NIC/TOE,iWARP,
UM(Agent,Client,WebGUI)
High capacity
TOE
Configures adapters to establish a large
number of TOE connections.
NIC/TOE, vNIC, Bonding, MAFO,IPv6,
UM(Agent,Client,WebGUI), Filtering,
TM
iSCSI
Performance*
Improves iSCSI performance on T5
adapters.
NIC/TOE,iSCSI Target, iSCSI Initiator,
UM(Agent,Client,WebGUI)
UDP Seg.Offload
& Pacing
Configures adapters to establish a large
number of UDP Segmentation Offload
connections.
NIC/TOE, IPv6, USO,
UM(Agent,Client,WebGUI), Filtering, TM
T5 Wire Direct
Latency*
Configures T5 adapters to provide low Wire
Direct latency.
NIC/TOE,iWARP,WD-UDP,WD-TOE,
UM(Agent,Client,WebGUI)
T5 High Capacity
WD*
Configures T5 adapters to establish a large
number of WD-UDP connections.
NIC/TOE,WD-UDP,WD-TOE,
UM(Agent,Client,WebGUI)
T5 Hash Filter*
Configures T5 adapters to create more
filters.
NIC,Filtering, UM(Agent,Client,WebGUI)
3. Software/Driver Installation
There are two main methods to install the Chelsio Unified Wire package: from source and RPM.
If you decide to use source, you can install the package using CLI or GUI mode.
If you decide to use RPM, you can install the package using Menu or CLI mode.
Irrespective of the method chosen for installation, the machine needs to be rebooted for
changes to take effect.
The following table describes the various configuration tuning options available during
installation and drivers/software installed with each option by default:
* Supported only on T5 adapters.
Chelsio T5/T4 Unified Wire For Linux Page 20
Chapter I. Chelsio Unified Wire
3.1. Pre-requisites
Depending on the component you choose to install, please ensure that the following
requirements are met, before proceeding with the installation.
If you want to install OFED with NFS-RDMA support, please refer “Setting up NFS-RDMA”
in iWARP (RDMA) (Click here).
If you’re planning to install iSCSI PDU Offload Initiator, please install openssl-devel
package.
IPv6 should be enabled in the machine to use the RPM Packages.
3.2. Installing Chelsio Unified Wire from source
3.2.1. GUI mode (with Dialog utility)
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
ii. Untar the tarball using the following command:
[root@host]# tar -zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the GUI installer:
[root@host]# ./install.py
iv. If Dialog utility is present, you can skip to step (v). If not, press ‘y’ to install it when the
installer prompts for input.
Chelsio T5/T4 Unified Wire For Linux Page 21
Chapter I. Chelsio Unified Wire
The tuning options may vary depending on the Linux distribution.
Note
v. Select “install” under “Choose an action”
vi. Select Enable IPv6-Offload to install drivers with IPv6 Offload support or Disable IPv6-
offload to continue installation without IPv6 offload support.
vii. Select the required T5/T4 configuration tuning option:
Chelsio T5/T4 Unified Wire For Linux Page 22
Chapter I. Chelsio Unified Wire
For more information on the Performance tuning options, please refer to
Software/Driver Configuration and Fine-tuning section of the Network
(NIC/TOE) chapter.
Note
To install Bypass or FCoE PDU Offload Target drivers, please select Unified
Wire in step (ix). Then select “custom” option.
Important
viii. Under “Choose install components”, select “all” to install all the related components for the
option chosen in step (ix) or select “custom” to install specific components.
ix. Select the required performance tuning option.
a. Enable Binding IRQs to CPUs: Bind MSI-X interrupts to different CPUs and disable
IRQ balance daemon.
b. Retain IRQ balance daemon: Do not disable IRQ balance daemon.
c. TX-Coalasce: Write tx_coal=2 to modprobe.d/conf.
Chelsio T5/T4 Unified Wire For Linux Page 23
Chapter I. Chelsio Unified Wire
OFED is currently not supported on RHEL 6.5
Note
x. If you already have the required version of OFED software installed, you can skip this step.
To install OFED-3.5-2 choose the Install-OFED option. To install a different version, select
Choose-OFED-Version and then select the appropriate version. To skip this step, select
Skip-OFED.
xi. The selected components will now be installed:
Chelsio T5/T4 Unified Wire For Linux Page 24
Chapter I. Chelsio Unified Wire
Press Esc or Ctrl+C to exit the installer at any point of time.
Note
xii. After successful installation, summary of installed components will be displayed.
xiii. Select “View log” to view the installation log or “Exit” to continue.
xiv. Select “Yes” to exit the installer or “No” to go back.
xv. Reboot your machine for changes to take effect.
Chelsio T5/T4 Unified Wire For Linux Page 25
Chapter I. Chelsio Unified Wire
3.2.1.1. Installation on updated kernels
If the kernel version on your Linux distribution is updated, follow the steps mentioned below to
install the Unified Wire package:
i. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the GUI installer:
[root@host]# ./install.py
ii. Select “Yes” to continue with the installation on the updated kernel or “No” to exit.
iii. Select the nearest supported kernel version from the list and select “OK”.
iv. Follow steps (xv) to (xvii) mentioned in the previous section.
3.2.2. CLI mode (without Dialog utility)
If your system does not have Dialog or you choose not to install it, follow the steps mentioned
below to install the Unified Wire package:
Chelsio T5/T4 Unified Wire For Linux Page 26
Chapter I. Chelsio Unified Wire
To install Bypass or WD-TOE driver, run
[root@host]# ./install.py -c <target> and follow steps (iv) - (vi)
mentioned above.
To customize the installation, view the help by typing
[root@host]#./install.py –h
Important
OFED is currently not supported on RHEL 6.5.
Note
Please make sure that you have enabled password less authentication with ssh
on the peer nodes for this feature to work.
Important
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
ii. Untar the tarball using the following command:
[root@host]# tar -zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and run the
following script to start the installer:
[root@host]# ./install.py
iv. When the installer prompts you for your input, press ‘n’ to continue installation without the
Dialog utility.
v. Enter the number corresponding to the Configuration tuning option in the Input field and
press Enter.
vi. If you already have the required version of OFED software installed, you can skip this step.
To install OFED-3.5-2 choose the Install-OFED option. To install a different version, select
Choose-OFED-Version and then select the appropriate version. To skip this step, select
Skip-OFED.
vii. The selected components will now be installed.
After successful installation you can press 1 to view the installation log. Press any other key
to exit from the installer.
viii. Reboot your machine for changes to take effect.
3.2.2.1. iWARP driver installation on Cluster nodes
Chelsio’s Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a
single command. Follow the procedure mentioned below:
Chelsio T5/T4 Unified Wire For Linux Page 27
Chapter I. Chelsio Unified Wire
OFED is currently not supported on RHEL 6.5.
Note
i. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in
the cluster. You can view the sample file, sample_machinefile, provided in the package to
view the format in which the nodes have to be listed.
ii. Now, execute the following command:
[root@host] # ./install.py -C -m <machinefilename>
iii. Select the required T5/T4 configuration tuning option. The tuning options may vary
depending on the Linux distribution.
iv. Select the required Cluster Configuration.
v. If you already have the required version of OFED software installed, you can skip this step.
To install OFED-3.5-2 choose the Install-OFED option. To install a different version, select
Choose-OFED-Version and then select the appropriate version. To skip this step, select
Skip-OFED.
vi. The selected components will now be installed.
3.2.3. CLI mode
i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz from Chelsio Download Center,
http://service.chelsio.com/
ii. Untar the tarball using the following command:
[root@host]# tar -zxvfm ChelsioUwire-x.xx.x.x.tar.gz
iii. Change your current working directory to Chelsio Unified Wire package directory and build
the source using :
[root@host]# make
iv. Install the drivers, tools and libraries using the following command:
[root@host]# make install
Chelsio T5/T4 Unified Wire For Linux Page 28
Chapter I. Chelsio Unified Wire
To view the different configuration tuning options, view help by typing
[root@host]#make help
Note
Steps (iv)and (v) mentioned above will NOT install Bypass, FCoE PDU offload
target drivers and benchmark tools.They will have to be installed manually.
Please refer to section CLI mode (individual drivers) for instructions on
installing them.
Important
v. The default configuration tuning option is Unified Wire. The configuration tuning can be
selected using the following commands:
[root@host]# make CONF=<T5/T4 configuration>
[root@host]# make CONF=<T5/T4 configuration> install
vi. Reboot your machine for changes to take effect.
3.2.3.1. Installation on updated kernels
If the kernel version on your Linux distribution is updated, please execute the following
command to install the Unified Wire package:
[root@host]# make UNAME_R=<kernel_version>
Where kernel_version is the nearest supported kernel version.
For example, if you want to install the package on a RHEL 6 distribution updated to 2.6.32-
279.2.1.el6 kernel, run the following commands:
[root@host]# make UNAME_R=2.6.32-279.el6
[root@host]# make UNAME_R=2.6.32-279.el6 install
To view the list of the supported kernel versions, run the following command:
[root@host]# make list_kernels
Reboot your machine for changes to take effect.
Chelsio T5/T4 Unified Wire For Linux Page 29
Chapter I. Chelsio Unified Wire
3.2.4. CLI mode (individual drivers)
You can also choose to install drivers individually. Provided here are steps to build and install
NIC, TOE, iWARP, Bypass, WD-UDP, WD-TOE, UDP Segmentation Offload, FCoE PDU
Offload target drivers and benchmarking tools. To know about other drivers, view help by
running make help.
To build and install NIC driver without offload support :
[root@host]# make nic
[root@host]# make nic_install
To build and install NIC driver with offload support and Offload drivers:
[root@host]# make toe
[root@host]# make toe_install
To build and install Offload drivers without IPv6 support:
[root@host]# make toe_ipv4
[root@host]# make toe_ipv4_install
To build and install iWARP driver against outbox OFED:
[root@host]# make iwarp
[root@host]# make iwarp_install
To build and install all drivers without IPv6 support:
[root@host]# make ipv6_disable=1
[root@host]# make ipv6_disable=1 install
The above step will not install Bypass and WD-TOE drivers.
To build and install Bypass driver:
[root@host]# make bypass
[root@host]# make bypass_install
Chelsio T5/T4 Unified Wire For Linux Page 30
Loading...