Chelsio Communications Terminator Series, Terminator 6 Installation And User Manual

Chelsio Unified Wire for Linux i

Chelsio Unified Wire for Linux ii

This document and related products are distributed under licenses restricting their use, copying, distribution,
and reverse-engineering.

No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.

All third-party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED

WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO COMMUNICATIONS,
INC.

Sales

For all sales inquiries please send email to [e-mail]

Support

For all support related questions please send email to [e-mail]
Copyright © 2018. Chelsio Communications. All Rights Reserved.

Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.

Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A

www.chelsio.com

Tel: [phone]
Fax: [phone]

Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India

Tel: [phone]

Chelsio KK (Japan)

Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio Unified Wire for Linux iii

Document History

Version

Revision Date

1.0.0

12/08/2011

1.0.1

01/09/2013

1.0.2

01/27/2013

1.0.3

03/26/2013

1.0.4

04/12/2013

1.0.5

06/20/2013

1.0.6

08/17/2013

1.0.7

10/22/2013

1.0.8

03/08/2013

1.0.9

05/15/2013

1.1.0

07/26/2013

1.1.1

08/14/2013

1.1.2

12/06/2013

1.1.3

12/19/2013

1.1.4

03/13/2014

1.1.5

05/02/2014

1.1.6

06/30/2014

1.1.7

10/22/2014

1.1.8

11/04/2014

1.1.9

02/05/2015

1.2.0

03/04/2015

1.2.1

03/25/2015

1.2.2

06/03/2015

1.2.3

08/05/2015

1.2.4

02/29/2016

1.2.5

04/27/2016

1.2.6

08/25/2016

1.2.7

10/07/2016

1.2.8

10/18/2016

1.2.9

11/11/2016

1.3.0

12/05/2016

1.3.1

12/30/2016

1.3.2

01/30/2017

1.3.3

02/27/2017

1.3.4

05/11/2017

1.3.5

09/05/2017

Chelsio Unified Wire for Linux iv

1.3.6

09/29/2017

1.3.7

12/29/2017

1.3.8

02/28/2018

1.3.9

03/30/2018

1.4.0

04/18/2018

1.4.1

07/05/2018

1.4.2

07/18/2018

1.4.3

10/01/2018

Chelsio Unified Wire for Linux v

TABLE OF CONTENTS

I. CHELSIO UNIFIED WIRE 15

Introduction 16

1.1. Features 16

1.2. Hardware Requirements 17

1.3. Software Requirements 17

1.4. Package Contents 17

Hardware Installation 21
Software/Driver Installation 23

3.1. Pre-requisites 23

3.2. Enabling RDMA on ARM Platforms 24

3.3. Mounting debugfs 24

3.4. Allowing unsupported modules on SLES 25

3.5. Installing Chelsio Unified Wire from source 25

3.6. Installing Chelsio Unified Wire from RPM 32

3.7. Firmware Update 36

3.8. Removing Drivers from initramfs 36

Configuring Chelsio Network Interfaces 37

4.1. Configuring Adapters 37

4.2. Configuring network-scripts 41

4.3. Creating network-scripts 41

4.4. Checking Link 42

Performance Tuning 43

5.1. Generic 43

5.2. Throughput 43

5.3. Latency 43

Software/Driver Update 45
Software/Driver Uninstallation 46

7.1. Uninstalling Chelsio Unified Wire from source 46

7.2. Uninstalling Chelsio Unified Wire from RPM 50

II. NETWORK (NIC/TOE) 51

Introduction 52

1.1. Hardware Requirements 52

1.2. Software Requirements 53

Software/Driver Installation 54
Software/Driver Loading 55

3.1. Loading in NIC mode (without full offload support) 55

3.2. Loading in TOE mode (with full offload support) 55

Software/Driver Configuration 56

4.1. Enabling TCP Offload 56

Chelsio Unified Wire for Linux vi

4.2. Enabling Busy waiting 56

4.3. Precision Time Protocol (PTP) 57

4.4. VXLAN Offload 59

4.5. Performance Tuning 62

Software/Driver Unloading 68

5.1. Unloading the NIC Driver 68

5.2. Unloading the TOE Driver 68

III. VIRTUAL FUNCTION NETWORK (VNIC) 70

Introduction 71

1.1. Hardware Requirements 71

1.2. Software Requirements 72

Software/Driver Installation 73

2.1. Pre-requisites 73

2.2. Installation 73

Software/Driver Loading 74

3.1. Instantiate Virtual Functions (SR-IOV) 74

3.2. Loading the Driver 74

Software/Driver Configuration and Fine-tuning 76

4.1. VF Rate Limiting 76

4.2. Bonding 77

4.3. High Capacity VF Configuration 78

Software/Driver Unloading 81

5.1. Unloading the Driver 81

IV. IWARP (RDMA) 82

Introduction 83

1.1. Hardware Requirements 83

1.2. Software Requirements 84

Software/Driver Installation 85

2.1. Pre-requisites 85

2.2. Installation 85

Software/Driver Loading 86

3.1. Loading iWARP Driver 86

Software/Driver Configuration and Fine-tuning 87

4.1. Testing connectivity with ping and rping 87

4.2. Enabling various MPIs 88

4.3. Setting up NFS-RDMA 96

4.4. Performance Tuning 97

Software/Driver Unloading 98

V. ISER 99

Chelsio Unified Wire for Linux vii

Introduction 100

1.1. Hardware Requirements 100

1.2. Software Requirements 100

Kernel Installation 101
Software/Driver Installation 102

3.1. Pre-requisites 102

3.2. Installation 102

Software/Driver Loading 103
Software/Driver Configuration and Fine-tuning 104

5.1. HMA 105

5.2. Performance Tuning 105

Software/Driver Unloading 106

VI. WD-UDP 107

Introduction 108

1.1. Hardware Requirements 108

1.2. Software Requirements 108

Software/Driver Installation 110
Software/Driver Loading 111
Software/Driver Configuration and Fine-tuning 112

4.1. Accelerating UDP Socket Communications 112

Software/Driver Unloading 118

VII. WD-TOE 119

Introduction 120

1.1. Hardware Requirements 120

1.2. Software Requirements 120

Software/Driver Installation 121

2.1. Pre-requisites 121

2.2. Installation 121

Software/Driver Loading 122
Software/Driver Configuration and Fine-tuning 123

4.1. Running the Application 123

Software/Driver Unloading 124

VIII. NVME-OF 125

Introduction 126

1.1. Hardware Requirements 126

1.2. Software Requirements 126

Kernel Installation 128
Software/Driver Installation 129

3.1. Pre-requisites 129

Chelsio Unified Wire for Linux viii

3.2. Installation 129

Software/Driver Loading 130
Software/Driver Configuration and Fine-tuning 131

5.1. Target 131

5.2. Initiator 132

5.3. HMA 132

5.4. Performance Tuning 133

Software/Driver Unloading 134

IX. LIO ISCSI TARGET OFFLOAD 135

Introduction 136

1.1. Hardware Requirements 136

1.2. Software Requirements 136

Kernel Configuration 138
Software/Driver Installation 141

3.1. Pre-requisites 141

3.2. Installation 141

Software/Driver Loading 143
Software/Driver Configuration and Fine-tuning 144

5.1. Configuring LIO iSCSI Target 144

5.2. Offloading LIO iSCSI Connection 144

5.3. Running LIO iSCSI and Network Traffic Concurrently 145

5.4. Performance Tuning 146

Software/Driver Unloading 147

6.1. Unloading the LIO iSCSI Target Offload Driver 147

6.2. Unloading the NIC Driver 147

X. ISCSI PDU OFFLOAD TARGET 148

Introduction 149

1.1. Features 149

1.2. Hardware Requirements 150

1.3. Software Requirements 151

Software/Driver Installation 153
Software/Driver Loading 154

3.1. Latest iSCSI Software Stack Driver Software 154

Software/Driver Configuration and Fine-tuning 156

4.1. Command Line Tools 156

4.2. iSCSI Configuration File 156

4.3. A Quick Start Guide for Target 157

4.4. The iSCSI Configuration File 159

4.5. Challenge-Handshake Authenticate Protocol (CHAP) 170

4.6. Target Access Control List (ACL) Configuration 172

Chelsio Unified Wire for Linux ix

4.7. Target Storage Device Configuration 174

4.8. Target Redirection Support 176

4.9. The command line interface tools “iscsictl” & “chisns” 178

4.10. Rules of Target Reload (i.e. “on the fly” changes) 183

4.11. System Wide Parameters 184

4.12. Performance Tuning 185

Software/Driver Unloading 186

XI. ISCSI PDU OFFLOAD INITIATOR 187

Introduction 188

1.1. Hardware Requirements 188

1.2. Software Requirements 189

Software/Driver Installation 190

2.1. Pre-requisites 190

2.2. Installation 190

Software/Driver Loading 191
Software/Driver Configuration and Fine-tuning 192

4.1. Accelerating open-iSCSI Initiator 192

4.2. HMA 194

4.3. Auto login from cxgb4i initiator at OS bootup 195

4.4. Performance Tuning 196

Software/Driver Unloading 197

XII. CRYPTO OFFLOAD 198

Introduction 199

1.1. Hardware Requirements 199

1.2. Software Requirements 199

Kernel Configuration 200
Software/Driver Installation 203

3.1. Pre-requisites 203

3.2. Installation 203

Software/Driver Loading 204

4.1. Co-processor 204

4.2. Inline 204

Software/Driver Configuration and Fine-tuning 205

5.1. Co-processor 205

5.2. Inline 206

5.3. Performance Tuning 209

Software/Driver Unloading 210

XIII. DATA CENTER BRIDGING (DCB) 211

Introduction 212

Chelsio Unified Wire for Linux x

1.1. Hardware Requirements 212

1.2. Software Requirements 212

Software/Driver Installation 214
Software/Driver Loading 215
Software/Driver Configuration and Fine-tuning 217

4.1. Configuring Cisco Nexus 5010 switch 217

4.2. Configuring the Brocade 8000 switch 220

Running NIC & iSCSI Traffic together with DCBx 222

XIV. FCOE FULL OFFLOAD INITIATOR 223

Introduction 224

1.1. Hardware Requirements 224

1.2. Software Requirements 224

Software/Driver Installation 225
Software/Driver Loading 226
Software/Driver Configuration and Fine-tuning 227

4.1. Configuring Cisco Nexus 5010 and Brocade switch 227

4.2. FCoE fabric discovery verification 227

4.3. Formatting the LUNs and Mounting the Filesystem 231

4.4. Creating Filesystem 232

4.5. Mounting the formatted LUN 233

Software/Driver Unloading 234

XV. OFFLOAD BONDING 235

Introduction 236

1.1. Hardware Requirements 236

1.2. Software Requirements 236

Software/Driver Installation 238
Software/Driver Loading 239
Software/Driver Configuration and Fine-tuning 240

4.1. Offloading TCP traffic over a bonded interface 240

Software/Driver Unloading 241

XVI. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) 242

Introduction 243

1.1. Hardware Requirements 243

1.2. Software Requirements 244

Software/Driver Installation 245
Software/Driver Loading 246
Software/Driver Configuration and Fine-tuning 247

4.1. Offloading TCP traffic over a bonded interface 247

Software/Driver Unloading 248

Chelsio Unified Wire for Linux xi

XVII. UDP SEGMENTATION OFFLOAD AND PACING 249

Introduction 250

1.1. Hardware Requirements 251

1.2. Software Requirements 251

Software/Driver Installation 252
Software/Driver Loading 253
Software/Driver Configuration and Fine-tuning 254

4.1. Modifying the Application 254

4.2. Configuring UDP Pacing 255

Software/Driver Unloading 257

XVIII.OFFLOAD IPV6 258

Introduction 259

1.1. Hardware Requirements 259

1.2. Software Requirements 259

Software/Driver Installation 261

2.1. Pre-requisites 261

2.2. Installation 261

Software/Driver Loading 262
Software/Driver Configuration and Fine-tuning 263
Software/Driver Unloading 264

5.1. Unloading the NIC Driver 264

5.2. Unloading the TOE Driver 264

XIX. WD SNIFFING AND TRACING 265

Theory of Operation 266

1.1. Hardware Requirements 267

1.2. Software Requirements 268

Software/Driver Installation 269
Usage 270

3.1. Installing Basic Support 270

3.2. Using Sniffer (wd_sniffer) 270

3.3. Using Tracer (wd_tcpdump_trace) 270

XX. CLASSIFICATION AND FILTERING 272

Introduction 273

1.1. Hardware Requirements 273

1.2. Software Requirements 274

LE-TCAM Filters 275

2.1. Configuration 275

2.2. Creating Filter Rules 278

2.3. Listing Filter Rules 279

Chelsio Unified Wire for Linux xii

2.4. Removing Filter Rules 280

2.5. Layer 3 Example 280

2.6. Layer 2 Example 282

2.7. Filtering VF traffic 284

Hash/DDR Filters 286

3.1. Configuration 286

3.2. Creating Filter Rules 288

3.3. Listing Filter Rules 291

3.4. Removing Filter Rules 291

3.5. Filter Priority 291

3.6. Swap MAC Feature 292

3.7. Traffic Mirroring 292

3.8. Packet Tracing and Hit Counters 294

NAT Filtering 296

XXI. OVS KERNEL DATAPATH OFFLOAD 297

Introduction 298

1.1. Hardware Requirements 298

1.2. Software Requirements 299

Software/Driver Installation 300

2.1. Pre-requisites 300

2.2. Installation 300

Software/Driver Configuration and Fine Tuning 301

3.1. Configuring OVS Machine 302

3.2. Creating OVS flows 304

3.3. Verifying OVS Flow Dump 308

3.4. Setting up ODL with OVS 308

Software/Driver Uninstallation 310

XXII. RING BACKBONE 311

Introduction 312

1.1. Hardware Requirements 312

1.2. Software Requirements 312

1.3. Ring Connectivity 313

Software/Driver Installation 314
Software/Driver Configuration and Fine-tuning 315

XXIII. TRAFFIC MANAGEMENT 317

Introduction 318

1.1. Hardware Requirements 318

1.2. Software Requirements 319

Software/Driver Loading 320

Chelsio Unified Wire for Linux xiii

Software/Driver Configuration and Fine-tuning 321

3.1. Traffic Management Rules 321

3.2. Configuring Traffic Management 323

Usage 326

4.1. Non-Offloaded Connections 326

4.2. Offloaded Connections 326

4.3. Offloaded Connections with Modified Application 327

Software/Driver Unloading 328

XXIV. DPDK DRIVER 329

Introduction 330

1.1. Hardware Requirements 330

1.2. Software Requirements 331

Software/Driver Installation 332

2.1. Pre-requisites 332

2.2. Installation 332

Flashing Firmware Configuration File 333
Software/Driver Loading 334
Software/Driver Configuration and Fine Tuning 335

5.1. Huge Pages 335

5.2. Binding Network Ports 336

5.3. Unbinding Network Ports 338

5.4. Performance Tuning 338

Running DPDK Test Applications 339

6.1. Testpmd application 339

6.2. Pktgen Application 341

6.3. Runtime Options 344

Software/Driver Unloading 345
Software/Driver Uninstallation 346

XXV. UNIFIED BOOT 347

Introduction 348

1.1. Hardware Requirements 348

1.2. Software Requirements 349

1.3. Pre-requisites 350

Secure Boot 351
Flashing firmware and option ROM 352

3.1. Preparing USB flash drive 352

3.2. Legacy 353

3.3. uEFI 356

3.4. HP Firmware Management Protocol (FMP) 362

3.5. Default Option ROM Settings 366

Chelsio Unified Wire for Linux xiv

Configuring PXE Server 368
PXE Boot Process 369

5.1. Legacy PXE Boot 369

5.2. uEFI PXE Boot 372

FCoE Boot Process 376

6.1. Legacy FCoE Boot 376

6.2. uEFI FCoE Boot 381

iSCSI Boot Process 387

7.1. Legacy iSCSI Boot 387

7.2. uEFI iSCSI Boot 395

Creating Driver Update Disk (DUD) 404

8.1. Creating DUD for RedHat Enterprise Linux 404

8.2. Creating DUD for Suse Enterprise Linux 404

OS Installation 406

9.1. Installation using Chelsio DUD 406

9.2. Installation on FCoE LUN 417

9.3. Installation on iSCSI LUN 420

XXVI. APPENDIX A 427

Troubleshooting 428
Chelsio End-User License Agreement (EULA) 430

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 15

I. Chelsio Unified Wire

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 16

Introduction

Thank you for choosing Chelsio Unified Wire adapters. These high speed, single chip, single
firmware cards provide enterprises and data centers with high performance solutions for various
Network and Storage related requirements.

The Terminator series is Chelsio’s next generation of highly integrated, hyper-virtualized
1/10/25/40/50/100GbE controllers. The adapters are built around a programmable protocol­processing engine, with full offload of a complete Unified Wire solution comprising NIC, TOE,
iWARP RDMA, iSCSI, FCoE and NAT support. It scales to true 100Gb line rate operation from a
single TCP connection to thousands of connections, and allows simultaneous low latency and
high bandwidth operation thanks to multiple physical channels through the ASIC.

Ideal for all data, storage and high-performance clustering applications, the Unified Wire
adapters enable a unified fabric over a single wire by simultaneously running all unmodified IP
sockets, Fibre Channel and InfiniBand applications over Ethernet at line rate.

Designed for deployment in virtualized data centers, cloud service installations and high­performance computing environments, Chelsio adapters bring a new level of performance metrics
and functional capabilities to the computer networking industry.

Chelsio Unified Wire software comes in two formats: Source code and RPM package forms.
Installing from source requires compiling the package to generate the necessary binaries. You
can choose this method when you are using a custom-built kernel. You can also install the
package using the interactive GUI installer. In other cases, download the RPM package specific
to your operating system and follow the steps mentioned to install the package. Please note that
the OFED software required to install Chelsio iWARP driver comes bundled in both source as well
as RPM packages.

This document describes the installation, use and maintenance of Unified Wire software and
its various components.

1.1. Features

The Chelsio Unified Wire package uses a single command to install various drivers and utilities.
It consists of the following software:

Network (NIC/TOE)
Virtual Function Network (vNIC)
iWARP (RDMA)
iSER
WD-UDP
WD-TOE
NVMe-oF

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 17

LIO iSCSI Target Offload
iSCSI PDU Offload Target
iSCSI PDU Offload Initiator
Crypto Offload
Data Center Bridging (DCB)
FCoE full offload Initiator
Offload Bonding
Offload Multi-Adapter Failover(MAFO)
UDP Segmentation Offload and Pacing
Offload IPv6
Classification and Filtering feature
OVS Kernel Datapath Offload
Ring Backbone
Traffic Management feature (TM)
DPDK
Unified Boot Software
Utility Tools (cop, cxgbtool, t4_perftune, benchmark tools, sniffer & tracer)
libs (iWARP, WD-UDP and WD-TOE libraries)

For detailed instructions on loading, unloading and configuring the drivers/tools please refer to
their respective sections.

1.2. Hardware Requirements

The Chelsio Unified Wire software supports Chelsio Terminator series of Unified Wire adapters.
To know more about the list of adapters supported by each driver, please refer to their respective
sections.

1.3. Software Requirements

The Chelsio Unified Wire software has been developed to run on 64-bit Linux based platforms
and therefore it is a base requirement for running the driver. To know more about the complete
list of operating systems supported by each driver, please refer to their respective sections.

1.4. Package Contents

Source Package

The Chelsio Unified Wire source package consists of the following files/directories:

debrules: This directory contains packaging specification files required for building Debian

packages.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 18

docs: This directory contains support documents - README, Release Notes and User’s

Guide (this document) for the software.
kernels: This directory contains kernel.org-4.9.105 installation files.
libs: This directory is for libraries required to install the WD-UDP, WD-TOE and iWARP

drivers. The libibverbs library has implementation of RDMA verbs which will be used by

iWARP applications for data transfers. The librdmacm library works as an RDMA

connection manager. The libcxgb4 library works as an interface between the above

mentioned generic libraries and Chelsio iWARP driver. The libcxgb4_sock library is a

LD_PRELOAD-able library that accelerates UDP Socket communications transparently

and without recompilation of the user application.
OFED: This directory contains supported OFED packages.
RPM-Manager: This directory contains support scripts used for cluster deployment.
scripts: Support scripts used by the Unified Wire Installer.
specs: The packaging specification files required for building RPM packages.
src: Source code for different drivers.
support: This directory contains source files for the dialog utility.
tools:

• autoconf-x.xx: This directory contains the source for Autoconf tool needed for
WD-UDP and iWARP libraries.

• benchmarks: This directory contains various benchmarking tools to measure
throughput and latency of various networks.

• chelsio_adapter_config: This directory contains scripts and binaries needed to
configure Chelsio 40G adapters.

• cop: The cop tool compiles offload policies into a simple program form that can be
loaded into the kernel and interpreted. These offload policies are used to determine
the settings to be used for various connections. The connections to which the
settings are applied are based on matching filter specifications. Please find more
details on this tool in its manual page (run man cop command).

• cudbg: Chelsio Unified Debug tool which facilitates collection and viewing of
various debug entities like register dump, Devlog, CIM LA, etc.

• cxgbtool: The cxgbtool queries or sets various aspects of Chelsio network
interface cards. It complements standard tools used to configure network settings
and provides functionality not available through such tools. Please find more
details on this tool in its manual page (run man cxgbtool command). To use cxbtool
for FCoE Initiator driver, use [root@host~]# cxgbtool stor -h

• nvme_utils: This directory contains nvmecli, nvmetcli and targetcli installation
files, and dependent components.

• rdma_tools: This directory contains iWARP benchmarking tools.

• t4_sniffer: This directory contains sniffer tracing and filtering libraries. See WD

Sniffing and Tracing chapter for more information.

• 90-rdma.rules: This file contains udev rules needed for running RDMA
applications as a non-root user.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 19

• chdebug: This script collects operating system environment details and debug
information which can be sent to the support team, to troubleshoot Chelsio
hardware/software related issues.

• chiscsi_set_affinity.sh: This shell script is used for mapping iSCSI Worker
threads to different CPUs.

• chsetup: The chsetup tool loads NIC, TOE and iWARP drivers, and creates WD-
UDP configuration file.

• chstatus: This utility provides status information on any Chelsio NIC in the system.

• Makefile: The Makefile for building and installing tools.

• t4_latencytune.sh: Script used for latency tuning of Chelsio adapters.

• t4_perftune.sh: This shell script is to tune the system for higher performance. It

achieves it through modifying the IRQ-CPU binding. This script can also be used
to change Tx coalescing settings.

• t4-forward.sh: RFC2544 Forward test tuning script.

• uname_r: This file is used by chstatus script to verify if the Linux platform is

supported or not.

• wdload: UDP acceleration tool.

• wdunload: Used to unload all the loaded Chelsio drivers.

Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD.

The OptionROM directory contains Unified Boot Option ROM image (cubt4.bin), uEFI
driver (ChelsioUD.efi), default boot configuration file (bootcfg) and a legacy flash utility
(cfut4.exe), which can be used to flash the option ROM onto Chelsio adapters (CNAs).
The LinuxDUD directory contains image (.img) files required to update drivers for Linux

distributions.
chelsio-dkms.conf: DKMS configuration files for Ubuntu 16.04.1
install.py, dialog.py: Python scripts needed for the GUI installer.
EULA: Chelsio’s End User License Agreement.
install-dkms.sh: Installs necessary drivers to DKMS tree for Ubuntu 16.04.1
install.log: File containing installation summary.
Makefile: The Makefile for building and installing from the source.
sample_machinefile: Sample file used during iWARP installation on cluster nodes.

RPM Package

The Chelsio Unified Wire RPM package consists of the following:

config: This directory contains firmware configuration files.
docs: This directory contains support documents i.e. README, Release Notes and User’s

Guide (this document) for the software.
DRIVER-RPMS: RPM packages of Chelsio drivers.
OFED-RPMS: OFED RPM packages required to install iWARP driver.
scripts: Support scripts used by the Unified Wire Installer.
EULA: Chelsio’s End User License Agreement.
install.py: Python script that installs the RPM package. See Chelsio Unified Wire’s

Software/Driver Installation section for more information.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 20

uninstall.py: Python script that uninstalls the RPM package. See Chelsio Unified Wire’s

Software/Driver Uninstallation section for more information.

Uboot: There are two sub-directories in the Uboot directory: OptionROM and LinuxDUD.

The OptionROM directory contains Unified Boot Option ROM image (cubt4.bin), uEFI

driver (ChelsioUD.efi), default boot configuration file (bootcfg) and a legacy flash utility

(cfut4.exe), which can be used to flash the option ROM onto Chelsio adapters (CNAs).

The LinuxDUD directory contains image (.img) files required to update drivers for Linux

distributions.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 21

Hardware Installation

Follow these steps to install Chelsio adapter in your system:
i. Shutdown/power off your system.

ii. Power off all remaining peripherals attached to your system.
iii. Unpack the Chelsio adapter and place it on an anti-static surface.
iv. Remove the system case cover as per the system manufacturer’s instructions.
v. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
vi. For maximum performance, it is highly recommended to install the adapter into a PCIe x8/x16

slot.

vii. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector

on the motherboard. Apply even pressure on both edges until the card is firmly seated. It may
be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.

viii. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the

system manufacturer’s instructions. Replace the case cover.

ix. After securing the card, ensure that the card is still fully seated in the PCIE x8/x16 slot as

sometimes the process of securing the card causes the card to become unseated.

x. Connect a fiber/twinax cable, multi-mode for short range (SR) optics or single-mode for long

range (LR) optics, to the Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet

adapter.
xi. Power on your system.
xii. Run update-pciids command to download the current version of PCI ID list

xiii. Verify if the adapter was installed successfully by using the lspci command

For Chelsio adapters, the physical functions are currently assigned as:

Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI

All 4-ports of T6425-CR adapter will be functional only if PCIe x8 -> 2x PCIe x4 slot
bifurcation is supported by the system and enabled in BIOS. Otherwise, only 2-ports
will be functional.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 22

Physical function 6: for FCoE
Physical function 7: Currently not assigned

Once Unified Wire package is installed and loaded, examine the output of dmesg to see if the card
is discovered. You should see a similar output:

The above outputs indicate the hardware configuration of the adapter as well as serial number.

Network device names for Chelsio’s physical ports are assigned using the following

convention: the port farthest from the motherboard will appear as the first network
interface. However, for T5 40G and T420-BT adapters, the association of physical
Ethernet ports and their corresponding network device names is opposite. For these
adapters, the port nearest to the motherboard will appear as the first network
interface.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 23

Software/Driver Installation

There are two main methods to install the Chelsio Unified Wire package: from source and RPM.
If you decide to use source, you can install the package using CLI or GUI mode.
If you decide to use RPM, you can install the package using Menu or CLI mode.

RPM packages support only distro base kernels. In case of updated/custom kernels, use source
package. Irrespective of the method chosen for installation, the machine needs to be rebooted for
changes to take effect.

The following table describes the various configuration tuning options available during
installation and drivers/software installed with each option by default:

Configuration

Tuning Option

Description

Driver/Software installed

Unified

Wire(Default)

Default Configuration. Configures

adapters to run all protocols

simultaneously.

NIC/TOE, vNIC, iWARP, iSER, WD-UDP,

NVMe-oF, LIO iSCSI Target, iSCSI Target,

iSCSI Initiator, FCoE Initiator, Bonding,

MAFO, IPv6, Sniffer & Tracer, Filtering, TM

Low latency

Networking

Configures adapters to run TOE and

iWARP traffic with low latency.

TOE, iWARP, WD-UDP, WD-TOE, IPv6,

Bonding, MAFO

High capacity RDMA

Configures adapters to establish a large

number of iWARP connections.

iWARP

RDMA Performance

Improves iWARP performance.

iWARP, iSER, NVMe-oF

High capacity TOE

Configures adapters to establish a large

number of TOE connections.

TOE, Bonding, MAFO, IPv6

iSCSI Performance+

Improves iSCSI performance.

LIO iSCSI Target, iSCSI Target, iSCSI

Initiator, Bonding, DCB

UDP Seg.Offload &

Pacing*

Configures adapters to establish a large

number of UDP Segmentation Offload

connections.

UDP-SO, Bonding

Wire Direct Latency#

Configures adapters to provide low Wire

Direct latency.

TOE, iWARP, WD-UDP, WD-TOE

High Capacity WD

Configures adapters to establish a large

number of WD-UDP connections.

WD-UDP, WD-TOE

Hash Filter#

Configures adapters to create more filters.

Filtering

Ring Backbone#

Configures adapters in a ring backbone.

NIC/TOE, iWARP, iSER, WD-UDP, NVMe-

oF, LIO iSCSI Target, iSCSI Target, iSCSI

Initiator, IPv6, Filtering, TM

NVMe Performance^

Improves NVMe-oF performance.

iWARP, NVMe-oF

High Capacity VF#

Configures adapters to support more VFs.

NIC, vNIC

* Supported on T4/T5
+ Supported only on T5

#

Supported on T5/T6

^ Supported only on T6

3.1. Pre-requisites

To install Unifided Wire using GUI mode (with Dialog utility), ncurses-devel package must be
installed.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 24

3.2. Enabling RDMA on ARM Platforms

RDMA is disabled by default in RHEL 7.3 build of ARM architecture. To enable this feature, follow
the steps mentioned below:

i. Download the kernel source package and extract it.
ii. Create a kernel configuration file.

[root@host~]# make oldconfig

iii. The above command will create a configuration file .config in the same location. Edit the file

and enable the following parameters:

CONFIG_NET_VENDOR_CHELSIO=y
CONFIG_INFINIBAND=y

iv. Compile the kernel.
v. During kernel compilation, please ensure that the following parameters are set as follows:

CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
CONFIG_CHELSIO_T4VF=m

CONFIG_INFINIBAND_USER_MAD=m
CONFIG_INFINIBAND_USER_ACCESS=m
CONFIG_INFINIBAND_USER_MEM=y

CONFIG_INFINIBAND_CXGB3=m
CONFIG_INFINIBAND_CXGB3_DEBUG=y
CONFIG_INFINIBAND_CXGB4=m

CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m

vi. Install the kernel.
vii. Reboot into the newly installed kernel.

3.3. Mounting debugfs

All driver debug data is stored in debugfs, which will be mounted in most cases. If not, mount it
manually using:

[root@host~]# mount -t debugfs none /sys/kernel/debug

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 25

3.4. Allowing unsupported modules on SLES

On SLES11 SPx platforms, edit the /etc/modprobe.d/unsupported-modules file and change
allow_unsupported_modules to 1.

On SLES12 SPx platforms, edit the /etc/modprobe.d/10-unsupported-modules.conf file and
change allow_unsupported_modules to 1.

3.5. Installing Chelsio Unified Wire from source

GUI mode (with Dialog utility)

i. Download the Unified Wire driver package (tarball) from Chelsio Download Center.
ii. Untar the tarball using the following command:

[root@host~]# tar zxvf <driver_package>.tar.gz

iii. Change your current working directory to Chelsio Unified Wire package directory:

[root@host~]# cd ChelsioUwire-x.x.x.x

iv. Run the following script to start the GUI installer:

[root@host~]# ./install.py

v. If Dialog utility is present, you can skip to step (vi). If not, press ‘y’ to install it when the installer

prompts for input.

vi. Select “install” under “Choose an action”

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 26

vii. Select Enable IPv6-Offload to install drivers with IPv6 Offload support or Disable IPv6-offload

to continue installation without IPv6 offload support.

viii. Select the required configuration tuning option:

ix. Under “Choose install components”, select “all” to install all the related components for the

option chosen in step (viii) or select “custom” to install specific components.

To install Crypto Offload, WD-TOE, OVS, DPDK drivers and benchmark tools,
please select “custom option”.

Important

The tuning options may vary depending on the Linux distribution.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 27

x. Select the required performance tuning option.

a. Enable Binding IRQs to CPUs: Bind MSI-X interrupts to different CPUs and disable

IRQ balance daemon.
b. Retain IRQ balance daemon: Do not disable IRQ balance daemon.
c. TX-Coalasce: Write tx_coal=2 to modprobe.d/conf.

xi. If you already have the required version of OFED software installed, select Skip-OFED. To

install OFED 4.8-2 choose the Install-OFED option.

xii. The selected components will now be installed:

For more information on the Performance tuning options, please refer to

Performance Tuning section of the Network (NIC/TOE) chapter.

Note

This step will be prompted only for OFED supported platforms.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 28

xiii. After successful installation, summary of installed components will be displayed.

xiv. Select “View log” to view the installation log or “Exit” to continue.

xv. Select “Yes” to exit the installer or “No” to go back.

xvi. Reboot your machine for changes to take effect.

Press Esc or Ctrl+C to exit the installer at any point of time.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 29

CLI mode (without Dialog utility)

If your system does not have Dialog or you choose not to install it, follow the steps mentioned
below to install the Unified Wire package:

i. Download the Unified Wire driver package from Chelsio Download Center.
ii. Untar the tarball using the following command:

[root@host~]# tar zxvf <driver_package>.tar.gz

iii. Change your current working directory to Chelsio Unified Wire package directory:

[root@host~]# cd ChelsioUwire-x.x.x.x

iv. Run the following script to start the installer

[root@host~]# ./install.py -c <target>

v. Enter the number corresponding to the Configuration tuning option in the Input field and

press Enter.

vi. If you already have the required version of OFED software installed, select Skip-OFED. To

install OFED 4.8-2 choose the Install-OFED option.

vii. The selected components will now be installed.

After successful installation you can press 1 to view the installation log. Press any other key
to exit from the installer.

viii. Reboot your machine for changes to take effect.

iWARP driver installation on Cluster nodes

To customize the installation, view the help by typing

[root@host~]#./install.py –h

Important

Please make sure that you have enabled password less authentication with ssh
on the peer nodes for this feature to work.

Important

This step will be prompted only for OFED supported platforms.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Linux 30

Chelsio’s Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a

single command. Follow the procedure mentioned below:
i. Change your current working directory to Chelsio Unified Wire package directory:

[root@host~]# cd ChelsioUwire-x.x.x.x

ii. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in

the cluster. You can view the sample file, sample_machinefile, provided in the package to
view the format in which the nodes have to be listed.

iii. Now, execute the following command:

[root@host~]# ./install.py -C -m <machinefilename>

iv. Select the required configuration tuning option. The tuning options may vary depending on

the Linux distribution.

v. Select the required Cluster Configuration.

vi. If you already have the required version of OFED software installed, select Skip-OFED. To

install OFED 4.8-2 choose the Install-OFED option.

vii. The selected components will now be installed.
The above commands will install iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes

listed in the machinefilename file.

CLI mode

i. Download the Unified Wire driver package from Chelsio Download Center.

ii. Untar the tarball using the following command:

[root@host~]# tar zxvf ChelsioUwire-x.x.x.x.tar.gz

iii. Change your current working directory to Chelsio Unified Wire package directory and build

the source:

[root@host~]# cd ChelsioUwire-x.x.x.x
[root@host~]# make