Chelsio Communications T5, T4 Installation And User Manual

Chelsio Communications (Headquarters)
370 San Aleso Ave.
Suite 100
Sunnyvale, CA 94085
U.S.A

www.chelsio.com

Tel: [phone]
Fax: [phone]

Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India

Tel: [phone]
Fax: [phone]

Chelsio KK (Japan)

SHIMA Akasaka Bldg.
Minato-ku, Tokyo
Japan 107-0052

Tel: [phone]

This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.

No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.

All third party trademarks are copyright of their respective owners.

THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED

WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE

“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO

COMMUNICATIONS, INC.

Sales

For all sales inquiries please send email to [e-mail]

Support

For all support related questions please send email to [e-mail]
Copyright © 2014.Chelsio Communications. All Rights Reserved.

Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.

Chelsio T5/T4 Unified Wire For XenServer ii

Version

Revision Date

1.0.0

03/15/2013

1.0.1

04/02/2013

1.0.2

06/06/2014

Document History

Chelsio T5/T4 Unified Wire For XenServer iii

TABLE OF CONTENTS

I. CHELSIO UNIFIED WIRE 5

1. Introduction 6

1.1. Features 6

1.2. Hardware Requirements 6

1.3. Software Requirements 6

1.4. Package Contents 7

2. Hardware Installation 8

3. Software/Driver Installation 11

3.1. During OS installation 11

3.2. Runtime Installation 12

3.3. Firmware Update 13

4. Software/Driver Uninstallation 14

5. Software/Driver Update 15

II. NETWORK (NIC) 16

1. Introduction 17

1.1. Hardware Requirements 17

2. Software/Driver Loading 18

3. Software/Driver Configuration and Fine-tuning 19

3.1. Instantiate Virtual Functions (SR-IOV) 19

III. ISCSI PDU OFFLOAD INITIATOR 20

1. Introduction 21

1.1. Hardware Requirements 21

2. Software/Driver Loading 23

3. Software/Driver Unloading 24

4. Software/Driver Configuration and Fine-tuning 25

4.1. Accelerating open-iSCSI Initiator 25

4.2. Auto login from cxgb4i initiator at OS bootup 27

IV. APPENDIX 29

1. Chelsio End-User License Agreement (EULA) 30

Chelsio T5/T4 Unified Wire For XenServer iv

Chapter I. Chelsio Unified Wire

I. Chelsio Unified Wire

Chelsio T5/T4 Unified Wire For XenServer Page 5

Chapter I. Chelsio Unified Wire

1. Introduction

Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.

The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, ISCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.

The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified
Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and
Filtering.

Ideal for all data, storage and high performance clustering applications, the T5/T4 Adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.

Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5/T4 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.

1.1. Features

Chelsio’s Unified Wire software is an ISO image which installs Network and iSCSI PDU Offload
Initiator drivers for Chelsio’s T5 and T4 adapters on XenServer.

1.2. Hardware Requirements

Chelsio’s Unified Wire software supports Chelsio T5 and T4 Series of Unified Wire Adapters. To
know more about the list of adapters supported by each driver, please refer to their respective
sections.

1.3. Software Requirements

1.3.1. XenServer Requirements

Chelsio’s Unified Wire has been developed to run on XenServer platforms. Currently the
software is available for the following version:

Chelsio T5/T4 Unified Wire For XenServer Page 6

Chapter I. Chelsio Unified Wire

Citrix XenServer Host 6.2.0

Kernel version: 2.6.32.43-0.4.1.xs1.8.0.835.170778xen
Other versions have not been tested and are not guaranteed to work.

1.3.2. Resource Requirements

Control Domain (Dom0) memory: Based on the number of 40G, 10G and 1G ports, driver

allocates memory enough to run at the desired rate. It is suggested to increase the control

domain memory to 2GB using the procedure explained in

http://support.citrix.com/article/CTX134951

SW-IOMMU Size: Increasing SW-IOMMU size by appending swiotlb=128 to

/boot/extlinux.conf avoids PCI-DMA: Out of SW-IOMMU space for <size> bytes

at device <BDF> messages.

1.4. Package Contents

The software supplied is an ISO image. It consists of the following:

RPM packages containing drivers and associated tools, firmware, etc.
Shell script to install the driver package.

Chelsio T5/T4 Unified Wire For XenServer Page 7

Chapter I. Chelsio Unified Wire

2. Hardware Installation

Follow the steps to install Chelsio Adapter in your system:

1. Shutdown and power off your system.

2. Power off all remaining peripherals attached to your system.

3. Unpack the Chelsio adapter and place it on an anti-static surface.

4. Remove the system case cover according to the system manufacturer’s instructions.

5. Remove the PCI filler plate from the slot where you will install the 10Gb Ethernet adapter.

6. For maximum performance, it is highly recommended to install the adapter into a PCIE x8

slot.

7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector

on the motherboard. Apply even pressure on both edges until the card is firmly seated. It

may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.

8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the

system manufacturer’s instructions. Replace the case cover.

9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as

sometimes the process of securing the card causes the card to become unseated.

10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range

(LR) optics, to the 10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet

adapter.

11. Power on your system.

12. Verify if the adapter was installed successfully by using the lspci command

a. For T5 adapters :

[root@host]# lspci |grep -i Chelsio

07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.3 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR
Unified Wire Storage Controller
07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire
Storage Controller

Chelsio T5/T4 Unified Wire For XenServer Page 8

Chapter I. Chelsio Unified Wire

b. And for T4 adapters :

[root@host]# lspci | grep –i Chelsio
03:00.0 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.1 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.2 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.3 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.4 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.5 SCSI storage controller: Chelsio Communications Inc T420-CR Unified
Wire Storage Controller
03:00.6 Fibre Channel: Chelsio Communications Inc T420-CR Unified Wire
Storage Controller
03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000

For Chelsio T5/T4 adapters, the physical functions are currently assigned as:

Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned

Once Unified Wire package is installed and loaded, examine the output of dmesg to see if the
card is discovered.

 For T5 adapters:

eth4: Chelsio T520-LL rev 0 10GBASE-SFP RNIC PCIe 5 GT/s x8 MSI-X, Offload
capable
0000:07:00.4: S/N: PT03140177, P/N: 110116750C0

 And, for T4 adapters:

eth0: Chelsio T420-CR rev 2 10GBASE-SFP RNIC PCIe x8 MSI-X
0000:04:00.4: S/N: PT18111226, P/N: 110112140D0

Chelsio T5/T4 Unified Wire For XenServer Page 9

Chapter I. Chelsio Unified Wire

Network device names for Chelsio’s physical ports are assigned using the

following convention: the port farthest from the motherboard will appear as the
first network interface. However, for T5 40G and T420-BT adapters, the
association of physical Ethernet ports and their corresponding network device
names is opposite. For these adapters, the port nearest to the motherboard will
appear as the first network interface.

Note

The above outputs indicate the hardware configuration of the adapters as well as the Serial
numbers. As observed by the x8, the card is properly installed in an x8 slot on the machine and
using MSI-X interrupts.

Chelsio T5/T4 Unified Wire For XenServer Page 10