Chelsio Communications Terminator Series Installation And User Manual

Chelsio Unified Wire for Windows ii

This document and related products are distributed under licenses restricting their use, copying, distribution,
and reverse-engineering.

No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.

All third-party trademarks are copyright of their respective owners.

THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED

WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE

“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO COMMUNICATIONS,

INC.

Sales

For all sales inquiries please send email to [e-mail]

Support

For all support related questions please send email to [e-mail]
Copyright © 2019. Chelsio Communications. All Rights Reserved.

Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.

Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A

www.chelsio.com

Tel: [phone]
Fax: [phone]

Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India

Tel: [phone]

Chelsio KK (Japan)

Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio Unified Wire for Windows iii

Document History

Version

Revision Date

1.5.2

01/05/2017

1.5.3

02/07/2017

1.5.4

02/24/2017

1.5.5

03/10/2017

1.5.6

03/31/2017

1.5.7

04/11/2017

1.5.8

05/09/2017

1.5.9

05/29/2017

1.6.0

06/26/2017

1.6.1

06/30/2017

1.6.2

08/18/2017

1.6.3

08/24/2017

1.6.4

09/28/2017

1.6.5

11/27/2017

1.6.6

12/04/2017

1.6.7

12/08/2017

1.6.8

12/27/2017

1.6.9

02/20/2018

1.7.0

02/26/2018

1.7.1

02/28/2018

1.7.2

07/06/2018

1.7.3

07/17/2018

1.7.4

08/13/2018

1.7.5

09/26/2018

1.7.6

12/27/2018

1.7.7

01/31/2019

1.7.8

03/29/2019

1.7.9

04/24/2019

Chelsio Unified Wire for Windows iv

TABLE OF CONTENTS

I. CHELSIO UNIFIED WIRE 5

1. Introduction 6

1.1. Features 6

1.2. Hardware Requirements 6

1.3. Software Requirements 7

1.4. Package Contents 7

2. Hardware Installation 8

3. Software/Driver Installation 9

3.1. Prerequisites 9

3.2. Unified Wire Installer 9

3.3. Zip Package 13

4. Firmware Update 16

5. cxgbtool help 17

6. Software/Driver Uninstallation 36

6.1. Installer 36

6.2. Zip Package 39

II. FCOE STORPORT MINIPORT 40

1. Introduction 41

1.1. Hardware Requirements 41

1.2. Software Requirements 41

2. Configuration and Fine-tuning 42

2.1. Configuring Cisco Nexus 5010 switch 42

2.2. FCoE fabric discovery verification 45

III. APPENDIX 50

1. Troubleshooting 51

2. Chelsio End-User License Agreement (EULA) 52

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 5

I. Chelsio Unified Wire

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 6

1. Introduction

Thank you for choosing Chelsio Unified Wire adapters. These high speed, single chip, single
firmware cards provide enterprises and data centers with high performance solutions for various
Network and Storage related requirements.

The Terminator series is Chelsio’s next generation of highly integrated, hyper-virtualized
10/25/40/50/100GbE controllers. The adapters are built around a programmable protocol­processing engine, with full offload of a complete Unified Wire solution comprising NIC, TOE,
iWARP RDMA, iSCSI, FCoE and NAT support. It scales to true 100Gb line rate operation from a
single TCP connection to thousands of connections and allows simultaneous low latency and high
bandwidth operation thanks to multiple physical channels through the ASIC.

Ideal for all data, storage and high-performance clustering applications, the Unified wire
adapters enable a unified fabric over a single wire by simultaneously running all unmodified IP
sockets, Fibre Channel and InfiniBand applications over Ethernet at line rate.

Designed for deployment in virtualized datacenters, cloud service installations and high­performance computing environments, Chelsio Terminator adapters bring a new level of
performance metrics and functional capabilities to the computer networking industry.

1.1. Features

Chelsio Unified Wire for Windows is an easy to use utility developed to provide installation of 64­bit Windows based drivers and tools for Chelsio's Unified Wire adapters.

It consists of the following component(s):

• FCoE Storport Miniport Driver

• Virtual Bus Driver (VBD)

1.2. Hardware Requirements

The Chelsio Unified Wire supports all x64 architectures supporting PCIE (x4, x8, x16) slots.

AMD CPUs, 64-bit (x86_64/amd64)
Intel CPUs, 64-bit (x86_64)

The Chelsio Unified Wire supports 3.3v PCI bus only. Running an adapter on a
PCI x4 slot is not recommended as performance will be significantly reduced by
the limitations of PCI.

Note

T4 family of adapters are not supported.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 7

1.3. Software Requirements

The Chelsio Unified Wire software has been developed to run on Windows based platforms. To
know more about the complete list of versions supported by each driver/software, please refer
their respective sections.

1.4. Package Contents

Unified Wire Installer

Chelsio Unified Wire comes with an interactive installer and support documentation. The
documentation, consisting of README, Release Notes and User’s Guide (this document), can
be found in the <system_drive>\ChelsioUwire\docs\ directory after installing Chelsio Unified Wire.

Zip Package

The zip package contains driver files, firmware binaries, adapter configuration binaries and
support documentation (docs folder).

Drivers are WHQL certified for Windows Server 2016, 2012 R2 and 10 Client.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 8

2. Hardware Installation

i. Shutdown/power off your system.
ii. Power off all remaining peripherals attached to your system.
iii. Unpack the Chelsio adapter and place it on an anti-static surface.
iv. Remove the system case cover according to the system manufacturer’s instructions.
v. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
vi. For maximum performance, it is highly recommended to install the adapter into a PCIE x8/x16

slot.

vii. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector

on the motherboard. Apply even pressure on both edges until the card is firmly seated. It may

be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
viii. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the

system manufacturer’s instructions. Replace the case cover.
ix. After securing the card, ensure that the card is still fully seated in the PCIE x8/x16 slot as

sometimes the process of securing the card causes the card to become unseated.
x. Connect a fiber/twinax cable, multi-mode for short range (SR) optics or single-mode for long

range (LR) optics, to the Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet

adapter.
xi. Power on your system.
xii. Verify if the adapter was installed successfully. To do so, open Device Manager in Control

Panel.
xiii. Under Other devices section, Chelsio adapter should be listed as Ethernet Controller. If the

adapter is not listed, right-click on the system name or click on the Actions menu and select

Scan for hardware changes
For Chelsio adapters, the physical functions are currently assigned as:

Physical functions 0 - 3: for the SR-IOV functions
Physical function 4: for all NIC functions of the card
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned

xiv. Once the Unified Wire package is installed, open Device Manager again. Expand Network

adapters section and now Chelsio adapter should be listed.

Network device names for Chelsio’s physical ports are assigned using the

following convention: the port farthest from the motherboard will appear as the
first Ethernet interface. However, for T5 40G adapters, the association of physical
Ethernet ports and their corresponding network device names is opposite. For
these adapters, the port nearest to the motherboard will appear as the first

network interface.

Note

All 4-ports of T6425-CR adapter will be functional only if PCIe x8 -> 2x PCIe x4 slot
bifurcation is supported by the system and enabled in BIOS. Otherwise, only 2-ports
will be functional.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 9

3. Software/Driver Installation

Chelsio Unified Wire can be installed using the Installer or the zip package. Refer the relevant
section below depending on the method of installation selected.

3.1. Prerequisites

Please make sure that the following requirements are met before installation:

Latest Microsoft Visual C++ Redistributable packages for Visual Studio (x86 and x64) are

required if you are planning to use installer.

Windows update KB2685811 should be installed on Windows 7 Client and 2008 R2 Server. It

will be installed if Automatic Updates is configured or can be installed manually from

https://www.catalog.update.microsoft.com/Search.aspx?q=KB2685811

3.2. Unified Wire Installer

Chelsio Unified Wire Installer provides two methods of installation: GUI or CLI mode. GUI mode
provides an interactive GUI installer with customizable options. Whereas, CLI mode enables
unattended installation of Chelsio drivers and software thereby relieving the user from monitoring
the installation process and providing input via dialog boxes.

Enabling Test Signing

In case the drivers in the package are not WHQL certified, follow the steps mentioned below to
enable test signing on your system:

i. Goto Start->Run command option, enter "cmd" and press OK. This will open the command

prompt utility.
ii. Run the following command:

C:\Users\Administrator> bcdedit /set testsigning on

iii. Reboot the machine for the changes to take effect.

GUI mode (Installer)

i. Run the ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe installer application.
ii. Click Next for the Chelsio End User License Agreement Window.

Important

If you are planning to install Unified Wire in a cluster, it should be done on all
the members. Using different versions of Unified Wire is not supported.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 10

Figure 1 - Unified Wire installer welcome window

iii. Select the radio button I accept the terms of the license agreement and click Next

Figure 2 - Chelsio EULA window

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 11

iv. Select Complete for complete package installation.

Figure 3 - Select setup (installation) type

v. Click Install to start the installation.

Figure 4 - Start installation

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 12

vi. FCoE Storport Miniport and NIC drivers will now be installed.

Figure 5 - Installation in progress

vii. Click Finish to exit from the Unified Wire Installer.

Figure 6 - Finishing Unified Wire installation

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 13

CLI mode (Silent Installation)

To install the driver using this feature, open command prompt and execute the following
command:

C:\Users\Administrator> ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe –in all

The above command will install NIC and FCoE Storport Miniport drivers.
To know more about other parameters and options, execute the following command:

C:\Users\Administrator> ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe -help

The above command will create a help file, ChelsioUwire_x.x.x.x_WIN_yyy.y.yy_help.txt, in the
present working directory which contains the complete list of command line syntax required for
performing all the necessary CLI operations.

3.3. Zip Package

Chelsio Unified Wire zip package provides two methods of installation: Manual and using Offline
Windows Image. In Manual method, the driver components must be individually installed using

the Device Manager. Using the second method, you can add Chelsio driver components to an
offline OS image. The image can then be used to install Windows on your system.

Enabling Test Signing

In case the drivers in the package are not WHQL certified, please follow the steps mentioned
below to enable test signing before installing Unified Wire:

i. Open command prompt and run the following command to enable test signing:

C:\Users\Administrator>bcdedit /set testsigning on

ii. Change your working directory to ChelsioUwire_x.x.x.x_WIN_yyy.y.yy/Selfsign/ and run the

following commands:

C:\ ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\Selfsign> .\certmgr.exe -add
.\chelsiocert1.cer -s -r LocalMachine root -all
C:\ ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\Selfsign> .\certmgr.exe -add
.\chelsiocert1.cer -s -r LocalMachine trustedpublisher -all

iii. Reboot the machine for the changes to take effect.

A log file, ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.log, is created in the same directory
which keeps a record of all the commands executed and their results.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 14

Drivers location

The driver files should be manually picked from:

• ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\drivers\Win10 for Server 2016 and 10 Client.

• ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\drivers\Win7 for Server 2012 R2.

Manual

Virtual Bus Driver

i. Open Device Manager (Control Panel -> System & Security-> System -> Device Manager),

click on System Devices, right click on Chelsio Bus Enumerator and select Update

Driver Software.
ii. Select Browse my computer for driver software.
iii. Select Let me pick from a list of device drivers on my computer and click Next.
iv. Click on Have Disk Button and on the next screen browse for chvbdx64.inf file in the zip

package. Click Open and then OK.
v. Click Next and driver installation will progress. Click Close once the installation is complete.

FCoE Storport Miniport driver

i. Open Device Manager (Control Panel -> System & Security-> System -> Device Manager),

click on System Devices, right click on Chelsio Generic function with function 6 and select

Properties. In case the adapter is not listed, it may appear as Fibre Channel Controller

under Other devices.

Important

• The driver components need to be installed strictly in the order described
here. Using any other order is not supported.

• Before proceeding, open Device Manager and disable NDIS Miniport driver
(Network adapters->Chelsio Network Adapter) followed by Virtual Bus Driver
(System devices->Chelsio Bus Enumerator).

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 15

ii. Under Driver tab, select Update Driver…
iii. Select Browse my computer for driver software.
iv. Select Let me pick from a list of device drivers on my computer and click Next.
v. Click on Have Disk Button and on the next screen, browse for csiofcoe.inf file in the zip

package. Click Open and then OK.

vi. Click Next and driver installation will progress. Click Close once the installation is complete.

NDIS Miniport Driver

i. Open Device Manager (Control Panel -> System & Security-> System -> Device Manager),

click on Network Adapters, right click on the Chelsio network adapter and select Update

Driver Software.
ii. Select Browse my computer for driver software.
iii. Select Let me pick from a list of device drivers on my computer.
iv. Click on Have Disk Button and on the next screen browse for chnetx64.inf file in the zip

package. Click Open and then OK.
v. Click Next and driver installation will progress. Click Close once the installation is complete.

Repeat the above steps for the other adapters.

Generic Function

i. Open Device Manager (Control Panel -> System & Security-> System -> Device Manager),

click on System Devices, right click on Chelsio Generic Function and select Update

Driver Software.
ii. Select Browse my computer for driver software.
iii. Select Let me pick from a list of device drivers on my computer.
iv. Click on Have Disk Button and on the next screen browse for chnulx64.inf file in the zip

package. Click Open and then OK.
v. Click Next and driver installation will progress. Click Close once the installation is complete.

Offline Windows Image

You can use Windows Deployment Image Servicing and Management (DISM) tool to add Chelsio
driver components (.inf files) to an offline Windows image. For more information, visit the following
links:

• http://technet.microsoft.com/en-us/library/hh825070.aspx

• http://blogs.technet.com/b/heyscriptingguy/archive/2012/09/27/use-the-powershell-dism-

cmdlets-to-manage-windows-8.aspx

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 16

4. Firmware Update

The driver will auto-load the firmware if an update is required. The firmware version can be verified
using cxgbtool:

C:\Users\Administrator>cxgbtool nic0 debug version

In case of installer, the firmware binaries will be copied to <system_drive>\ChelsioUwire
\Firmware\ folder during installation.

For zip package, the binaries will be present in ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\firmware\
folder.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 17

5. cxgbtool help

The cxgbtool command queries or sets various aspects of Chelsio network interface cards. It
complements standard tools used to configure network settings and provides functionality not
available through such tools.

Some of the commands provided can be used to query running statistics to aid in debugging.

Syntax

cxgbtool <vbdIface|nicIface|iSCSIIface|rdma> <category> <command> <options>

Definitions

nicIface is the name of the network device to work on, given in the format nic0, nic1.
vbdIface is the name of the Chelsio Bus Enumerator instance to work on, given in the format

vbd0, vbd1
iscsIface is the name of the Chelsio iSCSI interface to work on, given in the format iSCSI0,

iSCSI1
rdma is the name of the Chelsio RDMA interface to work on.
fcoeIface is the name of the Chelsio FCoE interface to work on, given in the format

Scsi1,Scsi2
category is one of the cxgbtool categories, i.e., boot, cudbg, debug, firmware, hardware
command is the action you wish to perform on the adapter or property you wish to view/change.
option is the optional parameter to be provided with command.

Help

o Displaying cxgbtool help: cxgbtool

o Displaying list of available cxgbtool commands: cxgbtool -h all

o Displaying category specific help:

cxgbtool <vbdIface|nicIface|iSCSIIface> <category> -h

o Displaying list of cudbg entities: cxgbtool <vbdIface> cudbg dbg -h

o Displaying RDMA help: cxgbtool rdma -h

o Displaying FCoE help: cxgbtool fcoe -h

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 18

FCoE

Description: Display FCoE information.
Syntax: cxgbtool fcoe -a [FCoEAdapterIndex] <options>
Examples:

o Scanning for available Chelsio adapters.

o Printing adapter status and parameters.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 19

o Printing adapter statistics.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 20

o Printing physical port statistics of adapter.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 21

o Printing DCB parameters.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 22

o Printing driver parameters.

o Printing SCSI statistics.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 23

o Printing FCFs information connected to the adapter.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 24

o Printing local nodes information.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 25

o Printing local nodes statistics.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 26

o Printing remote nodes information.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 27

o Printing remote nodes statistics.

o Displaying FCoE statistics.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 28

o Displaying internal memory information.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 29

o Displaying firmware debug logs.

o Creating NPIV ports for the parent WWN.

o Deleting NPIV ports for the parent WWN.

o Printing NPIV ports for the parent WWPN.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 30

Debug

▪ version
Description: Display adapter information like part number, serial number, device ID, firmware

version, TP, etc.

Syntax: cxgbtool <vbdIface|nicIface|iscsiIface> debug version
Example:

▪ inst
Description: Install driver package specified in the setup information file (.inf).

Syntax: cxgbtool debug inst <file>
Example:

▪ update
Description: Update driver package.

Syntax: cxgbtool debug update <file>
Example:

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 31

▪ uninst
Description: Uninstall driver package specified in the setup information file (.inf).

Syntax: cxgbtool debug uninst <file>
Example:

▪ rescan
Description: Scan for hardware changes in the device manager.

Syntax: cxgbtool debug rescan
Example:

▪ qstats
Description: Display statistics for each Tx & Rx queue.

Syntax: cxgbtool <nicIface|iscsiIface> debug qstats <queueType|clr>

clr: Clear queue statistics.

Queue Type

Description

txeth

Tx tunnel queue statistics.

rxeth

Rx tunnel queue statistics.

txvirt

Tx VM queue statistics.

rxvirt

Rx VM queue statistics.

txtoe

Tx Chimney queue statistics.

rxtoe

Rx Chimney queue statistics.

txrdma

Tx RDMA queue statistics.

rxrdma

Rx RDMA queue statistics.

txctrl

Chimney control queue statistics.

txfwd

Chimney forwarding queue statistics.

txnvgre *

Tx NVGRE statistics.

rxnvgre *

Rx NVGRE statistics.

txiscsi

Tx iSCSI queue statistics.

rxiscsi

Rx iSCSI queue statistics.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 32

txpd

Tx PacketDirect queue statistics.

rxpd

Rx PacketDirect queue statistics.

* not supported on T6 adapters

Qstats Parameter

Description

BcBytes

Broadcast packet size in bytes.

BcPkts

Number of broadcast packets.

CoalBytes

Coalesced packet size in bytes.

CoalPkts

Number of coalesced packets.

DropPkts

Number of dropped packets.

EncapBytes

#

NVGRE/VXLAN packet size in bytes.

EncapPkts

#

Number of NVGRE/VXLAN packets.

VlanMismatch

Number of packets with VLAN mismatch.

Fw4Ack

Number of SGE update requests.

LargeRxDropPkts

Dropped packets due to large size counter.

LSO

Number of large send offloaded packets.

McBytes

Multicast packet size in bytes.

McPkts

Number of multicast packets.

NblsRecv

Total Nbls received.

NblsSent

Total Nbls sent.

NoRxBufs

Packets couldn't process because of lack of memory.

NvPkts

Send NVGRE packets.

PeerAbort

Peer abort request.

CqComp

Number of CQ completions.

PktsDefrag

Number of defragged packets.

PktsDrop

Total packets dropped.

PktsQued

Total packets queued.

PktsSent

Total packets sent.

Quefull

Number of time queue full happened.

RecvNbls

Total Nbls received.

RssGroup

RSS group number allocated to queue.

VPortId

Vport ID of queue.

RssProc

RSS processor number allocated to queue.

RxCsumErr

Number of received packets with checksum error.

RxCsumErrIp

Number of received IP packets with checksum error.

RxCsumErrTcp

Number of received TCP packets with checksum error.

SysThreadEvts

Number of system thread events.

TxCsumOfld

Number of checksum offloaded Tx packets.

UcBytes

Unicast packets size in bytes.

UcPkts

Number of unicast packets.

VlanEx

Number of VLAN extracted packets.

VLANin

Number of VLAN inserted packets.

#

supported only on T6 adapters

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 33

Example of rxeth qstats

hardware

▪ sensor
Description: Display sensor data.

Syntax: cxgbtool <nicIface> hardware sensor
Example:

▪ fec
Description: Set/Display Forward Error Correction (FEC) parameters.

Syntax:

cxgbtool <nicIface> hardware fec [<config params>]

Example:

o Displaying currently set FEC mode.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 34

o Setting FEC mode:

▪ link_cfg
Description: Set/Display link configuration parameters.

Syntax:

cxgbtool <nicIface> hardware link_cfg <options>

Example:

o Displaying link configuration information:

o Enabling/Disabling auto-negotiation:

Auto-negotiation is enabled by default. To disable it, use the autoneg off parameter on all

connected machines of the link.

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 35

o Changing link speed:

25G port can be set to 10G and 100G port can be set to 25G/40G/50G. Auto-negotiation

should be disabled on all connected machines while changing link speed.

cudbg

▪ collect
Description: Collect Chelsio adapter debug logs to a compressed file.

Syntax: cxgbtool <vbdIface> cudbg collect <entities> <outputFile> <options>
Examples:

o Collecting debug log for all entities

o Collecting debug log skipping specific debug entities

o Collecting debug log avoiding entities that can affect running traffic

Before setting 40G or 10G speeds, FEC should be disabled on the port using:

C:\Users\Administrator>cxgbtool nic0 hardware fec off

Important

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 36

6. Software/Driver Uninstallation

Similar to installation, Chelsio Unified Wire can be uninstalled using the Installer or zip package.
Refer the relevant section depending on the method of installation used to install drivers.

6.1. Installer

Chelsio Unified Wire Installer can be uninstalled using two methods: GUI or CLI mode. GUI mode
requires user interaction and uninstallation occurs with options specified by the user. Whereas,
CLI mode does not require any user input.

GUI mode (Installer)

i. Run the ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe application.
ii. Select Modify to add or remove features. Select Repair to repair the previous installation.

Select Remove to uninstall the application. After you have selected the appropriate option,

click Next.

Figure 7 - Adding or removing features using ‘Modify’ option

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 37

Figure 8 - Repairing previous installation using ‘Repair’ option

Figure 9 – Uninstalling Unified Wire using ‘Remove’ option

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 38

iii. Click on the Finish button to exit from the installer.

Figure 10 - Finishing uninstallation

CLI Mode

To uninstall FCoE Storport Miniport driver, execute the following command:

C:\Users\Administrator> ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe –un all

This method of uninstallation is possible only if the drivers were installed using
Unified Wire Installer.

Note

Chapter I. Chelsio Unified Wire

Chelsio Unified Wire for Windows 39

6.2. Zip Package

If the driver components were installed using the zip package, they will have to be uninstalled
manually. The following section lists the various components and their respective methods of
uninstallation.

NDIS Miniport driver

i. Open the Device Manager (Control Panel -> System & Security-> System -> Device

Manager), click on the Network Adapters, right click on the Chelsio Network Adapter and

select Uninstall.
ii. Select the Delete the driver software for this device checkbox when Confirm Device

Uninstall is prompted and click OK.
Repeat the same procedure for the other Chelsio Network Adapters.

FCoE Storport Miniport

Open the Device Manager (Control Panel -> System & Security-> System -> Device Manager),
click on Storage controllers, right-click on Chelsio [FCoE] function and select Uninstall.

Virtual Bus Driver

i. Open the Device Manager (Control Panel -> System & Security-> System -> Device

Manager), click on the System Devices, right click on the Chelsio Bus Enumerator and

select Uninstall.
ii. Select the Delete the driver software for this device checkbox when Confirm Device

Uninstall is prompted and click OK.

Generic Function

i. Open the Device Manager (Control Panel -> System & Security-> System -> Device

Manager), click on the Network Adapters, right click on the Chelsio Generic Function and

select Uninstall.
ii. Select the Delete the driver software for this device checkbox when Confirm Device

Uninstall is prompted and click OK.
Repeat the same procedure for the other Generic Functions.

Important

The driver components need to be uninstalled strictly in the order described
here. Using any other order is not supported.

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 40

II. FCoE Storport Miniport

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 41

1. Introduction

Fibre Channel over Ethernet (FCoE) is a mapping of Fibre Channel over selected full duplex IEEE

802.3 networks. The goal is to provide I/O consolidation over Ethernet, reducing network
complexity in the Datacenter.

Chelsio FCoE initiator maps Fibre Channel directly over Ethernet while being independent of the
Ethernet forwarding scheme. The FCoE protocol specification replaces the FC0 and FC1 layers
of the Fibre Channel stack with Ethernet. By retaining the native Fibre Channel constructs, FCoE
will integrate with existing Fibre Channel networks and management software.

1.1. Hardware Requirements

Supported Adapters

The following are the currently shipping Chelsio adapters that are compatible with Chelsio FCoE
storport miniport driver:

T6225-CR
T6225-LL-CR
T520-CR
T520-LL-CR
T540-CR

1.2. Software Requirements

Windows Requirement

Currently Chelsio FCoE Storport Miniport driver is available for following Windows versions:

Server 2016
10 Client
Server 2012 R2

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 42

2. Configuration and Fine-tuning

2.1. Configuring Cisco Nexus 5010 switch

Configuring the DCBX parameters

In this procedure, you may need to adjust some of the parameters to suit your environment, such
as VLAN IDs, Ethernet interfaces, and virtual Fibre Channel interfaces.

To enable PFC, ETS, and DCB functions on a Cisco Nexus 5000 series switch:
i. Open a terminal configuration setting.

Switch# config terminal
switch(config)#

ii. Configure qos class-maps and set the traffic priorities: NIC uses priority 0 and FCoE uses

priority 3.

Switch(config)#class-map type qos class-nic
Switch(config-cmap-qos)# match cos 0
Switch(config-cmap-qos)# class-map type qos class-fcoe

Switch(config-cmap-qos)# match cos 3

iii. Configure queuing class-maps.

Switch(config)#class-map type queuing class-nic

Switch(config-cmap-que)#match qos-group 2

By default, the Cisco Nexus switch enables DCBX functionality and configures
PFC for FCoE traffic making it no drop with bandwidth of 50% assigned to FCoE
class of traffic and another 50% for the rest (like NIC). If you wish to configure
custom bandwidth, then follow the procedure below.

Note

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 43

iv. Configure network-qos class-maps.

Switch(config)#class-map type network-qos class-nic

Switch(config-cmap-nq)#match qos-group

v. Configure qos policy-maps.

Switch(config)#policy-map type qos policy-test
Switch(config-pmap-qos)#class type qos class-nic

Switch(config-pmap-c-qos)#set qos-group 2

vi. Configure queuing policy-maps and assign network bandwidth. Divide the network

bandwidth between FCoE and NIC traffic.

Switch(config)#policy-map type queuing policy-test
Switch(config-pmap-que)#class type queuing class-nic
Switch(config-pmap-c-que)#bandwidth percent 50
Switch(config-pmap-c-que)#class type queuing class-fcoe
Switch(config-pmap-c-que)#bandwidth percent 50
Switch(config-pmap-c-que)#class type queuing class-default

Switch(config-pmap-c-que)#bandwidth percent 0

vii. Configure network-qos policy maps and set up the PFC for no-drop traffic class.

Switch(config)#policy-map type network-qos policy-test
Switch (config-pmap-nq)#class type network-qos class-nic

Switch(config-pmap-nq-c)#pause no-drop

viii. Apply the new policy (PFC on NIC and FCoE traffic) to the entire system.

Switch(config)#system qos
Switch(config-sys-qos)#service-policy type qos input policy-test
Switch(config-sys-qos)#service-policy type queuing output policy-test
Switch(config-sys-qos)#service-policy type queuing input policy-test

Switch(config-sys-qos)#service-policy type network-qos policy-test

By default, FCoE is set to pause no drop. In such a tradeoff, one may want to set
NIC to drop instead.

Note

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 44

Configuring the FCoE/FC ports

In this procedure, you may need to adjust some of the parameters to suit your environment, such
as VLAN IDs, Ethernet interfaces, and virtual Fibre Channel interfaces

i. Following steps will enable FCoE services on a particular VLAN and does a VSAN-VLAN

mapping. These steps are not required to be done every time, unless a new mapping is

required.

Switch(config)# vlan 2
Switch(config-vlan)# fcoe vsan 2

Switch(config-vlan)#exit

ii. Following steps help in creating a virtual fibre channel(VFC) and binds that VFC to an

Ethernet interface so that the Ethernet port begins functioning as a FCoE port.

Switch(config)# interface vfc 13
Switch(config-if)#bind interface ethernet 1/13
Switch(config-if)# no shutdown
Switch(config-if)#exit
Switch(config)#vsan database
Switch(config-vsan-db)# vsan 2
Switch(config-vsan-db)#vsan 2 interface vfc 13

Switch(config-vsan-db)#exit

iii. Assign VLAN ID to the Ethernet port on which FCoE service was enabled in step1.

Switch(config)# interface ethernet 1/13
Switch(config-if)#switchport mode trunk
Switch(config-if)#switchport trunk allowed vlan 2
Switch(config-if)#no shutdown

Switch(config)#exit

If you are binding the VFC to a MAC address instead of an Ethernet port then
make sure the Ethernet port is part of both default VLAN and FCoE VLAN

Note

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 45

iv. On the FC Ports, if a FC target is connected then perform the following steps -

Switch(config)#vsan database
Switch(config-vsan-db)#vsan 2
Switch(config-vsan-db)# vsan 2 interface fc 2/2
Switch(config-vsan-db)#exit
Switch(config)interface fc 2/2
Switch(config-if)# switchport mode auto
Switch(config-if)# switchport speed auto

Switch(config-if)# no shutdown.

v. If you have not created a zone then make sure the default-zone permits the VSAN created,

otherwise the initiator and the target on that particular VSAN although FLOGI’d into the

switch will not talk to each other. To enable it, execute the below command.

Switch(config)# zone default-zone permit vsan 2

2.2. FCoE fabric discovery verification

Verifying Target Discovery

To check the summary of targets being discovered from FCoE ports, open device manger and
follow the steps given below:

i. Go to Start->Control Panel->Device Manager or launch device manager directly from

Start->Run->devmgmt.msc and hit [Enter].
ii. Go to Disk drives section under which you will find the list of all the LUNS that were

discovered.

Figure 11 - Discovered Target LUNs

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 46

iii. To bring the LUNs online, launch disk management by going to Start->Run-

>diskmgmt.msc. Right click on the LUN and select online.

Figure 12 - Bringing LUNs online

iv. Now format the LUN by right clicking on it and choosing any of the volume types. In this

example, we will choose New Simple Volume. This will invoke the New Simple Volume

Wizard.

Figure 13 - Formatting LUNs

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 47

v. Enter the volume size (the entire volume size is selected by default), and in the subsequent

windows assign a driver letter to the volume and choose the file system.

Figure 14 - Specifying Volume size

Figure 15 - Assigning drive letter or path

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 48

Figure 16 - Formatting the partition

vi. Click Finish to close the wizard.

Figure 17 - Completing Simple Volume Wizard

Chapter II. FCoE Storport Miniport

Chelsio Unified Wire for Windows 49

vii. The volume should appear in My Computer.

Figure 18 - The newly created volume in My Computer

Troubleshooting

i. Always enable LLDP on the interfaces as there won’t be FCoE link up until and unless a

successful LLDP negotiation happens.
ii. On switch, make sure priority-flow-control mode is always set to auto and flowcontrol is

disabled.
iii. Always configure the Ethernet interfaces on cisco switch in trunk mode.
iv. If you are binding the VFC to MAC address in case of cisco nexus, then make sure you

make the Ethernet interface part of both Ethernet VLAN and FCoE VLAN.
v. If in any case the switch-port on the cisco nexus switch is reporting

“pauseRateLimitErrDisable”, then perform an Ethernet port shut/no shut.
Also, note that only one FCoE function will be displayed under the storage controllers of the

device manager, irrespective of the number of ports a card possesses. All ports will be using the
same interface.

Chapter III. Appendix

Chelsio Unified Wire for Windows 50

III. Appendix

Chapter III. Appendix

Chelsio Unified Wire for Windows 51

1. Troubleshooting

Installer issues

In case of any failures while running the Chelsio Unified Wire Installer, please collect the below:

• Screenshot of the failure

• Installer Log file from C:\Windows\Temp\ChelsioUwire_x.x.x.x_WIN_y.y.y_devlog.log

Logs collection

In case of any driver/firmware issues, follow the below procedure to collect the standard logs:

• Download the script from https://github.com/Microsoft/SDN/blob/master/Diagnostics/Get-

NetView.PS1

• Run the script and the logs will be collected in a zip package.

PS C:\Users\Administrator\Desktop> .\Get-NetView.PS1

Please contact Chelsio support at [e-mail] with relevant logs for any issues.

Chapter III. Appendix

Chelsio Unified Wire for Windows 52

2. Chelsio End-User License Agreement (EULA)

Installation and use of the driver/software implies acceptance of the terms in the Chelsio End­User License Agreement (EULA).

IMPORTANT: PLEASE READ THIS SOFTWARE LICENSE CAREFULLY BEFORE DOWNLOADING OR OTHERWISE
USING THE SOFTWARE OR ANY ASSOCIATED DOCUMENTATION OR OTHER MATERIALS
(COLLECTIVELY, THE "SOFTWARE"). BY CLICKING ON THE "OK" OR "ACCEPT" BUTTON YOU AGREE
TO BE BOUND BY THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO THE TERMS OF THIS
AGREEMENT, CLICK THE "DO NOT ACCEPT" BUTTON TO TERMINATE THE INSTALLATION PROCESS.

1. License. Chelsio Communications, Inc. ("Chelsio") hereby grants you, the Licensee,
and you hereby accept, a limited, non-exclusive, non-transferable license to install and
use the Software with one or more Chelsio network adapters on a single server computer
for use in communicating with one or more other computers over a network. You may also
make one copy of the Software in machine readable form solely for back-up purposes,
provided you reproduce Chelsio's copyright notice and any proprietary legends included
with the Software or as otherwise required by Chelsio.

2. Restrictions. This license granted hereunder does not constitute a sale of the
Software or any copy thereof. Except as expressly permitted under this Agreement, you
may not:

(i) reproduce, modify, adapt, translate, rent, lease, loan, resell, distribute, or create
derivative works of or based upon, the Software or any part thereof; or

(ii) make available the Software, or any portion thereof, in any form, on the Internet.
The Software contains trade secrets and, in order to protect them, you may not decompile,
reverse engineer, disassemble, or otherwise reduce the Software to a human-perceivable
form. You assume full responsibility for the use of the Software and agree to use the
Software legally and responsibly.

3. Ownership of Software. As Licensee, you own only the media upon which the Software
is recorded or fixed, but Chelsio retains all right, title and interest in and to the
Software and all subsequent copies of the Software, regardless of the form or media in
or on which the Software may be embedded.

4. Confidentiality. You agree to maintain the Software in confidence and not to disclose
the Software, or any information or materials related thereto, to any third party without
the express written consent of Chelsio. You further agree to take all reasonable
precautions to limit access of the Software only to those of your employees who reasonably
require such access to perform their employment obligations and who are bound by
confidentiality agreements with you.

5. Term. This license is effective in perpetuity, unless terminated earlier. You may
terminate the license at any time by destroying the Software (including the related
documentation), together with all copies or modifications in any form. Chelsio may
terminate this license, and this license shall be deemed to have automatically
terminated, if you fail to comply with any term or condition of this Agreement. Upon any
termination, including termination by you, you must destroy the Software (including the
related documentation), together with all copies or modifications in any form.

6. Limited Warranty. If Chelsio furnishes the Software to you on media, Chelsio warrants
only that the media upon which the Software is furnished will be free from defects in

Chapter III. Appendix

Chelsio Unified Wire for Windows 53

material or workmanship under normal use and service for a period of thirty (30) days
from the date of delivery to you.

CHELSIO DOES NOT AND CANNOT WARRANT THE PERFORMANCE OR RESULTS YOU MAY OBTAIN BY USING
THE SOFTWARE OR ANY PART THEREOF. EXCEPT FOR THE FOREGOING LIMITED WARRANTY, CHELSIO
MAKES NO OTHER WARRANTIES, EXPRESS OR IMPLIED, AND HEREBY DISCLAIMS ALL OTHER WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, NON-INFRINGEMENT OF THIRD PARTY RIGHTS, MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow the exclusion of implied
warranties or limitations on how long an implied warranty may last, so the above
limitations may not apply to you. This warranty gives you specific legal rights and you
may also have other rights which vary from state to state.

7. Remedy for Breach of Warranty. The sole and exclusive liability of Chelsio and its
distributors, and your sole and exclusive remedy, for a breach of the above warranty,
shall be the replacement of any media furnished by Chelsio not meeting the above limited
warranty and which is returned to Chelsio. If Chelsio or its distributor is unable to
deliver replacement media which is free from defects in materials or workmanship, you
may terminate this Agreement by returning the Software.

8. Limitation of Liability. IN NO EVENT SHALL CHELSIO HAVE ANY LIABILITY TO YOU OR ANY
THIRD PARTY FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR PUNITIVE DAMAGES,
HOWEVER CAUSED, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF OR RELATED TO THE LICENSE
OR USE OF THE SOFTWARE, INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR LOSS OF ANTICIPATED
PROFITS, EVEN IF CHELSIO HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN NO
EVENT SHALL CHELSIO'S LIABILITY ARISING OUT OF OR RELATED TO THE LICENSE OR USE OF THE
SOFTWARE EXCEED THE AMOUNTS PAID BY YOU FOR THE LICENSE GRANTED HEREUNDER. THESE
LIMITATIONS SHALL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED
REMEDY.

9. High Risk Activities. The Software is not fault-tolerant and is not designed,
manufactured or intended for use or resale as online equipment control equipment in
hazardous environments requiring fail-safe performance, such as in the operation of
nuclear facilities, aircraft navigation or communication systems, air traffic control,
direct life support machines, or weapons systems, in which the failure of the Software
could lead directly to death, personal injury, or severe physical or environmental
damage. Chelsio specifically disclaims any express or implied warranty of fitness for
any high risk uses listed above.

10. Export. You acknowledge that the Software is of U.S. origin and subject to U.S.
export jurisdiction. You acknowledge that the laws and regulations of the United States
and other countries may restrict the export and re-export of the Software. You agree
that you will not export or re-export the Software or documentation in any form in
violation of applicable United States and foreign law. You agree to comply with all
applicable international and national laws that apply to the Software, including the
U.S.

Export Administration Regulations, as well as end-user, end-use, and destination
restrictions issued by U.S. and other governments.

11. Government Restricted Rights. The Software is subject to restricted rights as
follows. If the Software is acquired under the terms of a GSA contract: use, reproduction
or disclosure is subject to the restrictions set forth in the applicable ADP Schedule
contract. If the Software is acquired under the terms of a DoD or civilian agency
contract, use, duplication or disclosure by the Government is subject to the
restrictions of this Agreement in accordance with 48 C.F.R. 12.212 of the Federal

Chapter III. Appendix

Chelsio Unified Wire for Windows 54

Acquisition Regulations and its successors and 49 C.F.R. 227.7202-1 of the DoD FAR
Supplement and its successors.

12. General. You acknowledge that you have read this Agreement, understand it, and that
by using the Software you agree to be bound by its terms and conditions. You further
agree that it is the complete and exclusive statement of the agreement between Chelsio
and you, and supersedes any proposal or prior agreement, oral or written, and any other
communication between Chelsio and you relating to the subject matter of this Agreement.
No additional or any different terms will be enforceable against Chelsio unless Chelsio
gives its express consent, including an express waiver of the terms of this Agreement,
in writing signed by an officer of Chelsio. This Agreement shall be governed by
California law, except as to copyright matters, which are covered by Federal law. You
hereby irrevocably submit to the personal jurisdiction of, and irrevocably waive
objection to the laying of venue (including a waiver of any argument of forum non
conveniens or other principles of like effect) in, the state and federal courts located
in Santa Clara County, California, for the purposes of any litigation undertaken in
connection with this Agreement. Should any provision of this Agreement be declared
unenforceable in any jurisdiction, then such provision shall be deemed severable from
this Agreement and shall not affect the remainder hereof. All rights in the Software not
specifically granted in this Agreement are reserved by Chelsio. You may not assign or
transfer this Agreement (by merger, operation of law or in any other manner) without
the prior written consent of Chelsio and any attempt to do so without such consent
shall be void and shall constitute a material breach of this Agreement.

Should you have any questions concerning this Agreement, you may contact Chelsio by
writing to:

Chelsio Communications, Inc.
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A