Chelsio Communications Terminator 5 Installation And User Manual

Chelsio T5 Unified Wire for Windows ii
This document and related products are distributed under licenses restricting their use, copying, distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission by Chelsio Communications.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO COMMUNICATIONS,
INC.
Sales
For all sales inquiries please send email to sales@chelsio.com
Support
For all support related questions please send email to support@chelsio.com Copyright © 2016. Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications. All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters) 209 North Fair Oaks Avenue, Sunnyvale, CA 94085 U.S.A
www.chelsio.com
Tel: 408.962.3600 Fax: 408.962.3661
Chelsio (India) Private Limited Subramanya Arcade, Floor 3, Tower B No. 12, Bannerghatta Road, Bangalore-560029 Karnataka, India
Tel: +1-91-80-4039-6800
Chelsio KK (Japan)
Yamato Building 8F, 5-27-3 Sendagaya, Shibuya-ku, Tokyo 151-0051, Japan
Chelsio T5 Unified Wire for Windows iii
Document History
Version
Revision Date
1.3.6
11/19/2015
1.3.7
12/11/2015
1.3.8
01/28/2016
1.3.9
03/04/2016
1.4.0
04/21/2016
1.4.1
05/13/2016
1.4.2
06/27/2016
1.4.3
07/25/2016
1.4.4
07/29/2016
1.4.5
08/16/2016
1.4.6
08/26/2016
1.4.7
10/14/2016
Chelsio T5 Unified Wire for Windows iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 7
1. Introduction 8
1.1. Features 8
1.2. Hardware Requirements 9
1.3. Software Requirements 9
1.4. Package Contents 9
2. Hardware Installation 10
3. Software/Driver Installation 11
3.1. Prerequisites 11
3.2. Unified Wire Installer 12
3.3. Nano Server Installer 21
3.4. Zip Package 22
4. Configuring Chelsio Network Interfaces 26
4.1. Configuring 40G Adapters 26
4.2. Assigning IP address 29
5. Mass Deployment (Installer) 31
5.1. Pre-deployment Configuration 31
5.2. Flashing Option ROM 33
5.3. Erasing Option ROM 41
5.4. Configuring using UM CLI 42
6. cxgbtool/cxgbnano help 48
7. Firmware Update 93
8. Software/Driver Uninstallation 94
8.1. Installer 94
8.2. Zip Package 96
9. Software/Driver Update 97
II. NDIS FUNCTION 98
1. Introduction 99
1.1. Hardware Requirements 99
1.2. Software Requirements 99
2. Software/Driver Configuration and Fine-tuning 100
2.1. Advanced Configuration 100
2.2. NVGRE Offload 107
2.3. VXLAN Task Offload 107
2.4. Virtual RSS (vRSS) 107
III. IWARP (ND) 109
1. Introduction 110
1.1. Hardware Requirements 110
Chelsio T5 Unified Wire for Windows v
1.2. Software Requirements 110
2. Software/Driver Configuration and Fine-tuning 111
2.1. Registering iWARP(ND) driver 111
IV. SMB DIRECT 112
3. Introduction 113
3.1. Hardware Requirements 113
3.2. Software Requirements 113
4. Software/Driver Configuration and Fine-tuning 114
4.1. Enabling SMB Direct 114
4.2. Verifying RDMA 114
4.3. RDMA/NVGRE concurrent (Mode 2) 115
4.4. Troubleshooting 116
V. NDIS SR-IOV 117
1. Introduction 118
1.1. Hardware Requirements 118
1.2. Software Requirements 118
2. Software/Driver Configuration and Fine-tuning 119
2.1. Enabling SR-IOV 119
2.2. vSwitch Configuration 122
2.3. Guest (VM) Configuration 122
VI. DATA CENTER BRIDGING (DCB) 123
1. Introduction 124
1.1. Hardware Requirements 124
1.2. Software Requirements 124
2. Software/Driver Configuration and Fine-tuning 125
2.1. Installing DCB 125
2.2. Quality of Service (QoS) 125
2.3. Configuring Dell Force10 Switch 126
VII. UNIFIED WIRE MANAGER (UM) 129
1. Introduction 130
1.1. Features 130
1.2. Reference Architecture 131
1.3. Unified Wire Manager Components 131
1.4. Authentication and encryption 132
2. Hardware and Software 133
2.1. Supported Adapters 133
2.2. Platform/Component Matrix 133
2.3. Platform/Driver Matrix 133
Chelsio T5 Unified Wire for Windows vi
3. Installation (Zip Package) 134
4. Configuration 139
4.1. Configuring Management Station 139
5. Verifying UM components status 141
5.1. Verifying Management Agent 141
5.2. Verifying Management Client 142
5.3. Verifying Management Station 142
6. Management Agent 143
6.1. Communication 143
6.2. Service configuration 143
6.3. Firewall 143
7. CLI client 144
7.1. CLI Help system 144
7.2. Client conflict resolution 144
8. Web GUI client 145
8.1. Management Station 145
8.2. Accessing Web Management Interface 146
8.3. Layout and Navigation 148
8.4. Home page 149
8.5. System page 159
8.6. Network page 163
8.7. Storage 178
8.8. Hardware Features 202
9. Uninstallation 214
VIII. UM FOR WIN PE 218
1. Introduction 219
1.1. Hardware Requirements 219
1.2. Software Requirements 220
1.3. Examples 220
IX. APPENDIX 224
Chelsio End-User License Agreement (EULA) 225
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 7
I. Chelsio Unified Wire
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 8
1. Introduction
Thank you for choosing Chelsio T5 Unified Wire adapters. These high speed, single chip, single firmware cards provide enterprises and data centers with high performance solutions for various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized 40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, iSCSI, FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to thousands of connections, and allows simultaneous low latency and high bandwidth operation thanks to multiple physical channels through the ASIC.
Ideal for all data, storage and high performance clustering applications, the T5 adapters enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets, Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high performance computing environments, Chelsio T5 adapters bring a new level of performance metrics and functional capabilities to the computer networking industry.
1.1. Features
Chelsio Unified Wire for Windows is an easy to use utility developed to provide installation of 64­bit Windows based drivers and tools for Chelsio's T5 Unified Wire adapters.
It consists of the following components:
NDIS Function driver (NIC)  NVGRE Offload  VXLAN Task Offload  SMB Direct  RDMA/NVGRE concurrent (Mode 2)  iWARP (ND)  NDIS SR-IOV  Virtual RSS (vRSS)  Data Center Bridging (DCB)  Unified Wire Manager (UM)  UM for Win PE
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 9
1.2. Hardware Requirements
The Chelsio T5 Unified Wire supports all x64 architectures supporting PCIE (x4, x8) slots.
AMD CPUs, 64-bit (x86_64/amd64) Intel CPUs, 64-bit (x86_64)
1.3. Software Requirements
The Chelsio T5 Unified Wire software has been developed to run on Windows based platforms. To know more about the complete list of versions supported by each driver/software, please refer their respective sections.
1.4. Package Contents
Unified Wire Installer
Chelsio T5 Unified Wire comes with an interactive installer and support documentation. The documentation, consisting of README, Release Notes and User’s Guide (this document), can be found in the <system_drive>\ChelsioUwire\docs\ directory after installing Chelsio Unified Wire.
Zip Package
The zip package contains driver files, UM installer, UM for Win PE application, firmware binaries, adapter configuration binaries and support documentation (docs folder).
Nano Server Installer
The Nano Server Installer package contains driver files, firmware binaries, adapter configuration binaries and support documentation (docs folder)
The Chelsio Unified Wire supports 3.3v PCI bus only. Running an adapter on a PCI x4 slot is not recommended as performance will be significantly reduced by the limitations of PCI.
Note
T4 family of adapters are not supported on Windows 2016 Server & 10 AU Client versions.
Note
Drivers are WHQL certified.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 10
2. Hardware Installation
i. Shutdown/power off your system. ii. Power off all remaining peripherals attached to your system. iii. Unpack the Chelsio adapter and place it on an anti-static surface. iv. Remove the system case cover according to the system manufacturer’s instructions. v. Remove the PCI filler plate from the slot where you will install the Ethernet adapter. vi. For maximum performance, it is highly recommended to install the adapter into a PCIE x8
slot. vii. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It may
be necessary to remove the SFP (transceiver) modules prior to inserting the adapter. viii. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover. ix. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated. x. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter. xi. Power on your system. xii. Verify if the adapter was installed successfully. To do so, open Device Manager in Control
Panel. xiii. Under Other devices section, Chelsio adapter should be listed as Ethernet Controller. If the
adapter is not listed, right-click on the system name or click on the Actions menu and select
Scan for hardware changes For Chelsio adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions Physical function 4: for all NIC functions of the card Physical function 5: for iSCSI Physical function 6: for FCoE Physical function 7: Currently not assigned
xiv. Once the Unified Wire package is installed, open Device Manager again. Expand Network
adapters section and now Chelsio adapter should be listed.
Network device names for Chelsio’s physical ports are assigned using the
following convention: the port farthest from the motherboard will appear as the first Ethernet interface. However, for T5 40G adapters, the association of physical Ethernet ports and their corresponding network device names is opposite. For these adapters, the port nearest to the motherboard will appear as the first
network interface.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 11
3. Software/Driver Installation
Chelsio Unified Wire can be installed using the Installer or the zip package. Refer the relevant section below depending on the method of installation selected.
3.1. Prerequisites
Please ensure that the following requirements are met, before proceeding with the installation:
Unified Wire Installer
Download and install the latest Microsoft Visual C++ Redistributable Packages for Visual Studio (x86 and x64).
NDIS SR-IOV
Enable SR-IOV in the machine.
Unified Wire Manager (Installer & Zip Package)
The Unified Wire installer has been designed to install Unified Wire Manager (UM) along with driver components by default, hence no separate installation is required. In case of zip package, you will have to manually run the UM installer located in ChelsioUwire-x.x.x.xx/UM folder (See
Unified Wire Manager chapter for for more information). Based on the Windows version running
on the system, the three UM components, i.e. Management Agent, Client and Station will be installed. If an older version of UM exists, the installer will upgrade it to the version provided in the package.
Management Agent
If you wish to install Management Agent, please make sure that Microsoft .Net Framework 3.5 is installed before proceeding with the installation.
Management Station
If you wish to install Management Station, please make sure that the following requirements are met before proceeding with the installation:
i. Install Python 2.6.6 (32-bit). (Download from here) ii. Ensure that the path to python binary (typically C:\Python26), is added to PATH system
variable. iii. Install Apache HTTP Server 2.2 with SSL. (Download from here) iv. If the Apache Server is running, it should be stopped before starting the installation process.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 12
3.2. Unified Wire Installer
Chelsio Unified Wire Installer provides two methods of installation: GUI or CLI mode. GUI mode provides an interactive GUI installer with customizable options. Whereas, CLI mode enables unattended installation of Chelsio drivers and software thereby relieving the user from monitoring the installation process and providing input via dialog boxes.
Enabling Test Signing
In case the drivers in the package are not WHQL certified, follow the steps mentioned below to enable test signing on your system:
i. Goto Start->Run command option, enter "cmd" and press OK. This will open the command
prompt utility. ii. Run the following command:
C:\Users\Administrator> bcdedit /set testsigning on
iii. Reboot the machine for the changes to take effect.
GUI mode (Installer)
i. Run the ChelsioUwire-x.x.x.xx.exe installer application. ii. Click Next for the Chelsio End User License Agreement Window.
Figure 1 - Unified Wire installer welcome window
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 13
iii. Select the radio button I accept the terms of the license agreement and click Next.
Figure 2 - Chelsio EULA window
iv. Now, either select Complete for complete package installation or else select Custom radio
button to customize the installation.
Figure 3 - Select setup (installation) type
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 14
Figure 4 - Customizing the installation
v. Click Install to start the installation.
Figure 5 - Start installation
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 15
vi. Selected driver components will now be installed.
Figure 6 - Installation in progress
vii. After successful installation of driver(s), the Unified Wire Manager installer will be invoked.
Click Next for the Chelsio End User License Agreement Window.
Figure 7 - UM installer welcome window
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 16
viii. Select the radio button I accept the terms of the license agreement and click Next.
Figure 8 - Chelsio EULA window
ix. The next window will display the prerequisites for various UM components. Ensure that they
are met before proceeding. Click Next.
Figure 9 - UM prerequisites window
If prerequisites mentioned are not met, installation of UM will fail. Driver installation will however continue.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 17
x. Now, either select Complete for complete package installation or else select Custom radio
button to customize the installation. The Custom option provides the option to change the
installation path and which UM components to install. Click Next.
Figure 10 - Select setup (installation) type
If Management Station is selected, please make sure that all related prerequisites are met before proceeding (See Prerequisites) or else the component will be skipped during installation.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 18
Figure 11 - Changing UM installation path
Figure 12 - Selecting UM components to install
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 19
xi. Click Install to start the installation. Unified Wire Manager will now be installed with the
selected options.
Figure 13 - Starting UM installation
xii. Click Finish to exit from the UM Installer.
Figure 14 - Finishing UM installation
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 20
xiii. Click Finish to exit from the Unified Wire Installer.
Figure 15 - Finishing Unified Wire installation
CLI mode (Silent Installation)
To install drivers/software using this feature, open command prompt and execute the following command:
C:\Users\Administrator>ChelsioUwire-x.x.x.xx.exe –in <driver(s)>
E.g.:
C:\Users\Administrator>ChelsioUwire-x.x.x.xx.exe –in all
The above command will install all the drivers and UM Agent. To add a driver to an existing list of already installed drivers, use the following command:
C:\Users\Administrator>ChelsioUwire-x.x.x.x.exe –add <driver(s)>
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 21
To know more about other parameters and options, execute the following command:
C:\Users\Administrator>ChelsioUwire-x.x.x.xx.exe -help
The above command will create a help file, ChelsioUwire-x.x.x.xx_help.txt, in the present working directory which contains the complete list of command line syntax required for performing all the necessary CLI operations.
3.3. Nano Server Installer
Follow the steps mentioned below to install Unified Wire on a Nano Server machine: i. Download and unzip the driver package Chelsio-NANO-installer-x.x.x.xx.zip
ii. Connect to Nano Server machine and copy the package contents. iii. Open PowerShell with administrative privileges and change your working directory to Chelsio-
NANO-installer-x.x.x.xx
iv. Run the installer
[nanomachine-ip]: PS D:\Chelsio-NANO-installer-x.x.x.xx> .\install.ps1 -in all
A log file, Chelsio-installer.log, containing installation summary will be created in the same folder.
To know more about Installer parameters and options, view the help:
[nanomachine-ip]: PS D:\Chelsio-NANO-installer-x.x.x.xx> .\install.ps1 -h
A log file, ChelsioUwire-x.x.x.xx.log, is created in the same directory which keeps a record of all the commands executed and their results.
Note
In case of non-WHQLed drivers, if Nano Server is freshly installed, the installer will enable Test Signing. Reboot the machine and run the installer again.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 22
3.4. Zip Package
Chelsio Unified Wire zip package provides two methods of installation: Manual and using Offline Windows Image. In Manual method, the driver components will have to be individually installed
using the Device Manager. Using the second method, you can add Chelsio driver components to an offline OS image. The image can then be used to install Windows on your system.
Enabling Test Signing
In case the drivers in the package are not WHQL certified, please follow the steps mentioned below to enable test signing before installing Unified Wire:
1. Open command prompt and run the following command to enable test signing:
C:\Users\Administrator>bcdedit /set testsigning on
2. Change your working directory to ChelsioUwire-x.x.x.xx/Selfsign/ and run the following commands:
C:\ChelsioUwire-x.x.x.xx\Selfsign> .\certmgr.exe -add chelsiocert.cer -s -r localmachine root C:\ChelsioUwire-x.x.x.xx\Selfsign> .\certmgr.exe -add chelsiocert.cer -s -r localmachine trustedpublisher
3. Reboot the machine for the changes to take effect.
Manual
Virtual Bus Driver
i. Open Device Manager (Control Panel -> System & Security-> System -> Device Manager),
click on System Devices, right click on Chelsio T5 40G/10G Enumerator and select
Update Driver Software. ii. Select Browse my computer for driver software. iii. Select Let me pick from a list of device drivers on my computer and click Next. iv. Click on Have Disk Button and on the next screen browse for chvbdx64.inf file and click
Open and then OK.
Important
The driver components need to be installed strictly in the order described
here. Using any other order is not supported.
Before proceeding, open Device Manager and disable NDIS Miniport driver
(Network adapters->Chelsio Network Adapter) followed by Virtual Bus Driver (System devices->Chelsio T5 40G/10G Bus Enumerator).
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 23
v. Click Next and driver installation will progress. Click Finish once the installation is complete.
NDIS Miniport driver
i. Open Device Manager (Control Panel -> System & Security-> System -> Device Manager),
click on Network Adapters, right click on the Chelsio network adapter and select Update
Driver Software. ii. Select Browse my computer for driver software. iii. Select Let me pick from a list of device drivers on my computer. iv. Click on Have Disk Button and on the next screen browse for chnetx64.inf file and click
Open and then OK. Click Next and driver installation will progress. Click Finish once the
installation is complete. Repeat the above steps for the other adapters.
Generic Function
i. Open Device Manager (Control Panel -> System & Security-> System -> Device Manager),
click on Network Adapters, right click on Chelsio T5 40G/10G Generic Function and
select Update Driver Software. ii. Select Browse my computer for driver software. iii. Select Let me pick from a list of device drivers on my computer. iv. Click on Have Disk Button and on the next screen browse for chnullx64.inf file and click
Open and then OK. v. Click Next and driver installation will progress. Click Finish once the installation is complete.
Offline Windows Image
You can use Windows Deployment Image Servicing and Management (DISM) tool to add Chelsio driver components (.inf files) to an offline Windows image. For more information, visit the following links:
http://technet.microsoft.com/en-us/library/hh825070.aspx http://blogs.technet.com/b/heyscriptingguy/archive/2012/09/27/use-the-powershell-dism-
cmdlets-to-manage-windows-8.aspx
Nano Server: Driver Installation
Before installing Nano Server, the Chelsio driver components need to be added to the boot image file. The following steps explain step-by-step procedure to add Chelsio VBD, NDIS and Null drivers to the image file:
i. Copy NanoServer.wim image file from Windows Server 2016 (build>=10586) ISO to a
desired folder on your hard drive. ii. Open a command prompt with administrative privileges (elevated command prompt).
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 24
iii. Determine the index number of image file.
C:\Windows\system32>dism /get-wiminfo /wimfile:<path>\NanoServer.wim
iv. Mount the image file:
C:\Windows\system32>dism /mount-wim /wimfile:<path>\NanoServer.wim /index:<index_number> /mountdir:<mount_directory>
v. Copy the Microsoft OEM driver set package from NanoServer\Packages folder in the ISO to
your hard drive and add it to the image file:
C:\Windows\system32>dism /image:<path>\<mount_directory> /add-package /packagepath:<path>\Microsoft-NanoServer-OEM-Drivers-Package.cab
vi. Add Chelsio drivers to the image file:
C:\Windows\system32> dism /image:<path>\<mount_directory> /add-driver /driver:<path>\ChelsioUwire-x.x.x.xx\chvbdx64.inf
C:\Windows\system32> dism /image:<path>\<mount_directory> /add-driver /driver:<path>\ChelsioUwire-x.x.x.xx\chnetx64.inf
C:\Windows\system32> dism /image:<path>\<mount_directory> /add-driver /driver:<path>\ChelsioUwire-x.x.x.xx\chnulx64.inf
C:\Windows\system32> dism /image:<path>\<mount_directory> /add-driver /driver:<path>\ChelsioUwire-x.x.x.xx\chvbdx64.inf /forceunsigned
vii. Commit and save the image file:
C:\Windows\system32>dism /unmount-wim /mountdir:<mount_directory> /commit
In case of non-WHQLed drivers, run the above commands with the /forceunsigned option. Here’s an example for the first command:
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 25
viii. Finally, add the image file to WDS server and install Nano Server (with Chelsio drivers)
using PXE boot. ix. The Nano Server machine will reboot.
[nanomachine-ip]: PS D:\> bcdedit /set testsigning on
In case of non-WHQLed drivers, run the following command to enable test signing and reboot the machine again:
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 26
4. Configuring Chelsio Network Interfaces
In order to test Chelsio adapters’ features, it is required to use at least two machines, each with Chelsio’s network adapters. These machines can be connected directly (back-to-
back) or with a switch.
4.1. Configuring 40G Adapters
You can use the chelsio_adapter_config.ps1 script to configure Chelsio T5 40G adapters in any of the following three operational modes:
DEFAULT (2X40G): This is the default mode of operation where each port functions as
40Gbps link. The port nearest to the motherboard will appear as the first network interface
(Port 0).
SPIDER (4X10G): In this mode, port 0 functions as 4 10Gbps links and port 1 is disabled.  QSA (2X10G): This mode adds support for QSA (QSFP to SFP+) modules, enabling smooth,
cost-effective, connections between 40 Gigabit Ethernet adapters and 1 or 10 Gigabit
Ethernet networks using existing SFP+ based cabling. The port farthest from the motherboard
will appear as the first network interface (Port 0).
Windows GUI Machine (Installer & Zip Package)
i. In case of Installer, the configuration script and cxgbtool.exe will be copied to
<system_drive>\Windows\System32 folder during installation.
If you are using the zip package, copy cxgbtool.exe from ChelsioUwire-x.x.x.xx folder to
<system_drive>\Windows\System32 and change your working directory to ChelsioUwire-
x.x.x.xx\Adapter Configuration.
ii. Open PowerShell with administrative privileges. iii. Run the adapter configuration script and select Windows GUI (option 1) as the Windows
version. Hit [Enter]. Installer:
PS C:\Users\Administrator> chelsio_adapter_config.ps1
Figure 16 - Adapter configuration utility (Installer)
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 27
Zip Package:
PS D:\ChelsioUwire-x.x.x.xx\Adapter Configuration> .\chelsio_adapter_config.ps1
Figure 17 - Adapter configuration utility (zip package)
iv. Enter the index of the 40G adapter for which the configuration needs to be updated. Hit
[Enter].
Figure 18 - Selecting adapter
v. Select Port settings (option 3) as the configuration type. Hit [Enter].
Figure 19 - Port settings
vi. Select the operation mode and then enter y to confirm.
Figure 20 - Operation mode
vii. Open Device Manager, click on System Devices and right-click on the 40G Chelsio
adapter selected in step (iv). viii. Select Disable and then Enable for changes to take effect.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 28
Nano Server Machine (Nano Server Installer & Zip Package)
i. Connect to Nano Server Machine. ii. In case of Nano Server Installer, the configuration script and cxgbnano.exe utility will be
copied to <system_drive>\Windows\System32 during installation.
If you are using the zip package, copy cxgbnano.exe from ChelsioUwire-x.x.x.xx folder to
<system_drive>\Windows\System32 and change your working directory to ChelsioUwire-
x.x.x.xx\Adapter Configuration.
iii. Run the adapter configuration script and select Windows Nano Server (option 2) as the
Windows version. Hit [Enter].
Nano Server Installer:
[nanomachine-ip]: PS D:\> chelsio_adapter_config.ps1
Figure 21 - Adapter configuration utility (Nano Server Installer)
Zip Package:
[nanomachine-ip]: PS D:\ChelsioUwire-x.x.x.xx\Adapter Configuration> .\chelsio_adapter_config.ps1
Figure 22 - Adapter configuration utility (zip package)
iv. Enter the index of the 40G adapter for which the configuration needs to be updated
Figure 23 - Selecting adapter
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 29
v. Select Port settings (option 3) as the configuration type. Hit [Enter].
Figure 24 - Port settings
vi. Select the operation mode and then enter y to confirm.
Figure 25 - Operation mode
vii. Determine the Instance ID of the adapter using the Get-PnpDevice command:
Figure 26 - Adapter instance IDs
viii. Use the Instance ID obtained from the previous step to disable and enable the adapter for
changes to take effect.
Figure 27 - Committing changes
4.2. Assigning IP address
i. Double click on the Network Connections icon and choose the Chelsio card entry and
double click it. ii. Click on the Properties button from the Local Area Connection X Status. iii. Select “Internet Protocol (TCP/IP)” from the list and click on Properties button below it.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 30
iv. From the Internet Protocol (TCP/IP) Properties window, assign an IP Address (e.g.
192.169.1.10) and subnet mask (e.g. 255.255.255.0). v. Click on Ok and close on the other window. vi. Check to see if you can ping to some other address on this subnet.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 31
5. Mass Deployment (Installer)
Using Unified Wire Installer’s Mass Deployment feature, you can flash or erase Option ROM on multiple systems (nodes) simultaneously. The Installer utilizes Unified Wire Manager’s Agent and Client components to implement this feature. The Client component has to be installed on the host machine and the Agent component on the remote nodes.
5.1. Pre-deployment Configuration
A configuration file containing directives is required by the Installer during installation, flashing and erasing option ROM. Follow the steps mentioned below to generate and configure the file:
i. Copy the Unified Wire Installer (ChelsioUwire-x.x.x.xx.exe) to a shared location accessible
to all the nodes.
ii. Open Windows PowerShell with administrative privileges (right-click and select Run as
Administrator) and run the following command:
PS C:\Users\Administrator> .\ChelsioUwire-x.x.x.xx.exe -sampleconfig
The above command will generate a sample configuration file, RemoteMachineconfig.txt. It contains the following directives:
StartShareMachineInfo
ExecutablePath:\\<Machine-IP>\<Shared directory>\ChelsioUwire-x.x.x.xx.exe
User:<user>
Password:<password>
EndShareMachineInfo
StartOfCluster
<user>:<Password>
<IPAddress1>
<IPAddress2>
<IPAddress3>
<IPAddressN>
EndOfCluster
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 32
StartOfNonCluster
<IPAddress1>:<user1>:<Password1>
<IPAddress2>:<user2>:<Password2>
<IPAddress3>:<user3>:<Password3>
<IPAddressN>:<userN>:<PasswordN>
EndOfNonCluster
iii. Open the configuration file and provide the following values:
a. Enter absolute path of the shared location where installer is copied to, for the
ExecutablePath parameter.
b. Provide user credentials for the User and Password parameters of the machine where
Installer was copied to.
c. Provide IP addresses of remote nodes between the StartOfCluster and EndOfCluster
tags. Enter each node’s IP address per line. If same user credentials are set for all the nodes, enter them after the StartOfCluster tag, separated by a colon.
E.g.:
StartOfCluster admin:pass123
10.193.184.63
10.193.184.62 . . EndOfCluster
For machines with different user credentials, enter each node’s IP address and corresponding user credentials per line between the StartOfNonCluster and EndOfNonCluster tags in the following format.
E.g.:
10.193.184.76:admin:pass789
10.193.184.78:admin:pass_456
iv. Save the file to a desired location.
For successful deployment of UM components and flashing/erasing option ROM, please ensure that the host machine and remote nodes are in the same domain.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 33
5.2. Flashing Option ROM
Option ROM can be flashed using two methods. Both methods involve deployment of UM components.
Deploying components and Flashing Option ROM together. Deploying components and Flashing Option ROM separately.
Unified Wire Installer will flash Option ROM onto the first Chelsio adapter present in remote node (adapter with index 0. See Configuring using UM CLI to determine adapter index). Hence, ensure that you either use a non-Chelsio adapter for corporate/private network or ensure that the Chelsio adapter used is not installed as first on remote nodes.
Important
Both methods mentioned above will flash option ROM only onto the first Chelsio
adapter present in the remote node. If you plan to flash more adapters in the same node or add new nodes to the cluster, they will have to be added as members to a group and flashed using UM’s CLI component. See Configuring using UM CLI for instructions.
Flashing option ROM on remote nodes with inbox NDIS drivers will fail. Hence,
please ensure that all nodes are updated to the latest version using the Unified Wire Installer.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 34
Deploying UM components and Flashing Option ROM together
i. Run the following command to invoke the UM installer.
PS C:\Users\Administrator> .\ChelsioUwire-x.x.x.xx.exe -action flashoptionrom -config <config_file.txt>
ii. Click the Next button for the Chelsio End User License Agreement Window.
Figure 28 - UM Installer welcome window
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 35
iii. Select the radio button I accept the terms of the license agreement and click Next.
Figure 29 - Chelsio EULA window
iv. The next window will display the pre-requisites for various UM components. Ensure that they
are met before proceeding. Click Next.
Figure 30 - UM prerequisites window
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 36
v. Now, select Custom radio button to customize the installation. Click Next.
Figure 31 - UM prerequisites window
vi. The next window will display the location where UM will be installed by default. You can
change the location by using the Change button or click Next to continue with the default path.
Figure 32 - Changing UM installation path
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 37
vii. Next, ensure that only Client is selected under features to be installed. Click Next.
Figure 33 - Selecting UM components
viii. Click Finish to complete Client installation.
Figure 34 - Finishing UM installation
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 38
ix. Now, installation of UM Agent component on remote nodes will commence.
Figure 35 - Starting Agent installation
x. Click Agree on the PsExec License Agreement window that appears. This window will
appear only during first installation.
Figure 36 - PsEXec license agreement
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 39
xi. UM Agents will now be installed on remote nodes. A default group OptionROM will be created
and all the nodes on which UM Agent was successfully installed, will be added to it as
members.
Figure 37 - Agent installation on remote nodes
xii. Option ROM will now be flashed onto adapters on remote nodes present in the OptionROM
group.
Figure 38 - Flashing option ROM
xiii. After completion, a log file MassDeploy.log containing the summary of the process will be
created. Click OK on the dialog box that appears to exit the installer and view the log.
Figure 39 - Finishing option ROM flashing
Please ensure that the OptionROM group created here is not deleted. Also, UM Client component should not be un-installed. Doing so will cause flashing/erasing Option ROM to fail on remote nodes. UM components will have to be deployed again to fix this issue.
Important
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 40
Deploying UM components and Flashing Option ROM separately
Deploying UM Components
i. Run the following command to invoke the Unified Wire Installer.
PS C:\Users\Administrator> .\ChelsioUwire-x.x.x.xx.exe -action deployagent –config <config_file.txt>
ii. Follow steps (ii)-(xi) in the Deploying UM components and Flashing Option ROM
together section.
iii. After completion, a log file MassDeploy.log containing the summary of the process will be
created. Click OK on the dialog box that appears to exit the installer and view the log.
Figure 40 - Finishing UM components deployment
Flashing Option ROM
i. Run the following command to flash Option ROM:
PS C:\Users\Administrator> .\ChelsioUwire-x.x.x.xx.exe –action flashoptionrom
ii. Flashing Option ROM is possible only if UM Agents were installed on nodes using this
installer. Click Yes to continue.
Figure 41 - Starting option ROM flashing
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 41
iii. Follow steps xii and xiii in the Deploying UM components and Flashing Option ROM
together section.
5.3. Erasing Option ROM
i. To erase OptionROM from adapters on all the nodes, run the following command:
PS C:\Users\Administrator> .\ChelsioUwire-x.x.x.xx.exe -action eraseoptionrom
ii. Erasing OptionROM is possible only if the adapters were flashed using this installer. Click
Yes to continue.
Figure 42 - Starting option ROM erasing
iii. Option ROM will now be erased from adapters on nodes present in the OptionROM group.
Figure 43 - Erasing option ROM on remote nodes
iv. After completion, a log file MassDeploy.log containing the summary of the process will be
created. Click OK on the dialog box that appears to exit the installer and view the log.
Unified Wire Installer will erase option ROM only from the first Chelsio adapter present in the remote node. If you plan to erase from more adapters in the same node or add new nodes to the cluster, they will have to be added as members to a group and erased
using UM’s CLI component. See Configuring using UM CLI for instructions.
Note
Flashing option ROM on remote nodes with inbox NDIS drivers will fail. Hence, please ensure that all nodes are updated to the latest version using the Unified Wire Installer.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 42
Figure 44 - Finishing option ROM erasing
5.4. Configuring using UM CLI
Configuring groups and members
You can manage multiple groups and members using Unified Wire Manager’s CLI component,
chelsio_uwcli.
Create Group
To create a new group, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe –m creategroup grpname=<new_group_name> OSType=windows GroupType=t4adapter
E.g.
PS C:\Users\Administrator> chelsio_uwcli.exe –m creategroup grpname=flash OSType=windows GroupType=t4adapter
Figure 45 - Creating group
To use the CLI component, you will have to change your working directory to the location where UM Client is installed. Typically, this will be C:\Program Files (x86)\Chelsio Communications\UnifiedWireManager\client. To avoid this and run the command from any path, you will need to reboot the host. The following examples assume that the host machine was rebooted before running commands.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 43
Add member to group
To add a member to a group, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe –m addmember grpname=<group_name> details=”<IP_address>;<user_id>;<password>;*;*,*,<adapter_index>,*,*,*,*,*>”
E.g.
PS C:\Users\Administrator> chelsio_uwcli.exe m addmember grpname=OptionROM details=”10.193.185.107;administrator;cdrom888;*;*,*,0,*,*,*,*,*>”
Figure 46 - Adding member to group
View group and member details
To view details of all the groups and members created, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe –m list
Figure 47 - Viewing group and member details
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 44
Delete member from a group
To delete a member from a group, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe –m deletemember grpname=<group_name> index=<member_index>
E.g.
PS C:\Users\Administrator> chelsio_uwcli.exe –m deletemember grpname=OptionROM index=1
Figure 48 - Deleting member from group
Delete Group
To delete a group, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe –m deletegroup grpname=<group_name>
E.g.
PS C:\Users\Administrator> chelsio_uwcli.exe –m deletegroup grpname=OptionROM
Figure 49 - Deleting group
Flashing/Erasing Option ROM
Apart from Unified Wire Installer, you can also use Unified Wire Manager’s CLI component (chelsio_uwcli) to flash or erase Option ROM on multiple nodes. This is particularly useful when you need to flash/erase Option ROM on groups other than the default OptionROM group.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 45
Flashing Option ROM
To flash Option ROM using UM, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe chelsio flashrom adapter=BULKADAPTER path=<path_to_optionrom_image_file> –B <group_name>
E.g.
PS C:\Users\Administrator> chelsio_uwcli.exe chelsio flashrom adapter=BULKADAPTER path=C:\Users\Administrator\Desktop\cuwlbt4.bin –B OptionROM
Figure 50 - Flashing option ROM using UM CLI
Viewing status
You can list the adapters on local host and verify if Option ROM was successfully flashed using the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe chelsio listadapters –B <group_name>
E.g.
PS C:\Users\Administrator> chelsio_uwcli.exe chelsio listadapters –B OptionROM
Flashing option ROM on remote nodes with inbox NDIS drivers will fail. Hence, please ensure that all nodes are updated to the latest version using the Unified Wire Installer.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 46
Figure 51 - Viewing status
Erasing Option ROM
To erase Option ROM using UM, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe chelsio eraserom adapter=BULKADAPTER force=1 –B <group_name>
In addition to flashing and erasing Option ROM, you can perform additional bulk operations on remote nodes like setting MTU and VLAN ID, changing adapter and port parameters, etc. To know more about these options available, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe -
Figure 52 - Erasing option ROM using UM CLI
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 47
Help
To view Unified Wire Manager’s CLI help, run the following command:
PS C:\Users\Administrator> chelsio_uwcli.exe –m
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 48
6. cxgbtool/cxgbnano help
The cxgbtool (cxgbnano for Nano Server) command queries or sets various aspects of Chelsio network interface cards. It complements standard tools used to configure network settings and provides functionality not available through such tools.
Some of the commands provided can be used to query running statistics to aid in debugging.
Definitions
[nicInterface] is the name of the network device to work on, given in the format "nic[0,1,..n]" [vbdInterface] is the name of the Chelsio Bus Enumerator instance to work on, given in the format
"vbd[0,1,...n]"
Syntax
cxgbtool [vbdInterface|nicInterface][parameters][(optionalParameters)] In case of Nano Server, run the commands using cxgbnano instead of cxgbtool, i.e., cxgbnano [vbdInterface|nicInterface][parameters][(optionalParameters)]
-h
Description: Displays help Syntax: cxgbtool -h
cim_la
Decription: Displays results of logic analyzer trace. Syntax: cxbtool [vbdInterface] cim_la Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 49
context
Description: Shows an SGE context. Syntax: cxgbtool [vbdInterface] context [contextType] [queueId] Context Type Parameters:
Egress: Egress queue context. fl: Free list manager context. response: Response queue context. ingress: Ingress queue context. cq: RDMA completion queue context. cong: Congestion context.
Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 50
filter
Description: Displays list of configured hardware filters Syntax: cxgbtool [nicInterface] filter Example:
loadfw
Description: Loads firmware image. Syntax: cxgbtool [vbdInterface] loadfw [firmwareImage]
Example:
loadcfg
Description: Loads firmware configuration file or clears configuration flash region.
Syntax: cxgbtool [vbdInterface] loadcfg [[firmwareConfigurationFile]|clear] Example:
Loading T5 firmware configuration file:
Clearing the configuration file region in flash
The configuration file used must be in text format and not a header file.
Note
The Firmware input file used must be a binary and not a header file.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 51
loadphy
Description: Loads phy firmware. Syntax: cxgbtool [vbdInterface] loadphy [phyFile] Example:
loadboot
Description: Flashes the Option ROM image. Syntax: cxgbtool [vbdInterface] loadboot [bootImageFile] [pf {0|1|2|…|7}|offset {val}|clear] Parameters:
pf{val}: Flash Option ROM image to the offset defined by the PFs EXPROM_OFST in the serial configuration.
offset {val}: Flash Option ROM image to the offset specified by the user. clear: The flash area reserved for Option ROM image will be cleared. Example: Flashing Option ROM image to the offset defined by the PFs EXPROM_OFST:
Flashing Option ROM image to user specified offset:
loadboot-cfg
Description: Loads boot configuration file. Syntax: cxgbtool [vbdInterface] loadboot-cfg [[bootConfigImageFile]|clear] Parameters:
clear: The flash area reserved for boot configuration file will be cleared.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 52
Example: Loading boot configuration file:
Clearing the flash area:
mdio
Description: Reads/writes MDIO register. Syntax: cxgbtool [vbdInterface] mdio
[physicalAddress][manageableDevicesAddress][registerAddress] [(writeValue)] Example Read MDIO register
Write MDIO register currently not supported.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 53
meminfo
Description: Displays memory info. Syntax: cxgbtool [vbdInterface] meminfo Example:
mtus
Description: Prints hardware MTU table. Syntax: cxgtool [vbdInterface] mtus
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 54
Example:
qsets
Description: Reads # of qsets Syntax: cxgbtool [nicInterface] qsets Example:
qstats
Description: Displays statistics for each Tx & Rx queue. Syntax: cxgbtool [nicInterface] qstats [queueType [(clr)]] Queue Type parameters:
txeth: Tx tunnel queue statistics. rxeth: Rx tunnel queue statistics. txvmq: Tx VM queue statistics. rxvmq: Rx VM queue statistics. txrdma: RDMA Tx queue statistics. rxrdma: RDMA Rx queue statistics. txctrl: Chimney control queue statistics. txfwd: Chimney forwarding queue statistics. txnvgre: Tx NVGRE statistics. rxnvgre: Rx NVGRE statistics. clr: Clear Queue statistics.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 55
Example of rxeth qstats:
reg
Description: Reads/writes register. Syntax:
Register Read: cxgbtool [vbdInterface] reg [readAddress] Register Write: cxgbtool [vbdInterface] reg [[writeAddress] = {val}]
Example:
Register read:
Register write:
regdump
Description: Displays registers of a hardware module. Not specifying any parameter will display registers for all the modules. Not all modules are available on all adapters.
Syntax: Display registers for all available modules: cxgbtool [vbdInterface] regdump
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 56
Display registers for specific module: cxgbtool [vbdInterface] regdump [registerModule]
Table 1 – T5 register module parameters
Module Parameter
Description
sge
Scatter-Gather DMA Engine common register set.
pci
PCI Express Interface common register set. This module implements the PCI-Express SR-IOV physical logical, data link, and transaction layers.
dbg
Debug Engine Common register set.
mc0
Memory controller 0 common register set. This module implements the memory controller for the optional external DDR-II/DDR-III SDRAM.
mc1
Memory controller 1 common register set. This module implements the memory controller for the optional external DDR-II/DDR-III SDRAM.
ma
Memory Arbiter common register set. This module implements the arbitration of memory requests from the various on-chip sources to the memory hierarchy consisting of on-chip eDRAM, external DDR2/DDR3 memory, and host memory that is accessed through the PCIe.
edc0
eDRAM and Controller 0 common register set. This is the on-chip eDRAM and controller.
edc1
eDRAM and Controller 1 common register set. This is the on-chip eDRAM and controller.
cim
CIM common register set. This module implements the CPU interface and μP is the embedded microprocessor. The CIM incorporates functions to improve the performance of CPU accesses to external memory.
tp
Transport Protocol Engine common register set. This module implements the main packet processing pipeline.
ulp_rx
Ingress Upper Layer Protocol common register set. This module implements the upper layer protocol processing in the ingress direction for protocols that are layered on top of TCP, such as iSCSI and RDMA.
ulp_tx
Egress Upper Layer Protocol common register set. This module provides Upper Layer support for RDMA and iSCSI offload in the transmit direction, and also implements LSO/TSO functionality.
pmrx
Ingress Payload Manager common register set. These modules implement the payload manager for receive/ingress.
pmtx
Egress Payload Manager common register set. These modules implement the payload manager for transmit/egress.
mps
Multi-port support common register set. This module implements the multi-port support for T4, and switches egress packets to the ingress path when their Ethernet DA (Destination Address) matches an address in the exact match Ethernet Address database, or if the outer-VLAN indicates that the packet is destined to another virtual machine connected to the T4, or if another virtual machine is subscribing to an L2 multicast group that is the MAC destination address of the packet.
cplsw
CPL Switch common register set. This module implements a configurable switch for ingress CPL messages to the SGE and/or CIM.
The embedded μP is assigned a receive queue number and can be
assigned one or more MAC addresses and IP addresses, and any of these can be used to switch ingress packets to the μP for processing.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 57
smb
System Management Bus common register set. This module implements an SMBus Master/Slave for system management.
i2c
I2C-Bus Master common register set. This module implements an I2C­Bus Master for PHY management and I/O expansion.
mi
MI common register set. This module implements an MDIO Master for PHY management.
uart
pmu
Power Management Unit common register set. Power management unit.
sf
Serial Flash controller common register set. This module implements the serial flash controller. It interfaces to the external serial flash device.
pl
PIO Local Bus controller common register set. This module implements the PIO Local Bus controller. It is physically distributed across T4/T5.
le
Lookup Engine Common register set. The Lookup Engine implements the connection, filter and ACL databases. This module includes a TCAM Memory controller that interfaces with the on-chip TCAM memory array, and it implements the interface to the external memory that is used to scale the support of the connection database to 1M connections. The LE stores ACL rules, it stores routing information to handle routing for SYN-cookie mode offloaded listening servers, and it stores tuple information for offloaded connections, and FCoE exchanges.
ncsi
Network Controller sideband Interface common register set. The module implements the NCSI (Network Controller Sideband Interface) protocol.
xgmac
mac
MAC common register set.
hma
Example:
Displaying all available module registers:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 58
Displaying specific (mps) module registers:
tcb
Description: Reads hardware TCP Control Block, which contains details regarding all offloaded connections.
Syntax: cxgbtool [nicInterface] tcb [tid] Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 59
tpi
Description: Displays TP indirect registers. Syntax: cxgbtool [vbdInterface] tpi Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 60
sgedbg
Description: Displays sge debug indirect registers. Syntax: cxgbtool [vbdInterface] sgedbg Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 61
dumpctx
Description: Displays adapter context. Syntax: cxgbtool [nicInterface] dumpctx Example:
version
Description: Displays the adapter Part Number, Serial Number, Device ID, Firmware Version, TP, NDIS Driver version, and VBD version.
Syntax: cxgbtool [vbdInterface] version Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 62
fwtoc
Description: Converts firmware binary file to source file header. Syntax: cxgbtool [vbdInterface] fwtoc [firmwareFile] filename= [CFileName].bin Example:
inst
Description: Installs driver package specified in the setup information file (.inf). Syntax: cxgbtool [vbdInterface] inst [infFilePath] force Example:
unins
Description: Uninstalls driver package specified in the setup information file (.inf). Syntax: cxgbtool [vbdInterface] uninst [infFile] Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 63
update
Description: Updates driver package. Syntax: cxgbtool update [infFile] Example:
rescan all
Description: Scans for hardware changes in the device manager. Syntax: cxgbtool [vbdInterface] rescan all Example:
trace
Description: Enables driver debug prints. Syntax: trace [nicInterface|vbdInterface] trace [nic|vbd] [(flags)][(level)] Example:
seeprom
Description: Reads/Writes SEEPROM (init+VPD) data. Syntax:
Read SEEPROM data to a bin file: cxgbtool [vbdInterface] seeprom read Display SEEPROM data in console: cxgbtool [vbdInterface] seeprom dump Write SEEPROM data: cxgbtool [vbdInterface] seeprom write [seepromFile] Verify SEEPROM data: cxgbtool [vbdInterface] seeprom verify [seepromFile]
Important
Use this option with caution. Incorrect usage may render the adapter useless.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 64
Example: Read SEEPROM data:
Write SEEPROM data:
Verify SEEPROM data:
serialinit
Description: Reads/Writes serialinit data. Syntax:
Read serialinit data to a bin file: cxgbtool [vbdInterface] serialinit read Display serialinit data in console: cxgbtool [vbdInterface] serialinit dump Write serialinit data: cxgbtool [vbdInterface] serialinit write [initdataFile] Verify serialinit data: cxgbtool [vbdInterface] serialinit verify [initdataFile]
Important
Use this option with caution. Incorrect usage may render the adapter useless.
The seepromFile should be in binary format (.bin).
Note
The initdataFile should be in binary format (.bin).
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 65
Example:
Read serialinit data
Write serialinit data
Verify serialinit data
vpd
Description: Reads/writes VPD data. Syntax:
Read VPD data: cxgbtool [vpdInterface] vpd read Write VPD data to serial EEPROM on chip: cxgbtool [vpdInterface] vpd write [vpdFile]
Example:
Read VPD Data:
Important
Use this option with caution. Incorrect usage may render the adapter useless.
The vpdFile should be in binary format (.bin).
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 66
Write VPD Data:
vpdparams
Description: Displays the adapters Serial Number, Device ID, EC, Core Clock, Part Number, and Network Address (MAC).
Syntax: cxgbtool [vbdInterface] vpdparams Example:
wtp
Description: Traces the Ingress and Egress path of a packet through the Chelsio adapter. Syntax: cxgbtool [vbdInterface] wtp Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 67
stats
Description: Prints MAC statistics for a given port or TP MIB statistics. Syntax: cxgbtool [vbdInterface] stats [mac{portInstance}|mib] Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 68
cpl_stats
Description: Displays CPL Request and Response Statistics for all channels. Syntax: cxgbtool [vbdInterface] cpl_stats Example:
debugfs
Description: When used in conjunction with other options, debugfs displays useful information regarding Chelsio adapters.
Syntax: cxgbtool [vbdInterface] debugfs [option]
debugfs options:
cctrl
Description: Displays congestion control table. Syntax: cxgbtool [vbdInterface] debugfs cctrl Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 69
cim_pif_la
Description: Displays CIM PIF logic analyzer trace. Syntax: cxgbtool [vbdInterface] debugfs cim_pif_la Example:
cim_ma_la
Description: Displays results of CIM MA logic analyzer trace. Syntax: cxgbtool [vbdInterface] debugfs cim_ma_la Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 70
cim_qcfg
Description: Displays CIM queue configuration details. Syntax: cxgbtool [vbdInterface] debugfs cim_qcfg Example:
clk
Description: Displays the core clock. Syntax: cxgbtool [vbdInterface] debugfs clk Example:
sensor
Description: Displays sensor data. Syntax: cxgbtool [nicInterface] debugfs sensor
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 71
Example:
ddp_stats
Description: Displays DDP(Direct Data Placement) statistics. Syntax: cxgbtool [vbdInterface] debugfs ddp_stats Example:
edc0
Description: Redirects EDC0 memory details to a file. Syntax: cxgbtool [vbdInterface] debugfs edc0 [fileName]
Example:
edc1
Description: Redirects EDC1 memory details to a file. Syntax: cxgbtool [vbdInterface] debugfs edc1 [fileName]
It is recommended that the fileName be provided without any extension.
Note
It is recommended that the fileName should be provided without any extension.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 72
Example:
flash
Description: Redirects Flash memory details to a file. Syntax: cxgbtool [vbdInterface] debugfs flash [fileName]
Example:
ibq_tp
Description: Displays CIM TP inbound queue. Syntax: cxgbtool [vbdInterface] debugfs ibq_tp[{0|1}] Example:
It is recommended that the fileName should be provided without any extension.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 73
ibq_ulp
Description: Displays CIM ULP inbound queue. Syntax: cxgbtool [vbdInterface] debugfs ibq_ulp Example:
ibq_sge
Description: Displays CIM SGE inbound queue. Syntax: cxgbtool [vbdInterface] debugfs ibq_sge[{0|1}] Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 74
ibq_ncsi
Description: Displays CIM NCSI inbound queue. Syntax: cxgbtool [vbdInterface] debugfs ibq_ncsi Example:
mc
Description: Redirects MC memory details to a file. Syntax: cxgbtool [vbdInterface] debugfs mc[{0|1}] [dumpFileName] Example:
mps_tcam
Description: Displays MPS TCAM configuration. Syntax: cxgbtool [vbdInterface] debugfs mps_tcam
It is recommended that the dumpFileName should be provided without any extension.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 75
Example:
mps_trc_rd trace
Description: Reads MPS trace filter. Syntax: cxgbtool [vbdInterface] debugfs mps_trc_rd trace[traceQueueId] Example:
mps_trc_wr trace
Description: Sets MPS trace filter. Syntax:
cxgbtool [vbdInterface] debugfs mps_trc_wr trace[0|1|2|3] [tx[portInstance]|rx[portInstance]]
Example:
Setting up trace0 with tx0 (tx0 is port0 Tx path)
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 76
mbox
Description: Displays the last command in each mailbox. Syntax: cxgbtool [vbdInterface] debugfs mbox[{0|1|2|…|7}] Example:
obq_ulp
Description: Displays ULP outbound queue. Syntax: cxgbtool [vbdInterface] debugfs obq_ulp[{0|1|2|3}] Example:
obq_sge
Description: Displays SGE outbound queue. Syntax: cxgbtool [vbdInterface] debugfs obq_sge
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 77
Example:
obq_ncsi
Description: Displays NCSI outbound queue. Syntax: cxgbtool [vbdInterface] debugfs obq_ncsi Example:
obq_sge_rx_q
Description: Displays CIM SGE outbound queue. Syntax: cxgbtool [vbdInterface] debugfs obq_sge_rx_q[{0|1}]
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 78
Example:
pm_stats
Description: Displays page memory statistics. Syntax: cxgbtool [vbdInterface] debugfs pm_stats Example:
tcp_stats
Description: Displays IPv4/IPv6 TCP statistics. Syntax: cxgbtool [vbdInterface] debugfs tcp_stats Example:
tp_err_stats
Description: Displays TP error statistics for channel 0-3. Syntax: cxgbtool [vbdInterface] debugfs tp_err_stats
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 79
Example:
tp_la
Description: Dumps TP la. Syntax:
cxgbtool [vbdInterface] debugfs tp_la cxgbtool [vbdInterface] debugfs tp_la [{2|3} ] Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 80
tid_info
Description: Displays TID info. Syntax: cxgbtool [nicInterface] debugfs tid_info Example:
tx_rate
Description: Displays TX rate for NIC and offload traffic. Syntax: cxgbtool [vbdInterface] debugfs tx_rate Example:
ulprx_la
Description: Dumps ULP RX LA. Syntax: cxgbtool [vbdInterface] debugfs ulprx_la Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 81
cudbg
Description: Collects and processes Chelsio adapter debug logs. Syntax: cxgbtool [vbdInterface] debugfs cudbg [operation] [entity] [outfile][(optionalParams)] Examples:
o Collecting debug log for all entities
o Collecting debug log skipping specific debug entities
o Collecting debug log avoiding entities that can affect running traffic
o Displaying debug log on the screen.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 82
o Displaying debug log on the screen skipping specific entities
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 83
o Displaying summary of debug log present in a compressed dump file on screen
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 84
o Extracting the compressed debug log in a human readable format to the specified path
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 85
o Extracting the compressed debug log in a human readable format to the specified path
skipping specific entities
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 86
o Displaying debug log on the screen without storing them in any file.
devlog
Description: Prints firmware device log information. Syntax: cxgbtool [vbdInterface] devlog Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 87
rss
Description: Prints RSS info. Syntax: cxgbtool [vbdInterface] rss Example:
rss_config
Description: Prints RSS Configuration. Syntax: cxgbtool [vbdInterface] rss_config Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 88
rss_key
Description: Prints RSS Key. Syntax: cxgbtool [vbdInterface] rss_key Example:
rss_pf_config
Description: Prints RSS PF Configuration Syntax: cxgbtool [vbdInterface] rss_pf_config Example:
rss_vf_config
Description: Prints RSS VF Configuration. Syntax: cxgbtool [vbdInterface] rss_vf_config Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 89
coalesce
Description: Changes the coalescing settings for tunnel Rx queues of the specified Ethernet device.
Syntax: cxgbtool [
nicInterface
] coalesce [rx-usecs-irq {val}] [rx-frames-irq {val}] [(persistent)]
Parameters
rx-usecs-irq: Rx Coalescing Timer. Number of microseconds after which interrupt will be sent. rx-frames-irq: Rx Coalescing Threshold Packets. Number of packets after which interrupt will be
sent. Example: Setting Rx Coalescing Timer:
Setting Rx Coalescing Threshold Packets
eps
Description: Prints endpoints (eps) if NDK/ND is enabled. Syntax: cxgbtool [nicInterface] eps Example:
qps
Description: Prints queue paris (qps) if NDK/ND is enabled. Syntax: cxgbtool [nicInterface] qps Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 90
rdma_stats
Description: Prints RDMA statistics if NDK/ND is enabled Syntax: cxgbtool [nicInterface] rdma_stats Example:
stags
Description: Prints STAG contents if NDK/ND is enabled. Syntax: cxgbtool [nicInterface] stags Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 91
chim_sock
Description: Prints chimney statistics. Syntax: cxgbtool [nicInterface] chim [tcb {tid}|sock]
tcb: Prints Hardware TCB information for given tid of an offloaded connection. sock: Prints driver per socket statistics.
l2t
Description: Displays l2t table contents. Syntax: cxgbtool [nicInterface] l2t Example:
hw_sched
Description: Displays hardware schedule information Syntax: cxgbtool [vbdInterface] hw_sched Example:
Currently not supported.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 92
mbox_log
Description: Prints firmware mailbox command/reply log information. Syntax: cxgbtool [vbdInterface] mbox_log Example:
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 93
7. Firmware Update
The driver will auto-load the T5 firmware if an update is required. The firmware version can be verified using cxgbtool:
C:\Users\Administrator>cxgbtool nic0 version
In case of installer, the firmware binaries will be copied to <system_drive>\ChelsioUwire \Firmware\ folder during installation.
For zip package, the binaries will be present in ChelsioUwire-x.x.x.xx\firmware\ folder. For Nano Server Installer, the binaries will be present in Chelsio-NANO-installer-
x.x.x.xx\firmware\ folder.
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 94
8. Software/Driver Uninstallation
Similar to installation, Chelsio Unified Wire can be uninstalled using the Installer or zip package. Refer the relevant section depending on the method of installation used to install drivers.
8.1. Installer
Chelsio Unified Wire Installer can be uninstalled using two methods: GUI or CLI mode. GUI mode requires user interaction and uninstallation occurs with options specified by the user. Whereas, CLI mode does not require any user input.
GUI mode (Installer)
i. Run the ChelsioUwire-x.x.x.xx.exe application. ii. Select Modify to add or remove features. Select Repair to repair the previous installation.
Select Remove to uninstall the application. After you have selected the appropriate option, click Next.
Figure 53 - Selecting maintenance option
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 95
iii. Click on the Finish button to exit from the installer.
Figure 54 - Finishing uninstallation
CLI Mode
To uninstall all the drivers, execute the following command:
C:\Users\Administrator>ChelsioUwire-x.x.x.xx.exe –un all
To uninstall a particular driver, execute the following command:
C:\Users\Administrator>ChelsioUwire-x.x.x.x.exe –rm <driver(s)>
This method of uninstallation is possible only if the drivers were installed using
Unified Wire Installer.
Uninstalling Unified Wire package using the above method will not uninstall
Unified Wire Manager. See Uninstallation section of Unified Wire Manager chapter for more information.
Note
Uninstalling Unified Wire package using the above method will not uninstall
Unified Wire Manager. You will have to manually uninstall it using “Programs
and Features” in the Control Panel.
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 96
8.2. Zip Package
If the driver components were installed using the zip package, they will have to be uninstalled manually. The following section lists the various components and their respective methods of uninstallation.
NDIS Miniport driver
i. Open the Device Manager (Control Panel -> System & Security-> System -> Device
Manager), click on the Network Adapters, right click on the Chelsio Network Adapter and select Uninstall.
ii. Select the Delete the driver software for this device checkbox when Confirm Device
Uninstall is prompted and click OK.
iii. Repeat the same procedure for the other Chelsio Network Adapters.
Virtual Bus Driver
i. Open the Device Manager (Control Panel -> System & Security-> System -> Device
Manager), click on the System Devices, right click on the Chelsio T5 40G/10G Bus Enumerator and select Uninstall.
ii. Select the Delete the driver software for this device checkbox when Confirm Device
Uninstall is prompted and click OK.
Generic Function
i. Open the Device Manager (Control Panel -> System & Security-> System -> Device
Manager), click on the Network Adapters, right click on the Chelsio T5 40G/10G Generic Function and select Uninstall.
ii. Select the Delete the driver software for this device checkbox when Confirm Device
Uninstall is prompted and click OK.
iii. Repeat the same procedure for the other Generic Functions.
Important
The driver components need to be uninstalled strictly in the order described here. Using any other order is not supported.
In case of Virtual Functions in VM, use Chelsio VF Network Adapter.
Note
In case of Virtual Functions in VM, use Chelsio T5 40G/10G Bus Enumerator [Virtual Function].
Note
Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 97
9. Software/Driver Update
For any distribution specific problems, please check README and Release Notes included in the release for possible workaround.
Please visit Chelsio Download Center, http://service.chelsio.com/, for regular updates on various software/drivers. You can also subscribe to our newsletter for the latest software updates.
Chapter II. NDIS Function
Chelsio T5 Unified Wire for Windows 98
II. NDIS Function
Chapter II. NDIS Function
Chelsio T5 Unified Wire for Windows 99
1. Introduction
Chelsio’s T5 series of Unified Wire adapters provide extensive support for NIC operation, including all stateless offload mechanisms for both IPv4 and IPv6 (IP, TCP and UDP checksum offload, LSO - Large Send Offload aka TSO - TCP Segmentation Offload, Network Direct and assist mechanisms for accelerating LRO - Large Receive Offload).
1.1. Hardware Requirements
Supported Adapters
Following is the list of NDIS components and supported Chelsio adapters:
Table 2 - Chelsio adapters and supported NDIS components
Chelsio Adapter
NDIS Components
T580-CR
NIC, NVGRE, VXLAN, vRSS
T520-CR
T580-LP-CR
T580-OCP-SO
NIC, NVGRE, VXLAN
T520-OCP-SO
T580-SO-CR
NIC, NVGRE, VXLAN, vRSS
T520-SO-CR
T520-LL-CR
NIC, NVGRE, VXLAN, vRSS
T540-CR
T520-BT
1.2. Software Requirements
Windows Requirement
Currently Chelsio T5 NDIS function driver is available for the following Windows versions:
Server 2016 Nano Server 2016 10 AU Client Windows Server 2012 R2
Chapter II. NDIS Function
Chelsio T5 Unified Wire for Windows 100
2. Software/Driver Configuration and Fine-tuning
2.1. Advanced Configuration
The Chelsio network driver provides advanced configuration options under the Device Properties. In the Network Connections window, select the Chelsio adapter’s Local Area Connection interface (right-click -> properties). Click the Configure button.
VBD Driver Parameters
To see all tunable VBD driver parameters, open the Device Manager, click on the System Devices, double click on the Chelsio T5 40G/10G Bus Enumerator, and then click the Advanced tab.
iSCSI Instances
Description: Specify the number of iSCSI instances.  Value: 0-2 | default  Default: 0
Nic Instances:
Description: Specify the number of NIC instances.  Value: 0-8 | default  Default: default
NDIS Miniport Driver Parameters
To see all tunable NDIS Miniport Driver parameters, open the Device Manager, click on the Network adapters, double click on any Chelsio Network Adapter, and then click the Advanced
tab.
Encapsulated Task Offload
Description: Enable or disable stateless offloads (checksums, LSO, VMQ and filtering) of
encapsulated traffic.
Value: Disabled | Enabled  Default: Enabled
Encapsulated Overhead
Description: Specify the amount of overhead required in Ethernet frames due to virtual
network overlay encapsulation such as VXLAN and NVGRE.
Value:
o 32 o 64 o 96
Loading...