Teledyne IBTrainer User Manual

2403 Walsh Avenue, Santa Clara, CA 95051-1302 Tel: [phone] Fax: [phone]

CATC IBTrainer InfiniBand Exerciser

User’s Manual

For Software Version 2.0

Manual Version1.0

16 August, 2002

Important Notice

Manual PartNumber

730-0040-00

Revision

Revision 1.0, August 2002

Printed in U.S.A.

Computer Access Technology Corporation,

2403 Walsh

Avenue, Santa Clara CA 95951

USA

Document Disclaimer

The information in this document has been
carefully checked and is believed to be reliable.
However, no responsibility can be assumed for
inaccuracies that may not have been detected.

CATC reserves the right to revise the information
in this document without notice or penalty.

Trademarks and Servicemarks

CATC, Merlin, NetMate, Advisor, Chief,
FireInspector, Inspector, IBTracer, Detective,
Traffic Generator, BusEngine, USB4DOS, UPT,
HPT, and UHT are trademarks of Computer

Access Technology Corporation.

Microsoft, Windows, and Windows NT are
registered trademarks of Microsoft Inc.

All other trademarks are property of their
respective companies.

Portions of this documentation have been
licensed from Agilent Technologies, Inc.

Copyright

Copyright © 2002, Computer Access Technology
Corporation (CATC); All Rights Reserved.

This document may be printed and reproduced
without additional permission, but all copies
should contain this copyright notice.

Safety Notices

CAUTION

A CAUTION notice denotes a hazard. It calls
attention to an operating procedure, practice, or
the like that, if not correctly performed or adhered
to, could result in damage to the product or loss
of important data. Do not proceed beyond a
CAUTION notice until the indicated conditions
are fully understood and met.

WARNING

A WARNING notice denotes a hazard. It calls
attention to an operating procedure, practice, or
the like that, if not correctly performed or adhered
to, could result in personal injury or death. Do not
proceed beyond a WARNING notice until the
indicated conditions are fully understood and
met.

2 CATC IBTrainer InfiniBand Exerciser, August 2002

Safety Information

The following general safety precautions must be observed during all

phases of operation of this instrument. Failure to comply with these

precautions or with specific warnings elsewhere in this manual

violates safety standards of design, manufacture, and intended use of

the instrument.

CATC assumes no liability for the customer's failure to comply with

these requirements.

Before operation, review the instrument and manual for safety

markings and instructions. You must follow these to ensure safe

operation and to maintain the instrument in safe condition.

Safety Summary This product is a Safety Class 1 instrument (provided with a protective

earth terminal). The protective features of this product may be

impaired if it is used in a manner not specified in the operation

instructions.

All Light Emitting Diodes (LEDs) used in this product are Class 1

LEDs as per IEC 60825-1.

Environmental conditions This instrument is intended for indoor use in an installation category

II, pollution degree 2 environment. It is designed to operate at a

maximum relative humidity of 95% and at altitudes of up to 2000

meters.

Refer to the specifications tables for the ac mains voltage

requirements and ambient operating temperature range.

Before applying power Verify that the product is set to match the available line voltage, the

correct fuse is installed, and all safety precautions are taken.

Ground the instrument To minimize shock hazard, the instrument chassis and cover must be

connected to an electrical protective earth ground. The instrument

must be connected to the ac power mains through a grounded power

cable, with the ground wire firmly connected to an electrical ground

(safety ground) at the power outlet. Any interruption of the protective

(grounding) conductor or disconnection of the protective earth

terminal will cause a potential shock hazard that could result in

personal injury.

Do not operate in an explosive

atmosphere

CATC IBTrainer InfiniBand Exerciser, August 2002 3

Do not operate the instrument in the presence of flammable gases or

fumes.

Do not remove the instrument cover Operating personnel must not remove instrument covers. Component

replacement and internal adjustments must be made only by qualified

personnel.

Instruments that appear damaged or defective should be made

inoperative and secured against unintended operation until they can

be repaired by qualified service personnel.

EU Conformance Statement

This equipment complies with the EMC Directive 89/336/EEC and the

Low Voltage Directive 73/23/EEC, and their associated amendments

for Class A ISM Equipment. It has been tested and found to comply

with EN55022 and EN55011 (EN61000-4-2, EN61000-4-3, EN61000-4-4,

EN61000-4-5, EN61000-4-6, EN61000-4-11), and EN61010-1.

4 CATC IBTrainer InfiniBand Exerciser, August 2002

Contents

About this Manual 9

Getting Started with the IBTrainer 11

Contents

Document History 9

Installing the Software 11
Setting Up the IBTrainer 12
Performing a First Test 12

How to Connect to the DUT 13
Controlling the Link State of the IBTrainer 14
Sending an SMP Packet 15
Evaluating the Returned Packet 17

Overview of the IBTrainer 19

Description of the Hardware 20

Installation and Maintenance 22

InfiniBand 23
Description of the Software 24
Testing Capabilities of the IBTrainer 26

Physical Layer 26
Link Layer 27

Flow Control 27
Addressing 28
Error Detection 29
Packet Length 29

Network Layer 30
Transport Layer 30
Testing Capabilities in Terms of Concrete Tests 31

CATC IBTrainer InfiniBand Exerciser, August 2002 5

Contents

Testing with the User Interface 33

Testing Principles 34
Setting Up the Test 35

Configuring Outgoing Packets to Be Transmitted 36

How to Set Up Outgoing Packets 37
Advanced Attributes 39
How to Inject Errors in Outgoing Packets 40

Using the Transmit Memory for Generating Sequences of
Packets

How to Insert Packets in the Transmit Memory 42
How to Send a Single Packet Multiple Times 42

How to Control Skew Between the Four Lanes 43

Running the Test 44

How to Trigger the Transmission from an External Signal 45

Viewing the Results 46

41

Filtering Incoming Packets 47
Using Matching Patterns for Filtering Incoming Packets 48

Graphical Representation of the Pattern Matching Process 50
Example of Pattern Matching 51

Using Packet Handlers for Managing Incoming Packets 52

How to Write Your Own Handler 53
Managing Handlers 53

How to Configure Virtual Lanes for Rerouting Incoming
Packets

54

Setting Invalid Generator Properties 55

Using Auxiliary Features 57

Visualizing the Network Topology 58
Determining the Performance of InfiniBand Traffic 60

How to Launch a Performance Measurement 62

Sending TCL Commands to the IBTrainer 62
Extending the IBTrainer Software with Your Own TCL

Scripts

63

6 CATC IBTrainer InfiniBand Exerciser, August 2002

Contents

Using the InfiniBand Compliance Test Suite 65

Overview of the Compliance Test Suite 66
Testing with the Compliance Tester User Interface 68

How to Add or Update Tests 69
How to Select Tests 70
How to Run Tests 71
How to Set Up and View the Report 72

Example of Report 73

Test Description 74

Available Tests 74
Packet with Bad DLID 75
Packet with Bad ICRC 77
Packet with GRH and VL15 78
Packet with Bad VCRC 80
Packet Receive Statemachine MARKED BAD PKT 81
Packet Receive Statemachine BAD PKT 82
Packet with Bad LVer 84
Packet with Bad PktLen 85
The SL Field Shall be Ignored by DUT When Using VL15 86
LMC Check 87
Number of VLs Supported 88
State Transitions 89
SMP Send and Receive in States "Initialize" and "Arm" 90
Ignoring Invalid Link State Settings 91
EUI-64 GUID Assigned by Manufacturer 92
Four Running Disparity Errors in a Sequence 92
Basic Directed Route SMP Response 93
Correct SMP Packet - Expect Response 94
SLID Check 94

Specifications 95

Operating Characteristics 95
Certification 96
Power Requirements 96

Glossary 97

CATC IBTrainer InfiniBand Exerciser, August 2002 7

Contents

8 CATC IBTrainer InfiniBand Exerciser, August 2002

Document History About this Manual

About this Manual

Aims This document provides the information necessary to program tests for

the CATC InfiniBand Exerciser for InfiniBand.

Target Audience The target audience is as follows:

• Device designers

Device users use the IBTrainer for debugging InfiniBand devices.

• Software engineers

Software engineers set up test environments for validating InfiniBand
devices.

Both groups should have a good knowledge of the InfiniBand
specification. In addition, software engineers should have good working
knowledge of the TCL (Tools Command Language) language or C++.

Document History

All revisions and updates of this document are listed below.

Date Document Revision Description

August 2002 1.0 Initial version of the documentation

CATC IBTrainer InfiniBand Exerciser, August 2002 9

About this Manual Document History

10 CATC IBTrainer InfiniBand Exerciser, August 2002

Installing the Software Getting Started with the IBTrainer

Getting Started with the IBTrainer

The IBTrainer is an exerciser for InfiniBand, designed as a standalone
unit.

Its primary purpose is to generate, receive, and analyze arbitrary
InfiniBand packets. It can serve as an endnode within an InfiniBand
fabric, insert errors, and monitor protocol compliance and performance
data on the InfiniBand link.

It is controlled by an external PC over a USB connection.

This chapter shows you how to install and use the IBTrainer and its
software. It also provides instructions on how to perform your first test.

Installing the Software

When you insert the delivered CD-ROM, the installation program of the
software starts automatically. If the Autorun function is disabled on your
PC, you can start the installation program by double-clicking on the
setup.exe file located at the root of the CD-ROM.

If you purchased the IBTrainer Compliance Test Suite, you should have
also received the necessary key. During installation, the installation
program prompts you to enter the key. The Compliance Test Suite
provides you with a large number of easy-to-use tests for quickly
verifying InfiniBand devices.

NOTE If you did not purchase the IBTrainer Compliance Test Suite, the

installation programs still installs the user interface of the Compliance
Test Suite with a few free tests.

CATC IBTrainer InfiniBand Exerciser, August 2002 11

Getting Started with the IBTrainer Setting Up the IBTrainer

Setting Up the IBTrainer

Plug in the IBTrainer and use the delivered USB cable to connect it to
your PC.

NOTE The IBTrainer has a USB 1.1 port. The delivered USB cable has therefore

two different ends (one for the IBTrainer, one for the PC).

Now start the software to make sure that the IBTrainer is connected to
the PC:

♦ Press

The shortcut changes with the release installed. For example, if your
release is 2.0.1.0, then the name of the shortcut is

When the software starts, it looks for attached IBTrainers. The List of
Available Generators dialog box should contain the name of your
IBTra iner. If your IBTraine r is not listed, double-check the USB
connection. Please also keep in mind that Windows NT does not support
USB.

Start>Programs>CATC> IBTrainer(Release)>GUI

.

IBTrainer2.0.1.0>GUI

Performing a First Test

Once you have connected the IBTrainer to your PC, you can:

• Connect the DUT to the IBTrainer.

Refer to “How to Connect to the DUT” on page 13 for more
information.

.

• Set up the link state of the IBTrainer

• Set up and send an SMP packet to get the NodeInfo from the DUT.
Refer to “Sending an SMP Packet” on page 15 for more information.

• Evaluate the returned packet. Refer to “Evaluating the Returned
Packet” on page 17 for more information.

12 CATC IBTrainer InfiniBand Exerciser, August 2002

Performing a First Test Getting Started with the IBTrainer

How to Connect to the DUT

After you have:

• Connected the IBTrainer to your PC

• Started the software

• Turned on the DUT

you can establish the connection between the IBTra iner and the DUT:

1 Connect the DUT to the IBTrainer with the InfiniBand cable. The

InfiniBand port is located on the front of the IBTrainer.

The List of Available Generators dialog box should be opened.

2 Connect the IBTrainer to the DUT by clicking Connect.

The Link State dialog box opens. “Controlling the Link State of the
IBTrainer” on page 14 describes this dialog box in detail.

3 Click the button corresponding to the link state that you want to

reach.

CATC IBTrainer InfiniBand Exerciser, August 2002 13

Getting Started with the IBTrainer Performing a First Test

Controlling the Link State of the IBTrainer

The status bar shows you the current link state of the IBTrainer.

If the link state does not correspond to the desired transmission packet,
you can set the link state by selecting Generator>Link State. The Link
State dialog box opens.

This dialog box consists of two groups, reflecting the conventions given
in the InfiniBand specification:

• Link Training State Machine, corresponding to the Physical layer

Link initialization, configuration, and link error recovery operations
are performed by the link training state machine. The primary states
for the link training state machine that you can set in the Link State
dialog box are Disabled, Sleep or Poll.

• Link State Machine, corresponding to the Link layer

As described in the InfiniBand specification, the link layer handles the
sending and receiving of data across the links at the packet level. The
specification uses state machines to define the logical operation of the
link layer. The different stages of the link state machine that you can
set in the Link State dialog box are Down, Arm or Active.

The InfiniBand specification requires that the current link states of two
InfiniBand devices connected together must be identical before the link
state of one of the devices can be set a level higher. Here, you can
activate AutoNegotiate to let the IBTrai ner handle the intermediate
steps for reaching the desired link state for both the IBTrainer and the
DUT, provided that no Subnet Manager is attached to the network.

14 CATC IBTrainer InfiniBand Exerciser, August 2002

Performing a First Test Getting Started with the IBTrainer

For instance, if both devices are set to Initialized, you can select
AutoNegotiate and click Active to have both devices set as Active: the
IBTrainer sets the link state from Initialized to Armed and then from
Armed to Active for both devices.

Because we intend to only send Subnet Management packets (SMP) in
the current test, we only need to set the link training of the IBTrainer to
Poll. In that case, the status of the link training state machine eventually
becomes LinkUp and the link state machine Initialized.

Sending an SMP Packet

The software is delivered with three sample packets that you can use for
your first tests:

• Sample SMP packet which is a Subnet Management Packet

• Sample IB packet which is a standard InfiniBand packet

• Sample GMP packet which is a Management datagram (MAD) packet

CATC IBTrainer InfiniBand Exerciser, August 2002 15

Getting Started with the IBTrainer Performing a First Test

For our first test, we use the Sample SMP packet. To set up and send this
packet:

1 Check that this packet is set up so that the IBTrainer queries the

NodeInfo property of the DUT by double-clicking Sample SMP packet.
The SMP Packet: Sample SMP Packet dialog box opens.

This dialog box shows the properties of the selected packet. Following
default values should be set:

– Method is set to 0x01 - Get.

– Attribute ID is set to 0x11 - NodeInfo.

If not set, change these values to the ones above, and click Ok to
accept changes and close the window.

2 Right-click on Sample SMP, and select Transmit to send the packet.

16 CATC IBTrainer InfiniBand Exerciser, August 2002

Performing a First Test Getting Started with the IBTrainer

Evaluating the Returned Packet

After you have sent the SMP packet, an entry appears in blue in the
Packet Log window. The IBTrainer then listens for a response from the
DUT.

When the DUT answers, a new entry corresponding to the returned
packet appears in red in the Packet Log window, as shown in the figure
below.

Double-click the returned packet’s entry. The SMP packet: MAD VL=15
getResp NodeInfo dialog box opens. The NodeInfo properties are then
displayed in the SMP Data list in this dialog box, as shown in the
following figure.

You can compare the received NodeInfo properties with the DUT’s
values to ensure that the packet was correctly transmitted.

CATC IBTrainer InfiniBand Exerciser, August 2002 17

Getting Started with the IBTrainer Performing a First Test

18 CATC IBTrainer InfiniBand Exerciser, August 2002

Overview of the IBTrainer

The IBTrainer is a 4x Exerciser for InfiniBand, designed as standalone
units. It is controlled by an external PC using the USB connection and
can be programmed to generate arbitrary InfiniBand packets behaving as
endpoint or endnode within an InfiniBand network.

Overview of the IBTrainer

In the following, IBTrainer stands for both generators. This chapter
describes:

• “Description of the Hardware” on page 20

• “InfiniBand” on page 23

• “Description of the Software” on page 24

• “Testing Capabilities of the IBTrainer” on page 26

CATC IBTrainer InfiniBand Exerciser, August 2002 19

Overview of the IBTrainer Description of the Hardware

Description of the Hardware

The front panel of the IBTrainer is as follows:

The rear panel of the IBTrainer is as follows:

LEDs The front panel of the IBTrainer contains the following LEDs:

LED Lights when...

Power indicator Main power is switched on.

USB Link USB link to the control PC has been established.

IB XMIT Xmit LED is lit when InfiniBand link is up; flashes

when packets are transmitted.

IB RCVE Packets are being received.

Red Error An error has occurred.

20 CATC IBTrainer InfiniBand Exerciser, August 2002

Description of the Hardware Overview of the IBTrainer

Its front panel contains the following LEDs:

LED Lights when...

Pwr Main power is switched on.

Usb USB link to the control PC has been established.

Link Lights when InfiniBand link is up.

Tx Blinks when packets are transmitted.

Rx Blinks when packets are being received.

Err An error has occurred.

Push-button and Connector Additional elements on the front panel are:

• RST push-button to reset the InfiniBand link or run built-in self-test
(press the RST button for about 5 s).

• InfiniBand connector

NOTE Here is what happens when you press the RST push-button for about 5

seconds to run the built-in self-test:

• You hear four short beeps.

• all LEDs light and then turn off.

• RX, TX and Link LEDs light

If the test passes, two short high-pitch beeps sound, and eventually the
power LED comes on, all other LEDs are off (normal power-up state).

if the test fails, error LED comes on, three longer low-pitch beeps sound,
and RX, TX and link show specific pattern. You need to contact your
CATC representative.

Rear Panel Connectors The rear panel has various connectors:

• Wide-range AC connector module

Power socket with power on/off switch

• USB port

• External input (for triggering) with following cable specifications:
LVTTL level, low active, internal termination: 50 ohm to +2.5V, level
sensitive

The associated LED indicates trigger activity.

• External output (for triggering) with following cable specifications:
LVTTL level, low active

The associated LED indicates trigger activity.

CATC IBTrainer InfiniBand Exerciser, August 2002 21

Overview of the IBTrainer Description of the Hardware

WARNING • Do not open the unit. There are no operator serviceable parts inside.

For servicing contact your CATC representative.

• To prevent the influence of ESD on measurements, all tests must be
performed in an ESD-protected environment.

Installation and Maintenance

Initial Inspection Inspect the shipping container for damage. If the container or cushioning

material is damaged, keep it until the contents of the shipment have been
checked for completeness and the instrument has been verified both
mechanically and electrically.

WARNING To avoid hazardous electric shock, do not perform electrical tests when

there are signs of shipping damage to any part of the instrument's outer
covers or panels.

If the contents are incomplete, or there is mechanical damage, or if the
instrument does not work as expected within its specifications, notify
the nearest CATC office.

Power Requirements The instrument can operate from any single-phase AC power source

supplying 100 – 240 V in the frequency range from 50 – 60 Hz. The
maximum power consumption is 100 VA with all options installed. When
the instrument is switched on the power supply adapts automatically to
the applied AC power (Auto Selection) and monitors the AC power range
during operation.

Ventilation Requirements Make sure that there is adequate clearance of 30 mm at the left and the

right side to ensure adequate air flow. If the air flow is restricted, the
internal operating temperature will be higher, reducing the instrument's
reliability.

NOTE Do not cover the ventilation holes.

Cleaning Recommendation Use a dry cloth or one slightly dampened with water to clean external

case parts. Do not attempt to clean internally.

WARNING To prevent electrical shock, disconnect the instrument from mains

before cleaning.

22 CATC IBTrainer InfiniBand Exerciser, August 2002

InfiniBand Overview of the IBTrainer

InfiniBand

InfiniBand architecture The InfiniBand Architecture (IBA) is designed around a point-to-point,

switched I/O fabric, whereby end node devices (which can range from
very inexpensive I/O devices like single chip SCSI or ethernet adapters to
very complex host computers) are interconnected by cascaded switch
devices. The physical properties of the IBA interconnect support two
predominant environments, with bandwidth, distance and cost
optimizations appropriate for these environments:

• Module-to-module, as typified by computer systems that support I/O
module add-in slots

• Chassis-to-chassis, as typified by interconnecting computers, external
storage systems, and external LAN/WAN access devices (such as
switches, hubs, and routers) in a data-center environment.

InfiniBand specification The design and implementation of the IBTrainer and its test capabilities

is based on the InfiniBand Specification InfiniBand
Specification Volume 1 and 2 Release 1.0.a from June 19, 2001.

The IBTrainer 4x (by four) Exerciser for InfiniBand uses the electrical
4x connection as set down in the InfiniBand Specification.

• The signalling rate for encoded data on the media is 2.5 Gbits/sec,
which results in a data rate that can be considered to be 250
MBytes/second. The connections are point to point and full duplex,
unidirectional.

• The byte stream on each physical lane is encoded using the industry
standard 8B/10B coding.

• The following interface width is available for the IBTrainer:

8 differential pairs, 4 per direction for a total of 16 wires

TM

Architecture

CATC IBTrainer InfiniBand Exerciser, August 2002 23

Overview of the IBTrainer Description of the Software

Description of the Software

The IBTrainer can only be controlled by a PC. This can be done by
programming TCL scripts or C++ programs using the API or by using the
Graphical User Interface (GUI).

Graphical User Interface The Graphical User Interface provides an easy way of:

• Performing simple tests on InfiniBand devices.

• Determining and displaying the traffic performance as a diagram.

• Visualizing the topology of the InfiniBand network containing the
IBTrainer as end-node.

API For more extensive testing, such as the validation of an InfiniBand

device, programs written in TCL or C++ can be used to control the
IBTra iner.

NOTE The TCL interpreter is installed with the software.

24 CATC IBTrainer InfiniBand Exerciser, August 2002

Description of the Software Overview of the IBTrainer

Optional software module The IBTrainer comes with an optional InfiniBand Compliance test

module. This module can be called directly from the GUI. It consists of a
series of prewritten InfiniBand compliance tests that you can run to
automatically verify a large number of compliance statements defined in
the InfiniBand specification, providing a pass/fail result.

Requirements The IBTrainer works with any personal computer using:

• Windows 98SE ®

• Windows 2000 ®

• Windows XP ®

• Windows NT ® only in demo/offline mode

• Windows ME ® operating systems

The software can also be used offline for demonstration or test
development purposes.

Licensing No license is necessary to run the Graphical User Interface.

For the Compliance Test software, the standard shipment includes eight
free tests. To install additional tests, you have to purchase the IBTrainer
Compliance Test Suite option. You will receive a license key that you
have to enter during installation.

CATC IBTrainer InfiniBand Exerciser, August 2002 25

Overview of the IBTrainer Testing Capabilities of the IBTrainer

Testing Capabilities of the IBTrainer

This section lists the key test capabilities of the IBTrain er. They are
structured along the following ISO/OSI layers as described in the
InfiniBand specification:

• Physical layer

• Link layer

• Network layer

• Transport layer

“Testing Capabilities in Terms of Concrete Tests” on page 31 gives a list
of the capabilities of the IBTrainer as defined by the Compliance
Interoperability Work Group (CIWG).

Physical Layer

The IBTrainer is not capable of testing the physical layer.

As described in the IB specification, the physical layer is the physical
connection between two devices. Thus, there are certain aspects of
InfiniBand connections that the IBTra iner does not test:

• Signal level, quality, and frequency

• 8B/10B encoding and decoding

The IBTrainer makes comma and skip insertions according to the
InfiniBand specification (1 comma and 3 skip symbols occur every 4352
to 4608 clocks). The number of skips cannot be influenced. The
IBTra iner generates idle packets as described in the InfiniBand
specification.

You can, however, insert running disparity errors into the data stream.
Running disparity errors can be created only within packets. This is
controlled by the packet behavior setting. Such packet behavior is
controlled by the packet property setting.

26 CATC IBTrainer InfiniBand Exerciser, August 2002

Testing Capabilities of the IBTrainer Overview of the IBTrainer

Link Layer

Definition As described in the InfiniBand specification, services provided by the

link layer include:

•Flow control

Refer to

• Addressing

Refer to

• Error detection

Refer to

•Switching

Switching is not a capability of the IBTr ainer.

The IBTrainer automatically sets up the link. Its hardware sends and
receives training sequences and link packets. The link status (link up /
link down / no signal) can be read out of the status registers.

“Flow Control” on page 27 for more information.

“Addressing” on page 28 for more information.

“Error Detection” on page 29 for more information.

NOTE A behavior is a property setting that applies to a single packet.

Virtual Lanes The IBTrainer supports all the possible virtual lanes, including VL 15,

which is used for management purposes. The particular virtual lanes that
are to be supported are user-selectable using a resource and enabling
mechanism.

Flow Control

Flow control packets are generated automatically by the IBTrainer. The
packets also pass credits back and forth within the InfiniBand network.
You can influence the credit system of the IBTrainer in several ways.

Receive Credits Credits are given per virtual lane. Each virtual lane can be configured to

behave in two different ways:

• Credits can be given out automatically, which turns the IBTrainer into
an unlimited data sink. Maximum credit is given out for any virtual
lane set to this mode. This mode is called the data sink mode.

• Credits are given out until such time that the data receive memory is
full. It is then up to the software to process each packet, and you can
instruct the software to give out credits as soon as the receive memory
is read out (until such time that the memory is full again). This mode is
called the packet control mode.

CATC IBTrainer InfiniBand Exerciser, August 2002 27

Overview of the IBTrainer Testing Capabilities of the IBTrainer

The receive memory of the IBTrainer consists of a single block of 2
Mbytes. By selecting a virtual lane resource you decide which path the
data stream is to use. Refer to “Receiving Packets” on page 31 for more
information.

Transmit Credits You can set a behavior for each packet to either ignore credits or to

behave according to the specification.

Detailed organization of the IBTrainer credit system is as follows:

• The FCCL calculation is based on a 12-bit Adjusted Blocks Received
(ABR) counter maintained for each virtual lane at the receiver. The
ABR counter is set to zero when the link initializes. This is in
accordance with the specification. Upon receipt of each flow control
packet, the ABR sets the value of the FCTBS (Flow Control Total
Blocks Sent) field.

• Upon receipt of each data packet, the ABR is incremented by the
number of blocks received, modulo 4096. The ABR does not get
incremented due to received packets that have been discarded due to
the lack of receive capacity in the receiver. You can adjust this
capacity for the IBTrainer in the data sink mode.

• Upon transmission of a flow control packet, the FCCL is set to one of
the following:

– If the current buffer state of the receiver permits reception of 2048

or more blocks from all combinations of valid InfiniBand packets
without discarding them, the FCCL is set to ABR plus 2048 modulo

4096.

– Otherwise the FCCL is set to ABR plus the number of blocks the

receiver is capable of receiving from all combinations of valid
InfiniBand packets without discarding these, modulo 4096.

Addressing

Source addresses are generated automatically depending on the address
supplied to the IBTrainer by the subnet manager. Source addresses can
be overwritten. This way the IBTraine r can emulate a switch or a router.
The IBTrainer is capable of generating all types of InfiniBand packets as
allowed by the specification. This includes any MAD packets, any regular
InfiniBand packets, Raw or RawIPv6 packets. Refer to section 5.2 in the
InfiniBand specification (data packet format) for more details.

The destination address for the packets is always defined by the user.

28 CATC IBTrainer InfiniBand Exerciser, August 2002

Testing Capabilities of the IBTrainer Overview of the IBTrainer

Error Detection

The IBTrainer calculates the ICRC (Invariant CRC) and the VCRC
(Variant CRC) automatically. You can influence this by making one of the
following choices:

• Τhe behavior ‘bad packet’ causes the IBTrainer to end the packet with
a bad packet delimiter (EBP).

• The behavior ‘bad ICRC’ corrupts the ICRC.

• The behavior ‘bad VCRC’ corrupts the VCRC.

To corrupt a packet, the IBTrainer places the 1’s complement of the
appropriate CRC calculated for the transmitted packet in the CRC field.

Packet Length

The IBTrainer distinguishes between the real packet length and the
shown packet length. The real packet length is purely an internally used
value, which can, but does not have to, correlate with the packet length
shown in the packet header. You can create an illegal packet length by
setting the shown length to a value that is at variance with the real packet
length. You can set this behavior on a per-packet basis.

CATC IBTrainer InfiniBand Exerciser, August 2002 29

Overview of the IBTrainer Testing Capabilities of the IBTrainer

Network Layer

The network and link protocols deliver a packet to the desired
destination. The transport portion of the packet delivers the packet to
the proper Queue Pair and instructs the Queue Pair how to process the
packet’s data.

The receiver part of the IBTrainer can be programmed to either discard
or keep invalid packets that arrive at the generator (wrong IP version
field, wrong DGID and so on).

Transport Layer

The IBTrainer lets you set up or program all elements of the transport
layer. With the user interface delivered with the IBTrainer or self-written
TCL scripts or C++ applications you can control the IBTrainer to
generate all types of InfiniBand traffic.

Transmitting Packets The programming of the IBTra iner is packet based (not message based).

You have two basic choices of generating packets with the IBTrainer:

• Direct packet FIFO

This FIFO can be used to generate arbitrary packets directly out of
software. The direct software call

packet)

packet buffer and sends it out immediately. This method of sending
packets has priority over the transmit memory-based packet
generation. Its purpose is to generate MAD (Management Datagram)
packets (for subnet management) or any other high priority packets
whenever this is required, without having to wait for the transmit
memory to finish sending packets and get reprogrammed with the
next sequence.

• Memory-based packet generation

You can fill a block memory with up to 1024 InfiniBand packets which
are then executed in a row. There are additional properties that you
can set, such as the inter-packet delay. All behaviors are listed under

“Using the Transmit Memory for Generating Sequences of Packets”
on page 41. The transmit memory can be looped and offers a feature

to repeatedly send single packets from the programmed packet
sequence. Additionally the transmit memory can increase the packet
sequence number automatically to generate continuous streams of
packets (and can include very large data payloads).

; from the

IGCGenerator

void PacketSend (IGCPacket &

class passes a packet down to this

30 CATC IBTrainer InfiniBand Exerciser, August 2002