Dell 0J05YT, 0K6V3V User Manual

Download

Page 1

Dell and the Dell logo are trademarks of Dell Inc.

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s

Mezzanine Card for Dell User Manual

0J05YT, 0K6V3V

Rev 1.0

www.mellanox.com

Page 2

NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUS TOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CAN NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER CHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEM PLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale, CA 94085 U.S.A. www.mellanox.com Tel: (408) 970-3400

Mellanox Technologies, Ltd. PO Box 586 Hermon Building Yokneam 20692 Israel Tel: +972-4-909-7200 Fax: +972-4-959-3245

Fax: (408) 970-3403

© Copyright 2012. Mellanox Technologies. All rights reserved. Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniRISC®, InfiniScale®, InfiniPCI®, PhyX®, COREDirect®, Virtual Protocol Interconnect®, Voltaire® and SwitchX® are registered trademarks of Mellanox Technologies, Ltd. FabricIT™, MLNX-OS™ and Unbreakable-Link™ are trademarks of Mellanox Technologies, Ltd. All other marks and names mentioned herein may be trademarks of their respective companies.

Mellanox Technologies

Document Number: 3526

Page 3

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Chapter 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.1 Cards Covered in this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.2 Finding the GUID/MAC and Serial Number on the Mezzanine Cards . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Chapter 2 Card Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1 I/O Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.1 InfiniBand Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.2 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.1.3 I2C-compatible Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.2 Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.1 System Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.2 SPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.3 EEPROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Chapter 3 Card Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .TBD 18

3.2 Installation Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3 Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Chapter 4 Driver Software and Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1 Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1.1 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1.2 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.2 FlexBoot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.3 Updating Mezzanine Card Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Appendix A Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

AppendixB Avertissements de sécurité d’installation (French). . . . . . . . . . . . . . . . . . . . . . . . . . 31

AppendixC Installation - Sicherheitshinweise (German). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

AppendixD Advertencias de seguridad para la instalación (Warnings in Spanish) . . . . . . . . . 33

Mellanox Technologies

Page 4

Rev 1.0

List of Tables

Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Table 2: Documents List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Table 3: ConnectX-3 Mezzanine Card Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Table 4: Jumper Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Table 5: 0J05YT VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Table 6: 0K6V3V VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

Table 7: Jumper Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Table 8: ConnectX-3 0J05YT/0K6V3V Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Mellanox Technologies

Page 5

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

List of Figures

Figure 1: 0J05YT Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 2: Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 3: 0K6V3V Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 4: Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 5: 0J05YT/0K6V3V Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Figure 6: I2C Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12

Figure 7: Grounding Clip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Figure 8: Blade Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Figure 9: Protective Cover in Place . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

Figure 10: Remove the Cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Figure 11: Press on the Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Figure 12: Single Card in Slot C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Figure 13: Two Cards Installed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Figure 14: Device Manager (example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

Figure 15: Support Download Assistant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28

Figure 16: Mechanical Drawing of 0J05YT/0K6V3V Mezzanine Card . . . . . . . . . . . . . . . . . . . . . . 29

Mellanox Technologies

Page 6

Rev 1.0

Revision History

This document was first printed on February 14, 2012

Table 1 - Revision History Table

Date Rev Comments/Changes

February

2012

1.0 Initial Release

Mellanox Technologies

Page 7

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

About this Manual

This User Manual describes ConnectX®-3 Dual QSFP Port mezzanine card for the Dell Blade Servers. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the cards, installation instructions, and relevant documentation.

Intended Audience

This manual is intended for the installer and user of the mezzanine cards.

The manual assumes basic familiarity with the InfiniBand tions.

Online Resources

• Mellanox Technologies Web pages: http://www.mellanox.com

• Dell Support Web pages: http://support.dell.com

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega bytes. The use of Mb or Mbits (small b) indicates size in mega bits.

Specifications

These symbols indicate a situation, status, or condition that may cause harm to people or damage to the equipment.

Mellanox Technologies

Page 8

Rev 1.0

Mellanox Technologies

Page 9

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

1 Overview

This document is a User Manual for the Mellanox Technologies ConnectX®-3 40Gb/s and 56Gb/ s InfiniBand dual-port mezzanine cards for the Dell Blade Servers. The cards described in this manual have the following main features:

• InfiniBand Architecture Specifications v1.2.1 compliant

• PCI Express 3.0 (1.1 and 2.0 compatible) through an x8 connector up to 8GT/s

•1s MPI ping latency

• Up to 56Gb/s InfiniBand per port

• CPU off-load of transport operations

• Application Offload

• End-to-end QoS and congestion control

• Hardware-based I/O virtualization

• TCP/UDP/IP stateless offload

• RoHS-R6 compliant

1.1 Cards Covered in this Manual

Table 3 - ConnectX-3 Mezzanine Card Details

Ordering Part

Number (OPN)

0J05YT PCIe 3.0 x8 8GT/s / InfiniBand 40Gb/s QDR R-6 MT27508A1-FCCR-QV

0K6V3V PCIe 3.0 x8 8GT/s / InfiniBand 56Gb/s FDR R-6 MT27508A1-FCCR-FV

PCI Express SERDES

Speed

InfiniBand Data

Transmission Rate

RoHS Adapter IC Part Number

1.2 Finding the GUID/MAC and Serial Number on the Mezzanine Cards

All cards include a label that has the card serial number and the card GUID for InfiniBand protocol on the print side of the card.

Mellanox Technologies

Page 10

Figure 1: 0J05YT Label

Figure 2: Board Label

OverviewRev 1.0

Figure 3: 0K6V3V Label

Figure 4: Board Label

Mellanox Technologies

Page 11

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

2 Card Interfaces

2.1 I/O Interfaces

Each mezzanine card includes the following interfaces:

• PCI Express (PCIe)

•I2C-compatible connector (for debug)

Figure 5: 0J05YT/0K6V3V Card

2.1.1 InfiniBand Interface

The ConnectX®-3 mezzanine cards are compliant with the InfiniBand Architecture Specification, Release 1.2.1. It has two compliant InfiniBand ports with four Tx/Rx pairs of SerDes.

Mellanox Technologies

Page 12

2.1.2 PCI Express Interface

The ConnectX®-3 mezzanine cards support PCI Express 3.0 (1.1 and 2.0 compatible) . The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations.

2.1.3 I2C-compatible Interface

A three-pin header on the mezzanine cards is provided as the I2C-compatible interface.

Card InterfacesRev 1.0

Figure 6: I2C Connector

2.2 Power

All adapter cards receive 12V and 3.3V power from the PCI Express connector. All other required power voltages are generated by on-board switch mode regulators. See Appendix A.2,“Specifications,” on page 30.

2.3 Memory

The mezzanine cards support multiple memory devices through the PCI Express, SPI (Flash), and

C-compatible interfaces.

2.3.1 System Memory

The mezzanine cards utilize the PCI Express interface to store and access fabric connection information on the system memory.

2.3.2 SPI

The mezzanine cards include one SPI Flash device accessible via the Flash interface of the ConnectX®-3 device.

Mellanox Technologies

Page 13

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

A jumper inserted into the drill on each adapter card indicates to the device whether an on-board Flash device is to be used or not. Table 4 provides information on this jumper.

Table 4 - Jumper Configuration

Description Option

Flash present/

not present

2.3.3 EEPROM

The mezzanine cards incorporate an EEPROM that is accessible through the I2C-compatible interface. The EEPROM capacity is 4Kb. The EEPROM is used for storing the Vital Product Data (VPD) and FRU. plies with the format defined in the PCI 3.0 Specification,

Card Default

Configuration

connection open – Flash present connection shorted – Flash not present

connection open –

Flash present

The PCI VPD (Vital Product Data) layout for each of the mezzanine cards com-

Appendix I.

Table 5 - 0J05YT VPD Layout

Offset

(Decimal)

0 Large Resource Type ID

1 Length [7:0] LSB 0x14

2 Length [15:8] MSB 0x0

3 Data DELL ConnectX-3

23 Large Resource Type VPD-R

24 Length [7:0] LSB 0x51

25 Length [15:8] MSB 0x00

26 VPD Keyword PN STR Add in Card Part Number

28 Length 0x15

29 Part Number PN %STR_SPC

50 VPD Keyword EC STR Engineering Change Level of

52 Length 0x3

53 Revision RV %STR PCB revision

56 VPD Keyword SN STR Serial Number

58 Length 0x14

Item Value Format Description

0x82

String Tag (0x02)

STR

Mezz

0x90

Tag (0x10)

the card (rev)

Mellanox Technologies

Page 14

Table 5 - 0J05YT VPD Layout

Card InterfacesRev 1.0

Offset

(Decimal)

59 SerialNumber OO0J05YTMMM

79 VPD Keyword V0 STR Misc Information

81 Length 0x15

82 Data VPI QDR SFF

103 VPD Keyword RV STR

105 Length 0x1

106 Data 0,105 %CS0

107 Large Resource Type VPD-W

108 Length [7:0] LSB 0x91

109 Length [15:8] MSB 0xF

110 VPD Keyword V1 STR EFI Driver version

112 Length 0x6

113 Data N/A STR_SPC

119 VPD Keyword YA STR Asset Tag

121 Length 0x18

122 Data N/A STR_SPC “N/A”

146 VPD Keyword RW STR Remaining read/write area

148 Length 0x6b

149 Data STR_ZERO Reserved (0x00)

256 VPD Keyword RW STR Remaining read/write area

258 Length 0xfd

259 Data STR_ZERO Reserved (0x00)

512 VPD Keyword RW STR Remaining read/write area

514 Length 0xfd

515 Data STR_ZERO Reserved (0x00)

768 VPD Keyword RW STR Remaining read/write area

770 Length 0xfd

771 Data STR_ZERO Reserved (0x00)

1024 VPD Keyword RW STR Remaining read/write area

1026 Length 0xfd

1027 Data STR_ZERO Reserved (0x00)

1280 VPD Keyword RW STR Remaining read/write area

1282 Length 0xfd

1283 Data STR_ZERO Reserved (0x00)

1536 VPD Keyword RW STR Remaining read/write area

1538 Length 0xfd

1539 Data STR_ZERO Reserved (0x00)

1792 VPD Keyword RW STR Remaining read/write area

1794 Length 0xfd

1795 Data STR_ZERO Reserved (0x00)

2048 VPD Keyword RW STR Remaining read/write area

Item Value Format Description

%STR_SPC “00..00XXXX..XX”

MMYMDSSSS

STR_SPC

mezz

0x91

Tag (0x11)

Mellanox Technologies

Page 15

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Table 5 - 0J05YT VPD Layout

Offset

(Decimal)

2050 Length 0xfd

2051 Data STR_ZERO Reserved (0x00)

2304 VPD Keyword RW STR Remaining read/write area

2306 Length 0xfd

2307 Data STR_ZERO Reserved (0x00)

2560 VPD Keyword RW STR Remaining read/write area

2562 Length 0xfd

2563 Data STR_ZERO Reserved (0x00)

2816 VPD Keyword RW STR Remaining read/write area

2818 Length 0xfd

2819 Data STR_ZERO Reserved (0x00)

3072 VPD Keyword RW STR Remaining read/write area

3074 Length 0xfd

3075 Data STR_ZERO Reserved (0x00)

3328 VPD Keyword RW STR Remaining read/write area

3330 Length 0xfd

3331 Data STR_ZERO Reserved (0x00)

3584 VPD Keyword RW STR Remaining read/write area

3586 Length 0xfd

3587 Data STR_ZERO Reserved (0x00)

3840 VPD Keyword RW STR Remaining read/write area

3842 Length 0xfc

3843 Data STR_ZERO Reserved (0x00)

4095 Small Resource Type END

Item Value Format Description

0x78

Tag (0x11)

Table 6 - 0K6V3V VPD Layout

Offset

(Decimal)

0 Large Resource Type ID

1 Length [7:0] LSB 0x14

2 Length [15:8] MSB 0x0

3 Data DELL ConnectX-3

23 Large Resource Type VPD-R

24 Length [7:0] LSB 0x4D

25 Length [15:8] MSB 0x00

26 VPD Keyword PN STR Add in Card Part Number

28 Length 0x15

29 Part Number PN %STR_SPC

50 VPD Keyword EC STR Engineering Change Level of

52 Length 0x3

Item Value Format Description

0x82

String Tag (0x02)

STR

Mezz

0x90

Tag (0x10)

the card (rev)

Mellanox Technologies

Page 16

Table 6 - 0K6V3V VPD Layout

Card InterfacesRev 1.0

Offset

(Decimal)

53 Revision RV %STR PCB revision

56 VPD Keyword SN STR Serial Number

58 Length 0x14

59 SerialNumber OO0K6V3VMMM

79 VPD Keyword V0 STR Misc Information

81 Length 0x11

82 Data VPI FDR SFF

99 VPD Keyword RV STR

101 Length 0x1

102 Data 0,101 %CS0

103 Large Resource Type VPD-W

104 Length [7:0] LSB 0x95

105 Length [15:8] MSB 0xF

106 VPD Keyword V1 STR EFI Driver version

108 Length 0x6

109 Data N/A STR_SPC

115 VPD Keyword YA STR Asset Tag

117 Length 0x18

118 Data N/A STR_SPC “N/A”

142 VPD Keyword RW STR Remaining read/write area

144 Length 0x6f

145 Data STR_ZERO Reserved (0x00)

256 VPD Keyword RW STR Remaining read/write area

258 Length 0xfd

259 Data STR_ZERO Reserved (0x00)

512 VPD Keyword RW STR Remaining read/write area

514 Length 0xfd

515 Data STR_ZERO Reserved (0x00)

768 VPD Keyword RW STR Remaining read/write area

770 Length 0xfd

771 Data STR_ZERO Reserved (0x00)

1024 VPD Keyword RW STR Remaining read/write area

1026 Length 0xfd

1027 Data STR_ZERO Reserved (0x00)

1280 VPD Keyword RW STR Remaining read/write area

1282 Length 0xfd

1283 Data STR_ZERO Reserved (0x00)

1536 VPD Keyword RW STR Remaining read/write area

1538 Length 0xfd

1539 Data STR_ZERO Reserved (0x00)

1792 VPD Keyword RW STR Remaining read/write area

Item Value Format Description

%STR_SPC “00..00XXXX..XX”

MMYMDSSSS

STR_SPC

mezz

0x91

Tag (0x11)

Mellanox Technologies

Page 17

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Table 6 - 0K6V3V VPD Layout

Offset

(Decimal)

1794 Length 0xfd

1795 Data STR_ZERO Reserved (0x00)

2048 VPD Keyword RW STR Remaining read/write area

2050 Length 0xfd

2051 Data STR_ZERO Reserved (0x00)

2304 VPD Keyword RW STR Remaining read/write area

2306 Length 0xfd

2307 Data STR_ZERO Reserved (0x00)

2560 VPD Keyword RW STR Remaining read/write area

2562 Length 0xfd

2563 Data STR_ZERO Reserved (0x00)

2816 VPD Keyword RW STR Remaining read/write area

2818 Length 0xfd

2819 Data STR_ZERO Reserved (0x00)

3072 VPD Keyword RW STR Remaining read/write area

3074 Length 0xfd

3075 Data STR_ZERO Reserved (0x00)

3328 VPD Keyword RW STR Remaining read/write area

3330 Length 0xfd

3331 Data STR_ZERO Reserved (0x00)

3584 VPD Keyword RW STR Remaining read/write area

3586 Length 0xfd

3587 Data STR_ZERO Reserved (0x00)

3840 VPD Keyword RW STR Remaining read/write area

3842 Length 0xfc

3843 Data STR_ZERO Reserved (0x00)

4095 Small Resource Type END

Item Value Format Description

0x78

Tag (0x11)

Mellanox Technologies

Page 18

Card InstallationRev 1.0

3 Card Installation

3.1 Hardware RequirementsTBD

Before installing the mezzanine card, make sure that the system meets the hardware and software requirements listed in

Table 7 - Jumper Configuration

Requirement Description

Hardware Used with Dell Blade Servers

Software Operating Sys-

tems/Distributions

3.2 Installation Kit

Make sure all of the parts are in the kit before you start the installation. If any parts are damaged or missing, call your supplier immediately.

Installation Instructions

Refer to the Dell Blade Servers Manuals

Installation of this mezzanine card should only be done by a properly qualified technician or engineer. Installation or service not authorized by Dell or performed by unqualified personnel may void guarantees and warranties. Read and follow all safety precautions specified in this document and in the PowerEdge documentation.

Mellanox Technologies

Page 19

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Use the documentation supplied with the Dell Blade Servers to remove and replace the blade from the chassis.

The card shown below has grounding clips that must come in contact with the chassis after the card is installed.

Figure 7: Grounding Clip

Make sure that you are properly grounded. Make sure that the equipment, both while in the chassis and while removed from the chassis, is properly grounded to prevent ESD.

Figure 8: Blade Server

DIMMS

Mellanox Technologies

Page 20

DIMMS can be hot. Allow sufficient time for the blade components to cool before starting this procedure.

1. Remove the blade from the chassis.

2. Remove the cover from the blade.

3. Open the retaining clip.

The card can be placed in either of the two available slots.

4. Expose the socket to be used for the new card.

a. When replacing an existing card, remove the card from the socket. Grab the card on the edge

on the side with UPC number and pull up while gently rocking the card back and forth.

For a new installation remove the protective cover enclosing the socket for the card.

Figure 9: Protective Cover in Place

Card InstallationRev 1.0

Mellanox Technologies

Page 21

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Figure 10: Remove the Cover

Hold the mezzanine card by the edges only.

Line up the mezzanine card so that the pins of the mezzanine card are over the sockets in the blade server.

5. Plug the card into the socket by placing your thumb over the UPC symbol and pressing down until the card reaches the bottom.

Figure 11: Press on the Card

6. Close the retaining clip.

Mellanox Technologies

Page 22

Figure 12: Single Card in Slot C

Figure 13: Two Cards Installed

Card InstallationRev 1.0

7. Replace the blade server cover.

8. Replace the blade server into the chassis

Mellanox Technologies

Page 23

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

3.3 Safety Warnings

1. Installation Instructions

Read all installation instructions before connecting the equipment to the power source.

2. Over Temperature

This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: °C (°F).

To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation

openings.

3. Lightening – Electrical Hazard

During periods of lightning activity, do not work on the equipment.

4. Installation of Equipment

This equipment should be installed, replaced, or serviced only by trained and qualified personnel.

5. Disposal of Equipment

Disposal of this equipment should be in accordance to all national laws and regulations.

6. Compliance with Local and National Codes

This equipment should be installed in compliance with local and national electrical codes.

Mellanox Technologies

Page 24

4 Driver Software and Firmware

4.1 Driver Software

4.1.1 Linux

For Linux, download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the Mellanox web site at: http://www.mellanox.com => Products => Soft- ware/Drivers => Infiniband & VPI Software/Drivers => Mellanox OFED => Download. Follow the installation instructions included in the download package (also available from the download

To ensure that communication has been established follow the instructions below.

page).

Check the link status

To check the IB link status, for IB cards, run “ibstat” and focus on the Physical state attributes.

Driver Software and FirmwareRev 1.0

Example:

> Host# ibstat

LCA 'mlx4_0'

CA type: MT4099

Number of ports: 2

Firmware version: 2.10.2000

Hardware version: 0

Node GUID: 0x0002c903002fefe0

System image GUID: 0x0002c903002fefe3

Port 1:

State: Active

Physical state: LinkUp

Rate: 56

Base lid: 4

LMC: 0

SM lid: 12

Capability mask: 0x02514868

Port GUID: 0x0002c903002fefe1

Link layer: InfiniBand

Port 2:

State: Active

Physical state: LinkUp

Mellanox Technologies

Page 25

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Rate: 56

Base lid: 8

LMC: 0

SM lid: 12

Capability mask: 0x02514868

Port GUID: 0x0002c903002fefe2

Link layer: InfiniBand

Check the OFED version

To get the version of the running Mellanox OFED/BXOFED, run the following command:

# ofed_info |head -1

MLNX_OFED_LINUX-1.5.2-2.0.0 (OFED-1.5.2-20101219-1546):

Troubleshooting MLNX_OFED Installation

For troubleshooting driver installation, please check Mellanox OFED driver user manual ar

http://www.mellanox.com => Support => InfiniBand Products => Mellanox OFED.

4.1.2 Windows

For Windows, download and install the latest Mellanox WinOF VPI for Windows software package available via the Mellanox web site at: http://www Drivers => Infiniband & VPI Software/Drivers => Mellanox WinOF VPI => Download. Follow the installation instructions included in the download package (also available from the download page). To ensure that communication has been established follow the instructions below.

Displaying the Device Manager will show the Mellanox adapter devices and an IPoIB (network) device for each port.

MLNX_OFED_LINUX-1.5.3-0.1.6 (OFED-1.5.3-0.1.6):

.mellanox.com => Products => Software/

Mellanox Technologies

Page 26

Driver Software and FirmwareRev 1.0

Figure 14: Device Manager (example)

Note: If the cards are connected to a managed switch, there is no need to run openSM. Only

one OpenSM should run per subnet.In InfiniBand interfaces, OpenSM is installed as a disabled Windows service. To enable it, enter at the command line:

> sc start opensm

4.2 FlexBoot

FlexBoot enables remote boot over Boot over InfiniBand (BoIB), or Boot over iSCSI (Bo-iSCSI). This technology is based on the Preboot Execution Environment (PXE) standard specification, and FlexBoot software is based on the open source EtherBoot/gPXE project (see www.etherboot.org). For more information go to http://www.mellanox.com => Products => InfiniBand & VPI Soft- ware/Drivers => FlexBoot => User’s Manual.

4.3 Updating Mezzanine Card Firmware

Each card is shipped with the latest version of qualified firmware at the time of manufacturing. Firmware is updated occasionally, and the most recent firmware can be obtained from:

http://www.mellanox.com => Support => Download Firmware. Check that the firmware on your

card is the latest found on the Mellanox site, if not update to the latest version found on the Mellanox web site.

Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firm- ware Tools (MFT) package. from the Mellanox Firmware Tools page. See http://www.mellanox.com => Support => Download Firmware Tools.

This package is available for download, along with its user manual,

A firmware binaries table lists a binary file per mezzanine card. The file name of each such binary is composed by combining the firmware name, the firmware release version, and the card part

Mellanox Technologies

Page 27

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

number. Please contact Mellanox System Support if you cannot find the firmware binary for your mezzanine card.

The following steps describe how to retrieve the PSID (firmware identification) and programmed firmware version of your mezzanine card. They also describe how to update the card with the lat

est firmware version available.

1. Retrieve the PSID and firmware version:

a. Install the MFT package. The package is available at www.mellanox.com => Products =>

Software/Drivers => InfiniBand & VPI Software/Drivers => Firmware Tools. Make sure to download the package corresponding to your computer’s operating system.

b. Enter: mst start

c. Get the Mellanox mst device name using the command "mst status". The mst device name will be of

the form: /dev/mst/mt4099_pci_cr0

d. Get the PSID (firmware identification) and programmed firmware version using the command

> flint -d /dev/mst/mt4099_pci_cr0 q Image type: ConnectX FW Version: 2.9.4000 Device ID: 4099 Chip Revision: 0 Description: Node Port1 Port2 Sys image GUIDs: 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs: 000002c90200 000002c90201 Board ID: (MT_1020110019) VSD: PSID: MT_1020110019

2. Compare the programmed firmware version with the latest available.

a. Go to Mellanox’s web site: http://www.mellanox.com/supportdownloader. See Figure 15

b. Enter your card PSID to display the latest firmware

Mellanox Technologies

Page 28

Figure 15: Support Download Assistant

Driver Software and FirmwareRev 1.0

3. If a newer firmware version exists for your mezzanine card, update the firmware as follows:

a. Download the firmware image zip file from the Download Center (see Step 2a above)

Unzip the firmware image

c. Burn the firmware image. Enter:

> flint -d /dev/mst/mt4099_pci_cr0 -i <binary image> burn

d. Reboot the computer

Enter: mst start

f. Verify that the card firmware was updated successfully

> flint -d /dev/mst/mt4099_pci_cr0 q Image type: ConnectX FW Version: 2.9.4100 Device ID: 4099 ...

Mellanox Technologies

Page 29

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Appendix A: Specifications

A.1 Board Mechanical Drawing and Dimensions

The ConnectX®-3 mezzanine card mechanical drawing is depicted in Figure 16.

All dimensions are in millimeters.

All the mechanical tolerances are +/- 0.1mm

Figure 16: Mechanical Drawing of 0J05YT/0K6V3V Mezzanine Card

23.92

57.08

64.08

72.36

78.91

Mellanox Technologies

Page 30

Rev 1.0

A.2 Specifications

Table 8 - ConnectX-3 0J05YT/0K6V3V Specifications Table

Physical

Protocol Support

Power and

Environmental

Regulatory

Size: 3.11in. x3.46 in. (78.91mm X 88mm)

InfiniBand:

QDR (10Gb/s per lane) or FDR (14.0625Gb/s per lane) port

Data Rate:

PCI Express Gen3: 3.0 SERDES @ 8.0 GT/s (1.1 and 2.0 compatible)

Voltage: 12V, 3.3V, 3.3VAUX

0J05YT Typ Power: 5.75W

0K6V3V Typ Power: 8.84W

Temperature: 0°C to 55°C

Air Flow: 200LFM

EMC: Refer to the following link:

www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.pdf

Safety: IEC/EN 60950-1:2006

IBTA v1.2.1

Up to 56Gb/s FDR– InfiniBand

Max Power: 7.56W

Max Power: 9.02W

ETSI EN 300 019-2-2 IEC 60068-2- 64, 29, 32

RoHS: RoHS-R6

Cable Support

Please refer to "Mellanox Products Approved Cable Lists" at www.mellanox.com/related-

docs/user_manuals/Mellanox_approved_cables.pdf

a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet.

Mellanox Technologies

Page 31

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Appendix B: Avertissements de sécurité d’installation

(French)

1. Instructions d’installation

Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique.

2. Température excessive

Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 55°C (131°F). Un flux d’air de 200LFM à cette température ambiante maximale est nécessaire. En outre, pour garantir un bon écoulement de l’air libre autour des ouvertures de ventilation.

3. Orages – dangers électriques

Pendant un orage, il ne faut pas utiliser le matériel.

4. Installation du matériel

, laissez au moins 8 cm (3 pouces) d’espace

Ce matériel ne doit être installé, remplacé ou entretenu que par du personnel formé et qualifié.

5. Elimination du matériel

L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur.

6. Codes électriques locaux et nationaux

Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux.

Mellanox Technologies

Page 32

Rev 1.0

Appendix C: Installation - Sicherheitshinweise (German)

1. Installationsanleitungen

Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen.

2. Übertemperatur

Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Außerdem sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen einwandfreien Luftstrom zu gewährleisten.

3. Bei Gewitter - Elektrische Gefahr

Arbeiten Sie während eines Gewitters und Blitzschlag nicht am Gerät.

4. Geräteinstallation

Diese Gerät sollte nur von geschultem und qualifiziertem Personal installiert, ausgetauscht oder gewartet werden.

5. Geräteentsorgung

Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen.

6. Regionale und nationale elektrische Bestimmungen

Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden.

Mellanox Technologies

Page 33

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Appendix D: Advertencias de seguridad para la instalación

(Warnings in Spanish)

1. Instrucciones de instalación

Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de instalación.

2. Sobrecalentamiento

No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 55°C(131°F). Además, para garantizar una circulación de aire adecuada, se debe dejar como mínimo un espacio de 8 cm (3 pulgadas) alrededor de las aberturas de ventilación.

3. Cuando hay rayos: peligro de descarga eléctrica

No utilizar el equipo ni conectar o desconectar cables durante períodos de actividad de rayos.

4. Instalación de equipos

La instalación, el reemplazo y el mantenimiento de este equipo estarán a cargo únicamente de personal capacitado y competente.

5. Eliminación de equipos

La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales.

6. Códigos eléctricos locales y nacionales

Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales.

Mellanox Technologies