Dell 0J05YT, 0K6V3V User Manual

Dell and the Dell logo are trademarks of Dell Inc.

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s

Mezzanine Card for Dell User Manual

0J05YT, 0K6V3V

Rev 1.0

www.mellanox.com

NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUS TOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CAN NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER CHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEM PLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale, CA 94085 U.S.A. www.mellanox.com Tel: (408) 970-3400

Mellanox Technologies, Ltd. PO Box 586 Hermon Building Yokneam 20692 Israel Tel: +972-4-909-7200 Fax: +972-4-959-3245

Fax: (408) 970-3403

© Copyright 2012. Mellanox Technologies. All rights reserved. Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniRISC®, InfiniScale®, InfiniPCI®, PhyX®, COREDirect®, Virtual Protocol Interconnect®, Voltaire® and SwitchX® are registered trademarks of Mellanox Technologies, Ltd. FabricIT™, MLNX-OS™ and Unbreakable-Link™ are trademarks of Mellanox Technologies, Ltd. All other marks and names mentioned herein may be trademarks of their respective companies.

Mellanox Technologies

Document Number: 3526

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Chapter 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.1 Cards Covered in this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.2 Finding the GUID/MAC and Serial Number on the Mezzanine Cards . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Chapter 2 Card Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1 I/O Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.1 InfiniBand Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.2 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.1.3 I2C-compatible Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.2 Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.1 System Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.2 SPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.3 EEPROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Chapter 3 Card Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .TBD 18

3.2 Installation Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3 Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Chapter 4 Driver Software and Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1 Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1.1 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1.2 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.2 FlexBoot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.3 Updating Mezzanine Card Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Appendix A Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

AppendixB Avertissements de sécurité d’installation (French). . . . . . . . . . . . . . . . . . . . . . . . . . 31

AppendixC Installation - Sicherheitshinweise (German). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

AppendixD Advertencias de seguridad para la instalación (Warnings in Spanish) . . . . . . . . . 33

Mellanox Technologies

Rev 1.0

List of Tables

Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Table 2: Documents List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Table 3: ConnectX-3 Mezzanine Card Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Table 4: Jumper Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Table 5: 0J05YT VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Table 6: 0K6V3V VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

Table 7: Jumper Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Table 8: ConnectX-3 0J05YT/0K6V3V Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Mellanox Technologies

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

List of Figures

Figure 1: 0J05YT Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 2: Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 3: 0K6V3V Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 4: Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 5: 0J05YT/0K6V3V Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Figure 6: I2C Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12

Figure 7: Grounding Clip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Figure 8: Blade Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Figure 9: Protective Cover in Place . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

Figure 10: Remove the Cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Figure 11: Press on the Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Figure 12: Single Card in Slot C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Figure 13: Two Cards Installed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Figure 14: Device Manager (example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

Figure 15: Support Download Assistant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28

Figure 16: Mechanical Drawing of 0J05YT/0K6V3V Mezzanine Card . . . . . . . . . . . . . . . . . . . . . . 29

Mellanox Technologies

Rev 1.0

Revision History

This document was first printed on February 14, 2012

Table 1 - Revision History Table

Date Rev Comments/Changes

February

2012

1.0 Initial Release

Mellanox Technologies

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

About this Manual

This User Manual describes ConnectX®-3 Dual QSFP Port mezzanine card for the Dell Blade Servers. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the cards, installation instructions, and relevant documentation.

Intended Audience

This manual is intended for the installer and user of the mezzanine cards.

The manual assumes basic familiarity with the InfiniBand tions.

Online Resources

• Mellanox Technologies Web pages: http://www.mellanox.com

• Dell Support Web pages: http://support.dell.com

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega bytes. The use of Mb or Mbits (small b) indicates size in mega bits.

Specifications

These symbols indicate a situation, status, or condition that may cause harm to people or damage to the equipment.

Mellanox Technologies

Rev 1.0

Mellanox Technologies

ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0

1 Overview

This document is a User Manual for the Mellanox Technologies ConnectX®-3 40Gb/s and 56Gb/ s InfiniBand dual-port mezzanine cards for the Dell Blade Servers. The cards described in this manual have the following main features:

• InfiniBand Architecture Specifications v1.2.1 compliant

• PCI Express 3.0 (1.1 and 2.0 compatible) through an x8 connector up to 8GT/s

•1s MPI ping latency

• Up to 56Gb/s InfiniBand per port

• CPU off-load of transport operations

• Application Offload

• End-to-end QoS and congestion control

• Hardware-based I/O virtualization

• TCP/UDP/IP stateless offload

• RoHS-R6 compliant

1.1 Cards Covered in this Manual

Table 3 - ConnectX-3 Mezzanine Card Details

Ordering Part

Number (OPN)

0J05YT PCIe 3.0 x8 8GT/s / InfiniBand 40Gb/s QDR R-6 MT27508A1-FCCR-QV

0K6V3V PCIe 3.0 x8 8GT/s / InfiniBand 56Gb/s FDR R-6 MT27508A1-FCCR-FV

PCI Express SERDES

Speed

InfiniBand Data

Transmission Rate

RoHS Adapter IC Part Number

1.2 Finding the GUID/MAC and Serial Number on the Mezzanine Cards

All cards include a label that has the card serial number and the card GUID for InfiniBand protocol on the print side of the card.

Mellanox Technologies

Figure 1: 0J05YT Label

Figure 2: Board Label

OverviewRev 1.0

Figure 3: 0K6V3V Label

Figure 4: Board Label

Mellanox Technologies

+ 23 hidden pages