Dell M2401G User Manual
Third party information brought
to you courtesy of Dell.
InfiniScale® III 16+8 Port 20Gb/s InfiniBand
Switch for Dell PowerEdge M1000e-series
Blade Enclosures User Manual
PN: M2401G
Rev 1.7
Mellanox Technologies
2
Copyright 2008. Mellanox Technologies, Inc. All Rights Reserved.
Mellanox Technologies ConnectX, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale,
and InfiniPCI are registered trademarks of Mellanox Technologies, Ltd. Virtual Protocol Intercon
nect is a trademark of Mellanox Technologies, Ltd. Dell and the Dell logo are trademarks of Dell
Inc. All other marks and names mentioned herein may be trademarks of their respective companies.
Third party information brought to you courtesy of Dell.
InfiniScale® III 16+8 Port 20Gb/s InfiniBand S witch for Dell PowerEdge M1000e-series Blade
Enclosures User Manual
Document Number: 2948
Mellanox Technologies, Inc.
350 Oakmead Parkway
Sunnyvale, CA 94086
U.S.A.
www.Mellanox.com
Tel: (408) 970-3400
Fax: (408) 970-3403
Mellanox Technologies Ltd
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: +972-4-909-7200
Fax: +972-4-959-3245
NOTE:
THIS INFORMATION IS PROVIDED BY MELLANOX FOR INFORMATIONAL PURPOSES ONLY AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
MELLANOX BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSE
QUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLI
GENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS HARDWARE, EVEN IF ADVISE D
OF THE POSSIBILITY OF SUCH DAMAGE.
-
-
Rev 1.7 Mellanox Technologies
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
Contents
Contents 3
About this Manual 4
Revision History 5
Chapter 1 Overview 6
1.1 InfiniBand Connectors 7
1.2 Switch Status Lights 7
Chapter 2 Installation and Basic Operation 9
2.1 Installation Safety Warnings 10
2.2 Mechanical Installation 11
2.2.1 Removing an Old Switch From the Chassis 11
2.2.2 Removing a Blank Cover Plate From the Chassis 12
2.2.3 Installing the Mellanox M2401G Switch Into the Dell Chassis 12
2.3 Power Connections and Initial Power On 12
2.4 InfiniBand Cable Installation 13
Chapter 3 Cluster Management and Firmware 14
3.1 Network Management and Clustering Software 14
3.2 Updating Firmware 14
3.2.1 Instructions for Reprogramming Over the InfiniBand Network 14
Chapter 4 Troubleshooting 17
3
Appendix A Specifications 18
Appendix B Mechanical Drawing 19
Appendix C EMC Certification Statements 20
Appendix D InfiniBand Connector Pinout 22
Appendix E Avertissements de sécurité d’installation 24
Appendix F Installation - Sicherheitshinweise 26
Mellanox Technologies Rev 1.7
4
About this Manual
This manual describes the installation and basic use of the Mel lanox M2401G 16+8 Port 20Gb/s
InfiniBand Switch for Dell PowerEdge M1000e-series Blade Enclosures.
Intended Audience
This manual is intended for users and system administrators responsible for installing and setting
up the switch platforms listed above.
The manual assumes familiarity with the InfiniBand® Architecture Specification.
Related Documentation
The documentation set accompanying the Mellanox M2401G includes the following:
Table 1 - Reference Documents
InfiniBand Architecture Specification Volume 1 Release 1.2.1 and Volume 2 release
1.2.1
Mellanox OFED Stack The Mellanox OFED stack provides a tool set to manage InfiniBand fabric clusters. The
Mellanox Firmware Tools (MFT)
User’s Manual
Document # 2329
Online Resources
• Mellanox Technologies Web pages: http://www.mellanox.com
• Dell Support W eb pages: http://support.dell.com
Conventions
Throughout this manual, the name M2401G and the term switch are used to describe the 16+8 port
20Gb/s InfiniBand switch for the Dell M1000e, unless explicitly indicated otherwise.
InfiniBand architecture specification descriptions
stack with user’s manual can be downloaded at:
http://www.mellanox.com > Downloads > InfiniBand SW/Drivers,
Be sure to download the stack that matches your OS.
The MFT (Mellanox Firmware Tools) package is a set of firmware tools. The manual
supplied with this package provides an overview of the firmware its installation and
replacement. The MFT can be downloaded with its documentation at:
http://www.mellanox.com > Downloads > Firmware Tools
Rev 1.7 Mellanox Technologies
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
Revision History
Table 2 - Revision History Table
Date Revision Description
Mar. 2009 Rev 1.7 Added Dell flat black Logo
Dec. 2008 Rev 1.6 Minor fixes to Rev 1.5
Dec. 2008 Rev 1.5 Minor fixes to Rev 1.4
Added Instructions for Reprogramming Over the InfiniBand Network
5
Nov. 2008 Rev 1.4 Added Dell logo and Dell 3rd party info statement
Aug. 2008 Rev 1.3 Added Certs and Link to FW
Aug. 2008 Rev 1.2 Added appendix with safety warnings in French for Canada Certification
July 2008 Rev 1.1 Fixed Max Ambient Temperature
June 2008 Rev 1.0 First Draft
Replaced Mellanox address
Fixed Links to new web site
Mellanox Technologies Rev 1.7
6
Locking Mechanism
Switch Status Lights
1 Overview
The Mellanox M2401G 20Gb/s InfiniBand Switch Blade for Dell PowerEdge M1000e-series Blade
Enclosures provides a high bandwidth, low latency, low power fabric for Enterprise Data Center
and High-Performance Computing environments. Based on the third generat ion InfiniScale® III
InfiniBand switch device, the platform delivers an ideal balance between cost and performance.
When used in conjunction with ConnectX® InfiniBand dual port I/O cards clustered data bases,
parallelized applications and transactional services applications will achieve significant perfor
mance improvements resulting in reduced completion time and lower cost per operation.
The switch platform comes pre-installed with all necessary firmware, and configured for standard
operation within an InfiniBand fabric, and requires an InfiniBand compl iant Subnet Manager run
ning from one of the hosts. All that is required for normal operation is to follow the usual precautions for installation and connection from the switch to the HCAs or other switches. Once
connected, the Subnet Management software automatically configures and begins utilizing the
switch.
It is recommended that Mellanox OpenFabrics software package be installed on all nodes connected to the M2401G. The software package provides a subnet manager and network management
tools as well as connectivity software for servers and storage, and is available on the Mellanox web
site. See
“Cluster Management and Firmware” on page 14. for more information.
-
-
Basic installation, hot-swapping components and hardware main tenance is covered in “Installa-
tion and Basic Operation” on page 9.
The M2401G switch has a Hot Swap controller and a PSOC Management IC.
Figure 1: Switch Front Panel and Locking Mechanism
Rev 1.7 Mellanox Technologies
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
LED Name Connection Status
Physical Link - Green Off – No Physical Link
ON – Physical Link
Data Activity - Yellow Blinking – indicates Data Transfer
Constant on – indicates Link exists
with no Data Transfer taking place
Off with green LED lit – indicates that
the Subnet Manager may not be
running
The I/O Module is on and
ready when both the blue and
green LEDs are lit.
This LED can be blue or
amber. A fault is indicated
when the amber LED is blinking. See Table 3.
1.1 InfiniBand Connectors
InfiniBand connectivity has 8 microGiGa connectors through the front panel. The remai ning 16
interfaces are through the AirMax Midplane Connector out the back of the switch.
the front 8 ports.
Each of the InfiniBand ports has two LEDs located next to the connector. The green LED, when lit,
indicates that a valid physical connection to the other system (switch or HCA port) exists. The yel
low LED when lit, indicates that the Subnet Manager is running and a valid data link exists. The
yellow LED illuminates when the InfiniBand network is discovered over th e ph ysical lin k. A valid
data activity link without data transfer is designated by a constant yellow LED indication. A valid
data activity link with data transfer is designated by a blinking yellow LED indication. If the LEDs
are not active, either the physical link or the logical link (or both) connections have not been estab
lished.
Figure 2: Physical and Logical Link Indication LEDs
7
Figure 1 shows
-
-
1.2 Switch Status Lights
The switch Status lights indicate whether the switch is receiving power from the chassis, and the
state of the switch.
Figure 3: Indicator LEDs
Mellanox Technologies Rev 1.7
8
Table 3 - IOM states and LED configurations:
LED
Switch Status
Indication Green Status
OFF OFF OFF
ON OFF Boot in Progress
Switch not ready
Blinking Blue
ON BLINKING BLUE The CMC is identifying
the newly installed switch
ON ON BLUE Switch is on and operating Normally
ON ON or BLINKING
AMBER
OFF ON or BLINKING
AMBER
Fault in System
Self-diagnosed
Fault in System
CMC-detected
Rev 1.7 Mellanox Technologies
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
2 Installation and Basic Operation
Installation and initialization of the switch platform are straightforward processes, requirin g attention to the normal mechanical, power, and thermal precautions for rack-mounted equipment. The
switch platform does not require any programming or configuration to operate as a basic Infini
Band switch and includes all of the necessary functionality to operate with external standard InfiniBand Subnet Management software.
This section describes the installation process and basic operation of the switch platform. Please
first read the warnings sub-section carefully before carrying on with instal lati on procedures.
Suitable electrical, mechanical and fire enclosure shall be provided by the end user.
9
-
Mellanox Technologies Rev 1.7
Loading...