IBM 24R9718 IB User Manual

IB Switch Module for IBM eServer BladeCenter User Guide

Release 2.2.0 Update 2

Part Number: 10-00092-02-A0 IBM Part number: 24R9718

The Topspin Switched Computing System, Topspin Host Channel Adapter, Topspin Element Manager, and collateral software programs and documentation are subject to and made available only pursuant to the license agreement signed by you and Topspin, Communications, Inc. (or if no signed license agreement exists, the license agreement included with your original media) and may only be used in accordance with the terms of that agreement. Making copies, modifications, or compilation works of the software except as specifically allowed in that agreement is prohibited by law and constitutes a punishable violation of the law. This documentation may not be copied, modified or reduced to any electronic or machine-readable form without Topspin Communication, Inc.'s prior written consent.

As defined in FAR section 2.101, DFAR section 252.227-7014(a)(1) and DFAR section 252.227-7014(a)(5) or otherwise, the collateral software programs provided to you are “commercial items,” “commercial computer software” and/or “commercial computer software documentation.” Consistent with DFAR section 227.7202, FAR section 12.212 and other sections, any use, modification, reproduction, release, performance, display, disclosure or distribution thereof by or for the U.S. Government shall be governed solely by the terms of the license agreement and shall be prohibited except to the extent expressly permitted by the terms of that agreement. Any technical data provided that is not covered by the above provisions shall be deemed “technical data-commercial items” pursuant to DFAR section 227.7015(a) or equivalent. Any use, modification, reproduction, release, performance, display or disclosure of such technical data shall be governed by the terms of DFAR section 227.7015(b) or equivalent.

This documentation may include technical inaccuracies or typographical errors and is subject to correction and other revision without notice. TOPSPIN PROVIDES THIS DOCUMENTATION “AS IS” AND WITHOUT ANY WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states or jurisdictions do not allow disclaimer of express or implied warranties in certain transactions; therefore, this statement may not apply to you.

©Copyright 2004, Topspin Communications, Inc. All rights reserved. Topspin is a registered trademark and the Topspin logo, TopspinOS, and Topspin Switched Computing System are trademarks of Topspin Communications, Inc. Other company, product, or service names are the property of their respective owners.

Document Version: 2.2.0 Update 2

Part Number: 10-00092-02-A0

May, 2005

Printed in the United States of America.

Important Notice

License information for IB Switch Module for IBM eServer BladeCenter

IMPORTANT NOTICE: This hardware product was manufactured for IBM by Topspin Communications and contains software and/or firmware provided to you by Topspin Communications under provided license agreements. By using the enclosed hardware and/or software products, you are agreeing to the terms of these license agreements, and agree that the terms of such agreement are extended to you solely by Topspin Communications.

First Edition (November 2004)

IBM®, Eserver, BladeCenter, and the e-business logo are trademarks of the IBM Corporation in the United States, other countries, or both.

US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Statement of Limited Warranty Preamble

Thank you for your purchase. IBM provides hardware warranty service for this product equivalent to that described in the IBM Statement of Limited Warranty ("SOLW"), a copy of which is included (“IBM Statement of Limited Warranty Z125-4753-08 04/2004”

on page xv), notwithstanding the section entitled, "What this Warranty Does not Cover" of the SOLW. All references to "Machine" in

the SOLW apply to this product (except for its firmware or accompanying software) for purposes of the warranty. References to "IBM's Official Published Specifications" mean the specifications for the product designated by IBM. Any firmware or software included in or which accompanies this product is subject to the terms and conditions of the manufacturer's end user license agreement."

Contents

Important Notice ........................................................................................................................................ iii

Statement of Limited Warranty Preamble ................................................................................................. iii

Regulatory Notices......................................... ix

FCC Statement........................................................................................................................................... ix

Safety Information ..................................................................................................................................... ix

Electrical Cautions........................................................................................................................ ix

General Cautions............................................................................................................................. x

IBM Support Contact Information...............................................................................................................x

Before You Call ............................................................................................................................ xi

Topspin Contact Information.................................................................................................................... xii

IBM Statement of Limited Warranty Z125-4753-08

04/2004 ......................................................... xv

Part 1 - General Terms...............................................................................................................................xv

What this Warranty Covers........................................................................................................... xv

What this Warranty Does not Cover........................................................................................... xvi

How to Obtain Warranty Service................................................................................................ xvi

What IBM Will Do to Correct Problems .................................................................................... xvi

Exchange of a Machine or Part.................................................................................................. xvii

Your Additional Responsibilities............................................................................................... xvii

Limitation of Liability ............................................................................................................... xvii

Governing Law ......................................................................................................................... xviii

Jurisdiction................................................................................................................................ xviii

Part 2 - Country-unique Terms .............................................................................................................. xviii

AMERICAS.............................................................................................................................. xviii

NORTH AMERICA ................................................................................................................... xix

ASIA PACIFIC............................................................................................................................. xx

EUROPE, MIDDLE EAST, AFRICA (EMEA)........................................................................ xxii

EUROPEAN UNION (EU) ...................................................................................................... xxiv

CONSUMERS .......................................................................................................................... xxiv

THE FOLLOWING TERMS APPLY TO THE COUNTRY SPECIFIED:............................... xxv

Part 3 - Warranty Information.............................................................................................................. xxviii

Warranty Period...................................................................................................................... xxviii

Types of Warranty Service ....................................................................................................... xxix

EU Country Telephone List........................................................................................................ xxx

Notices ....................................................... xxxi

Edition notice........................................................................................................................................ xxxii

Trademarks ........................................................................................................................................... xxxii

Important notes .................................................................................................................................... xxxiii

Product recycling and disposal ............................................................................................................ xxxiii

Battery return program......................................................................................................................... xxxiii

Electronic emission notices ................................................................................................................. xxxiv

Federal Communications Commission (FCC) statement ....................................................... xxxiv

Industry Canada Class A emission compliance statement...................................................... xxxiv

Australia and New Zealand Class A statement......................................................................... xxxv

United Kingdom telecommunications safety requirement ....................................................... xxxv

European Union EMC Directive conformance statement ........................................................ xxxv

Taiwanese Class A warning statement ..................................................................................... xxxv

Chinese Class A warning statement.......................................................................................... xxxv

Japanese Voluntary Control Council for Interference (VCCI) statement............................... xxxvi

WEEE Notice ........................................... xxxvii

Safety....................................................... xxxix

1: About the Server Switch Module................ 1

Introduction..................................................................................................................................................1

Server Switch Module and HCA Expansion Cards ........................................................................ 2

Topologies ...................................................................................................................................................2

Single-Switch Topology ................................................................................................................. 2

Dual-Switch Topology.................................................................................................................... 2

Management.................................................................................................................................................3

LEDs ............................................................................................................................................................3

Server Switch Module Status LEDs ............................................................................................... 4

Port Status LEDs............................................................................................................................. 4

External InfiniBand Ports ............................................................................................................................4

2: Getting Started........................................... 5

Install the Server Switch Module.................................................................................................................5

Configure the Server Switch Module ..........................................................................................................5

Change the Default Password ......................................................................................................... 6

Configure the System Clock ........................................................................................................... 7

Connect Beyond BladeCenter......................................................................................................................8

Connecting InfiniBand Cables...................................................................................................... 10

3: Specifications and Compliance................ 13

Safety Certifications ..................................................................................................................................13

EMC/EMI Certifications............................................................................................................................13

Electrical Requirements.............................................................................................................................14

Environmental Requirements ....................................................................................................................14

vii

viii

Regulatory Notices

FCC Statement

This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense.

Safety Information

• “Electrical Cautions” on page ix

• “General Cautions” on page x

Electrical Cautions

CAUTION: Always ground yourself before touching any internal IB Switch Module component to avoid damage from electrostatic discharge (ESD).

CAUTION: The IB Switch Module contains lithium batteries. Do not attempt to replace or discard these batteries. The batteries may only be serviced by Topspin service personnel.

CAUTION: Observe and follow service markings. Do not service the IB Switch Module, except as explained in the Topspin documentation supplied with your IB Switch Module. Opening the chassis or removing the enclosure cover exposes you to electrical shock and may damage IB Switch Module components.

If one or more of the following situations occurs, do not install the IB Switch Module in a chassis and contact your Topspin support representative:

• The IB Switch Module has been exposed to water.

• The IB Switch Module has been dropped or damaged in any way.

• The IB Switch Module does not operate correctly after following the installation instructions.

CAUTION: You must ensure the operating environment meets the IB Switch Module’s electrical and environmental operating requirements.

CAUTION: Use the Topspin IB Switch Module only with an approved chassis.

CAUTION: Never place your hand inside an empty card or module bay. You should never have cause

to place a hand anywhere inside the chassis. Unused card and module bays should always have a cover over the bay to ensure proper safety, ventilation, and cooling.

General Cautions

CAUTION: Opening and/or modifying the expansion modules may void your warranty. Refer to your warranty to determine access restrictions.

In general

• Do not spill food or liquids on your IB Switch Module components.

• Place cables appropriately so that they do not obstruct any egress within the data center and they do not block any ventilation inlets or outlets.

IBM Support Contact Information

For technical support, refer to the following Web site:

http://www.ibm.com/support

To contact technical support, send email to support@ibm.com or refer to one of the telephone numbers below.

To obtain warranty service contact IBM or your IBM reseller. In Canada or the United States, call 1-800-IBM-SERV (426-7378). In the EU countries, see the telephone numbers in Table iii-1.

Table iii-1: EU Country Telephone List

Austria -- 43-1-24592-5901 Italy -- 39-02-482-9202

Belgium -- 02-718-4339 Luxembourg -- 352-360385-1

Denmark -- 4520-8200 Netherlands -- 020-514-5770

Finland -- 358-9-4591 Portugal -- 351-21-7915-147

France -- 0238-557-450 Spain -- 34-91-662-4916

Germany -- 07032-15-4920 Sweden -- 46-8-477-4420

Greece -- 30-210-688-1220 United Kingdom -- 01475-555-055

Ireland -- 353-1-815-4000

Phone numbers are subject to change without notice.

NOTE: IBM and Topspin are both available for more information. For support, contact IBM before Topspin.

Before You Call

• (Linux only; leave blank for Windows) Enter the rpm -qa | grep topspin command at your host to view your RPMs, then provide this output.

• Provide all relevant messages from system and console logs.

• Provide the output from the show running-status all command.

• Provide the firmware version of the host. For a Linux host, refer to step a; for a Windows host, refer to step b.

a. For Linux hosts, enter the /usr/local/topspin/sbin/tvflash -i command to view the firmware

version. In the example that follows, the image is v3.01.0000 build 2.0.0.491.

Example

# /usr/local/topspin/sbin/tvflash -i HCA #0: Found MT23108, Cougar, revision A1 (firmware autoupgrade) Primary image is v3.01.0000 build 2.0.0.491, with label 'HCA.Cougar.A1.Boot' Secondary image is valid, unknown source [root@enclus2 root]#

b. For Windows hosts, perform the following steps:

1. Log in to your Windows host.

2: Click Start > Programs > Topspin IB SDK > Utilities. The Utilities window opens and

displays a list of utilities.

3: Enter tvflash -i command to display the firmware version.

xii

• (Linux only) Run the dmesg command on your Linux host and provide the output.

Example

# dmesg 0 0 0 1 1 71 0a 000 00 1 0 0 0 0 0 0 00 0b 000 00 1 0 0 0 0 0 0 00 0c 00F 0F 0 0 0 0 0 1 1 79 0d 00F 0F 0 0 0 0 0 1 1 81 0e 00F 0F 0 0 0 0 0 1 1 89 0f 00F 0F 0 0 0 0 0 1 1 91 10 00F 0F 1 1 0 1 0 1 1 99 11 000 00 1 0 0 0 0 0 0 00 12 00F 0F 1 1 0 1 0 1 1 A1 13 00F 0F 1 1 0 1 0 1 1 A9 14 000 00 1 0 0 0 0 0 0 00 15 000 00 1 0 0 0 0 0 0 00 16 000 00 1 0 0 0 0 0 0 00 17 000 00 1 0 0 0 0 0 0 00

//content truncated

scsi : 2 hosts left. scsi2 : srp initializing plug timer for queue f69fce18 Starting timer : 1 1 blk: queue f69fce18, I/O limit 4095Mb (mask 0xffffffff) Vendor: SEAGATE Model: ST336607FC Rev: 0004 Type: Direct-Access ANSI SCSI revision: 03 initializing plug timer for queue f69fd018 Starting timer : 1 1 blk: queue f69fd018, I/O limit 4095Mb (mask 0xffffffff) Attached scsi disk sdb at scsi2, channel 0, id 6, lun 0 SCSI device sdb: 71687372 512-byte hdwr sectors (36704 MB) sdb: sdb1 sdb2 srp_host: target_bindings=22000004cfe7a949.0:22000004cfe7aa7a.1:22000004cfe7aac7.2:2200000 4cfe7aadd.3:22000004cfe7ab01.4:22000004cfe7ab2f.5:22000004cfe7ab35.6:22000004cfe 7ab6c.7:22000004cfe7ab7b.8:22000004cfe7ab89.9:22000004cfe7ab8d.a:22000004cfe7ac1 f.b:22000004cfe7ac30.c:22000004cfe7ac69.d divert: allocating divert_blk for ib0 divert: allocating divert_blk for ib1 nfs: server 10.5 not responding, still trying nfs: server 10.5 OK

Topspin Contact Information

Topspin contact information appears in Table iii-2

Table iii-2: Contact Information

Address Topspin Communications, Inc.

515 Ellis St.

Mountain View, CA 94043

Table iii-2: Contact Information (Continued)

Telephone, Corporate Headquarters (866) TOPSPIN or (650) 316-3300

Fax (650) 316-3269

Telephone, Technical Support (800) 499-1473

Email, Technical Support support@ibm.com (contact IBM first)

support@topspin.com

Web site, Support http://www.ibm.com/support

http://support.topspin.com

(contact IBM first)

xiii

xiv

IBM Statement of Limited Warranty Z125-4753-08 04/2004

Part 1 - General Terms

This Statement of Limited Warranty includes Part 1 - General Terms, Part 2 - Country-unique Terms, and Part 3 - Warranty Information. The terms of Part 2 replace or modify those of Part 1. The warranties provided by IBM in this Statement of Limited Warranty apply only to Machines you purchase for your use, and not for resale. The term “Machine” means an IBM machine, its features, conversions, upgrades, elements, or accessories, or any combination of them. The term “Machine” does not include any software programs, whether pre-loaded with the Machine, installed subsequently or otherwise. Nothing in this Statement of Limited Warranty affects any statutory rights of

consumers that cannot be waived or limited by contract.

What this Warranty Covers

IBM warrants that each Machine 1) is free from defects in materials and workmanship and 2) conforms to IBM's Official Published Specifications (“Specifications”) which are available on request. The warranty period for the Machine starts on the original Date of Installation and is specified in Part 3 Warranty Information. The date on your invoice or sales receipt is the Date of Installation unless IBM or your reseller informs you otherwise. Many features, conversions, or upgrades involve the removal of parts and their return to IBM. A part that replaces a removed part will assume the warranty service status of the removed part. Unless IBM specifies otherwise, these warranties apply only in the country or region in which you purchased the Machine.

THESE WARRANTIES ARE YOUR EXCLUSIVE WARRANTIES AND REPLACE ALL OTHER WARRANTIES OR CONDITIONS, EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OR CONDITIONS OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. SOME STATES OR

xvi

JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF EXPRESS OR IMPLIED WARRANTIES, SO THE ABOVE EXCLUSION MAY NOT APPLY TO YOU. IN THAT EVENT, SUCH WARRANTIES ARE LIMITED IN DURATION TO THE WARRANTY PERIOD. NO WARRANTIES APPLY AFTER THAT PERIOD. SOME STATES OR JURISDICTIONS DO NOT ALLOW LIMITATIONS ON HOW LONG AN IMPLIED WARRANTY LASTS, SO THE ABOVE LIMITATION MAY NOT APPLY TO YOU.

What this Warranty Does not Cover

This warranty does not cover the following:

• any software programs, whether pre-loaded or shipped with the Machine, or installed subsequently;

• failure resulting from misuse (including but not limited to use of any Machine capacity or capability, other than that authorized by IBM in writing), accident, modification, unsuitable physical or operating environment, or improper maintenance by you;

• failure caused by a product for which IBM is not responsible; and

• any non-IBM products, including those that IBM may procure and provide with or integrate into an IBM Machine at your request.

The warranty is voided by removal or alteration of identification labels on the Machine or its parts.

IBM does not warrant uninterrupted or error-free operation of a Machine.

Any technical or other support provided for a Machine under warranty, such as assistance with “how-to” questions and those regarding Machine set-up and installation, is provided WITHOUT WARRANTIES OF ANY KIND.

How to Obtain Warranty Service

If the Machine does not function as warranted during the warranty period, contact IBM or your reseller to obtain warranty service. If you do not register the Machine with IBM, you may be required to present proof of purchase as evidence of your entitlement to warranty service.

What IBM Will Do to Correct Problems

When you contact IBM for service, you must follow the problem determination and resolution procedures that IBM specifies. An initial diagnosis of your problem can be made either by a technician over the telephone or electronically by access to an IBM website.

The type of warranty service applicable to your Machine is specified in Part 3 - Warranty Information.

You are responsible for downloading and installing designated Machine Code (microcode, basic input/output system code (called “BIOS”), utility programs, device drivers, and diagnostics delivered with an IBM Machine) and other software updates from an IBM Internet Web site or from other electronic media, and following the instructions that IBM provides.

If your problem can be resolved with a Customer Replaceable Unit (“CRU”) (e.g., keyboard, mouse, speaker, memory, hard disk drive), IBM will ship the CRU to you for you to install.

If the Machine does not function as warranted during the warranty period and your problem cannot be resolved over the telephone or electronically, through your application of Machine Code or software updates, or with a CRU, IBM or your reseller, if approved by IBM to provide warranty service, will either, at its discretion, 1) repair it to make it function as warranted, or 2) replace it with one that is at least functionally equivalent. If IBM is unable to do either, you may return the Machine to your place of purchase and your money will be refunded.

IBM or your reseller will also manage and install selected engineering changes that apply to the Machine.

xvii

Exchange of a Machine or Part

When the warranty service involves the exchange of a Machine or part, the item IBM or your reseller replaces becomes its property and the replacement becomes yours. You represent that all removed items are genuine and unaltered. The replacement may not be new, but will be in good working order and at least functionally equivalent to the item replaced. The replacement assumes the warranty service status of the replaced item.

Your Additional Responsibilities

Before IBM or your reseller exchanges a Machine or part, you agree to remove all features, parts, options, alterations, and attachments not under warranty service.

You also agree to:

1. ensure that the Machine is free of any legal obligations or restrictions that prevent its exchange;

2. obtain authorization from the owner to have IBM or your reseller service a Machine that you do not own; and

3. where applicable, before service is provided:

a. follow the service request procedures that IBM or your reseller provides;

b. backup or secure all programs, data, and funds contained in the Machine;

c. provide IBM or your reseller with sufficient, free, and safe access to your facilities to permit

IBM to fulfill its obligations; and

d. inform IBM or your reseller of changes in the Machine's location.

4. (a) ensure all information about identified or identifiable individuals (Personal Data) is deleted from the Machine (to the extent technically possible), (b) allow IBM, your reseller or an IBM supplier to process on your behalf any remaining Personal Data as IBM or your reseller considers necessary to fulfill its obligations under this Statement of Limited Warranty (which may include shipping the Machine for such processing to other IBM service locations around the world), and (c) ensure that such processing complies with any laws applicable to such Personal Data.

Limitation of Liability

IBM is responsible for loss of, or damage to, your Machine only while it is 1) in IBM's possession or 2) in transit in those cases where IBM is responsible for the transportation charges.

Neither IBM nor your reseller are responsible for any of your confidential, proprietary or personal information contained in a Machine which you return to IBM for any reason. You should remove all such information from the Machine prior to its return.

Circumstances may arise where, because of a default on IBM's part or other liability, you are entitled to recover damages from IBM. In each such instance, regardless of the basis on which you are entitled to claim damages from IBM (including fundamental breach, negligence, misrepresentation, or other contract or tort claim), except for any liability that cannot be waived or limited by applicable laws, IBM is liable for no more than

1. damages for bodily injury (including death) and damage to real property and tangible personal property; and

2. the amount of any other actual direct damages, up to the charges (if recurring, 12 months' charges apply) for the Machine that is subject of the claim. For purposes of this item, the term “Machine” includes Machine Code and Licensed Internal Code (“LIC”).

This limit also applies to IBM's suppliers and your reseller. It is the maximum for which IBM, its suppliers, and your reseller are collectively responsible.

UNDER NO CIRCUMSTANCES IS IBM, ITS SUPPLIERS OR RESELLERS LIABLE FOR ANY OF THE FOLLOWING EVEN IF INFORMED OF THEIR POSSIBILITY: 1) THIRD

xviii

PARTY CLAIMS AGAINST YOU FOR DAMAGES (OTHER THAN THOSE UNDER THE FIRST ITEM LISTED ABOVE); 2) LOSS OF, OR DAMAGE TO, DATA; 3) SPECIAL, INCIDENTAL, OR INDIRECT DAMAGES OR FOR ANY ECONOMIC CONSEQUENTIAL DAMAGES; OR 4) LOST PROFITS, BUSINESS REVENUE, GOODWILL OR ANTICIPATED SAVINGS. SOME STATES OR JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THE ABOVE LIMITATION OR EXCLUSION MAY NOT APPLY TO YOU. SOME STATES OR JURISDICTIONS DO NOT ALLOW LIMITATIONS ON HOW LONG AN IMPLIED WARRANTY LASTS, SO THE ABOVE LIMITATION MAY NOT APPLY TO YOU.

Governing Law

Both you and IBM consent to the application of the laws of the country in which you acquired the Machine to govern, interpret, and enforce all of your and IBM's rights, duties, and obligations arising from, or relating in any manner to, the subject matter of this Statement of Limited Warranty, without regard to conflict of law principles.

THESE WARRANTIES GIVE YOU SPECIFIC LEGAL RIGHTS AND YOU MAY ALSO HAVE OTHER RIGHTS WHICH VARY FROM STATE TO STATE OR JURISDICTION TO JURISDICTION.

Jurisdiction

All of our rights, duties, and obligations are subject to the courts of the country in which you acquired the Machine.

Part 2 - Country-unique Terms

AMERICAS

ARGENTINA

Jurisdiction: The following is added after the first sentence:

Any litigation arising from this Statement of Limited Warranty will be settled exclusively by the Ordinary Commercial Court of the city of Buenos Aires.

BOLIVIA

Jurisdiction: The following is added after the first sentence:

Any litigation arising from this Statement of Limited Warranty will be settled exclusively by the courts of the city of La Paz.

BRAZIL

Jurisdiction: The following is added after the first sentence:

Any litigation arising from this Statement of Limited Warranty will be settled exclusively by the court of Rio de Janeiro, RJ.

CHILE

Jurisdiction: The following is added after the first sentence:

Any litigation arising from this Statement of Limited Warranty will be settled exclusively by the Civil Courts of Justice of Santiago.

+ 41 hidden pages