Sun Microsystems Datacenter 3456 820-4733-10 User Manual

Sun

™

Datacenter Switch 3456

Service Manual

Sun Microsystems, Inc. www.sun.com

Part No. 820-4733-10 August 2008, Revision A

Submit comments about this document at: http://www.sun.com/hwdocs/feedback

Sun Microsystems,Inc. has intellectual property rights relating to technology embodiedin the product that is described inthis document. In particular,and without limitation, these intellectualproperty rightsmay include one or more ofthe U.S. patents listed at http://www.sun.com/patents and oneor more additional patents orpending patent applications in the U.S. andin other countries.Unpublished -rights reserved under the CopyrightLaws of the United States..

This distributionmay include materials developed bythird parties.Parts of the product may bederived fromBerkeley BSD systems, licensed from theUniversity of California. UNIX isa registered trademarkin the U.S. and inother countries, exclusively licensed through X/Open Company, Ltd.

Sun, SunMicrosystems, the Sun logo, SunService,and The Network is the Computer aretrademarks or registered trademarksof Sun Microsystems, Inc.in the U.S. and othercountries.The OPEN LOOK and Sun(TM) Graphical UserInterface was developed by Sun Microsystems, Inc.for its users and licensees.

Sun acknowledgesthe pioneering efforts of Xerox inresearching anddeveloping the concept of visualor graphical user interfaces for the computer industry.Sun holds a non-exclusive licensefrom Xerox to the Xerox Graphical User Interface,which license also covers Sun’s licensees whoimplement OPEN LOOK GUIs andotherwise comply with Sun’s writtenlicense agreements.

Use ofany spare or replacement CPUs is limitedto repairor one-for-one replacement of CPUs in products exportedin compliance with U.S. export laws.Use of CPUs as productupgrades unless authorized by the U.S. Governmentis strictly prohibited.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULARPURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Sun Microsystems,Inc. détient les droits de propriété intellectuels relatifs àla technologie incorporée dans le produit quiest décrit dans ce document. Enparticulier, et ce sanslimitation, ces droits de propriété intellectuelle peuvent inclure unou plus des brevets américains listés à l’adresse http://www.sun.com/patent set un ou les brevets supplémentairesou les applications de brevet enattente aux Etats - Uniset dans lesautres pays.Non publie - droits réservésselon la législation des Etats-Unissur le droit d’auteur.

Cette distributionpeut comprendredes composants développés par destierces parties.Desparties de ce produit pourront être dérivées des systèmes BerkeleyBSD licenciés par l’Université deCalifornie. UNIX est une marque déposée aux Etats-Unis et dans d’autrespays et licenciée exclusivement parX/Open Company, Ltd.

Sun, Sun Microsystems, le logo Sun, SunService, et The Network is the Computer sont desmarques de fabrique ou des marques déposéesde Sun Microsystems,Inc. aux Etats-Unis et dansd’autres pays.L’interface d’utilisation graphique OPEN LOOK etSun(TM) a été développée par Sun Microsystems,Inc. pour ses utilisateurs etlicenciés.

Sun reconnaîtles effortsde pionniers de Xerox pour la recherche etle développement du concept desinterfaces d’utilisation visuelle ou graphique pourl’industrie de l’informatique. Sun détientune license non exclusive deXerox surl’interface d’utilisation graphique Xerox, cette licence couvrantégalement les licenciés de Sunqui mettent en place l’interfaced’utilisation graphique OPEN LOOK etqui, en outre, se conforment auxlicences écrites de Sun.

L’utilisationde pieces detachees ou d’unitescentrales de remplacement est limitee aux reparations oua l’echange standard d’unites centrales pour lesproduits exportes, conformement a lalegislation americaine en matiere d’exportation. Sauf autorisationpar les autorites des EtatsUnis, l’utilisationd’unites centrales pour proceder ades mises a jour de produits estrigoureusement interdite.

LA DOCUMENTATION ESTFOURNIE "EN L’ETAT" ET TOUTES AUTRES CONDITIONS, DECLARATIONS ETGARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES,DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRISNOTAMMENT TOUTE GARANTIEIMPLICITE RELATIVE A LA QUALITEMARCHANDE, A L’APTITUDE AUNE UTILISATION PARTICULIEREOU A L’ABSENCEDE CONTREFACON.

Please

Recycle

Preface vii

1. Sun Datacenter Switch 3456 Overview 1–1

1.1 Switch Chassis 1–1

1.2 Power Supplies 1–2

1.3 Fan Units 1–3

1.4 Chassis Management Controllers 1–4

1.5 Fabric Cards 1–5

1.6 Line Cards 1–6

2. Chassis Service Procedures 2–1

2.1 Filler Panels 2–1

▼ Removing the Filler Panel 1

▼ Installing the Filler Panel 3

2.2 Midplane Connector Pins 2–6

2.2.1 Tools 2–6

▼ Inspecting the Pins on the Rear Side of the Midplane 7

▼ Inspecting the Pins on the Front Side of the Midplane 10

▼ Removing Pins 13

▼ Removing a Pin With Needle-Nose Pliers 14

iii

▼ Inserting Pins 15

2.3 Cable Management Hardware 2–16

▼ Removing Cable Plates 16

▼ Installing Cable Plates 16

▼ Removing Cable Trees 17

▼ Installing Cable Trees 18

▼ Removing Cable Trays 20

▼ Installing Cable Trays 20

2.4 InfiniBand Cables 2–21

2.4.1 Cable Cautions 2–21

▼ Removing Cables From the Sun Datacenter Switch 3456 22

▼ Attaching Cables to the Sun Datacenter Switch 3456 23

3. Power Supply Service Procedures 3–1

3.1 Troubleshooting Power Supplies 3–2

3.2 Powering Off Power Supplies 3–2

3.3 Removing Power Supplies 3–3

▼ Removing a Power Supply 3

3.4 Installing Power Supplies 3–5

▼ Installing a Power Supply 5

3.5 Powering On Power Supplies 3–7

4. Fan Service Procedures 4–1

4.1 Troubleshooting Fans 4–1

4.2 Removing Fans 4–1

▼ Removing a Fan 1

4.3 Installing Fans 4–2

▼ Installing a Fan 3

5. Chassis Management Controller Service Procedures 5–1

iv Sun Datacenter Switch 3456 Service Manual • August 2008

5.1 Troubleshooting CMCs 5–2

5.2 Powering Off CMCs 5–2

5.3 Removing CMCs 5–3

▼ Removing a CMC 3

5.4 Installing CMCs 5–4

▼ Installing a CMC 4

5.5 Powering On CMCs 5–6

6. Fabric Card Service Procedures 6–1

6.1 Fabric Card Considerations 6–1

6.2 Troubleshooting Fabric Cards 6–2

6.3 Inspecting Fabric Cards 6–3

▼ Inspecting the Fabric Card iTRAC Connectors 3

▼ Inspecting the Fans 5

▼ Inspecting the Retainers 5

6.4 Powering Off Fabric Cards 6–6

6.5 Removing Fabric Cards 6–7

▼ Removing a Fabric Card 7

6.6 Installing Fabric Cards 6–8

▼ Installing a Fabric Card 8

6.7 Powering On Fabric Cards 6–10

7. Line Card Service Procedures 7–1

7.1 Line Card Considerations 7–1

7.2 Troubleshooting Line Cards 7–2

7.3 Inspecting Line Cards 7–3

▼ Inspecting the Line Card iTRAC Connectors 3

▼ Inspecting the Line Card iPASS Connectors 5

▼ Inspecting the Retainers 5

Contents v

7.4 Powering Off Line Cards 7–6

7.5 Removing Line Cards 7–7

▼ Removing a Line Card 7

7.6 Installing Line Cards 7–12

▼ Installing a Line Card 12

7.7 Powering On Line Cards 7–15

Index Index–1

vi Sun Datacenter Switch 3456 Service Manual • August 2008

Preface

This service manual provides service procedures for the Sun Datacenter Switch 3456.

This document is written for service personnel and users who have advanced knowledge and experience managing an InfiniBand network.

Using UNIX Commands

This document might not contain information about basic UNIX®commands and procedures such as shutting down the system, booting the system, and configuring devices. Refer to the following for this information:

■ Software documentation that you received with your system

■ Solaris™ Operating System documentation, which is at:

http://docs.sun.com

vii

Shell Prompts

Shell Prompt

C shell machine-name%

C shell superuser machine-name#

Bourne shell and Korn shell $

Bourne shell and Korn shell superuser #

Typographic Conventions

Typeface

AaBbCc123 The names of commands, files,

AaBbCc123 What you type, when contrasted

AaBbCc123 Book titles, new words or terms,

* The settings on your browser might differ from these settings.

Meaning Examples

Edit your.login file. and directories; on-screen computer output

with on-screen computer output

words to be emphasized. Replace command-line variables with real names or values.

Use ls -a to list all files.

% You have mail.

% su

Password:

Read Chapter 6 in the User’s Guide.

These are called class options.

Yo u must be superuser to do this.

To delete a file, type rm filename.

Sun Datacenter Switch 3456 Overview

This chapter provides a visual overview of the Sun Datacenter Switch 3456. Topics include:

■ Section 1.1 “Switch Chassis” on page 1-1

■ Section 1.2 “Power Supplies” on page 1-2

■ Section 1.3 “Fan Units” on page 1-3

■ Section 1.4 “Chassis Management Controllers” on page 1-4

■ Section 1.5 “Fabric Cards” on page 1-5

■ Section 1.6 “Line Cards” on page 1-6

1.1 Switch Chassis

FIGURE 1-1 describes the various parts of the switch chassis.

1-1

FIGURE 1-1 Parts of the Switch Chassis

8 9 10

Figure Legend

1 Fabric cards (18) 6 Line cards (24)

2 Power supplies (16) 7 Cable trays (24)

3 Cable plates (left) 8 Cable tree (left)

4 Cable plates (right) 9 Chassis

5 Cable tree (right) 10 CMCs (2)

1.2 Power Supplies

FIGURE 1-2 describes the various parts of the power supplies.

1-2 Sun Datacenter Switch 3456 Service Manual • August 2008

FIGURE 1-2 Parts of the Power Supplies

Figure Legend

1 Power Distribution Board connector

2 Power cord connector

3 Status LEDs

4 Fan

5 Release lever

1.3 Fan Units

FIGURE 1-3 describes the various parts of the fan units.

Chapter 1 Sun Datacenter Switch 3456 Overview 1-3

FIGURE 1-3 Parts of the Fan Units

Figure Legend

1 Fabric card connector

2 Thumbscrews

3 Status LED

1.4 Chassis Management Controllers

FIGURE 1-4 describes the various parts of the chassis management controllers (CMCs).

1-4 Sun Datacenter Switch 3456 Service Manual • August 2008

FIGURE 1-4 Parts of the CMCs

Figure Legend

1 Release levers

2 Network management por ts

3 Serial management ports

4 Status LEDs

TABLE 1-1 lists the IPMB addresses for the CMCs.

TABLE 1-1 IPMB Addresses for the CMCs

Component IPMB Address

CMC0 10

CMC1 12

1.5 Fabric Cards

FIGURE 1-5 describes the various parts of the fabric cards.

Chapter 1 Sun Datacenter Switch 3456 Overview 1-5

FIGURE 1-5 Parts of the Fabric Cards

Figure Legend

1 iTRAC connectors

2 Retainers

3 Fans

4 Status LEDs

TABLE 1-2 lists the IPMB addresses for the fabric cards.

TABLE 1-2 IPMB Addresses for Fabric Cards

Fabric card

IPMB

address

FC0 FC1 FC2 FC3 FC4 FC5 FC6 FC7 FC8 FC9 FC10 FC11 FC12 FC13 FC14 FC15 FC16 FC17

b2 b4 b6 b8 ba bc be c0 c2 c4 c6 c8 ca cc ce d0 d2 d4

1.6 Line Cards

FIGURE 1-6 describes the various parts of the line cards.

1-6 Sun Datacenter Switch 3456 Service Manual • August 2008

FIGURE 1-6 Parts of the Line Cards

Figure Legend

1 Status LEDs

2 Link LEDs

3 iTRAC connectors

4 iPASS connectors

5 Retainers

TABLE 1-3 lists the IPMB addresses for the line cards.

TABLE 1-3 IPMB Addresses for Line Cards

Line card Address Line card Address Line card Address Line card Address

LC5 8c LC11 98 LC17 a4 LC23 b0

LC4 8a LC10 96 LC16 a2 LC22 ae

LC3 88 LC9 94 LC15 a0 LC21 ac

LC2 86 LC8 92 LC14 9e LC20 aa

LC1 84 LC7 90 LC13 9c LC19 a8

LC0 82 LC6 8e LC12 9a LC18 a6

Chapter 1 Sun Datacenter Switch 3456 Overview 1-7

1-8 Sun Datacenter Switch 3456 Service Manual • August 2008

CHAPTER

Chassis Service Procedures

This chapter describes procedures performed on the Sun Datacenter Switch 3456 chassis. Topic include:

■ Section 2.1 “Filler Panels” on page 2-1

■ Section 2.2 “Midplane Connector Pins” on page 2-6

■ Section 2.3 “Cable Management Hardware” on page 2-16

2.1 Filler Panels

▼ Removing the Filler Panel

1. Grasp the locking levers at each end of the filler panel.

FIGURE 2-1.

See

2-1

FIGURE 2-1 Turning Filler Panel Lock Levers

2. Turn the levers counter-clockwise to the unlock position.

FIGURE 2-1.

See

3. Lift the filler panel away from the chassis by the handles.

See

FIGURE 2-2.

2-2 Sun Datacenter Switch 3456 Service Manual • August 2008

FIGURE 2-2 Removing the Filler Panel

4. Repeat from Step 1 for all remaining filler panels.

▼ Installing the Filler Panel

After you remove fabric cards and line cards, you must install filler panels into the vacant slots to maintain proper airflow and thermal management.

1. Grasp the handles at each end of the filler panel and lift the panel to the vacant slot.

2. Turn the levers counter-clockwise to the unlock position.

See

FIGURE 2-3.

Chapter 2 Chassis Service Procedures 2-3

FIGURE 2-3 Turning Filler Panel Lock Levers Counter-Clockwise

3. Slide the filler panel into the chassis until it stops.

FIGURE 2-4.

See

FIGURE 2-4 Installing the Filler Panel

2-4 Sun Datacenter Switch 3456 Service Manual • August 2008

4. Turn the levers clockwise to the lock position.

FIGURE 2-5.

See

FIGURE 2-5 Turning Filler Panel Lock Levers Clockwise

5. Repeat from Step 1 for any remaining vacant slots.

2.2 Midplane Connector Pins

Caution – All midplane connector pin operations require the Sun Datacenter Switch

3456 to be fully powered off.

2.2.1 Tools

For inspecting pins, you need the following tools:

■ Pin gauge block

■ Bright flashlight

■ Magnifying glass or loupe

■ Notepad and pen

Chapter 2 Chassis Service Procedures 2-5

To repair or replace a pin, you need the following tools:

■ Molex pin replacement tool

■ Replacement pins (extracted from a spare iTRAC connector)

■ Head-mounted magnifier

■ Flashlight

Note – Another person to assist you can make the task much easier.

▼ Inspecting the Pins on the Rear Side of the

Midplane

Note – This procedure assumes that a fabric card has been removed from the

chassis.

The primary tool used to check the midplane pins is the pin gauge block. This tool effortlessly slides over the straight pins of an iTRAC connector that is in good condition. If there is any resistance, a pin is bent and requires further examination.

1. Using the pin gauge block, gently insert it into the iTRAC connector at the upper end of the open slot.

2. If you feel any resistance to insertion, note the connector number on the notepad.

See

FIGURE 2-6.

2-6 Sun Datacenter Switch 3456 Service Manual • August 2008

FIGURE 2-6 Fabric Card Connector Nomenclature

The connector number is located above each connector and is unique. For example, the upper-left connector number is FC-R23-F0. The lower-right connector number is FC-R0-F17. The connector number is decoded as follows:

■ FC – Fabric card

■ Ra – Row a, where a is 0 through 23

■ Fb – Frame b, where b is 0 through 17

The origin of this coordinate system is the lower-left corner, FC-R0-F0. In

FIGURE 2-6, the connector identified is FC-R13-F15.

3. Continue down to the next connector and repeat from Step 1.

4. After you have checked the bottom connector, visually inspect any connectors you noted previously.

Chapter 2 Chassis Service Procedures 2-7

5. Using the magnifying glass and the flashlight positioned above you, look at the individual pins of the connector.

Look at the pins straight on. See

FIGURE 2-7 Inspecting the Pins (Fabric Card)

FIGURE 2-7.

6. Look for any light reflection inconsistencies.

A bent pin reflects light differently, either brighter or darker than the surrounding pins.

Note – Do not look at the individual pins, rather look at all of the connector’s pins

as a whole. A bent pin will be apparent.

7. Identify the pin that needs repair or replacement and note this.

8. Look for any contaminants that might be present.

These contaminants might appear as light dots near the pins.

9. Move your head to the next suspect connector and inspect its pins, repeating

Step 5 through Step 8.

10. Continue in this manner for all suspect connectors.

11. If a pin needs repair or replacement, see “Removing Pins” on page 2-13.

2-8 Sun Datacenter Switch 3456 Service Manual • August 2008

▼ Inspecting the Pins on the Front Side of the

Midplane

Note – This procedure assumes that a line card has been removed from the chassis.

1. Using the pin gauge block, gently insert it into the iTRAC connector at the left end of the open slot.

2. If you feel any resistance to insertion, note the connector number on the notepad.

See

FIGURE 2-8.

Chapter 2 Chassis Service Procedures 2-9

FIGURE 2-8 Line Card Connector Nomenclature

The connector number is located above each connector and is unique. For example, the lower-left connector number is LC-C17-L0. The upper-right connector number is LC-C0-L23. The connector number is decoded as follows:

■ LC – Line card

■ Cc – Column c, where c is 0 through 17

■ Ld – Level d, where d is 0 through 23

The origin of this coordinate system is the lower right corner, LC-C0-L0. In

FIGURE 2-8, the connector identified is LC-C14-L10.

3. Continue right to the next connector and repeat from Step 1.

4. After you have checked the far right connector, visually inspect any connectors you noted previously.

2-10 Sun Datacenter Switch 3456 Service Manual • August 2008

5. Using the magnifying glass and the flashlight positioned to the side of you, look at the individual pins of the connector.

Look at the pins as straight on. See

FIGURE 2-9 Inspecting the Pins (Line Card)

FIGURE 2-9.

6. Look for any light reflection inconsistencies.

A bent pin reflects light differently, either brighter or darker than the surrounding pins.

Note – Do not look at the individual pins, rather look at all of the connector’s pins

as a whole. A bent pin will be apparent.

7. Identify the pin that needs repair or replacement and note this.

8. Look for any contaminants that might be present.

These contaminants might appear as light dots near the pins.

9. Move your head to the next suspect connector and inspect its pins, repeating

Step 5 through Step 8.

10. Continue in this manner for all suspect connectors.

11. If a pin needs repair or replacement, see “Removing Pins” on page 2-13.

Chapter 2 Chassis Service Procedures 2-11

+ 65 hidden pages

Sun Microsystems Datacenter 3456 820-4733-10 User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Contents

Preface

Sun Datacenter Switch 3456 Overview

1.1 Switch Chassis

1.2 Power Supplies

1.3 Fan Units

1.4 Chassis Management Controllers

1.5 Fabric Cards

1.6 Line Cards

Chassis Service Procedures

2.1 Filler Panels

▼ Removing the Filler Panel

▼ Installing the Filler Panel

2.2 Midplane Connector Pins

2.2.1 Tools