Cisco Systems SFS 7012 User Manual

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Corporate Headquarters

Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000

800 553-NETS (6387)

Fax: 408 526-4100

Text Part Number: OL-8787-05

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The following inform ation is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense.

The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not installed in accordance with Cisco’s installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable protection against such interference in a residential installation. However, there is no guarantee that interference will not occur in a particular installation.

Modifying the equipment without Cisco’s written authorization may result in the equipment no longer complying with FCC requirements for Class A or Class B digital devices. In that event, your right to use the equipment may be limited by FCC regulations, and you may be required to correct any interference to radio or television communications at your own expense.

You can determine whether your equipment is causing interference by turning it off. If the interference stops, it was probably caused by the Cisco equipment or one of its peripheral devices. If the equipment causes interference to radio or television reception, try to correct the interference by using one or more of the following measures:

• Turn the television or radio antenna until the interference stops.

• Move the equipment to one side or the other of the television or radio.

• Move the equipment farther away from the television or radio.

• Plug the equipment into an outlet that is on a different circuit from the television or radio. (That is, make certain the equipment and the television or radio are on circuits controlled by different circuit breakers or fuses.)

Modifications to this product not authorized by Cisco Systems, Inc. could void the FCC approval and negate your authority to operate the product.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR

IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

CCVP, the Cisco logo, and Welcome to the Human Network are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco Systems, Cisco Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys, MeetingPlace, MGX, Networkers, Networking Academy, Network Registrar, PIX, ProConnect, ScriptShare, SMARTnet, StackWise, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0711R)

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Inc.; and Access Registrar, Aironet, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS,

Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me

New and Changed Information v

Preface vii

Audience vii

Organization vii

Conventions viii

New and Changed Information

The Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide applies to the SFS 7012 Release

3.1 or later.

Table 1 lists the new and changed features available with each supported SFS 7012 release.

Ta b l e 1 Documented Features for the Cisco SFS 7012 InfiniBand Server Switch Hardware

Users Guide

Feature Description

Initial release of the Cisco SFS 7012

InfiniBand Server Switch Hardware

Users Guide

Redundant Management

Double Data Rate (DDR) capabilities

Added redundant management information.

Added DDR information

Changed in Release

3.3 Installing the Spine

3.4 Product Overview,

Where Documented

and Leaf Modules, page 1-17

Updating Management Spine IP Addresses in a Redundant Management Configuration, page 1-25

page 1-1

Port Statistics Field Descriptions, page 2-33

OL-8787-05

CLI command ismPortSetSpeed in

the section

IbSwitchInfo, page B-55 of

Appendix B: Command Line Interface.

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Table 1 Documented Features for the Cisco SFS 7012 InfiniBand Server Switch Hardware

Users Guide (continued)

Changed in

Feature Description

User authentication via command line

User login and password required to

access the SFS 7024 interface (CLI), serial console and Chassis Viewer GUI

Release Where Documented

4.1.1.1.11 LDAP Configuration,

page 2-18

HTTP/CLI Session Configuration, page 2-19

Accessing the CLI, page B-3

Added the following CLI commands:

• userAdd

• userAdd allows multiple user

accounts to be created.

• userRem allows user accounts to be

removed.

4.1.1.1.11 Refer to the CLI command group

General, page B-4

• userRem

• userListShow

• userListShow allows the list of

user accounts to be displayed.

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-05

Preface

This preface describes the audience, organization, and conventions of the Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide . It also provides information on how to obtain related documentation.

Audience

The intended audience for this document are network administrators responsible for configuring and operating network equipment. It assumes a basic working knowledge of:

• Local Area Networks (LANs)

• Ethernet concepts

• Simple Network Management Protocol (SNMP)

• InfiniBand

Organization

OL-8787-05

This guide is organized as follows:

Chapter Title Description

Chapter 1 Installation Task-oriented information for installing the Cisco SFS 7012™

Chapter 2 Operations and

Administration

Appendix A Technical

Specifications

Appendix B Command Line

Interface

Appendix C Troubleshooting Troubleshooting symptoms and resolutions for the SFS 7012

Task-oriented information for configuring and monitoring the SFS 7012

SFS 7012 technical specifications

Reference information for the SFS 7012 command line interface (CLI)

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

vii

Conventions

This document uses the following conventions for notes, cautions, and safety warnings.

Notes and Cautions contain important information that you should be aware of.

Note Means reader take note. Notes contain helpful suggestions or references to material not

covered in the publication.

Caution Means reader be careful. You are capable of doing something that might result in equipment damage or

loss of data.

Safety warnings appear throughout this publication in procedures that, if performed incorrectly, may harm you. A warning symbol precedes each warning statement.

Warning

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. To see translations of the warnings that appear in this publication, refer to the Regulatory Compliance and Safety Information document that accompanied this device.

Obtaining Documentation, Obtaining Support, and Security Guidelines

viii

For information on obtaining documentation, obtaining support, providing documentation feedback, security guidelines, and also recommended aliases and general Cisco

What’s

New in Cisco Product Documentation, which also lists all new and revised Cisco technical

documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

documents, see the monthly

OL-8787-05

CHAPTER

Installation

This chapter describes how to install the Cisco SFS 7012™ and its components, and it includes the following information:

• Planning the Installation, page 1-7

• Installation Tasks, page 1-12

• Hot Swapping Components, page 1-30

Note Before you install, operate, or service the system, read the Regulatory Compliance and Safety

Information for the Cisco Server Fabric Switches: 7000P, 7000D, 7008P, 7012, 7024, and 3012R for important safety information.

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-1

Warning

IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device.

Statement 1071

SAVE THESE INSTRUCTIONS

Warning

This unit is intended for installation in restricted access areas. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security.

Statement 1017

Warning

Only trained and qualified personnel should be allowed to install, replace, or service this equipment.

Statement 1030

Warning

A readily accessible two-poled disconnect device must be incorporated in the fixed wiring.

1022

Statement 1019—Main Disconnecting Device

Warning

Waarschuwing

The plug-socket combination must be accessible at all times, because it serves as the main disconnecting device.

Statement 1019

De combinatie van de stekker en het elektrisch contactpunt moet te allen tijde toegankelijk zijn omdat deze het hoofdmechanisme vormt voor verbreking van de aansluiting.

Statement

1-2

Varoitus

Attention

Pistoke/liitinkohta toimii pääkatkaisumekanismina. Pääsy siihen on pidettävä aina esteettömänä.

La combinaison de prise de courant doit être accessible à tout moment parce qu'elle fait office de système principal de déconnexion.

Warnung

Der Netzkabelanschluß am Gerät muß jederzeit zugänglich sein, weil er als primäre Ausschaltvorrichtung dient.

Avvertenza

Il gruppo spina-presa deve essere sempre accessibile, poiché viene utilizzato come dispositivo di scollegamento principale.

Advarsel

Kombinasjonen støpsel/uttak må alltid være tilgjengelig ettersom den fungerer som hovedfrakoplingsenhet.

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-05

Aviso

A combinação ficha-tomada deverá ser sempre acessível, porque funciona como interruptor principal.

¡Advertencia!

Varning!

El conjunto de clavija y toma ha de encontrarse siempre accesible ya que hace las veces de dispositivo de desconexión principal.

Man måste alltid kunna komma åt stickproppen i uttaget, eftersom denna koppling utgör den huvudsakliga frånkopplingsanordningen.

Statement 1045—Short-circuit Protection

Warning

Waarschuwing

Varoitus

Attention

Warnung

Avvertenza

Advarsel

This product requires short-circuit (overcurrent) protection, to be provided as part of the building installation. Install only in accordance with national and local wiring regulations.

Voor dit product moet kortsluitbeveiliging (overstroombeveiliging) deel uitmaken van de installatie in het gebouw. De installatie moet voldoen aan de nationale en lokale bedradingvoorschriften.

Tämä tuote vaatii suojauksen oikosulkuja (ylivirtaa) vastaan osana asennusta rakennukseen. Asenna ainoastaan kansallisten ja paikallisten johdotussäännösten mukaisesti.

La protection de ce produit contre les courts-circuits (surtensions) doit être assurée par la configuration électrique du bâtiment. Vérifiez que l'installation a lieu uniquement en conformité avec les normes de câblage en vigueur au niveau national et local.

Für dieses Produkt ist eine Kurzschlußsicherung (Überstromsicherung) erforderlich, die als Teil der Gebäudeinstallation zur Verfügung gestellt wird. Die Installation sollte nur in Übereinstimmung mit den nationalen und regionalen Vorschriften zur Verkabelung erfolgen.

Questo prodotto richiede una protezione contro i cortocircuiti, da fornirsi come parte integrante delle dotazioni presenti nell’edificio. Effettuare l’installazione rispettando le Norme CEI pertinenti.

Dette produktet krever beskyttelse mot kortslutninger (overspenninger) som en del av installasjonen. Bare installer utstyret i henhold til nasjonale og lokale krav til ledningsnett.

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-3

Aviso

Este produto requer proteção contra curto-circuitos (sobreintensidade de corrente), que deve estar instalada nos edifícios. Instale apenas de acordo com as normas de instalação elétrica nacionais

locais.

Advertencia

Varning!

Este producto necesita estar conectado a la protección frente a cortacircuitos (sobretensiones) que exista en el edificio. Instálelo únicamente en conformidad con las regulaciones sobre cableado, tanto locales como nacionales, a las que se tenga que atener.

Denna produkt kräver att kortslutningsskydd (överström) tillhandahålles som en del av byggnadsinstallationen. Installera bara i enlighet med nationella och lokala kabeldragningsbestämmelser.

Aviso

Advarsel

Este produto requer uma proteção contra curto-circuito (sobrecorrente) que deve fazer parte da instalação do edifício. Faça a instalação somente de acordo com as regulamentações de cabeamento nacionais e locais.

Dette produkt kræver beskyttelse mod kortslutning (overstrøm). Dette skal være en del elinstallationen i bygningen. Installation skal ske i overensstemmelse med nationale og lokale ledningsregler.

1-4

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-05

Statement 1074—Comply with Local and National Electrical Codes

Warning

Waarschuwing

Varoitus

Attention

Warnung

Avvertenza

Advarsel

Aviso

¡Advertencia!

Varning!

Installation of the equipment must comply with local and national electrical codes.

Bij installatie van de apparatuur moet worden voldaan aan de lokale en nationale elektriciteitsvoorschriften.

Laitteisto tulee asentaa paikallisten ja kansallisten sähkömääräysten mukaisesti.

L'équipement doit être installé conformément aux normes électriques nationales et locales.

Die Installation der Geräte muss den Sicherheitsstandards entsprechen.

L'installazione dell'impianto deve essere conforme ai codici elettrici locali e nazionali.

Installasjon av utstyret må samsvare med lokale og nasjonale elektrisitetsforskrifter.

A instalação do equipamento tem de estar em conformidade com os códigos eléctricos locais e nacionais.

La instalación del equipo debe cumplir con las normativas de electricidad locales y nacionales.

Installation av utrustningen måste ske i enlighet med gällande elinstallationsföreskrifter.

Statement 1074

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-5

Statement 1075—Hazardous Voltage or Energy Present on DC Power Terminals

Warning

Waarschuwing

Varoitus

Attention

Warnung

Avvertenza

Advarsel

Hazardous voltage or energy may be present on DC power terminals. Always replace cover when terminals are not in service. Be sure uninsulated conductors are not accessible when cover is in

Statement 1075

place.

Op DC-aansluitingspunten kunnen zich gevaarlijke voltages of energieën voordoen. Plaats altijd de afsluiting wanneer de aansluitingspunten niet worden gebruikt Zorg ervoor dat blootliggende contactpunten niet toegankelijk zijn wanneer de afsluiting is geplaatst.

Tasavirtaliittimissä saattaa olla huomattava jännite tai teho. Sulje suojus aina, kun liittimet eivät ole käytössä. Suojuksen ollessa suljettuna varmista, että kohde on suojattu eristämättömiltä johtimilta.

Le voltage ou l'énergie électrique des terminaux à courant continu peuvent être dangereux. Veillez à toujours replacer le couvercle lors les terminaux ne sont pas en service. Assurez-vous que les conducteurs non isolés ne sont pas accessibles lorsque le couvercle est en place.

In mit Gleichstrom betriebenen Terminals kann es zu gefählicher Spannung kommen. Die Terminals müssen abgedeckt werden, wenn sie nicht in Betrieb sind. Stellen Sie bei Benutzung der Abdeckung sicher, dass alle nicht isolierten, stromführenden Kabel abgedeckt sind.

I terminali di alimentazione DC potrebbero contenere voltaggio o energia pericolosi. Accertarsi di sostituire il coperchio ogni qualvolta i terminali non sono operativi. Accertarsi che i conduttori scoperti non siano accessibili quando il coperchio è inserito.

Det kan forekomme farlig spenning eller energi i likestrømsterminaler. Sett alltid dekselet på plass når terminalene ikke er i bruk. Kontroller at uisolerte ledere ikke er tilgjengelige når dekselet er på plass.

1-6

Aviso

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Os terminais de corrente contínua podem fornecer tensão ou energia perigosa. Volte a colocar a tampa, sempre que os terminais não estiverem a ser utilizados. Certifique-se de que os condutores sem isolamento não estão acessíveis, quando a tampa estiver colocada.

OL-8787-05

¡Advertencia!

Puede haber energía o voltaje peligrosos en los terminales eléctricos de CC. Reemplace siempre la cubierta cuando no estén utilizándose los terminales. Asegúrese de que no haya acceso a conductores descubiertos cuando la cubierta esté colocada.

Varning!

Farlig spänning eller skadlig energi kan finnas i likströmsterminalerna. Sätt alltid tillbaka höljet när terminalerna inte används. Försäkra att inga oisolerade ledare kan nås när höljet sitter på plats.

Planning the Installation

Rack Specifications and Recommendations

The SFS 7012 switch is designed to be installed in an existing 19-inch equipment rack or server rack.

Note The SFS 7024 switch is designed for a four-post server cabinet. It is not designed for a two-post telco

cabinet.

Racks should conform to conventional standards. In the United States, use American National Standards Institute (ANSI)/Electronic Industries Association (EIA) standard ANSI/EIA-310-D-92, and International Electrotechnical Commission (IEC) 297

• Racks should meet the following mechanical recommendations:

–

Four-post, 19" rack to facilitate easy maintenance

–

Universal mounting rail hole pattern identified in IEC Standard 297

–

Mounting holes flush with the rails to accommodate the switch

• Use a rack grounding kit and a ground conductor that is carried back to earth or to another suitable

building ground. Ground the equipment rack to earth ground.

• Provide enough room to work on the equipment. Clear the work site of any unnecessary materials.

Make sure the equipment will have enough clearance for front and rear access.

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-7

Cable Requirements

Cable Distances

When planning the location of the switches, consider the distance limitations for signaling, EMI, and connector compatibility. It is recommended that the user does not exceed specified transmission rate and distance limits.

Note Building and electrical codes vary depending on the location. Comply with all code specifications when

planning the site and installing cable.

When running cables to the equipment, consider the following:

• Do not run cables where they can be stepped on or rolled over.

• Be sure cables are intact with no cuts, bends, or nicks.

• If the user is making a cable, ensure that the cable is properly crimped.

• Provide proper strain relief for standard IB cables.

• Support cable using a cable manager mounted above connectors to avoid unnecessary weight on the

• Bundle cable using velcro straps to avoid injuring cables.

• Keep all ports and connectors free of dust.

• Untwisted Pair (UTP) cables can build up Electrostatic Discharge (ESD) charges when being pulled

cable bundles.

into a new installation. Before installing category 5 UTP cables, discharge ESD from the cable by plugging it into a port on a system that is not powered on.

• When required for safety and fire rating requirements, plenum-rated cable can be used. Check the

local building codes to determine when it is appropriate to use plenum-rated cable, or refer to IEC standard 850.

Uninterruptible Power Supply

Consider the following when selecting Uninterruptible Power Supply (UPS) equipment:

• The minimum amperage requirements for a UPS:

–

Calculate VA (Volt-Amps): Locate the voltage and amperage requirements for each piece of equipment (usually located on a sticker on the back or bottom of the equipment). Multiply the numbers together to get VA.

–

Add the VA from each piece of equipment together to find the total VA requirement. Then add 30% to determine the minimum amperage requirements for the UPS.

• Transition time (the time necessary for the UPS to transfer from utility power to full-load battery

power).

• The longest potential time period the UPS might be required to supply backup power.

• Whether or not the UPS unit also provides online protection.

1-8

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-05

Installation Tasks Checklist

To perform the actual switch installation, the site implementation engineer must perform the following tasks, which are detailed in this section.

Caution Be sure to review the Safety Information on page 1-9 before starting the installation and during the

installation process. Check the installation site to verify the installation of cabinet power feeds, rails, and grounding.

Step 1 Unpack the equipment and inspect for any shipping damage. Any shipping damage should be reported

to the shipping company.

Step 2 Verify that the equipment shipped matches the packing list.

Step 3 Mark the rack and install the mounting rails.

Step 4 Physically install the switch in the rack.

Safety Information

The following safety guidelines are provided to ensure both personal safety for the user and to protect the system from potential damage. These precautions cover the following categories:

• Precautions for Rack-Mountable Products

• Protecting Against Electrostatic Discharge

• Electrical Safety Precautions

Precautions fit into one of three categories:

Note A textual callout designed to emphasize:

–

Tasks of particular importance.

–

Tips and reminders to maximize the use of the equipment.

Caution Potential for damage to system equipment. Damage to the system caused by the user may have potential

warranty implications.

Warning

Potential for personal injury.

Precautions for Rack-Mountable Products

Warning

Installing system components in a rack without the front and side stabilizers installed could cause the rack to tip over. Therefore, always install the stabilizers before installing components in the rack.

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-9

Warning

After installing system components in a rack never pull more than one component at one time out of the rack on its slide assemblies. The weight of more than one extended component could cause the rack to tip over.

Warning

Do not step on or stand on any component when servicing other components in a rack.

The chassis, when fully populated with leaf modules, spine modules, power and fan supplies, is very heavy (approximately 100 lbs.). It is recommended that a lifting device be used to handle a fully loaded chassis.

Caution Always load the rack from the bottom up, loading the heaviest item first.

Caution Make sure the rack is level and stable before extending any component from the rack.

Note Ensure that proper airflow is provided to components of the rack.

Protecting Against Electrostatic Discharge

Caution Use a grounded wrist strap designed to prevent static discharge.

Caution Static electricity can harm delicate components inside the system. To prevent ESD damage, users need

discharge any static electricity from their bodies before touching any electronic components. Touching an unpainted metal surface will discharge static electricity.

Caution When transporting an ESD sensitive component, first place it in an antistatic container or packaging.

Electrical Safety Precautions

Warning

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Do not work alone when working with high voltage components.

This unit may have more than one power cord. To reduce the risk of electrical shock, disconnect both cords before servicing the unit.

1-10

OL-8787-05

Warning

To avoid potential electrical shock, operate this unit only when the cover is in place.

Warning

Caution Do not overload the power supply branch circuit providing power to the rack. The total rack load should

To avoid potential electrical shock, use only a grounded (three wire) electrical outlet.

Keep objects that might damage this unit and liquids that might spill clear from this unit. Liquids and foreign objects that come into contact with voltage points could create the risk of fire or electrical shock.

not exceed 80 percent of the branch circuit rating.

Caution Keep power cord and connection cables clear of obstructions that might cause damage.

Caution Do not attempt to service the unit yourself. The first course of action is to contact Technical Support.

Note Unplug this unit from the electrical outlet and refer servicing to a qualified service center if any of the

following conditions occur:

–

The power cord is damaged or frayed.

–

The unit has been dropped or the case has been damaged.

–

The unit has been exposed to any liquids.

–

The unit does not operate normally when all operating instructions have been followed.

–

The unit exhibits a distinct change in performance, indicating a need for service.

Tools, Equipment and Personnel Required

• An ESD wrist strap

• A #2 Phillips screwdriver

• Pen (felt-tip) to mark the mounting holes

• Three people to complete the installation (highly recommended). The SFS 7012 switch chassis

weighs up to 100 pounds fully loaded.

Check the Installation Site

The SFS 7012 switch is designed to be installed in an existing server cabinet (not a telco cabinet), where it can be mounted in a standard equipment rack. Mounting brackets are integrated with the switch.

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-11

Be sure of the following:

• The cabinet has a full earth ground to provide reliable grounding.

• There is enough room to work on the equipment.

• The equipment will have enough clearance for front and rear access.

• The IB cables can be accessed easily.

• Water or moisture cannot enter the switch.

• The ambient temperature stays between 50° - 113°F (5° - 45° C).

• Cabinet doors do not interfere with front-to-back air flow.

The cabinet should have its own power distribution (with switch). If the switch has two power supplies, it is suggested that a cabinet with dual power distribution units is used.

It is recommended that cabinet anti-tip devices are used. This is especially true if installing or removing an SFS 7012 switch in the upper half of the cabinet when the lower half is empty.

Unpack the Equipment

Warning

Step 1 Carefully open the box and unpack the SFS 7012 switch. The SFS 7012 is shipped fully populated and

Read all installation instructions before connecting the system to its power source.

should contain.

–

SFS 7012 Chassis

–

Leaf modules (up to 12)

–

Leaf module blanks (up to 11). These blanks populate unused leaf module slots to help maintain the thermal integrity of the chassis.

–

Spine modules (up to 3)

–

Spine module blanks (up to 2). These blanks populate unused spine module slots to help maintain the thermal integrity of the chassis.

–

Power Supplies: up to six (6)

–

Power supply blanks (up to 5). These blanks populate unused power supply slots to help maintain the thermal integrity of the chassis.

–

Fans (4)

–

Mounting hardware kits

Step 2 Inspect the equipment for any shipping damage and report any problems to the shipping company.

Step 3 Verify that the equipment serial numbers match those on the packing slip.

Step 4 Resolve any issues with incorrect serial numbers or missing/incorrect parts before installing the

equipment.

Installation Tasks

The following is an overview of the installation tasks detailed in this section:

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-12

OL-8787-05

Step 1 If applicable, remove the doors of the rack.

Step 2 Mark the rack, allowing 7 U (12.25 inches) of vertical space to install the SFS 7012 switch.

Step 3 Install the support rails.

Step 4 Rack mount the switch.

Step 5 If applicable, replace the doors of the rack.

Mounting Kit

The mounting kit hardware contains all of the necessary parts for installing and mounting the SFS 7012 switch into a rack. These kits are intended for use in cabinets with a depth ranging from 28 - 34 inches.

Note If the rack is less than 28”, or more than 34” deep, instead of using the rails, a support shelf able to

support 100 lbs is required.

Mounting Hardware Kit Contents:

• Kit Mounting Rails containing:

–

One pair (left and right) of bottom support rail assemblies

–

One pair of lower rear mounting brackets

–

Two heyclips

• Kit Mounting Hardware 3/8” Square Hole containing:

–

Eighteen 10/32” x 0.375” pan-head Phillips screws

–

Fourteen caged nut adapters for square-holed racks

Mark the Rack

Allow 7U (12.25 inches) of vertical space to install the SFS 7012 switch in the rack.

Step 1 Determine the location on the rack of the bottom of the switch.

Step 2 Mark the upper and lower mounting positions on the vertical rails on the front of the rack.

Step 3 Mark the upper and lower mounting positions on the vertical rails on the back of the rack.

Install the Rails in the Rack

Note The front flange (chassis fan side) of the rail has 3 holes. The back flange of the rail (chassis leaf module

side) has two holes, which correspond to the bottom two holes on the front flange.

Step 1 Install the caged nuts into the 2 back holes (chassis leaf module side) and the top and bottom holes

(chassis fan side) in the front of the rack.

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-13

Note The 2 front holes should match up with the top and bottom holes of the rail front flange.

All holes should correspond to the rail mounting positions (i.e., the holes marked with pen or tape).

Step 2 Fasten the rail back flange (chassis leaf module side) to the rack by installing two screws into the rail

and rack.

Figure 1 Rail Back Flange

Install 2 screws to

top and bottom holes

Step 3 Fasten the rail front flange (chassis fan side) to the rack by installing one screw into the top hole of each

rail.

Figure 2 Rail Front Flange

Install the second rail on the other side of the rack by repeating steps 1 through 3.

Rack-Mount the Switch

Caution Because of its size and weight, it is recommended that either a lifting device or three (3) people install

the SFS 7012.

Install 2 screws to

top and bottom holes

1-14

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-05

To install the switch into the rack, perform the following steps:

Step 1 Clear the area of any unnecessary materials.

Step 2 Attach the clip of the ESD wristband (strap) to bare metal on the cabinet. Put the wristband around one

wrist with the metal button against the skin.

Step 3 Lift the switch and from the front of the cabinet, slide it onto the rails. The fans and power supplies are

on the front of the chassis; leaf modules are to the rear.

Caution Never lift the switch with the handles on the spine modules, leaf modules, power supplies, and fan trays.

These handles are not designed to support the weight of the SFS 7012.

Step 4 Tighten the screw on the bottom of each side of the chassis into the corresponding bottom hole of the

front flange of each rail.

Figure 3 Secure Chassis to Rail

Install 1 screw into

bottom hole on each rail

Step 5 On each side of the chassis, install a screw and caged nut into the holes above and below the handle.

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-15

Figure 4 Secure Chassis to Rack

Install 1 screw and

caged nut into each

hole above and below

the handle

Step 6 Using two screws, install the lower mounting bracket to the rail assembly and chassis as shown in

Figure 5.

Figure 5 Install the Lower Mounting Bracket and Heyclip

Install the

heyclip here

Step 7 Install the heyclip to the rail assembly.

Step 8 If applicable, reinstall the chassis fascia(s).

Install the lower rear

mounting bracket to the rail

assembly and chassis

Installing the Switch Face Plate

To install the switch face plate(s):

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-16

OL-8787-05

Step 1 On the switch fan side, insert the notches on the top of the fascia into the two slots on the chassis frame.

Snap the bottom of the faceplate in place.

Removing a Module or Blank

The handles are self-locking. To unlock, push up on the handles to disengage from the lock notch. Then gently pull the handles out and slide the module out of the slot.

Note If removing, but not replacing a module, remember to replace with a module blank. All slots must be

either populated with a module or have blanks for EMI and thermal integrity.

Installing the Spine and Leaf Modules

Note The purchased configuration for the SFS 7012 is shipped fully populated. Follow these steps when it

becomes necessary to install or replace spine modules and leaf modules.

Step 1 Remove the necessary spine modules, spine module blanks, leaf modules, and leaf module blanks. For

detailed instructions, please refer to the section

Removing a Module or Blank, page 1-17.

Note If the user is only adding additional modules, remove only the blank(s) for the slot(s) to be populated.

These will not be replaced.

Step 1 When placing the spine modules and leaf modules into chassis slots, the following recommendations

apply:

a. Spine Modules — It is recommended that the spine module(s) be installed into:

–

Slot 1 for managed. For redundant management, populate slots 1 and 2 with management-capable spines.

–

Slot 3 for unmanaged.

Refer to Figure 1-6 below:

OL-8787-05

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-17

Figure 1-6 SFS 7012 Chassis — Spine Module Slot Numbering

b. Leaf Modules— Leaf modules should be populated beginning with slot 1, then slot 2, then slots 3

through 12 respectively. Refer to

Figure 1-7 SFS 7012 Chassis — Leaf Module Slot Numbering

Figure 1-7:

Leaf 11

Leaf 9

Leaf 7

Leaf 5

Leaf 3

Leaf 1 Leaf 2

Step 2 To install a module or filler, hold it so that the ejector handles are on the bottom.

Step 3 Pull the handles out to extend them. Slide the module into the appropriate slot of the chassis until it

makes contact with the backplane. As the module seats in the chassis, the handles will begin to close.

Step 4 Push the handles in to fully close.

Note Be sure that all cards are fully inserted in their respective chassis slots, and that the handles are in the

locked position. This prevents accidental removal, provides proper grounding for the system, and helps to seat the bus connectors in the backplane receptacles.

Connect Equipment to the Ports and Power On the System

Leaf 12

Leaf 10

Leaf 8

Leaf 6

Leaf 4

1-18

Note Before performing the tasks in this section, take a few minutes to review Figure 1-8 and Figure 1-9

below. It is important to understand the slot numbering for the leaf modules and spine modules (and the corresponding spine RJ-45 ports). It is also important to understand the slot numbering for the fans and power supplies (and the corresponding AC power inlets for the power supplies).

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-05

Figure 1-8 SFS 7012 Chassis Front View

Figure 1-9 SFS 7012 Chassis Back View

Leaf 11

Leaf 9

Leaf 7

Leaf 5

Leaf 3

Leaf 1

AC power inlet for power

supplies 1 to 3

RJ-45 for Spine 1 RJ-45 for Spine 2

Step 1 Connect a Category 5 or 6 (Cat 5/6) Ethernet cable to one of the RJ-45 connectors on the SFS 7012.

Connect the other end of the Cat 5/6 to the OOB LAN workstation. Refer to

Leaf 12

Leaf 10

Leaf 8

Leaf 6

Leaf 4

Leaf 2

AC power inlet for power

supplies 4 to 6

Figure 1-9 for which RJ-45

connector(s) to use:

Step 2 Connect the switch to IB-enabled hosts using 4X-to-4X IB cables.

Note Make sure all cables latch securely into the corresponding port connectors. If the IB cable connector is

not properly oriented to fit onto the port receptacle (i.e., while attempting to insert the cable in the port), do not twist the connector to achieve the correct orientation. Instead, reach back a few feet on the cable, and twist the bulk cable to allow the connector to rotate to the proper orientation. Doing this prevents all of the rotational forces from acting right at the connector terminations.

OL-8787-05

Caution It is important to provide strain relief for the IB cable connector.

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

1-19

Note When handling IB connectors, make certain to remove the connectors by pulling on the center of the

Connecting Power

lanyard only as shown in

Figure 1-10 below. Pulling abruptly on the lanyard, or pulling on only one side

of the lanyard will prevent the latch/unlatch operation from occurring, and could damage or break the lanyard.

Figure 1-10 4X IB Cable Connector

Step 1 Provide strain relief for the power cable(s) by feeding them into the heyclips on the lower rail assemblies.

Note Be certain that the power cords are firmly seated into the SFS 7012 AC power inlets. Depending upon

the purchased configuration, refer to Figure 1-9 to determine the correct AC power inlet to use.

Step 2 If necessary, replace the faceplates over the switch fans.

Step 3 Connect the power cables to an AC power outlet.

Step 4 When the SFS 7012 switch is plugged into an AC power outlet:

a. The system powers up.

b. The fans start.

c. The system performs a power-on self test (POST).

Step 5 The switch, power supply, and fan LEDs light up.

Bringing Up the System For the First Time

Start-up Procedures

Step 1 Power up the SFS 7012.

Step 2 From its flash image on the CMU spine module, the switch begins its boot process.

1-20

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-05

+ 146 hidden pages

Cisco Systems SFS 7012 User Manual

New and Changed Information

Preface

Audience

Organization

Conventions

Related Documentation

Obtaining Documentation, Obtaining Support, and Security Guidelines

Installation

Planning the Installation

Rack Specifications and Recommendations

Cable Requirements

Uninterruptible Power Supply

Installation Tasks Checklist

Safety Information

Tools, Equipment and Personnel Required

Check the Installation Site

Unpack the Equipment

Installation Tasks

Mounting Kit

Mark the Rack

Install the Rails in the Rack

Rack-Mount the Switch

Installing the Switch Face Plate

Removing a Module or Blank

Installing the Spine and Leaf Modules

Connect Equipment to the Ports and Power On the System

Bringing Up the System For the First Time