Cisco Systems 78-17434-01 User Manual

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

Product number: PID=78-17434-01 Rev. A0

This installation and configuration note describes how to install and configure the Cisco SFS 7008P switch. This publication contains these sections:

• Safety Overview, page 2

• Parts List, page 2

• Required Tools, Equipment, and Personnel, page 2

• Before You Install, page 3

• Installing the Cisco SFS 7008P Switch into a Rack, page 6

• Troubleshooting, page 20

• Translated Safety Warnings, page 21

• Obtaining Documentation, page 27

• Documentation Feedback, page 27

• Cisco Product Security Overview, page 28

• Obtaining Technical Assistance, page 29

• Obtaining Additional Publications and Information, page 31

Corporate Headquarters: Cisco Systems, Inc., 170 West Tasman Drive, San Jose, CA 95134-1706 USA

Safety Overview

Safety warnings appear throughout this publication in procedures that may harm you if performed incorrectly. A warning symbol precedes each warning statement.

Warning

Parts List

IMPORTANT SAFETY INSTRUCTIONS This warningsymbol means danger.You are in a situation that could cause bodily injury. Before you

work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device. Statement 1071

SAVE THESE INSTRUCTIONS

The following items are included in your SFS 7008 package:

• Chassis with TopSpin OS v2.5.0 preloaded

• 2 Serial Cable Kits

–

Serial Cable

–

Serial cable document

–

DB-9 connector male to RJ-45

–

DB-9 connector female to RJ-45

–

MX-9-1 roll-over cable

• Power cord for each power supply

• Bezel

• Parts bag

• Regulation and Compliance Statement

• 1 year warranty package (Cisco PN 83-1190-01)

Required Tools, Equipment, and Personnel

In addition to the accessories provided with the switch, you should have the following:

• A #2 Phillips screw driver

• Screws (not provided) to attach the mounting brackets to the rack

• Any associated mounting clips to secure the brackets to your rack (two for each rail of the rack)

• A rack with enough clearance for the Cisco SFS 7008P switch and cables.

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

78-17434-01 Rev. A0

• Three people to complete the installation (highly recommended). The Cisco SFS 7008P switch

chassis weighs up to 106 pounds fully loaded.

Before You Install

Before you begin the installation, do the following:

• Make sure that you have the right cables and sufficient ventilation.

• Open the Cisco SFS 7008P package and remove the chassis, rack brackets, parts bag, and

documentation from the box. Remove the packaging from around the chassis. After you have removed the straps, lift the box up and away from the switch and pallet. Do not cut the packaging, as you could damage the switch. Save the box and packaging.

• Prepare a management workstation, such as a PC or laptop running a terminal program with a

rollover M/F DB-9 serial cable (included).

Verifying Basic Hardware

Before You Install

If you have not purchased a fully loaded chassis with high-availability components installed, verify that you have the minimum components necessary to run the chassis. Specific components in the Cisco SFS 7008P switch (see Figure 1 and Figure 2) have interdependencies on other components.

Check for basic hardware by following these steps:

Step 1 Verify that you have at least onemanagement interface module installed in slot 15 at the back of the unit.

(See Figure 1.) Slot 15 is paired with the Fabric Controller in slot 11. (See Figure 2.)

78-17434-01 Rev. A0

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

Verifying Basic Hardware

Figure 1 Back of Unit

Power

plug

Management I.O

modules

Chassis ID

Power

plug

4x copper line interface modules

Main air exhaust

144988

Step 2 Verify that you have a Chassis ID module installed. (See Figure 1). Step 3 Verify that you have a Fabric Controller installed in the Core slot (see slot 11 in Figure 2). A second

Fabric Controller module (installed in slot 12) is not required for the chassis to function, but is required to have a non-blocking fabric.

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

78-17434-01 Rev. A0

Figure 2 Cisco SFS 7008P Switch Front ViewWithout Bezel

Status LED

Power Supply

identity LED

System

Status LED

Power supplies

Fan Fan

Node slot 1 Slot 9 Node slot 2

Core slot 1

Core slot 2 Node slot 3 Node slot 4

Fan tray LED

Verifying Basic Hardware

Slot 10 Slot 11 Slot 12

Slot 13 Slot 14

154497

Step 4 Verify that you have at least one power supply installed. (See Figure 2.) Step 5 Verify that you have at least one fan module installed. (See Figure 2.) Step 6 Verify that you have at least one fabric module installed in one of the node slots. The node slots are 9,

10, 13 and 14. (See Figure 2.)

Step 7 Verify that you have two LIMs installed for each fabric module in the node slots. See Figure 3 for an

example of the connections between the LIMs and the fabric modules in the node slots:

78-17434-01 Rev. A0

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

Installing the Cisco SFS 7008P Switch into a Rack

Figure 3 LIM and Fabric Module Connections

Fabric card in slot 9

Fabric card in slot 10

Fabric card in slot 13

LIM card in slot 1

LIM card in slot 2

LIM card in slot 3

LIM card in slot 4

LIM card in slot 5

LIM card in slot 6

LIM card in slot 7

Fabric card in slot 14

LIM card in slot 8

154550

Installing the Cisco SFS 7008P Switch into a Rack

This section describes how to install the Cisco SFS 7008Pswitch in an equipment rack. If you purchased the custom Cisco SFS 7008P switch shelf, follow the directions in the Cisco SFS 7008 InfiniBand Server Switch Hardware Installation Guide. To mount the Cisco SFS 7008P switch chassis in an industry-standard rack, follow these steps:

Step 1 Install a shelf for the chassis as shown in Figure 4, if you have not already done so.

The shelf must support the weight of the chassis, which can be up to 106 pounds if all possible components are installed.

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

78-17434-01 Rev. A0

Figure 4 Install Appropriate Shelf

Installing the Cisco SFS 7008P Switch into a Rack

144964

Step 2 Ground yourself using an approved ground wrist-strap. Step 3 Determine the direction that the switch will be installedand justified in the rack.

The direction that the switch will be justified determines which set of mounting brackets need to be removed in the next step.

Step 4 Remove one set of mounting brackets, as shown in Figure 5:

• If the switch will be justified towardthe bezel-side in the rack, then unscrew the mounting brackets

from the port-side of the switch.

• If the switch will be justified toward the port-side in the rack, then unscrew the mounting brackets

from the bezel-side of the switch.

78-17434-01 Rev. A0

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

Installing the Cisco SFS 7008P Switch into a Rack

Figure 5 Unscrew Extra Mounting Brackets from the Chassis

Step 5 Separate the mounting brackets from the chassis when all screws have been loosened. Step 6 With 2 or 3 people working together, lift the switch into the rack as shown in Figure 6.

144965

Figure 6 Chassis Set onto Shelf

144966

Step 7 Push the chassis into the rack until the mounting brackets are flush with the rails. Step 8 Use screws that fit your rack to attach both mounting brackets to the rack posts.

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

78-17434-01 Rev. A0

Installing Optional High-Availability Components

If you purchased a High-Availability (HA) package, install the following components:

• Redundant Power Supply; see the power supply locations in Figure 7. Install the second power

supply according to the directions in the Power Supply Module Installation Guide in the package.

Figure 7 Power Supplies and Fans Located in the Front of the Unit with the Bezel Removed

Power Supply Status LED

Power Supply

identity LED

System

Status LED

Fan Fan

Node slot 1 Slot 9 Node slot 2

Core slot 1

Core slot 2 Node slot 3 Node slot 4

Power supplies

Installing the Cisco SFS 7008P Switch into a Rack

Slot 10 Slot 11 Slot 12

Slot 13 Slot 14

154497

Fan tray LED

• Redundant Fan Tray; see the fan locations in Figure 7. Install the second fan according to the Fan

Module Installation Guide in the package.

• Redundant Management Interface Module; see the location of the Management I/O Modules in

Figure 8. Install the second I/O Module according to the directions in the Management Interface

Module Installation Guide in the package.

78-17434-01 Rev. A0

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

Installing the Cisco SFS 7008P Switch into a Rack

Figure 8 Management I/O Modules Located in the Back of the Unit

Management I.O

Power

plug

Chassis ID

Power

plug

modules

4x copper line interface modules

Main air exhaust

144988

ConfiguringandConnectingHostChannelExpansionCardforIBMBladeCenter

This section briefly describes configuring your InfiniBand host. Download the configuration software and an integrated suite of drivers including IPoIB, SRP, SDP, uDAPL and MPI from the Cisco Software Center at this URL:

http://www.cisco.com/public/sw-center

Install the Host Channel Expansion Card for IBM BladeCenter by doing the following:

Step 1 Power down the host system following proper grounding procedures. Step 2 Unpack the HCA module and insert it into an open 64-bit PCI or PCI-X slot. Step 3 Power on the host system.

Installing Host Channel Expansion Card Linux Drivers

Make sure your Linux kernel is supported. See www.cisco.com for a list of supported versions.

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

78-17434-01 Rev. A0

Installing the Cisco SFS 7008P Switch into a Rack

To install the Host Channel Expansion Card for IBM BladeCenter drivers after you have installed the HCA Expansion Card, follow these steps:

Step 1 Locate your downloaded software direcory. Step 2 Open a terminal or graphic window and access the top-level directory. Step 3 Execute the command ./tsinstall in a terminal window.

# cdmount ./tsinstall

This program automatically detects the available kernel and installs the appropriate drivers as RPM packages. By default, the files are saved in the /usr/local/cisco directory.

Step 4 Configure the IPoIB driver by assigning an IP address to interface ib0 and/or ib1, which corresponds to

the two HCA ports. IPoIB drivers load when the interfaces are used for the first time. To assign ib interfaces, enter ifconfig ib# ip address netmask netmask.

This example configures one HCA:

# ifconfig ib0 192.168.0.1 netmask 255.255.255.0 # # ifconfig ib1 192.168.1.1 netmask 255.255.255.0 #

This example configures two HCAs:

# ifconfig ib0 192.168.0.1 netmask 255.255.255.0 # # ifconfig ib1 192.168.1.1 netmask 255.255.255.0 # # ifconfig ib2 192.168.2.1 netmask 255.255.255.0

# ifconfig ib3 192.168.3.1 netmask 255.255.255.0 #

Step 5 Merge ib0 and ib1 for port-to-port redundancy, if desired. Step 6 (Optional) Configure MPI by following these steps:

• Make sure that your connections to the HCA uses port 1 of the first HCA. MPI requires the use of

HCA 1, port 1.

• Set up rsh or ssh on two nodes so that you can run commands between a remote and local node

without entering a login or password. This is required to use the MPI protocol.

• Either add paths to your environment PATH:

/usr/local/cisco/mpi/mpich/bin /usr/local/cisco/bin

Add directories to your PATH variable. For example:

export PATH=$PATH:/usr/local/cisco

Step 7 Configure all other drivers, including SDP.

78-17434-01 Rev. A0

Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note

+ 23 hidden pages