Sun Oracle Sun Network, Sun Network QDR InfiniBand Gateway Switch Getting Started Manual

Page 1

Sun Network QDR InfiniBand Gateway Switch

Getting Started Guide

This document describes the minimum steps you must perform to bring the Sun Network QDR InfiniBand Gateway Switch from Oracle online for the first time.

Detailed installation information about all tasks in this document can be found in the Sun Network QDR

InfiniBand Gateway Switch Installation Guide or Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.0. For information about these documents, see “Related Documentation” on page 6.

Note – You must read the installation information before installing your switch.

Contents

1 Gateway switch 2 Front mounting bracket, long 3 Front mounting brackets, short 4 C-bracket 5 Documentation 6 Hardware 7 Extenders 8 Long rails

9 Cable management cover 10 Cable management assembly 11 Attachment brackets 12 Attachment plates

Safety and Compliance Information

Before performing an installation, refer to the following documents for safety information:

■ Important Safety Information for Sun Hardware Systems – Printed document included in the shipping kit.

■ Sun Network QDR InfiniBand Gateway Switch Safety and Compliance Guide – Available at the Oracle

documentation website.

Page 2

▼ Mount the Gateway Chassis in a Rack

Caution – The front of the gateway chassis (fan end) intakes from the cold aisle, and the rear of the

gateway chassis (connector end) exhausts to the hot aisle. This flow direction requires you to install the gateway in an orientation that is opposite of what you might assume.

1. Open the rack doors.

2. Assemble the cable management extenders. Perform Steps 2a to 2d for each extender.

a. Slide the attachment bracket over the extender, so that the tab on the bracket is opposite the flange on

the extender.

The open end of the tab is toward the flange. The flat end of the tab is toward the rear of the extender.

b. Place the attachment plate on the flange side of the extender, opposite the attachment bracket.

c. Use two screws to sandwich the bracket and plate to the extender, in the position farthest from the

flange.

d. Using a No. 2 Phillips screwdriver, tighten the two screws.

3. Attach the cable management extenders and long rails to the rear of the rack. Perform Steps 3a to 3c for each side of the rack.

a. Place the long rail at the mounting location on the rear rack post.

b. Butt the flange of the extender to the flange of the long rail.

c. Secure the assembly to the post with two captive nuts and two screws.

4. Attach the long front brackets to the front of the gateway with four screws on each side and the C- brackets to the rear of the gateway with four screws on each side.

5. Route the power cords through the rack with the female end at the front of the rack where the switch will install.

Ensure that there is 24 in. (610 mm) of power cord slack at the front of the rack to provide an adequate service loop when you remove the gateway from the rack.

6. Carefully lift the gateway into the front of the rack and slide the rear of the gateway onto the long rails.

Ensure that the ends of the long rails slide into the C-brackets at the rear of the gateway chassis and that the power cords lie in the cut-outs of the long front mounting brackets.

7. Mount the front of the gateway chassis to the rack posts with two captive nuts and two screws at each side.

Tighten the screws securely.

8. Install the cable management bracket to the tabs of the attachment brackets, tightening the thumbscrews on each side of the cable management bracket.

Page 3

▼ Connect the Management Cables

1. Route the management cables to the gateway so that they do not interfere with other cables, with servicing the gateway, or with other systems.

2. Connect an Ethernet cable between the gateway NET0 port and the network that is configured with a DHCP server.

Connections to the management controller are made through DHCP.

3. Configure your DHCP server to provide a host name and address to the gateway.

Note – If a DHCP server is unavailable, the gateway will default to static IP 169.254.0.36/16. Alternatively,

you can connect a USB-to-serial adapter cable between the gateway’s USB port and a terminal device. This connection also provides communication with the management controller. The terminal device must be configured 115200 baud, 8 bit, no parity, 1 stop bit.

▼ Power On the Gateway

1. Route the power cords so that they do not interfere with other cables, with servicing the gateway, or with other systems.

2. Plug the power cords into the receptacles at the rear of the gateway chassis and into separate power sources.

Note – To provide redundancy, connect each power supply to a separate power source. The system can operate

with only one power connection, but there is no redundancy in that case.

The AC LED on each power supply lights green. In a moment, the OK LED lights green, power is supplied, the fans spin up, and the management controller boots.

■ If the AC LED does not light, there is something wrong with supplied power.

■ If the OK LED does not light, there is something wrong with the power supply.

■ If the Attention LED on a power supply lights amber, there is a fault in the power supply.

■ If the Attention LED on a fan lights amber, there is a fault with that fan.

3. Begin network communication with the management controller.

% ssh -l ilom-admin gateway_name ilom-admin@gateway_name’s password: ilom-admin

where gateway_name is the host name provided by DHCP. The gateway-name might be the IP address.

If you do not see this output or prompt, there is a problem with the DHCP configuration, network communication, or the management controller.

Page 4

4. (Optional) Begin serial communication with the management controller.

... CentOS release 5.2 (Final) Kernel 2.6.27.13-nm2 on an i686

gateway_name login:ilom-admin Password: ilom-admin

If you do not see this output or prompt, there is a problem with the serial configuration, the USB-to-serial adapter, or the management controller.

5. Change to the restricted Linux shell.

-> show /SYS/Fabric_Mgmt Fabman@gateway_name->

6. Check the status of the management controller, power supplies, fans, and switch chip. Verify that the gateway voltage and temperature values are within specification.

Fabman@gateway_name-> showunhealthy Fabman@gateway_name-> env_test

7. Look for OK in the showunhealthy command output and PASSED in the last line of the env_test command output.

An unfavorable output from these commands means a hardware fault with that particular component. A voltage or temperature deviating more than 10% from the provided specification means a problem with the respective component.

▼ Start the Subnet Manager

Note – If your InfiniBand fabric is already active from a Subnet Manager external to the gateway, set that

Subnet Manager’s priority to higher than 0. You do not have to start another Subnet Manager.

1. Start the Subnet Manager.

Fabman@gateway_name-> enablesm

2. (Optional) Set the Subnet Manager Priority.

Fabman@gateway_name-> setsmpriority priority Fabman@gateway_name-> disablesm Fabman@gateway_name-> enablesm

where priority is 0 (lowest) to 13 (highest).

Page 5

▼ Connect the Cables

1. Route the InfiniBand cables and Ethernet cables so that they do not interfere with other cables, with servicing the gateway, or with other systems.

2. Loosen the two thumbscrews that secure the cover to the cable management bracket and lift the cover off.

3. Connect InfiniBand cables to ports 0A - 15A (top row) and 0B - 15B (bottom row) as necessary. Perform Steps 2a to 2e for each InfiniBand cable.

As you connect InfiniBand cables, the Subnet Manager begins establishing links within and routes through the gateway.

Caution – Do not allow any InfiniBand cable to bend tighter than a 5 in. (127 mm) radius. Tight bends

can damage the cable internally.

a. Remove the protective cap from the connector and visually inspect the InfiniBand cable.

If the QSFP connector is bent or damaged, use a different cable.

b. Orient the cable’s QSFP connector to the horizontal, ensuring that the L-groove or cut-out at the tip is

up for the top row of receptacles (0A - 15A) or is down for the bottom row of receptacles (0B - 15B).

c. Slowly slide the connector into the receptacle.

d. Continue to push the connector in until you feel a detent.

e. Place the cable into the open slot on the cable management bar.

4. (Optional) Install QSPF transceivers in Eth ports 0A - 1A (top row) and connect the optical splitter cables into the transceivers.

Place the cables into the open slot on the cable management bar.

5. Replace the cover for the cable management bracket and tighten the thumbscrews.

6. Check that the Link LEDs for cabled links are lit green.

If the Link LED is dark, the link is down. If the Link LED flashes, there are symbol errors.

7. Check the link status of the cables.

Fabman@gateway_name-> listlinkup

■ If the link for a connector is reported as Not present, there is no cable attached or the link at either end

of the cable is down.

■ If a port is down, use the enableswitchport 0 portnumber command to bring the port up.

8. Close the rack doors, if possible. Remove doors if necessary.

Page 6

▼ Verify the InﬁniBand Fabric

♦ Perform diagnostics on the InfiniBand fabric and report the fabric’s status.

Fabman@gateway_name-> ibdiagnet

▼ Conﬁgure the Gateway

♦ Refer to the Sun Network QDR InfiniBand Gateway Switch Administration Guide.

Sun Oracle Sun Network, Sun Network QDR InfiniBand Gateway Switch Getting Started Manual

Specifications and Main Features

Frequently Asked Questions

User Manual