Cisco Systems SFS 7012 User Manual 2

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Corporate Headquarters

Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000

800 553-NETS (6387)

Fax: 408 526-4100

Text Part Number: OL-8787-04

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL

C I P F M Y

A b

STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The following inform ation is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense.

The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not installed in accordance with Cisco’s installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable protection against such interference in a residential installation. However, there is no guarantee that interference will not occur in a particular installation.

Modifying the equipment without Cisco’s written authorization may result in the equipment no longer complying with FCC requirements for Class A or Class B digital devices. In that event, your right to use the equipment may be limited by FCC regulations, and you may be required to correct any interference to radio or television communications at your own expense.

You can determine whether your equipment is causing interference by turning it off. If the interference stops, it was probably caused by the Cisco equipment or one of its peripheral devices. If the equipment causes interference to radio or television reception, try to correct the interference by using one or more of the following measures:

• Turn the television or radio antenna until the interference stops.

• Move the equipment to one side or the other of the television or radio.

• Move the equipment farther away from the television or radio.

• Plug the equipment into an outlet that is on a different circuit from the television or radio. (That is, make certain the equipment and the television or radio are on circuits controlled by different circuit breakers or fuses.)

Modifications to this product not authorized by Cisco Systems, Inc. could void the FCC approval and negate your authority to operate the product.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

IMPLIED, INCLUDING, WITHOUT

CVP, the Cisco Logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco Systems,

nc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco

ress, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me Browsing,

ormShare, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys,

eetingPlace, MGX, Networking Academy, Network Registrar, Pack e t , PIX, ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet, StackWise, The Fastest Way to Increase

our Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

ll other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship

etween Cisco and any other company. (0609R)

New and Changed Information vii

Preface ix

Audience ix

Organization ix

Conventions x

New and Changed Information

The Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide applies to the SFS 7012 Release

3.1 or later.

Table 1 lists the new and changed features available with each supported SFS 7012 release.

Ta b l e 1 Documented Features for the Cisco SFS 7012 InfiniBand Server Switch Hardware

Users Guide

Feature Description

Initial release of the Cisco SFS 7012

InfiniBand Server Switch Hardware

Users Guide

Changed in Release

Where Documented

OL-8787-04

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

vii

Table 1 Documented Features for the Cisco SFS 7012 InfiniBand Server Switch Hardware

Users Guide (continued)

Changed in

Feature Description

Redundant

Added redundant management information.

Management

Release Where Documented

3.3 Installing the Spine

and Leaf Modules, page 2-26

Updating Management Spine IP Addresses in a Redundant Management Configuration, page 2-31

Rebooting Multiple Managed Spines, page 2-38

Double Data Rate (DDR) capabilities

Added DDR information

3.4 Product Overview,

page 1-1

Port Statistics Field Descriptions, page 3-68

CLI command ismPortSetSpeed in

the section

IbSwitchInfo, page 5-83 of

Appendix B: Command Line Interface.

viii

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

Preface

This preface describes the audience, organization, and conventions of the Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide . It also provides information on how to obtain related documentation.

Audience

The intended audience for this document are network administrators responsible for configuring and operating network equipment. It assumes a basic working knowledge of:

• Local Area Networks (LANs)

• Ethernet concepts

• Simple Network Management Protocol (SNMP)

• InfiniBand

Organization

OL-8787-04

This guide is organized as follows:

Chapter Title Description

Chapter 1 Product Overview High-level information about the Cisco SFS 7012™

Chapter 2 Installation Task-oriented information for installing the SFS 7012

Chapter 3 Operations and

Administration

Appendix A Technical

Specifications

Appendix B Command Line

Interface

Appendix C Troubleshooting Troubleshooting symptoms and resolutions for the SFS 7012

Task-oriented information for configuring and monitoring the SFS 7012

SFS 7012 technical specifications

Reference information for the SFS 7012 command line interface (CLI)

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Conventions

This document uses the following conventions for notes, cautions, and safety warnings.

Notes and Cautions contain important information that you should be aware of.

Note Means reader take note. Notes contain helpful suggestions or references to material not

covered in the publication.

Caution Means reader be careful. You are capable of doing something that might result in equipment damage or

loss of data.

Safety warnings appear throughout this publication in procedures that, if performed incorrectly, may harm you. A warning symbol precedes each warning statement.

Warning

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. To see translations of the warnings that appear in this publication, refer to the Regulatory Compliance and Safety Information document that accompanied this device.

Obtaining Documentation

Cisco documentation and additional literature are available on Cisco.com. This section explains the product documentation resources that Cisco offers.

Cisco.com

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

You can access the most current Cisco documentation at this URL:

http://www.cisco.com/techsupport

You can access the Cisco website at this URL:

http://www.cisco.com

OL-8787-04

You can access international Cisco websites at this URL:

http://www.cisco.com/public/countries_languages.shtml

Product Documentation DVD

The Product Documentation DVD is a library of technical product documentation on a portable medium. The DVD enables you to access installation, configuration, and command guides for Cisco software products. With the DVD, you have access to the HTML documentation and some of the PDF

files found on the Cisco website at this URL:

http://www.cisco.com/univercd/home/home.htm

The Product Documentation DVD is created and released regularly. DVDs are available singly or by subscription. Registered Cisco.com DOC-DOCDVD= or DOC-DOCDVD=SUB) from Cisco Store at this URL:

http://www.cisco.com/go/marketplace/docstore

Ordering Documentation

You must be a registered Cisco.com user to access Cisco Marketplace. Registered users may order Cisco

documentation at the Product Documentation Store at this URL:

hardware and

users can order a Product Documentation DVD (product number

Marketplace at the Product Documentation

http://www.cisco.com/go/marketplace/docstore

If you do not have a user ID or password, you can register at this URL:

http://tools.cisco.com/RPF/register/register.do

Documentation Feedback

You can provide feedback about Cisco technical documentation on the Cisco Technical Support & Documentation site area by entering your comments in the feedback form available in every online document.

Cisco Product Security Overview

Cisco provides a free online Security Vulnerability Policy portal at this URL:

http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

From this site, you will find information about how to do the following:

• Report security vulnerabilities in Cisco products

• Obtain assistance with security incidents that involve Cisco products

• Register to receive security information from Cisco

OL-8787-04

A current list of security advisories, security notices, and security responses for Cisco products is available at this

http://www.cisco.com/go/psirt

URL:

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

To see security advisories, security notices, and security responses as they are updated in real time, you can subscribe to the Product Security Incident Response Team Really Simple Syndication (PSIRT RSS) feed. Information about how to subscribe to the PSIRT RSS feed is found at this

http://www.cisco.com/en/US/products/products_psirt_rss_feed.html

Reporting Security Problems in Cisco Products

Cisco is committed to delivering secure products. We test our products internally before we release them, and we strive to correct all vulnerabilities quickly. If you think that you have identified a vulnerability in a Cisco

• For emergencies only — security-alert@cisco.com

• For nonemergencies — psirt@cisco.com

In an emergency, you can also reach PSIRT by telephone:

• 1 877 228-7302

• 1 408 525-6532

Tip We encourage you to use Pretty Good Privacy (PGP) or a compatible product (for example, GnuPG) to

encrypt any sensitive information that you send to Cisco. PSIRT can work with information that has been encrypted with PGP versions

Never use a revoked encryption key or an expired encryption key. The correct public key to use in your correspondence with PSIRT is the one linked in the Contact Summary section of the Security Vulnerability Policy page at this

http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

The link on this page has the current PGP key ID in use.

If you do not have or use PGP, contact PSIRT to find other means of encrypting the data before sending any sensitive material.

product, contact PSIRT:

An emergency is either a condition in which a system is under active attack or a condition for which a severe and urgent security vulnerability should be reported. All other conditions are considered nonemergencies.

2.x through 9.x.

URL:

Product Alerts and Field Notices

Modifications to or updates about Cisco products are announced in Cisco Product Alerts and Cisco Field Notices. You can receive Cisco Product Alerts and Cisco Field Notices by using the Product Alert Tool on Cisco.com. This tool enables you to create a profile and choose those products for which you want to receive information.

To access the Product Alert Tool, you must be a registered Cisco.com user. (To register as a Cisco.com user, go to this URL: tool at this URL: http://tools.cisco.com/Support/PAT/do/ViewMyProfiles.do?local=en

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

xii

http://tools.cisco.com/RPF/register/register.do) Registered users can access the

OL-8787-04

Obtaining Technical Assistance

Cisco Technical Support provides 24-hour-a-day award-winning technical assistance. The Cisco

Technical Support & Documentation website on Cisco.com features extensive online support resources. In addition, if you have a valid Cisco (TAC) engineers provide telephone support. If you do not have a valid Cisco your reseller.

Cisco Technical Support & Documentation Website

The Cisco Technical Support & Documentation website provides online documents and tools for troubleshooting and resolving technical issues with Cisco available 24

http://www.cisco.com/techsupport

Access to all tools on the Cisco Technical Support & Documentation website requires a Cisco.com user

ID and password. If you have a valid service contract but do not have a user ID or password, you

can register at this

http://tools.cisco.com/RPF/register/register.do

Note Use the Cisco Product Identification Tool to locate your product serial number before submitting a

request for service online or by phone. You can access this tool from the Cisco Documentation website by clicking the Tools & Resources link, clicking the All Tools (A-Z) tab, and then choosing Cisco options: by product ID or model name; by tree view; or, for certain products, by copying and pasting show command output. Search results show an illustration of your product with the serial number label location highlighted. Locate the serial number label on your product and record the information before placing a service call.

hours a day at this URL:

URL:

Product Identification Tool from the alphabetical list. This tool offers three search

service contract, Cisco Technical Assistance Center

products and technologies. The website is

service contract, contact

Technical Support &

OL-8787-04

Tip Displaying and Searching on Cisco.com

If you suspect that the browser is not refreshing a web page, force the browser to update the web page by holding down the Ctrl key while pressing F5.

To find technical information, narrow your search to look in technical documentation, not the entire Cisco.com website. On the Cisco.com home page, click the Advanced Search link under the Search box and then click the Technical Support & Documentation radio button.

To provide feedback about the Cisco.com website or a particular technical document, click Contacts & Feedback at the top of any Cisco.com web page.

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

xiii

Submitting a Service Request

Using the online TAC Service Request Tool is the fastest way to open S3 and S4 service requests. (S3 and S4 service requests are those in which your network is minimally impaired or for which you require product information.) After you describe your situation, the TAC recommended solutions. If your issue is not resolved using the recommended resources, your service request is assigned to a Cisco

http://www.cisco.com/techsupport/servicerequest

For S1 or S2 service requests, or if you do not have Internet access, contact the Cisco TAC by telephone. (S1 or S2 service requests are those in which your production network is down or severely degraded.) Cisco

engineers are assigned immediately to S1 and S2 service requests to help keep your business

operations running smoothly.

To open a service request by telephone, use one of the following numbers:

Asia-Pacific: +61 2 8446 7411 Australia: 1 800 805 227 EMEA: +32 2 704 55 55 USA: 1 800 553 2447

For a complete list of Cisco TAC contacts, go to this URL:

http://www.cisco.com/techsupport/contacts

engineer. The TAC Service Request Tool is located at this URL:

Definitions of Service Request Severity

Service Request Tool provides

To ensure that all service requests are reported in a standard format, Cisco has established severity definitions.

Severity 1 (S1)—An existing network is “down” or there is a critical impact to your business operations. You and Cisco

Severity 2 (S2)—Operation of an existing network is severely degraded, or significant aspects of your business operations are negatively affected by inadequate performance of Cisco Cisco

will commit full-time resources during normal business hours to resolve the situation.

Severity 3 (S3)—Operational performance of the network is impaired while most business operations remain functional. You and Cisco to satisfactory levels.

Severity 4 (S4)—You require information or assistance with Cisco product capabilities, installation, or configuration. There is little or no effect on your business operations.

will commit all necessary resources around the clock to resolve the situation.

will commit resources during normal business hours to restore service

Obtaining Additional Publications and Information

Information about Cisco products, technologies, and network solutions is available from various online and printed sources.

• The Cisco Online Subscription Center is the website where you can sign up for a variety of

Cisco

e-mail newsletters and other communications. Create a profile and then select the subscriptions that you would like to receive. To visit the Cisco Online Subscription Center, go

to this URL:

products. You and

xiv

http://www.cisco.com/offer/subscribe

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

• The Cisco Product Quick Reference Guide is a handy, compact reference tool that includes brief

product overviews, key features, sample part numbers, and abbreviated technical specifications for many Cisco the latest Cisco channel product offerings. To order and find out more about the Cisco

products that are sold through channel partners. It is updated twice a year and includes

Product Quick

Reference Guide, go to this URL:

http://www.cisco.com/go/guide

• Cisco Marketplace provides a variety of Cisco books, reference guides, documentation, and logo

merchandise. Visit Cisco

Marketplace, the company store, at this URL:

http://www.cisco.com/go/marketplace/

• Cisco Press publishes a wide range of general networking, training, and certification titles. Both new

and experienced users will benefit from these publications. For current Cisco information, go to Cisco

Press at this URL:

Press titles and other

http://www.ciscopress.com

• Internet Protocol Journal is a quarterly journal published by Cisco Systems for engineering

professionals involved in designing, developing, and operating public and private internets and intranets. You can access the Internet Protocol Journal at this

URL:

http://www.cisco.com/ipj

• Networking products offered by Cisco Systems, as well as customer support services, can be

obtained at this

URL:

http://www.cisco.com/en/US/products/index.html

• Networking Professionals Connection is an interactive website where networking professionals

share questions, suggestions, and information about networking products and technologies with Cisco

experts and other networking professionals. Join a discussion at this URL:

http://www.cisco.com/discuss/networking

• “What’s New in Cisco Documentation” is an online publication that provides information about the

latest documentation releases for Cisco products. Updated monthly, this online publication is organized by product category to direct you quickly to the documentation for your products. You can view the latest release of “What’s New in Cisco Documentation” at this URL:

http://www.cisco.com/univercd/cc/td/doc/abtunicd/136957.htm

• World-class networking training is available from Cisco. You can view current offerings at

this

URL:

http://www.cisco.com/en/US/learning/index.html

OL-8787-04

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

xvi

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

CHAPTER

Product Overview

The Cisco SFS 7012™ is an industry leading modular system used for creating large, single-system Grid/Cluster server fabrics, or as a building block for larger fabrics. The SFS 7012 is designed to maximize performance, streamline operations, and ensure uptime by providing full hardware and software reliability, availability, and serviceability (RAS) features.

Leveraging InfiniBand (IB) — an industry-standard interconnect — the SFS 7012 provides high performance, offering a full bisectional bandwidth (FBB) fabric (10Gb-30Gb) and ultra-low switching latency. Additionally, the SFS 7012 can be configured to run at 2.5 Gbps single data rate (SDR), 5.0 Gbps double data rate (DDR), or both.

All major SFS 7012 components and expansion modules are field replaceable and hot pluggable. To allow easy scaling, the SFS 7012 provides 12 expansion slots. Each slot can support expansion modules with twelve 10 Gbit/sec (or 4X) IB ports, meaning the SFS 7012 can scale to 144 (4X) InfiniBand ports. The SFS 7000

Series switches make possible the highest density of InfiniBand fabrics available today.

SFS 7012 Feature Set

Strategic Benefits

• Creates the industry’s simplest way to build medium to large server fabrics

–

Modular systems up to 144 ports (4X)

• Eliminates requirement for parallel storage networks

• High-availability design for mission critical needs

• Lowers Total Cost of Ownership (TCO)

Key Design Features

• All field-replaceable units (FRU’s) can be replaced while under power

• Redundant management, power and cooling

• High performance 10Gb/s

• Full Bisectional Bandwidth InfiniBand switching fabric

• Twelve (12) expansion slots for InfiniBand (4X)

• SDR and DDR support

• Embedded system management

OL-8787-04

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

SFS 7012 Product Specifications

Mechanical/Thermal/Power Specifications

• 7U - 19" rack mount environment

• Integrated thermal management

• Front-to-back airflow

• Hot plug, redundant cooling

• Hot plug, redundant power supplies

• 90/264 VAC operation

• 50/60 Hz Frequency

• Redundant, isolated AC inputs

• FCC and VCCI compliant

• U.L. Listed

Chapter

Thermal Management Features

• Four (4) fan trays, hot plug, N+1 fans

• Two (2) fans per tray, 60mm, 12VDC

• Available air flow: 200 cfm @ .20 in H2O

• Expected air velocity on the switch chip: 400 lfpm

• Fan rotation monitor

• I2C interface to CME

• Thermal input from ambient air sensor and Switch chip via IBML links

Power Design Features

• Up to six (6), 350 Watt power supplies

• 3.30" Width x 10.67" Depth x 1U height

• 12V bulk power

• Status indicators

• Two (2) independent AC input lines

Switch Characteristics

• 7U form factor

• Full module enclosure

• Available power (all copper): 745 Watts max

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

Chapter

• Available power (all fiber): 879 Watts max

• Available air flow: 200 cfm

• Thermal status reporting to CME via IBML

• Up to one hundred forty-four (144) 4X IB external copper ports

• Two (2) RJ-45 connectors for 100BASE Ethernet connection to management LAN.

• One (1) RJ-11 serial port per spine module for configuration.

Chassis Viewer Software

The Chassis Viewer is Cisco”s browser-based management software. Primary functions of Chassis Viewer

for the SFS 7012 switch and its associated components include:

• Management

• Configuration

• Monitoring

• Diagnostics

Figure 1-1 Chassis Viewer Home Page

• The Chassis Viewer runs on the Chassis Management Unit (CMU) processor of the SFS 7012 spine

module(s), and is accessed through an OOB LAN workstation connected to the Ethernet port of the switches.

Chassis Viewer Functionality

For the SFS 7012, Chassis Viewer provides an interface for performing the following management, configuration, and monitoring tasks:

• Manage and view user-defined data

• Monitor component status

OL-8787-04

• Monitor Switch-level detailed information

• Configure the OOB LAN IP address

• Manage and monitor log files, including:

–

Set debug levels determining the amount of information to be logged

–

Reset the logs

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

–

Perform maintenance tasks

–

Manage firmware updates, including uploading new firmware and setting the active version.

Chapter

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

Chapter

Subnet Manager

Every InfiniBand fabric subnet requires a single active Subnet Manager (SM) to initialize and maintain the subnet. The fabric may also have one or more standby Subnet Managers. One of the standby Subnet Managers can be configured to become the active SM in the event of a failure.

As Figure 1-2 shows, InfiniBand-enabled fabrics consist of one or more subnets. Each subnet consists of a number of InfiniBand-capable servers connected to target devices. InfiniBand-capable routers provide connectivity between InfiniBand subnets.

Figure 1-2 Subnet Manager

Subnet A

InfiniBand Subnet Manager for Subnet A

SFS 7012

InfiniBand Subnet Manager

Router

Subnet B

InfiniBand Capable Server

InfiniBand

Capable Server

When a network is initially powered up, the Subnet Manager queries the subnet management agents running on the network devices, for information about node/port addressing schemes, routing tables and partitions. From that point on, the Subnet Manager periodically sweeps the network for changes. Once the InfiniBand network is up and running, the InfiniBand General Services Managers (GSMs) begin to interface with General Services Agents (GSAs).

HCA

for Subnet B

IB Port

Subnet Management

Subnet Management Agents

A Subnet Management Agent processes management requests from the Subnet Manager. Subnet Management Agents are implemented on all nodes within the InfiniBand fabric. These nodes include IB-capable switches, routers, and channel adapters. Each node implements a Subnet Management Agent. Within the SFS 7012 Switch, a Subnet Management Agent (SMA) is implemented within its firmware.

OL-8787-04

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Chapter

An SMA also runs on every InfiniBand-capable server that is equipped with an HCA. As shown in Figure 1-3 the Subnet Manager interacts with these SMAs to discover information about the nodes on the fabric.

Figure 1-3 InfiniBand Managers and Agents

Subnet Manager

Subnet

Management

Agent

Subnet

Management

Agent

InfiniBand

Capable Server

SFS

7012

The Subnet Manager utilizes the agent information to discover and manage connections from the SFS 7012 to the InfiniBand fabric. The Subnet Manager maintains a database tracking subnet topology information as a result of its interaction with the SMAs.

InfiniBand General Services Managers and Agents

The InfiniBand specification defines the InfiniBand General Services Managers and their associated agents. General Services Managers exchange messages with agents to manage and monitor the performance and the physical environment of devices on the network.

Once the InfiniBand network is up and running, the InfiniBand General Services Managers begin to interface with General Services Agents. The General Services Managers and Agents include:

• Subnet Administration (SA)

SA provides InfiniBand fabric nodes with an interface to the Subnet Manager. This interface is used by the nodes on the fabric to interact with the Master Subnet Manager and to discover information about the fabric. Every InfiniBand fabric subnet requires a SA.

• The Device Manager (DM) and Device Management Agent (DMA)

Subnet

Management

Agent

InfiniBand

Capable Server

Subnet

Management

Agent

InfiniBand

Capable Server

The DM and DMA discover and manage the association between hosts and devices behind the Target Channel Adapters.

• The Communications Manager (CM)

The CM establishes and manages communication channels between nodes.

• The Baseboard Manager (BM) and Baseboard Management Agent (BMA)

The BM and BMA exchange messages relating to items such as temperature monitoring and hardware control to manage hardware on the fabric.

• The Performance Manager (PM) and Performance Management Agent (PMA)

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

Chapter

SNMP Support

SNMP MIBs

The PM and PMA exchange messages about performance statistics and error information of InfiniBand devices on the fabric.

Built-in support for Simple Network Management Protocol (SNMP) allows users to integrate a SFS 7012 into their existing management frameworks. SNMP allows users access to all statistics, trend analysis, alarm handling, filtering, and performance monitoring capabilities supported by these management frameworks.

SNMP Management Information Bases (MIBs) are management elements that are used by industry frameworks to monitor information about the SFS 7012 switch. The SFS 7012 switch supports MIBs from the following sources:

• Switch-sourced MIBs: MIBs from the CMU of the SFS 7012 provide status information regarding

the physical environment of the switch. Additionally, the CMU MIBs provide status and activity information for all line cards residing in the switch.

OL-8787-04

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Chapter

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

CHAPTER

Installation

This chapter describes how to install the Cisco SFS 7012™ and its components, and it includes the following information:

• Planning the Installation, page 2-15

• Installation Tasks, page 2-22

• Hot Swapping Components, page 2-39

Note Before you install, operate, or service the system, read the Regulatory Compliance and Safety

Information for the Cisco SFS 7012 and 7024 for important safety information.

OL-8787-04

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Chapter

Warning

IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device.

Statement 1071

SAVE THESE INSTRUCTIONS

This unit is intended for installation in restricted access areas. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security.

Statement 1017

Only trained and qualified personnel should be allowed to install, replace, or service this equipment.

Statement 1030

A readily accessible two-poled disconnect device must be incorporated in the fixed wiring.

Statement

1022

Statement 1019—Main Disconnecting Device

Warning

Waarschuwing

Varoitus

Attention

Warnung

Avvertenza

Advarsel

The plug-socket combination must be accessible at all times, because it serves as the main disconnecting device.

Statement 1019

De combinatie van de stekker en het elektrisch contactpunt moet te allen tijde toegankelijk zijn omdat deze het hoofdmechanisme vormt voor verbreking van de aansluiting.

Pistoke/liitinkohta toimii pääkatkaisumekanismina. Pääsy siihen on pidettävä aina esteettömänä.

La combinaison de prise de courant doit être accessible à tout moment parce qu'elle fait office de système principal de déconnexion.

Der Netzkabelanschluß am Gerät muß jederzeit zugänglich sein, weil er als primäre Ausschaltvorrichtung dient.

Il gruppo spina-presa deve essere sempre accessibile, poiché viene utilizzato come dispositivo di scollegamento principale.

Kombinasjonen støpsel/uttak må alltid være tilgjengelig ettersom den fungerer som hovedfrakoplingsenhet.

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

Chapter

Aviso

¡Advertencia!

Varning!

A combinação ficha-tomada deverá ser sempre acessível, porque funciona como interruptor principal.

El conjunto de clavija y toma ha de encontrarse siempre accesible ya que hace las veces de dispositivo de desconexión principal.

Man måste alltid kunna komma åt stickproppen i uttaget, eftersom denna koppling utgör den huvudsakliga frånkopplingsanordningen.

Statement 1045—Short-circuit Protection

Warning

Waarschuwing

Varoitus

Attention

Warnung

Avvertenza

This product requires short-circuit (overcurrent) protection, to be provided as part of the building installation. Install only in accordance with national and local wiring regulations.

Voor dit product moet kortsluitbeveiliging (overstroombeveiliging) deel uitmaken van de installatie in het gebouw. De installatie moet voldoen aan de nationale en lokale bedradingvoorschriften.

Tämä tuote vaatii suojauksen oikosulkuja (ylivirtaa) vastaan osana asennusta rakennukseen. Asenna ainoastaan kansallisten ja paikallisten johdotussäännösten mukaisesti.

La protection de ce produit contre les courts-circuits (surtensions) doit être assurée par la configuration électrique du bâtiment. Vérifiez que l'installation a lieu uniquement en conformité avec les normes de câblage en vigueur au niveau national et local.

Für dieses Produkt ist eine Kurzschlußsicherung (Überstromsicherung) erforderlich, die als Teil der Gebäudeinstallation zur Verfügung gestellt wird. Die Installation sollte nur in Übereinstimmung mit den nationalen und regionalen Vorschriften zur Verkabelung erfolgen.

Questo prodotto richiede una protezione contro i cortocircuiti, da fornirsi come parte integrante delle dotazioni presenti nell’edificio. Effettuare l’installazione rispettando le Norme CEI pertinenti.

OL-8787-04

Advarsel

Dette produktet krever beskyttelse mot kortslutninger (overspenninger) som en del av installasjonen. Bare installer utstyret i henhold til nasjonale og lokale krav til ledningsnett.

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

Chapter

Aviso

Advertencia

Varning!

Este produto requer proteção contra curto-circuitos (sobreintensidade de corrente), que deve estar instalada nos edifícios. Instale apenas de acordo com as normas de instalação elétrica nacionais

locais.

Este producto necesita estar conectado a la protección frente a cortacircuitos (sobretensiones) que exista en el edificio. Instálelo únicamente en conformidad con las regulaciones sobre cableado, tanto locales como nacionales, a las que se tenga que atener.

Denna produkt kräver att kortslutningsskydd (överström) tillhandahålles som en del av byggnadsinstallationen. Installera bara i enlighet med nationella och lokala kabeldragningsbestämmelser.

Aviso

Advarsel

Este produto requer uma proteção contra curto-circuito (sobrecorrente) que deve fazer parte da instalação do edifício. Faça a instalação somente de acordo com as regulamentações de cabeamento nacionais e locais.

Dette produkt kræver beskyttelse mod kortslutning (overstrøm). Dette skal være en del elinstallationen i bygningen. Installation skal ske i overensstemmelse med nationale og lokale ledningsregler.

Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide

OL-8787-04

+ 86 hidden pages

Cisco Systems SFS 7012 User Manual 2

New and Changed Information

Preface

Audience

Organization

Conventions

Related Documentation

Obtaining Documentation

Cisco.com

Product Documentation DVD

Ordering Documentation

Documentation Feedback

Cisco Product Security Overview

Reporting Security Problems in Cisco Products

Product Alerts and Field Notices

Obtaining Technical Assistance

Cisco Technical Support & Documentation Website

Submitting a Service Request

Definitions of Service Request Severity

Obtaining Additional Publications and Information

Product Overview

SFS 7012 Feature Set

Strategic Benefits

Key Design Features

SFS 7012 Product Specifications

Mechanical/Thermal/Power Specifications

Thermal Management Features

Power Design Features

Switch Characteristics

Chassis Viewer Software

Chassis Viewer Functionality

Subnet Manager

Subnet Management

InfiniBand General Services Managers and Agents

SNMP Support

SNMP MIBs

Installation