HP Fabric Copper User Manual
HP 24-Port 4x Fabric Copper Switch
User Guide
November 2004 (First Edition)
Part Number 377710-001
© Copyright 2004 Hewlett-Packard Development Company, L.P.
Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with
FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical
Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license.
The information contained herein is subject to change without notice. The only warranties for HP products and
services are set forth in the express warranty statements accompanying such products and services. Nothing
herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial
errors or omissions contained herein.
Microsoft®, MS Windows®, Windows®, and Windows NT® are U.S. registered marks of Microsoft Corporation.
HP 24-Port 4x Fabric Copper Switch User Guide
November 2004 (First Edition)
Part Number 377710-001
Contents
Intended Audience ..................................................................................................................................... ix
Typographical Conventions....................................................................................................................... ix
Contact Information.....................................................................................................................................x
1: Introducing the InfiniBand System ............................. 1
About the HP 24-Port 4x Fabric Copper Switch User Guide ......................................................................1
Maximize Server Networks .........................................................................................................................1
What is InfiniBand? .....................................................................................................................................2
How Does InfiniBand Work? ......................................................................................................................2
Possible Components...................................................................................................................... 2
Protocols ......................................................................................................................................... 2
Architectural Elements ................................................................................................................... 3
Understanding the Subnet Manager (SM) ...................................................................................... 4
Understanding Subnet Manager Routing........................................................................................ 5
i
2: Getting Started........................................................... 9
Overview of Set-up Procedures ...................................................................................................................9
Install HCAs and Drivers in Hosts ............................................................................................................10
Install and Power on the InfiniBand Chassis .............................................................................................10
Configure Basic Connectivity....................................................................................................................10
Configuring an Ethernet Management IP Address ....................................................................... 10
Configuring a Direct Serial-Console Connection ......................................................................... 11
Configuring an InfiniBand Management IP Address ................................................................... 12
Configuring the System Hostname ............................................................................................... 12
Test Network Connectivity........................................................................................................................13
Verify Communication Between Hosts ........................................................................................ 13
Verify Switch to Host Communication ........................................................................................ 14
Install the GUI (Element Manager) ...........................................................................................................14
Access a Management System...................................................................................................................14
Default User Name and Passwords............................................................................................... 14
CLI Management .......................................................................................................................... 14
GUI Management ......................................................................................................................... 15
SNMP Management...................................................................................................................... 15
(Optional) Enable Database Sync ..............................................................................................................15
Configure Partitions...................................................................................................................................15
Create a Partition ID (P_Key)....................................................................................................... 16
Specify Partition Members and the
Membership Type ......................................................................................................................... 17
Maintain Partition Key Information ............................................................................................. 17
Set User Levels and Passwords..................................................................................................................18
ii
Change Default User Name and Password ................................................................................... 18
3: Understanding the Management Options ................ 19
About the CLI ............................................................................................................................................19
Understanding the Command Modes ........................................................................................... 20
Using the CLI.............................................................................................................................................21
Entering the Sub-Command Mode ............................................................................................... 21
Exiting Command Modes ............................................................................................................. 21
Using Command Completion ....................................................................................................... 21
Displaying Command History ...................................................................................................... 22
Setting Terminal Parameters......................................................................................................... 22
Ending A CLI Session .................................................................................................................. 23
Quick Help.................................................................................................................................... 24
About Element Manager............................................................................................................................24
The Chassis Window .................................................................................................................... 25
The Tool Bar................................................................................................................................. 25
About Selecting Items................................................................................................................... 26
Using Element Manager ............................................................................................................................27
Installing the Element Manager Program ..................................................................................... 27
Starting the Element Manager ...................................................................................................... 29
Reading the Element Manager Status Colors ............................................................................... 30
About SNMP..............................................................................................................................................30
Supported MIBs............................................................................................................................ 30
Using SNMP ..............................................................................................................................................30
Configuring SNMP Settings ......................................................................................................... 30
4: Performing Admin Tasks Through the GUI.............. 33
Configuring the IB Interface Speed ...........................................................................................................33
Explicitly Configure IB Interface Speed ...................................................................................... 33
Set IB Interface Speed to Auto-Negotiate .................................................................................... 34
View the IB Interface Speed......................................................................................................... 35
Setting the System Clock...........................................................................................................................36
Setting Time Manually ................................................................................................................. 36
Synchronize the Clock to an NTP Server ..................................................................................... 36
Rebooting the System ................................................................................................................................37
Reboot a System with a Single Controller Card ........................................................................... 37
5: Performing Admin Tasks Through the CLI............... 39
Setting the IB Interface Speed ...................................................................................................................39
Explicitly Configure IB Interface Speed ...................................................................................... 39
Set IB Interface Speed to Auto-Negotiate .................................................................................... 40
View the IB Interface Speed......................................................................................................... 40
Notifying Users..........................................................................................................................................40
Broadcasting Messages to all Users.............................................................................................. 41
Sending Messages to Individual Users ......................................................................................... 41
Setting the System Clock...........................................................................................................................41
Setting Time.................................................................................................................................. 42
Synchronize the Clock to an NTP Server ..................................................................................... 42
Rebooting the System ................................................................................................................................43
Reboot a System with a Single Controller.................................................................................... 43
6: Setting Access and Security .................................... 45
Understanding Access and Accounts.........................................................................................................45
About User Accounts.................................................................................................................... 45
Elements of the Access System .................................................................................................... 46
Understanding Usernames and Passwords .................................................................................. 46
About Roles and Privileges........................................................................................................... 46
Managing Access and Accounts ................................................................................................................47
Setting or Changing a Password ................................................................................................... 47
Displaying User Information ........................................................................................................ 48
Adding New Users........................................................................................................................ 49
Deleting a User Account............................................................................................................... 50
User Account Configuration Commands...................................................................................... 50
Switching User Identity ................................................................................................................ 52
Changing Privilege Access-Levels ............................................................................................... 52
About Partitions .........................................................................................................................................53
How Partitions Work .................................................................................................................... 53
Partition Members......................................................................................................................... 54
Membership Types ....................................................................................................................... 54
Selecting a P_Key Value .............................................................................................................. 54
Understanding how P_Keys are Saved......................................................................................... 56
Create Partitions (CLI)...............................................................................................................................56
Create a Partition ID (P_Key)....................................................................................................... 57
Specify Partition Members and the
Membership Type ......................................................................................................................... 57
Create Partitions (GUI)..............................................................................................................................57
Create a Partition ID (P_Key)....................................................................................................... 57
Specify Partition Members and the
Membership Type ......................................................................................................................... 58
About SSH .................................................................................................................................................59
iii
7: Using the Subnet Manager Through the GUI .......... 61
The Subnet Manager (SM).........................................................................................................................61
Master Subnet Manager ................................................................................................................ 61
Standby Subnet Manager .............................................................................................................. 62
Viewing the Subnet Manager Configurations............................................................................................62
View a Summary of Subnet Management .................................................................................... 62
View Details of Subnet Management ........................................................................................... 62
Changing the Subnet Manager Configurations..........................................................................................64
Change the Priority of a SM ......................................................................................................... 64
Change the Sweep Interval of a SM ............................................................................................. 64
iv
Change the Response Timeout of a SM........................................................................................ 65
Managing Synchronization Between SMs.................................................................................................66
Enable/Disable Database Synchronization ................................................................................... 66
Set Configurations for the Master SM.......................................................................................... 67
Set Configurations for the Backup SM......................................................................................... 68
Adding a Subnet Manager .........................................................................................................................70
Viewing Partitions .....................................................................................................................................71
About InfiniBand Multicast Groups ..........................................................................................................72
Viewing Multicast Groups.........................................................................................................................72
View a Multicast Group Summary ............................................................................................... 72
View Multicast Group Details ...................................................................................................... 73
View the Subnet Manager Services ...........................................................................................................75
View a Summary of the SM Services........................................................................................... 75
View Details of the SM Services.................................................................................................. 76
Configure Subnet Manager Routing ..........................................................................................................77
Configure the LID Mask Control (LMC) ..................................................................................... 78
View InfiniBand Paths.................................................................................................................. 78
8: Using the Subnet Manager Through the CLI ........... 81
The Subnet Manager (SM).........................................................................................................................81
Master Subnet Manager ................................................................................................................ 81
Standby Subnet Manager .............................................................................................................. 82
Viewing the Subnet Manager Configurations............................................................................................82
View a Summary of Subnet Management .................................................................................... 82
View Details of Subnet Management ........................................................................................... 82
Changing the Subnet Manager Configurations..........................................................................................83
Change the Priority of a SM ......................................................................................................... 83
Change the Sweep Interval of a SM ............................................................................................. 84
Change the Response Timeout of a SM........................................................................................ 84
Managing Synchronization Between SMs.................................................................................................84
Enable/Disable Database Synchronization ................................................................................... 84
Set Configurations for the Master SM.......................................................................................... 85
Set Configurations for the Backup SM......................................................................................... 85
Adding a Subnet Manager .........................................................................................................................87
About InfiniBand Multicast Groups ..........................................................................................................87
Viewing Multicast Groups.........................................................................................................................88
View a Multicast Group Summary ............................................................................................... 88
View Multicast Group Details ...................................................................................................... 89
Viewing the SM Services ..........................................................................................................................90
View a Summary of the SM Services........................................................................................... 90
Configure Subnet Manager Routing ..........................................................................................................90
Configure the LID Mask Control (LMC) ..................................................................................... 91
View InfiniBand Paths.................................................................................................................. 91
9: Using Image Files .................................................... 93
Types of Image Upgrades ..........................................................................................................................93
TopspinOS Upgrades.................................................................................................................... 93
About the System Image............................................................................................................................93
What is a System Image?.............................................................................................................. 93
What is an Image File? ................................................................................................................. 94
About Copying/Downloading the Image...................................................................................................94
Card Status Requirements..........................................................................................................................95
Upgrade Procedure Overview....................................................................................................................95
Set-Up the Hardware Connection ..............................................................................................................95
Out-of-Band Connection............................................................................................................... 95
In-Band Connection...................................................................................................................... 95
Verify the Installed Image Version............................................................................................................96
Check the Image Version Through the GUI................................................................................. 96
Check the Image Version Through the CLI.................................................................................. 96
Copy/Download the Image ........................................................................................................................96
Copy/Download the Image Through the GUI .............................................................................. 97
Copy/Download an Image Through the CLI ................................................................................ 98
Activate an Image ....................................................................................................................................100
Specify a New Boot Image ......................................................................................................................101
Specify a New Boot Image Through the GUI ............................................................................ 101
Specify a New Boot Image Through the CLI ............................................................................. 102
Reboot the System ...................................................................................................................................102
Deleting Image Files................................................................................................................................103
Deleting Images Through the GUI ............................................................................................. 103
Deleting Images Through the CLI .............................................................................................. 103
v
10: Using Configuration Files..................................... 105
Understanding Configuration Files..........................................................................................................105
About the Startup-Config............................................................................................................ 105
About the Running-Config ......................................................................................................... 105
Listing Configuration Files......................................................................................................................106
List Config Files Through the CLI ............................................................................................. 106
List Config Files Through the GUI............................................................................................. 106
Export a Configuration File .....................................................................................................................106
Export a Config File Through the CLI ....................................................................................... 107
Export a Config File Through the GUI....................................................................................... 107
Import a Configuration File .....................................................................................................................108
Download a Config File Through the CLI.................................................................................. 108
11: Using Log Files .................................................... 111
Understanding Log Files..........................................................................................................................111
File Management and Storage .................................................................................................... 111
About Message Types................................................................................................................. 111
Listing Current Log File Names ..............................................................................................................112
Listing Current Logs Through the CLI....................................................................................... 112
Listing Current Logs Through the GUI ...................................................................................... 112
Viewing a Log File Through the CLI ......................................................................................................113
vi
Display Entire Log...................................................................................................................... 113
Show Most Recent Log Entries .................................................................................................. 113
Show Details of a Specific Log .................................................................................................. 114
Viewing a Log File Through the GUI......................................................................................................114
Filtering Logs.............................................................................................................................. 115
Configuring Remote Logging..................................................................................................................117
12: Viewing the IB Network Through the GUI ............ 119
About the Device Manager (DM)............................................................................................................119
Display the Device Manager....................................................................................................................119
View I/O Unit Information ......................................................................................................... 119
View I/O Controller Units .......................................................................................................... 120
View I/O Controller Units Services............................................................................................ 121
About the Topology View .......................................................................................................................121
Display the InfiniBand Topology ............................................................................................................122
View the Topology ..................................................................................................................... 122
View the Name of an HCA......................................................................................................... 123
View the GUID of an HCA ........................................................................................................ 124
Determine Which HCA Port is Connected to an IB Port ........................................................... 124
View the GUID of an IB Switch................................................................................................. 126
Add an Attached Device to the Topology View......................................................................... 126
View the Internal Chassis Topology........................................................................................................127
View Subnet Manager Details .................................................................................................................129
View Basic Node Information .................................................................................................... 129
View Advanced Node Information............................................................................................. 130
View Basic Port Information ...................................................................................................... 131
View Advanced Port Information............................................................................................... 133
13: Monitoring and Reporting Through the GUI......... 137
About Analyzing Network Data ..............................................................................................................137
Benefits ....................................................................................................................................... 137
Data Captured ............................................................................................................................. 138
About Tabular Formats ............................................................................................................................138
About Graph Formats ..............................................................................................................................138
Types of Graphs.......................................................................................................................... 138
Creating a Data Analysis Table ...............................................................................................................140
Create a Data Table..................................................................................................................... 140
Export a Data Table .................................................................................................................... 141
Print a Data Table ....................................................................................................................... 142
Creating a Data Analysis Graph ..............................................................................................................143
Modify a Graph........................................................................................................................... 145
Print a Graph............................................................................................................................... 146
About SNMP Traps .................................................................................................................................146
Events Sent to Trap Receivers .................................................................................................... 146
Configuring SNMP Settings ....................................................................................................................147
Viewing Current SNMP Trap Receivers .................................................................................... 147
Adding an SNMP Trap Receivers .............................................................................................. 147
Editing a Current SNMP Trap Receiver ..................................................................................... 148
14: Monitoring Through the CLI ................................. 149
About InfiniBand Events .........................................................................................................................149
About Tracing ..........................................................................................................................................149
Types of Traces........................................................................................................................... 150
Trace Levels................................................................................................................................ 150
About SNMP Traps .................................................................................................................................151
Events Sent to Trap Receivers .................................................................................................... 151
Configuring SNMP Settings ....................................................................................................................152
Viewing Current SNMP Trap Receivers .................................................................................... 152
Add an SNMP Trap Receiver ..................................................................................................... 152
vii
viii
Preface
ix
This document is a guide to the HP 24-Port 4x Fabric Copper Switch.
Intended Audience
The intended audience is the administrator responsible for installing, configuring, and managing your
equipment. This administrator should have experience administering similar networking or storage
equipment.
Typographical Conventions
The following typographic conventions are used in this manual to provide visual clues as to the purpose
or application of specific text.
• Bold text indicates a command.
• Courier text indicates example text as displayed on the computer screen or that you enter exactly as
shown.
• Italics indicate variable text that you replace with an actual value.
• Square angle-brackets ([data]) indicate an option that you choose to include or exclude. (Do not
include the brackets when supplying optional data.)
• Piping character (|) indicates an “or” choice. For example, a | b indicates “a or b”. [a] | [b] indicates
an optional choice between a or b.
• Menu1->Menu2->Item… indicates a pop-up menu sequence to open a form or execute a desired
function.
• Ellipses (…) indicate truncated text. You will see these in long examples depicting terminal output
that is too long to be shown in its entirety.
x
NOTE: Indicates an important point or aspect that you need to consider before continuing.
Contact Information
Table 2-1: Customer Contact Information
For the name of your nearest authorized
HP reseller:
For HP technical support: In the United States and Canada, call 1-800-HP-INVENT
In the United States, call 1-800-345-1518.
In Canada, call 1-800-263-5868.
(1-800-474-6836). This service is available 24 hours a day,
7 days a week. For continuous quality improvement, calls
may be recorded or monitored.
Outside the United States and Canada, refer to
www.hp.com
Introducing the InfiniBand System
This chapter gives an overview of the following:
• “About the HP 24-Port 4x Fabric Copper Switch User Guide” on page 1
• “Maximize Server Networks” on page 1
• “What is InfiniBand?” on page 2
• “How Does InfiniBand Work?” on page 2
1
1
About the HP 24-Port 4x Fabric Copper Switch User Guide
The HP 24-Port 4x Fabric Copper Switch User Guide is specifically intended to demonstrate the
processes involved in using and managing the InfiniBand
• For information regarding the Host Channel Adapter, refer to the HP Dual-port 4x Fabric Adapter
User Guide.
• For information regarding the switch, refer to the HP 24-Port 4x Fabric Copper Switch Hardware
User Guide.
™ switch technology.
Maximize Server Networks
The Topspin system uses InfiniBand as the underlying fabric that creates a scalable and efficient server
area network. The system also seamlessly interconnects with existing Fibre Channel and Ethernet
resources, extending the value of InfiniBand to the rest of the network.
2
What is InfiniBand?
InfiniBand (IB) is a high speed, high density serial interconnect that increases CPU utilization,
decreases latency, and eases the management pain of data centers.
The term “InfiniBand” refers to the entire hardware, communication, and management infrastructure.
Use of this technology increases the communication speed between:
•CPUs
• devices within servers
• subsystems located throughout a network.
How Does InfiniBand Work?
InfiniBand combines high-speed hardware, specialized protocols, and Remote Data Memory Access
(RDMA) techniques to achieve the objective of increased CPU utilization and decreased latency.
Operations of the InfiniBand Architecture are managed by the Subnet Manager.
Possible Components
One or more of the following hardware components may be used to maximize your server network.
• InfiniBand switch
• Host Channel Adapters (installed in host)
• Ethernet Gateway
• Fibre Channel Gateway
Protocols
InfiniBand requires a new set of protocols. For information on how to configure these protocols, refer to
the HP Dual-port 4x Fabric Adapter User Guide.
IPoIB
The IP over IB (IPoIB) link driver provides standardized Internet Protocol encapsulation over
InfiniBand fabrics. IPoIB can transparently use IP over InfiniBand technology, similar to the way that
IP runs over Ethernet.
The primary responsibilities of the IPoIB driver are to perform address resolution and the management
of multicast membership.
SDP
The Sockets Direct Protocol (SDP) is a transparent protocol used on InfiniBand networks to allow
sockets-based applications to take advantage of the RDMA performance over an InfiniBand network.
SDP provides:
• a reduction in the amount of software running inside a process context
• zero copy
SDP protocol support enables databases, application servers, and CPUs to operate more efficiently
because the databases spends less time waiting for work, the application servers spend less time waiting
for responses, and the CPUs have more cycles free for other work.
SRP
SCSI RDMA Protocol (SRP) is an upper-layer storage protocol for InfiniBand. It runs SCSI commands
across RDMA-capable networks for InfiniBand hosts to communicate with Fibre Channel storage
devices. This protocol allows InfiniBand hosts to natively send SCSI commands as if the storage was
direct attached.
The SRP protocol is designed to operate using an RDMA communication service. An RDMA
communication service provides communication between pairs of consumers; it uses messages for
control information and RDMA operations for data transfers.
The SRP protocol is only used if you have a Fibre Channel Gateway installed in your InfiniBand
system.
uDAPL
The user Direct Access Programming Library (uDAPL) is a standardized user mode API that natively
supports InfiniBand fabrics.
uDAPL performs name to address translations, establishes connections, and transfers data reliably.
The primary responsibilities of uDAPL are:
• Connection management
• Low latency data transfer and completion
MPI
The MPI protocol is bundled with the Upper Layer Protocol (ULP) suite. Topspin has taken the Ohio
State University’s (OSU’s) MVAPICH and created Topspin’s version of this release. However, in
addition, the HCAs also run using other popular InfiniBand MPI implementations.
Alternative MPI Implementations
Topspin customers have also deployed a variety of MPIs that use Mellanox’s VAPI layer. This includes
OSU, LAM-MPI, Verari Systems Software, Inc’s MPI/Pro (formerly Softech’s ), and LANL MPI.
Topspin products have also been used successfully with SCALI MPI, which is based on uDAPL.
Differences Between Topspin and Standard MPI
There are significant differences between the version of MPI provided, and OSU’s MPI.
• There is no restriction on which HCA port is used (OSU only supports Port 1)
• Support for Opteron 64 bit operation is provided
• Bug fixes have been provided for the purpose of improving stability
3
Architectural Elements
What is RDMA?
InfiniBand utilizes Remote Direct Memory Access (RDMA) technology. RDMA is a technology that
allows one computer to place information directly into the memory of another computer.
RDMA is specifically characterized by two important features:
• allows user space applications to directly access hardware
• zero-copy data movement
A combination of hardware and software allows user space applications to read and write the memory
of a remote system without kernel intervention or unnecessary data copies. This results in lower CPU
utilization per I/O operation and more efficient use of machine resources because applications place
most of the messaging burden upon InfiniBand’s high-speed network hardware.
Work Queues and Queue Pairs
A “verb” is the abstract description that is used to define the functionality of the Host Channel Adapter
(HCA). A “verb consumer” refers to the direct user of the verb.
A work queue provides a verb consumer with the ability to queue up a set of instructions that are
executed by the Channel Adapter. There are two types of Work Queues: Send Work Queue (outbound)
and a Receive Work Queue (inbound). Together these Work Queues create a Queue Pair.
4
The Queue Pair (QP) is one of the primary architectural elements of InfiniBand. In InfiniBand,
communication occurs between Queue Pairs, instead of between ports.
A Queue Pair (QP) in an addressable entity, and consists of two Work Queues: 1). Send Work Queue
and a 2). Receive Work Queue. (A work queue provides a verb consumer with the ability to queue up a
set of instructions that are executed by the Channel Adapter.) The Channel Adapter hardware takes over
the task of arbitrating communication - multiplexing access to the send queue or de-multiplexing
messages on the receive queue.
A connection is made by linking a local queue pair to a remote queue pair. Applications do not share
queue pairs; therefore, once you set them up, you can manage them at the application level without
incurring the overhead of system calls.
Send and Receive work queues are:
• always created as a pair
• always remain a pair
• known as Queue Pairs
• identified by a Queue Pair number, which is within the Channel Adapter.
Queue pairs have:
• a region of memory to be used as buffers (numbers of Queue Pairs are only limited by memory).
• a key that must match on each incoming packet (the Q_Key) to verify the validity of the packet
• (potentially) a partition key, which specifies the portion of the fabric that this queue pair may
access.
The queue pair is the mechanism by which you define quality of service, system protection, error
detection and response, and allowable services.
Types of Services
Each queue pair is independently configured for a particular type of service. These service types
provide different levels of service and different error-recovery characteristics.
The available transport-service types include:
• Reliable connection
• Unreliable connection
• Reliable Datagram
• Unreliable Datagram
Once the fabric connections are discovered, queue pairs and protection domains are established, and the
type and quality of service are defined for each queue pair, the fabric operates reliably and securely at
full performance without impact on system hardware or software resources.
Understanding the Subnet Manager (SM)
The Subnet Manager configures and maintains fabric operations. There can be multiple Subnet
Managers, but only one master.
For information regarding configuring the subnet managers, refer to “Using the Subnet Manager
Through the GUI” on page 61 or “Using the Subnet Manager Through the CLI” on page 81.
The Subnet Manager is the central repository of all information that is required to setup and bring up the
InfiniBand fabric.
The master Subnet Manager
• Discovers the fabric topology.
• Discovers endnodes.
• Configures switches and end nodes with their parameters, such as:
• Local Identifiers (LIDs)
• Global Unique Identifier (GUIDs)
• Partition Key (P_Keys)
• Configures switch forwarding tables.
• Receives traps from Subnet Management Agents (SMAs).
• Sweeps the subnet, discovering topology changes and managing changes as nodes are added and
deleted.
Understanding the Subnet Management Agents (SMAs)
Subnet Management Agents (SMA) are part of the Subnet Manager. A SMA is provided with each node
and process packets from the Subnet Manager.
If an Subnet Manager is elected master, all of its components, including SA, are implicitly elected
master. If a Subnet Manager ceases to be master, all of its components cease responding to messages
from clients.
Subnet Manager Hot Standby
The master and slave subnet managers can be synchronized so the information in the master is carried
over to the slave in the event of a fail-over. Refer to “Enable/Disable Database Synchronization” on
page 84 to configure SM hot standby.
The hot standby/database sync feature is used to synchronize the databases between subnet managers
running on separate chassis.
The Subnet Manager maintains a data base in the volatile memory of the master SM containing all
required information.
How is the synchronization done?
The database synchronization is accomplished in two stages:
• Cold Synchronization - This stage is initiated by the master SM when it is ready to start a
synchronization session with a standby SM. In this stage, all out of sync tables are copied from the
master SM to the standby SM.
• Transactional Synchronization - This stage is entered following successful completion of the cold
synchronization stage. In this stage, all database update transaction requests that are processed by
the master, are replicated to the standby.
What can cause a standby SM to become the master SM?
• A crash of the node running the current master SM.
• Partitioning of the subnet (e.g. due to link failure).
• Graceful shutdown of the master (e.g. for maintenance purposes).
What happens when a master subnet manager fails?
In the event of a failure:
• The standby subnet manager becomes the new master.
• The new master rebuilds the data base from information retrieved during the subnet discovery
phase.
• Existing LID assignments are retained, where possible.
• All ports are reset to force them to re-join multicast groups, re-advertise services, re-request event
forwarding, and re-establish connections.
• A “SlaveToMaster” event trap is generated to trigger any necessary processing by external
management applications.
5
Understanding Subnet Manager Routing
There are two different concepts associated with InfiniBand routing:
6
• Routing internally within a switch (hops between switch chips)
• Routing between whole switches (hops between nodes). This is also referred to as routing between
“switch elements.”
Internal switch routing can be configured to provide the highest performance in passing traffic, and to
minimize the threat of congestion within the switch.
The Routing Process Overview
1. The Subnet Manager (SM) first discovers all the InfiniBand switch chips in the network.
2. The SM groups the internal switch chips within each chassis into a “switch element.”
3. The SM process continues until all the InfiniBand switches are grouped into “switch elements.”
4. After all the switch chips are grouped, the SM will route the switch elements according to the
routing algorithm discussed in “Minimum Contention, Shortest Path & Load Balancing Algorithm”
on page 6.
5. The internal network of each InfiniBand switch is then routed based on the best algorithm for each
“switch element.”
Multiple Paths
The SM allows you to define the Logical Identifier Mask Control (LMC) value per subnet. The default
value of the LMC is 0, so by default only one Logical Identifier (LID) is assigned to each host port.
Once the LMC value has been assigned, the SM will route different paths for each LID associated with
the same host port. The result of these paths is based on the routing algorithm applied.
Understanding SM Routing Terms
The following terms are important to understand before distinguishing the various types of algorithms
that the Subnet Manager uses for routing:
Distance - Distance is defined as the number of hops (InfiniBand switches or “switch elements”)
between source and destination.
Contention - A contention is declared for every switch port on the path that is already used for routing
another LID associated with the same host port.
Minimum Contention, Shortest Path & Load Balancing
Algorithm
Minimum Contention, Shortest Path and Load Balancing is the algorithm that is used by default to route
between the “switch elements” and for routing between the internal InfiniBand switch chips within each
“switch element.”
The following algorithm is used for the calculation:
1. The shortest path for each of the host ports is calculated.
2. Contention is calculated for all the available paths that are within the (shortest path + tolerance)
distance.
a. The path with the least contention is selected.
b. If two paths have the same contention, the path with less distance is selected.
c. If two paths have the same contention and the same distance, the port usage count is used to
provide load balancing over the two paths. The usage count is a measure of how many LIDs have
been configured to use that particular port.
Configuring Your Network For Optimal Routing
Create Equal Paths Between Switch Elements
It is recommended that InfiniBand switch elements be connected so that all paths between any pair of
switch elements are the same distance (i.e. same number of hops), if possible. This enables you to
obtain the optimal paths using the default tolerance of 0.
Determine the First Path that will be Discovered
The SM Routing Algorithm selects the first best path that it finds. If multiple paths with the same
properties are available then the first of these paths found is the one that is selected. Therefore, it is
possible to setup the cabling between switch elements to force the algorithm to prioritize certain paths.
Depending on the network requirements, the prioritized paths can either be concentrated on a particular
switch element or spread across multiple switch elements to improve fault-tolerance.
7
8
Getting Started
The information in this chapter focuses on the software and firmware aspects of the initial set-up, and
assumes that you have additional documentation for the hardware.
This chapter provides the following information:
• Overview of entire system installation on page 9, with references to more detailed information.
9
2
• Setup procedures for the InfiniBand
™ switch.
Overview of Set-up Procedures
Follow the steps below to configure the InfiniBand server switch system.
1. Determine your hardware topology.
2. Install the Host Channel Adapter and drivers (page 10).
3. Install and power-on the InfiniBand Chassis (page 10).
4. Configure Basic Connectivity (page 10).
5. Test Network Connectivity (page 13).
6. Install the Element Manager GUI (page 14).
7. Access a Management System (page 14).
8. Configure Partitions (page 15).
9. Set User Level and Access (page 18).
10
Install HCAs and Drivers in Hosts
Refer to the HP Dual-port 4x Fabric Adapter Quick Setup Installation card and the HP Dual-port 4x
Fabric Adapter User Guide.
Install and Power on the InfiniBand Chassis
Refer to the HP 24-Port 4x Fabric Copper Switch Hardware Quick Setup Installation card and the HP
24-Port 4x Fabric Copper Switch Hardware User Guide for installation and power instructions.
Configure Basic Connectivity
The InfiniBand switch is not pre-configured with an IP address. You must configure the IP address of a
management port to administer and monitor the InfiniBand switch with the CLI and Element Manager.
A Management port is provided for a connected Ethernet host running TCP/IP or connected InfiniBand
hosts running IPoIB. Configure the Management port you wish to use.
• “Configuring an Ethernet Management IP Address” on page 10
• “Configuring an InfiniBand Management IP Address” on page 12
Login: super
Password: super
Topspin-360>
Configuring an Ethernet Management IP Address
To configure an out-of-band Ethernet Management IP address:
1. Make sure that the InfiniBand switch is attached to a PC or terminal via the serial port. Refer to the
HP 24-Port 4x Fabric Copper Switch Hardware Quick Setup Installation card and the HP 24-Port
4x Fabric Copper Switch Hardware User Guide.
2. Open a terminal emulation program, such as HyperTerminal for Windows
parameters as follows:
• Baud: 9600 b/s
• Data Bits: 8
•Parity: None
• Stop Bits: 1
• Flow control: None
3. At the Login: prompt, enter the username and password. The default is super and super.
Example
®, and set the session
4. At the CLI prompt, enter enable. This enters the privileged-execute mode.
Topspin-360> enable
5. Enter configure to enter the global-configuration mode.
Topspin-360# configure
Topspin-360(config)#
6. Set the IP address and netmask. The following address is an example.
Topspin-360(config)# interface mgmt-ethernet
Topspin-360(config mgmt-ethernet)# ip address 10.10.0.22 255.255.255.0
7. Set the default gateway address. This address is an example.
Topspin-360(config mgmt-ethernet)# gateway 10.10.0.1
8. Enable the management port
Topspin-360(config mgmt-ethernet)# no shutdown
9. Test IP connectivity by pinging the management station.
Topspin-360(config mgmt-ethernet)# exit all
Topspin-360> ping 10.10.0.3
sending 5 ICMP Echos to 10.10.0.3, 56 data bytes
!!!!!
Success rate is 100 percent (5/5)
round-trip min/avg/max = 0.000000/0.000000/0.000000 ms
Topspin-360>
copy
10. You must save the configuration persistently by using the
command.
11
Topspin-360(config mgmt-ethernet)# exit
Topspin-360# copy running-config startup-config
You are now ready to power down the chassis and mount it. Later, you can configure the box via
Telnet, SSH, Chassis Manager or Element Manager.
Configuring a Direct Serial-Console Connection
Refer to the HP Serial Management Cable Guide for information regarding setting up the physical
serial-console connection.
Remote Telnet Login
You can Telnet to the Management-Ethernet port on the box from a host on the same network as the
Management-Ethernet port, or from any host with a route to the Management-Ethernet network.
To run the CLI remotely:
1. Open a terminal or terminal emulator window.
For example: from the command line, enter the telnet command with the IP address, or network
name, of the Management-Ethernet port.
# telnet 10.0.0.47
The CLI login prompt (Login: ) is displayed.
2. Enter a CLI user name.
The CLI password prompt (Password:) is displayed.
3. Enter the CLI user password.
The prompt changes to indicate a successful login. The system is now ready to receive CLI
commands.
Remote SSH Login
TopspinOS supports SSH2 for secure, encrypted login to the CLI. SSH is enabled by default, and does
not require additional configuration.
12
Login: super
Password: super
Topspin-360>
Topspin-360> enable
Topspin-360#
Topspin-360# configure
Topspin-360(config)#
To login via SSH:
1. Use an SSH client (e.g. Putty) to port 22.
Configuring an InfiniBand Management IP Address
To configure an In-band InfiniBand management IP address:
1. At the Login: prompt, enter the username and password. The default is super and super.
2. At the CLI prompt, enter enable. This enters the privileged-execute mode, as indicated by the #
sign.
3. Enter configure to enter the configuration mode.
4. Enter the interface to be configured, and set the IP and mask addresses.
Topspin-360(config)# interface mgmt-ib
Topspin-360(config mgmt-ib)# ip address 10.3.102.20 255.255.255.0
5. Set the default gateway address. The gateway address refers to the address of the internal port.
Topspin-360(config mgmt-ib)# gateway 10.3.0.1
6. Enable the IB management port.
Topspin-360(config mgmt-ib)# no shutdown
7. Test IP connectivity by pinging an InfiniBand host on the other side of the gateway.
Topspin-360(config mgmt-ib)# exit all
Topspin-360> ping 10.3.102.34
sending 5 ICMP Echos to 10.3.102.34, 56 data bytes
!!!!!
Success rate is 100 percent (5/5)
round-trip min/avg/max = 0.000000/0.000000/0.000000 ms
Topspin-360>
8. Save the configuration by using the copy command, or wait until you execute the reload command.
You will be prompted to save the unsaved configuration changes. .
Topspin-360(config mgmt-ethernet)# exit
Topspin-360# copy running-config startup-config
You are now ready to power down the chassis and mount it. Later, you can configure the box via
Telnet, SSH, or the Element Manager.
Configuring the System Hostname
The Topspin system allows you to assign a hostname to the system for management purposes.
To assign a hostname name to the management port:
1. Start a CLI session.
2. Enter the privileged-user mode.
Topspin-360> enable
Topspin-360#
3. Enter the global-configuration mode.
Topspin-360# configure
Topspin-360(config)#
13
4. Enter the
The
prompt.
Topspin-360(config)# hostname MyHost
NOTE: This command also changes the CLI prompt. The new hostname is applied immediately,
however, the prompt does not change until you change modes. For example, the prompt changes
when you exit the global-configuration mode.
hostname
hostname
command with the name you wish to assign.
command assigns a convenient name to the system that shows up at the CLI
Test Network Connectivity
Refer to the HP 24-Port 4x Fabric Copper Switch Hardware User Guide for information regarding
connecting network devices.
After you install network cables, you can verify connectivity by pinging those connected devices from
the CLI or pinging between attached hosts.
Verify Communication Between Hosts
To verify the device recognizes and successfully links InfiniBand-attached hosts, enter the
command on one host and specify the IP address of another connected host.
# ping 10.2.65.50
PING 10.2.0.50 (10.2.0.50) from 10.2.0.41 : 56(84) bytes of data.
64 bytes from 10.2.0.50: icmp_seq=0 ttl=64 time=164 usec
64 bytes from 10.2.0.50: icmp_seq=1 ttl=64 time=144 usec
…
…
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max/mdev = 0.135/0.147/0.164/0.017 ms
#
ping
14
Verify Switch to Host Communication
To verify the InfiniBand device can reach a host on the network, enter either the user-execute or
privileged-execute mode on the InfiniBand device, then enter the
procedure only.
Topspin-360# ping 10.10.253.47
Sending 5 ICMP Echos to 10.10.253.47, 56 data bytes
!!!!!
Success rate is 100 percent (5/5)
round-trip min/avg/max = 0.000000/0.000000/0.000000 ms
Topspin-360#
Install the GUI (Element Manager)
HP 24-Port 4x Fabric Copper Switches can be managed visually through the Element Manager, which
provides a wide range of configuration, monitoring, and troubleshooting options.
Refer to “Installing the Element Manager Program” on page 27 for information regarding the Element
Manager installation.
ping
command. This is an In-band
Access a Management System
Default User Name and Passwords
For initial configuration, log in as the unrestricted user.
• The default unrestricted username for the CLI is
• The default community-string assigned to this user for the Element Manager is
Use the following methods to manage the Topspin system.
CLI Management
Refer to “About the CLI” on page 19 for more information about managing through the CLI.
Run the Command Line Interface (CLI) from one of the following methods:
• “Direct Serial-Console Connection” on page 14
• “Remote Telnet Login” on page 14
• “Remote SSH Login” on page 15
Direct Serial-Console Connection
Refer to the HP Serial Managment Cable Guide for information regarding setting up the physical
serial-console connection.
super
and the default password is
super
secret
.
.
Remote Telnet Login
You can Telnet to the Management-Ethernet port on the box from a host on the same network as the
Management-Ethernet port, or from any host with a route to the Management-Ethernet network.
To run the CLI remotely:
1. Open a terminal or terminal emulator window. For example: from the command line, enter the
telnet
# telnet 10.0.0.47
command with the IP address, or network name, of the Management-Ethernet port.
15
The CLI login prompt (
2. Enter a CLI user name.
The CLI password prompt (
3. Enter the CLI user password.
The prompt changes to indicate a successful login. The HP 24-Port 4x Fabric Copper Switch
system is now ready to receive CLI commands.
Login:
) is displayed.
Password:
) is displayed.
Remote SSH Login
TopspinOS supports SSH2 for secure, encrypted login to the switch CLI. SSH is enabled by default,
and does not require additional configuration.
1. To login via SSH, use an SSH client (e.g. Putty) to port 22.
GUI Management
1. Refer to “About Element Manager” on page 24 for more information about managing through the
CLI.
2. Run the Element Manager (GUI) over a TCP/IP network.
3. To log in to the GUI, refer to “Starting the Element Manager” on page 29.
SNMP Management
For more information regarding SNMP, refer to “About SNMP” on page 30.
Any Network Manager running the Simple Network Management Protocol (SNMP) can manage the
Topspin system, if the Management Information Base (MIB) is installed correctly. By default, the
Topspin GUI is a network manager and uses SNMP v2c as the protocol to communicate between the
chassis and the management workstation.
(Optional) Enable Database Sync
If you are configuring more than one InfiniBand chassis in your fabric, it is likely that you will want to
enable database synchronization of the subnet managers.
• To enable data synchronization with the Element Manager GUI, refer to “Enable/Disable Database
Synchronization” on page 66.
• To enable data synchronization with the CLI, refer to “Enable/Disable Database Synchronization”
on page 84.
Configure Partitions
Partitions are described in detail in “About Partitions” on page 53.
16
Create a Partition ID (P_Key)
A default partition is configured automatically. The members of a default partition include all connected
ports, and provide full membership. However, to create separation between traffic, you must configure
specific partitions.
1. Launch Element Manager, if you have not already done so.
2. Select InfiniBand --> Subnet Management.
The Subnet Management window appears.
3. Click open the Subnet Manager folders in the left window.
The Partitions folder appears.
4. Click on the Partitions folder in the left window. The Partitions Summary window appears.
5. Click the Add button.
The Add Partition dialog box appears.
Enter a Partition key (P_Key) to identify the new partition. For information regarding selecting
values, refer to the “Selecting a P_Key Value” on page 54.
00:01
a. Click the Add button.
The new Partition appears in the left window.
Specify Partition Members and the Membership Type
b. Click on the new Partition in the left window.
The available partition members appear in the right-side window.
p
17
Note that the “Available Members” refers only to members that are known to the Subnet Manager.
This includes HCAs and switches that are already plugged into the fabric as well as manually
configured entries.
If you know the GUID and port count of an HCA that has not yet been installed, you can configure
it before it is plugged in by using the “Add Other” button.
c. Click on a member from the Available Member list, and use the arrow button to move it to the Full
or Limited member columns.
For information regarding Membership Types, refer to the “Membership Types” on page 54
d. Click back to the Partitions folder (in the left-side window) when you have selected all of the
members for your Partition.
The new Partition appears in the Partition Summary window.
Maintain Partition Key Information
The configured p_keys will be needed in completing the configuration of the system.
e. Configured partition keys must be mapped to any of the following components that exist:
• Host Channel Adapters (HCAs). Refer to the HP Dual-port 4x Fabric Adapter User
Guide.
• Ethernet Gateway Bridge-groups.
18
• Fibre Channel gateways.
f. If you have multiple InfiniBand switches in your fabric:
• Exchange the partition configuration between switches by enabling database
synchronization, if you have not already done so. Refer to “Enable/Disable Database
Synchronization” on page 84.
Set User Levels and Passwords
Change Default User Name and Password
For security purposes, since multiple users exist on the system, it is highly recommended that you
change the default passwords after initial configuration.
See “Understanding Usernames and Passwords” on page 46 for more information.
1. Log in to the CLI as a super user. Use the default username (super) and the default password
(super) if they have not already been changed (refer to page 47).
2. Enter the privileged-user mode.
3. Enter the global-configuration mode.
4. Enter the
password.
Use the default user name and password if they have not already been changed (refer to page 47).
The user name and password are alphanumeric strings of up to 34 characters each.
username command and the
password
keyword to change the user account and user
5. Repeat step 4 to change additional usernames and passwords.
Example
Topspin-360# Login: super
Password: xxxx
Topspin-360>
Topspin-360#
Topspin-360(config)#
Topspin-360(config)# username ib-fc_admin communitystring
ibFc-commStr
enable
configure
username
ib-fc_admin
password
ibFcAdmin
6. Exit the global-configuration mode.
7. Use the
show user
command to verify changes.
Only a user with unrestricted privileges may view user information.
Topspin-90> show user all
=========================================================================
User Information
=========================================================================
username : admin
password : topspin
snmp-community : justatest
permission-level : ib-rw, ip-ethernet-rw, fc-rw
admin-status : enabled
num-logins : 0
num-unsuccessful-logins : 0
last-login :
last-unsuccessful-login :
Topspin-90>
Loading...