Cisco Systems GEM318P User Manual
Size:
852.97 Kb
Download

Cisco SFS InfiniBand Host Drivers User Guide for Linux

Release 3.2.0

June 2007

Americas Headquarters

Cisco Systems, Inc. 170 West Tasman Drive

San Jose, CA 95134-1706USAhttp://www.cisco.com Tel: 408526-4000

800 553-NETS(6387) Fax: 408527-0883

Text Part Number: OL-12309-01

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMEDSUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

CCVP, the Cisco logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco Systems, Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys, MeetingPlace, MGX, Networking Academy, Network Registrar, Packet, PIX, ProConnect, ScriptShare, SMARTnet, StackWise, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0705R)

Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.

Cisco SFS InfiniBand Host Drivers User Guide for Linux

© 2007 Cisco Systems, Inc. All rights reserved.

 

 

 

 

 

 

 

 

C O N T E N T S

 

 

 

Preface vii

 

 

 

 

 

 

 

 

 

 

 

Audience

vii

 

 

 

 

 

 

 

 

 

Organization

vii

 

 

 

 

 

 

 

 

 

Conventions

viii

 

 

 

 

 

 

 

 

 

Root and Non-root Conventions in Examples

ix

 

 

 

Related Documentation ix

 

 

 

 

 

 

 

Obtaining Documentation, Obtaining Support, and Security Guidelines ix

 

 

About Host Drivers

 

 

 

 

 

 

C H A P T E R

1

1-1

 

 

 

 

 

 

 

 

Introduction

1-1

 

 

 

 

 

 

 

 

 

Architecture

1-2

 

 

 

 

 

 

 

 

 

Supported Protocols

1-3

 

 

 

 

 

 

 

IPoIB

1-3

 

 

 

 

 

 

 

 

 

SRP

1-3

 

 

 

 

 

 

 

 

 

SDP

1-3

 

 

 

 

 

 

 

 

 

Supported APIs 1-4

 

 

 

 

 

 

 

 

MVAPICH MPI

1-4

 

 

 

 

 

 

 

uDAPL

 

1-4

 

 

 

 

 

 

 

 

 

Intel MPI

1-4

 

 

 

 

 

 

 

 

 

HP MPI

1-4

 

 

 

 

 

 

 

 

 

HCA Utilities and Diagnostics 1-4

 

 

 

 

 

 

Installing Host Drivers

 

 

 

 

 

C H A P T E R

2

2-1

 

 

 

 

 

 

 

Introduction

2-1

 

 

 

 

 

 

 

 

 

Contents of ISO Image

2-2

 

 

 

 

 

 

 

Installing Host Drivers from an ISO Image

2-2

 

 

 

 

 

 

Uninstalling Host Drivers from an ISO Image

2-3

 

 

 

 

 

IP over IB Protocol

 

 

 

 

 

 

C H A P T E R

3

3-1

 

 

 

 

 

 

 

 

Introduction

3-1

 

 

 

 

 

 

 

 

 

Manually Configuring IPoIB for Default IB Partition 3-2

 

 

 

 

 

 

 

 

Cisco SFS InfiniBand Host Drivers User Guide for Linux

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

OL-12309-01

 

 

 

 

 

 

 

iii

 

 

 

 

 

 

 

 

 

 

Contents

 

 

 

 

 

Subinterfaces

3-2

 

 

 

 

 

 

 

 

 

 

 

 

Creating a Subinterface Associated with a Specific IB Partition

3-3

 

 

 

 

 

 

Removing a Subinterface Associated with a Specific IB Partition

3-4

 

 

 

 

 

 

Verifying IPoIB Functionality

3-5

 

 

 

 

 

 

 

 

 

 

IPoIB Performance

3-6

 

 

 

 

 

 

 

 

 

 

 

 

Sample Startup Configuration File

3-8

 

 

 

 

 

 

 

 

 

IPoIB High Availability

3-8

 

 

 

 

 

 

 

 

 

 

 

Merging Physical Ports

3-8

 

 

 

 

 

 

 

 

 

 

Unmerging Physical Ports 3-9

 

 

 

 

 

 

 

SCSI RDMA Protocol

 

 

 

 

 

 

 

 

C H A P T E R

4

 

4-1

 

 

 

 

 

 

 

 

 

 

 

 

Introduction

4-1

 

 

 

 

 

 

 

 

 

 

 

 

 

Configuring SRP

4-1

 

 

 

 

 

 

 

 

 

 

 

 

Configuring ITLs when Using Fibre Channel Gateway 4-2

 

 

 

 

 

 

 

Configuring ITLs with Element Manager while No Global Policy Restrictions Apply 4-2

 

 

 

 

 

Configuring ITLs with Element Manager while Global Policy Restrictions Apply 4-4

 

 

 

 

 

Configuring SRP Host

4-6

 

 

 

 

 

 

 

 

 

 

Verifying SRP

4-7

 

 

 

 

 

 

 

 

 

 

 

 

Verifying SRP Functionality

4-7

 

 

 

 

 

 

 

 

 

Verifying with Element Manager

4-8

 

 

 

 

 

 

Sockets Direct Protocol

 

 

 

 

 

 

 

C H A P T E R

5

 

5-1

 

 

 

 

 

 

 

 

 

 

 

Introduction

5-1

 

 

 

 

 

 

 

 

 

 

 

 

 

Configuring IPoIB Interfaces

5-1

 

 

 

 

 

 

 

 

 

 

Converting Sockets-Based Application

5-2

 

 

 

 

 

 

 

 

Explicit/Source Code Conversion Type

5-2

 

 

 

 

 

 

 

Automatic Conversion Type

5-2

 

 

 

 

 

 

 

 

 

Log Statement

5-3

 

 

 

 

 

 

 

 

 

 

Match Statement

5-3

 

 

 

 

 

 

 

 

 

 

SDP Performance

5-4

 

 

 

 

 

 

 

 

 

 

 

 

Netperf Server with IPoIB and SDP 5-6

 

 

 

 

 

 

uDAPL 6-1

 

 

 

 

 

 

 

 

 

C H A P T E R

6

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Introduction

6-1

 

 

 

 

 

 

 

 

 

 

 

 

 

uDAPL Test Performance

6-1

 

 

 

 

 

 

 

 

 

 

uDAPL Throughput Test Performance

6-2

 

 

 

 

 

 

 

uDAPL Latency Test Performance

6-3

 

 

 

 

 

 

 

 

Compiling uDAPL Programs

6-4

 

 

 

 

 

 

 

 

 

Cisco SFS InfiniBand Host Drivers User Guide for Linux

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

iv

 

 

 

 

 

 

 

 

 

 

 

OL-12309-01

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Contents

C H A P T E R

7

MVAPICH MPI

7-1

 

 

 

 

 

 

Introduction

7-1

 

 

 

 

 

 

Initial Setup

7-2

 

 

 

 

 

 

Configuring SSH 7-2

 

 

 

 

 

 

Editing Environment Variables

7-5

 

 

 

 

Setting Environment Variables in System-Wide Startup Files

7-6

 

 

Editing Environment Variables in the Users Shell Startup Files

7-6

 

 

Editing Environment Variables Manually

7-7

 

 

 

MPI Bandwidth Test Performance 7-7

 

 

 

 

MPI Latency Test Performance

7-8

 

 

 

 

Intel MPI Benchmarks (IMB) Test Performance

7-9

 

 

 

Compiling MPI Programs

7-12

 

 

 

 

HCA Utilities and Diagnostics

 

 

 

C H A P T E R

8

8-1

 

 

 

 

Introduction

8-1

 

 

 

 

 

 

hca_self_test Utility 8-1

 

 

 

 

 

 

tvflash Utility

8-3

 

 

 

 

 

 

Viewing Card Type and Firmware Version

8-3

 

 

 

Upgrading Firmware

8-4

 

 

 

 

 

Diagnostics

8-5

 

 

 

 

 

 

Acronyms and Abbreviations

 

 

 

A P P E N D I X

A

A-1

 

 

 

 

 

 

 

 

 

 

I N D E X

 

 

 

 

 

 

 

Cisco SFS InfiniBand Host Drivers User Guide for Linux

 

OL-12309-01

v

 

Contents

Cisco SFS InfiniBand Host Drivers User Guide for Linux

 

vi

OL-12309-01