HP HyperFabric Setup and Install

Installing and Administering HyperFabric

HP-UX 11i v1 and HP-UX 11i v2

Edition 13

Manufacturing Part Number: B6257-90060

October 2006

Printed in U.S.A.

Legal Notices

The info rm a tion in this do cument is su b je ct to change without not i ce.

Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewl e tt-Packar d sh all not be he l d li able for e rrors con t ai ned herein or dire ct,

indirect, special, incidental or consequential damages in connection with the furnishing, perf o r m ance, or use of t h is mat e r i a l.

Warranty. A copy of the specific warranty terms applicable to your Hewlett- Packard product and replacement parts can be obtained from your loca l Sales and Serv ice Of fi ce.

U.S. Government License

Proprietary computer softw a r e. Valid lice nse f rom HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government un der vend o r's standard commercial license.



Copyright reserved. Rep roduction, adaptation , or translation of this document without prior written permis sio n is prohibited, except as allowe d und er the co pyr ight laws.

Trademark Notices

2003-2006 Hewlett-Packard Development Compan y L.P. All rights

Oracle UNIX

exclusiv el y thro ugh Th e Open Group.



is a registered trademark of Oracle Corporation.



is a registered trademark in the United States and other countries, licensed

1. Overview

Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

HyperFabric Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

HyperFabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Switches and Switch Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Other Product Elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

HyperFabric Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2. Planning the Fabric

Preliminary Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

HyperFabric Functionality for TCP/IP and HMP Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

TCP / IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Application Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

TCP/IP Supported Configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Point-to-Point Configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Switched. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

High Availability Switched. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Hybrid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Mixed HF1 / HF2 (Copper & fibre). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Hyper Messaging Protocol (HMP). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Application Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

HMP Supported Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Point to Point. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Enterprise (Database). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Technical Computing (Work Stations) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Contents

3. Installing HyperFabric

Checking HyperFabric Installation Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Installing Hyper Fabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Online Addition and Replacement—HP-UX 11i Only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Planning and Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Critical Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 7

Card Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Online Addition (OLA). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Online Replacement (OLR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Installing the Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

File Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Loading the Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Installing Hyper Fabric Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Before Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Steps for Installing the HF 1 Sw itch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Installing the H F2 Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Contents

With the Rail Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Installing the HF2 Switch With the Rail kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Without the Rail Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1

4. Configuring HyperFabric

Configuration Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Information You Need . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Configuration Information Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

Doing the Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Using the clic_init Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Examples of clic_init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3

Using SAM—HP-UX 11.0 and HP-UX 11i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

Deconfigu ri ng a HyperFabric Adapte r with SAM—HP-UX 11.0 and 11i 0nly. . . . . . . . . . . . . . . . . . . . 96

Configuring the HyperFabric EMS Mon itor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Configurin g HyperFabr ic with MC/ServiceGuard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

How HyperFabric Handles Adapter Fail ures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Configuring HyperFabric with the MC/ServiceGuard Resource Monitor . . . . . . . . . . . . . . . . . . . 104

Configuring MC/ServiceGuard with HyperFabric Usi ng the ASCII File. . . . . . . . . . . . . . . . . . . . 1 05

Configuring MC/ServiceGuard with HyperFabric Using SAM. . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Configuring MC/ServiceGuard for HyperFabric Relocatable IP Addresses. . . . . . . . . . . . . . . . . . 106

5. Managing HyperFabric

Starting HyperFabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Using the clic_start Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Using SAM—HP-UX 11.0 and 11i 0nly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Verifying Communications with in the Fabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

The clic_probe Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Examples of clic_probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Displaying Status and Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 15

The clic_stat Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 15

Examples of clic_stat. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Viewing man Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

Stopping HyperFabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 23

Using the clic_shutdown Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Using SAM—HP-UX 11.0 and 11i 0nly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

6. Troubleshooting HyperFabric

Running Diag nostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

The clic_diag Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Example of clic_diag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Using Support Tools Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Useful Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

LED Colors and Their Meanings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

Adapter LEDs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

HF1 Switch LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

HF2 Switch LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Contents

Determining Whether an Adapter or a Cable is Faulty. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

Determining Whether a Switch is Faulty. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

HF1 Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

HF2 Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Replacing a HyperFabric Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Replacing a HyperFabric Switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

Contents

HF1 Speed and Latency w/ TCP/IP Applications 30 HF2 Speed and Latency w/ TCP/IP Applications 30 Supported Configurations for A6386A HF2 Adapter On PCI (4X) 31 HF1 Speed and Latency w/ HMP Applications 41 HF2 Speed and Latency w/ HMP Applications 41 Supported Configurations for A6386A HF2 Adapter On PCI (4X) 42 Important OLAR Terms 56 LED Names (by Adapter) 135 HyperFabric Adapter LED Colors and Meanings 136 HF1 Switch LED Colors and Meanings 140 HF2 Switch LED Colors and Meanings 144

Tables

TCP/IP Point-To-Point Configurations 33 TCP/IP Basic Switched Configuration 34 TCP/IP High Availability Switched Configuration 35 TCP/IP Hybrid Configuration 36 TCP/IP Mixed HF1 & HF2 Configuration 37 HMP Point-To-Point Configurations 44 HMP Enterprise (Database) Configuration, Single Connection Between Nodes 46 HMP Enterprise (Database) Configuration, Multiple Connections Between Nodes 47 Technical Computing Configuration 49 Large Technical Computing Configuration 50 HyperFabric File Structure 60 Back of HF1 Switch 68 Front of HF2 Switch (A6388A Switch Module Installed) 76 Front of HF2 Switch (A6389A Switch Module Installed) 77 Parts of the Rail Kit 78

The Ends of the Rail Kit 79 Map for Configuration Information Example 88 An MC/ServiceGuard Configuration (with Two HyperFabric Switches) 100 Node with Two Active HyperFabric Adapters 102 Node with One Failed HyperFabric Adapter 103 When All HyperFabric Adapters Fail 104

Figures

Printing History

The manua l printing date and part num be r in di cate its c urr e nt ed itio n. Th e pr intin g date will change when a new edition is printed. Minor changes may be made at reprint without changing the printing date. The manual part number will change when extensive changes are made.

Manual updates may be issued between editi ons to corr ect err ors or documen t produc t changes . T o ensure that you receive the updated or new editi ons, you should subscribe to the appropriate product support service. See your HP sales representative for details.

First Edition: March 1998 Second Edition: June 1998 Third Edition: August 1998 Fourth Edition: October 1998 Fifth Edition: December 1998 Sixth Edition: February 1999 Seventh Edition: April 1999 Eighth Edition: March 2000 Ninth Edition: June 2000 Tenth Edition: December 20 00 Eleventh Edition: June 2001 Twelfth Edition: September 2002 Thirteenth Edition: March 2006

1 Overview

This chapter con tains the following sections that give general informatio n about HyperFabric:

• “Overview” on page 1 5

• “HyperF abric Products” on page 16

Chapter 1

Overview

•“HyperFabric Concepts” on page 19

Chapter 1

Overview

HyperFabric is a Hewlett-Packard high-speed, packet-based inter conne ct for node-to-node communi cat i ons. HyperFabric provides high er speed, lower ne twor k latency and less CPU usage than other industry standard protocols (e.g. Fibre Channel and Gigabit Ethernet). Instead of using a traditional bus based technology, HyperF abric is built around switched fabric architecture, providing the bandwidth necessary for high speed data transfer. This clustering solution delivers the performance, scalability and high availability requ ir ed by:

• Parallel Database Clusters: Oracle 9i Real Application Clusters (RAC)

Oracle 8i Parallel Servers (OPS)

• Parallel Computing Clusters

• Client/Server Archit e cture Interco nn e cts (e.g. SAP)

• Multi-Server Batch Applications (e.g. SAS Systems)

• Enterprise Resource Planning (ERP)

• Te ch nical Computing C lusters

• Omniback

• Network Backup

• NFS

• Data Center Network Consolidation

• E-services

Oracle RAC10g Support Notice

HyperFabric product suite was designed to optimize per formance of Oracle RAC9i database running on HP-U X cluster s. With the industry moving to standards-base d networking technologies for database clustering solutions, HP and Oracle have worked together to optimize features and performance of Oracle RAC10g database with standards-b ase d interc o nn ect te chn o logie s inc lud in g Gigabit Ethernet, 10Gigabit Ethernet and Infiniband.

To align with the market trend for standards-based interconnects, Oracle RAC10g database is not currently supported on configurations consisting of HyperFabric product suite and it will not be supported in the futur e either. As a result, customers must switch to Gigabit Ethe rnet, 10Gigab it Et hernet or Infiniba nd te chnology if they plan to use Oracle RAC10g.

Chapter 1

Please note that configurations comprising HyperF abric and Oracle 9i continue to be supported.

Overview

HyperFabric Products

Hyper Fabric Products

HyperFabric hardware consists of host-based interface adapter cards, interconnect cables and optional switches. HyperFabric software resides in ASICs and firmware on the adapter cards and includes user space components and HP-UX drivers.

Currently both copper and fibre base d HyperFabric hardware is available. There is also a hybrid switch that has 8 fibre ports and 4 copper ports to support mixed HF1 and HF2 clusters.

The various HyperFabric products are desc ribed belo w. Se e the HP HyperFabric Release Note for information about the HP 9000 systems these products ar e suppor te d on .

NOTE In this manual, the term HyperFabric (HF) is used in general to refer to the hardware

and software that form the HyperFabric cluster interconnect product. The term HyperFabric1 (HF1) refe rs to the copp e r b a sed hardwa re co mponents:

• The A4919A, A4920A, A4921A, and A6092A adapters.

• The A4891A switch.

• The A4892A cable.

The term HyperFabric2 (HF2) refers to the fibre based hardware components:

• The A6386A adapter.

• The A6384A switch chassis.

• The A6388A and A6389A switch modules. (Although the A6389A switch module has

4 copper ports it is still considered a HF2 component because it can only be used with the A6384A HF2 switch chassis).

• The C7524A, C7525A, C7526A, and C7527A cables.

HyperFabric Adapters

The HyperFabric adapters include the following:

• A4919A HF1 PCI (1X) adapter with a copper interface. (Discontinued...04-02)

• A4920A HF1 HSC adapter with a copper inter face. (Discontin ued ...09-02)

• A4921A HF1 EISA/HSC adapter with a copper interface. (Discontinued...09-02)

• A6092A HF1 PCI (4X) adapter with a copper interface.

• A6386A HF2 PCI (4X) adapter with a fibre interface.

The A4919A, A4920A, and A4921A HF1 adapters are supported beginning with the following Hyper Fabric software versio ns:

• HP-UX 10.20: HyperFabric software version B.10.20.02

• HP-UX 11.0: HyperFabric software version B.11.00.02

• HP-UX 11i: HyperFabric software version B.11.11.00

Chapter 1

Overview

HyperFabric Products

The A6092A HyperFabric adapter is supported beginning with the following HyperFabric sof t ware versions:

• HP-UX 10.20: HyperFabric software version B.10.20.09

• HP-UX 11.0: HyperFabric software version B.11.00.09

• HP-UX 11i: HyperFabric software version B.11.11.00

The A6386A HyperF abric2 adapter is supported beginning with the following HyperFabric sof t ware versions:

• HP-UX 11.0: HyperFabric software version B.11.00.11

• HP-UX 11i: HyperFabric software version B.11.11.01

Switches and Sw itc h M od ule s

The HyperFabric1 and HyperFabric2 switches are as follows:

• A4891A HF1 16-port copper switch with an Ethernet port.

• A6384A HF2 fibre switch chassis with one integrated Ethernet management LAN

adapte r card, one integrated 8- p o rt f ib r e card, and one expansi o n sl o t. For the chassis to be a functional switch, one of these two switch modules must be installed in the expansion slo t:

— The A6388A HF2 8-port fibre switch module. This gives the switch 16 fibre ports

(8 from the integrated fibre card and 8 from the A6388A).

— The A6389A HF2 4-port copper switch module. This gives the switch 12 ports—a

mixture of 8 fibre ports (from the integrated fibre card) and 4 copper ports (from the A6389A module). This swit ch modu le is com patible wit h HF1 com po nents making it possible to have a fabric comp osed of both HF1 and HF 2 com po nen ts.

The A4891A HF1 switch is supported beginning with the following HyperFabric software ve rsions:

• HP-UX 10.20: HyperFabric software version B.10.20.02

• HP-UX 11.0: HyperFabric software version B.11.00.02

• HP-UX 11i: HyperFabric software version B.11.11.00

The A6384A HF2 switch chassis with either module installed is supported beginning with the following Hyp erFabric software versio n s:

• HP-UX 11.0: HyperFabric software version B.11.00.11

• HP-UX 11i: HyperFabric software version B.11.11.01

NOTE In this manua l, the terms HyperFabric2 switch or HF2 switch refer to th e functional

switch (the A6384A switch chassis wit h one of the swit ch mod ules ins talle d) .

IMPORTANT HF1 and HF2 adapters and switches are not supported by software versions earlier than

those listed in “Hyper Fabric Adapters” on page 16 and “Switches and Switch Modules” on page 17.

Chapter 1

Overview

HyperFabric Products

To determine the version of HyperFabric you have, issue this command:

swlist | grep -i hyperfabric

Other Product Elements

The other elements of the HyperFabric product family are the following:

• A4892A HF1 copper cable (in 35-foot and 60-foot lengths).

• HF2 fibre cables:

— C7524A (2m length) — C7525A (16m length) — C7526A (50m length) — C7527A (200m length)

• The HyperFabric software: The software resides in ASICs and firmware on the

adapte r cards and in cl ud e s user space co m p o nents and HP - U X d rivers. HyperFabric supports the IP networ k proto c ol st ack, spe cific ally TCP /IP, UDP/IP,

and NFS. HyperFabric software includes HyperMessaging Protocol (HMP). HMP provides

higher bandwidth , lowe r CPU ove rhead , and lo wer late n cy (th e time it tak es a message to get from one point to another). Howev er, these HMP benefits are only available when applications that were deve lop ed on to p of HMP are ru nning. Note that HMP can only be used on HP 9000 systems ru nning HP-UX 11.0 or 11 i provided HyperFabric A6092A or A6386A (PCI 4X) adapter cards are installed on those systems.

In addition, running an HMP appl icati on disable s a node’s ability to inte rop e ra te with nodes that are using any HP-UX 10.20 version of HyperFabric, any HP-UX 11.0 HyperFabric versions earlier than B.11.00.11 or any HP-UX 11i HyperFabric versions earlier than B.11.11.01. If you use HMP on a node in the fabric, that node cann ot commu ni c ate wi th a n y o the r no de s t h at a re runn i ng t he a bo ve ve rsi on s o f th e HyperFabric software. See Chapter 2, “Planning th e Fabric,” on page 21 for detai ls on using HMP applic atio n s in a Hype r Fabric cluster.

Chapter 1

Overview

HyperFabric Concepts

HyperFabric Conc epts

Some basic HyperFabric concepts and terms are briefly described below. The fabric is the physical configuration that consists of all of the HyperF abric adapters,

the HyperFabric switches (if any are used) and th e Hy per Fabric cables conne cti ng t hem. The network software controls data tr ansfer over the fabric.

A HyperFabric configuration contains two or more HP 9000 systems and optional HyperFabric switches. Each HP 9000 acts as a node in the configuratio n . Each node has a minimum of one and a maximum of eigh t Hy perFabric adapters insta lled in it. (See Chapter 2, “Planning the Fabric,” on page 21for information about the maxi mum number of adapters that can be installed in each system.) Each HF1 switch has 16 ports; each HF2 sw itch can be conf igured wit h 12 or 16 po rts. HyperFabric su p p o rts a maximum of eight Hype r Fabric switches. HyperFabric switches can be meshed, and configuratio ns with up to fo ur levels of me shed switc hes are supported.

A HyperFabric cluster can be planned as a High Availability (HA) configuration, when it is necessary to ensure that each node can always participate in the fabric. This is done by using MC/ServiceGuard, MC/LockManager, and the Event Monitoring Service (EMS). Configurations of up to fou r nodes are supported under MC/ServiceGuard.

Beginning with HyperFabric software versions B.11.00.05 and B.11.11.00 (not HP-UX

10.20), relocatable IP addresses c an be used as part of an HA confi guratio n .

Relocatable IP addresses permit a client ap plicat ion to rero ute thro ug h an adap ter on a remote nod e, allowing that application to continue p ro cessing without inte rruption. The rerouting is transpar ent. Thi s functio n is ass ociate d with MC/ServiceGuard (se e “Configuring MC/ServiceGuard for HyperFabric Relocatable IP Addresses” on page 106). When the monitor for HyperFabric detects a failure and the backup adapter takes over, the relocatable IP address is transparently migrated to the backup adapter. Throughout this migration proc ess, the client application continues to exec ute norma l l y.

When you st art H yp e rFabric (w i th the clic_start co m m a n d , thr o ugh SAM [on HP- U X

11.0 or 11i only], or by booting the HP 9000 system), you start the management

process. This process must be active for HyperFabric to run. If the HyperFabric managem ent p rocess on a node stops runn ing f or so m e re ason ( for exam p le, if it is killed), all HyperFab ric-re lated communicat ions on that node are stopped i mmediately. This makes the node unreachab le by other compo nents in the fabric.

When you start Hyp erFabric, the fabric is, in effect, verifie d au tom atic a lly. This is because each node performs a self diagno sis and ve rifi cation over each ad apter i nst alled in the node. Also, the management process performs automatic routing and configuring for each switch (if switches are part of the fabric). You can, if you wish, run the clic_stat command to get a textual map of the fabr ic, whic h can be used as anothe r quick verification.

You might notice that the comm ands you use to administ er Hyp erFabric all have a prefix of clic_ , and some of the oth er c ompon ents have CLIC as part of their name (for example, the CLIC firmware and the CLIC software). CLIC stands for CLuster InterConnect, and it is used to differen t i ate those Hy perFabric commands/component s from ot he r co m m a nds/com p o nents. For example, the HyperFabric co m mand clic_init is different from the HP-UX init command.

Chapter 1

Overview

HyperFabric Concepts

Chapter 1

2 Planning the Fabric

This chapter contains the following sections offering general guidelines and protocol specific considerations for planning HyperF abric clusters that will run TCP/IP or HMP applications.

•“Preliminary Considerations” on page 23

Chapter 2

Planning the Fabric

•“HyperFabric Functionality for TCP/IP and HMP Applications” on page 24

•“TCP / IP” on page 25

•“Hyper Messaging Protocol (HMP)” on page 37

Chapter 2

Planning the Fabric

Preliminary Considerations

Before beginning to physically assemble a fa bric, follow th e st eps below to be sure all appropriate issues ha ve been consid er ed:

Step 1. Read Chapter 1, “Overview, ” on page 13 to get a basic understanding of HyperFabric and

its components.

Step 2. Read this chapter, Planning the Fab ric, to gain an un d e rst and ing of protocol specif ic

configuration guidelines for TCP/IP and HMP applications.

Step 3. Read “Configuration Ove rview” on page 85, “Information Y o u N e e d ” on page 86, and

“Configur at ion I nf ormation Examp le ” o n pa ge 88, to gain an understan d ing of the

information th at mu st be sp e cified whe n the fabric is con figured. Keep these configuration req uir ements in mind while following the rest of the steps belo w to plan and map the fabric. (See Figure 4-1 for an examp le of a graphical fabric map.)

Step 4. Deci d e the nu m be r of node s th at will be interconnected in the fabri c. Step 5. Deci d e the ty pe of HP 9000 system that each nod e will be (see the HP HyperFabric

Release Note for a list of the supported HP 9000 systems).

Step 6. Determine the network ban dwidth requirements for each node. Step 7. Determine the number of adapters needed for each node. Step 8. Determine if a High Availability (MC/ServiceGuard) configuration will be needed.

Remember, If M C/S ervice Gu ard is used ther e mu st be at least two adapt ers in each node.

Step 9. Deci d e what the to pol ogy o f the fabric wil l be. Step 10. Determine ho w ma ny switc he s will be use d based on the num ber of nodes in the fabric.

Remembe r, the only configuration tha t can be supp o rt ed wit ho ut a switch is the node-to-node configura tion (HA or non-HA). HyperFabric supports meshed switc hes up to a depth of four switches, starting with these versions of the HyperFabric software:

• F or HF switches: software versions B.10.20.05, B.11.00.05, and B.11.11.00.

• F or HF2 switches: software versions B.11.00.11 and B.11.11.01.

Step 11. Draw the cable connecti on s fro m eac h nod e to the sw itch es (if the f abri c will con tain

switches). If you will be us ing an HA configuration with switches, note that for full redundancy and to avoid a sin gle poin t of failure, your configuration will requ ire mor e than one switch. For example, each adapter can be connected to its own switch , or two switches can be connected to four adapters.

Chapter 2

Planning the Fabric

HyperFabric Functionality for TCP/IP and HMP Applications

The following sec tio ns in this ch apte r defin e Hype rFabric features, paramete r s, and supported configurations for TCP/IP applications and Hyper Messaging Protocol (HMP) applications. There are distinct differences in supported hardware, available features and performance, depending on which protocol is used by applications running on the HyperFabric.

Chapter 2

Planning the Fabric

TCP / IP

TCP/IP is supported on all HF1 (copper) and HF 2 (fibre) hardw are. Although so me of the HyperFabric adapter cards support both HMP and TCP/IP applications, our focus in this section will be on TCP/IP Hy p e rFabric applications.

Application Availability

All applications th at use the TC P/I P stack are supp o rted , inclu d ing Or acle 9i and HP-MPI.

NOTE There are distinct differences between the feature set that is supported for TCP/IP and

the feature set that is supported for HMP. Although TCP/IP and HMP applications are able to run simultaneousl y on the same HyperFabric cluster, for practical purposes, a HyperFabric cluster must run TCP/IP applications exclusively or HMP applications exclusively.

Features

• OnLine Addition and Replacement (OLAR): Supported

The OLAR feature allows the replacement or addition of HyperFabric adapter cards while the system (node) is running. HyperFabric supports this functionality on the rp54xx (L-class), rp74xx (N-class), rp8400 and Superd ome sys tems, running on the HP-UX 11i platform with patch PHN E_25485.

For more detailed informatio n on OL AR , includ ing instructions for imple me nting this feature, s ee “Online Additi on and Replac ement—HP-UX 11i Only” on page 55 in this manual, as well as Configuring HP-UX for Peripherals Part Number B2355-90698 November 2000 Edition .

• Event Monitoring Service (EMS): S upported

Starting with the December 2000 releases B.11.00.11 and B.11.11.01, the HyperFabric EMS monitor allows the system administrator to separately monitor each HyperFabric adapter on every node in the fa bric, in addition to monitoring the entire HyperF abric subsystem. The monitor can inform the user if the resource being monitored is UP or DO W N. The administr ato r define s the condition to trigger a notification (usually a change in interface status). Notification can be accomplished with a SNMP trap or by loggin g into the sys lo g file with a choic e of sev erity, or by email to a user defined em ail add ress.

For more detailed information on EMS, including instructions for implementing this feature, see “Configuring the HyperF abric EMS Monitor” on page 97 in this manual, as well as the EMS Hardware Monitors User’s Guide Part Number B6191-90028 September 2001 Edition.

Chapter 2

• MC Serv ice G uard: Sup p o rted

Within a cluster, MC/ServiceGuard groups application services (individual HP-UX processes) into packag es. In the event of a single servi ce fai l ure (node, network, or other res ourc e), EMS prov ides notif icati on and MC/ Servi ceGuar d transf ers con trol of

Planning the Fabric

TCP / IP

the package to another no de in the cluste r, allowing services to remain available with mi nim a l interrup ti o n. MC/ServiceGua r d via EMS, direct l y m onitors cluster nodes, LAN interfaces, and services (the individual processes within an application). MC/ServiceGuard uses a heartbeat LAN to monitor the no de s in a clus te r. It is not possible to use HyperFabric as a heartbeat LAN. Instead a separate LAN must be used for the hear tbe at.

For more detailed informatio n on co nfiguring MC ServiceG uard, see “Configuring HyperFabric with MC/ServiceGuard” on page 98 in this manual, as well as Managi ng MC /ServiceGu a rd Part Number B3936-90065 March 2002 Edition.

• High Availability (HA): Suppo r te d

To create a highly available HyperFabric cluster, there cannot be any singl e po int of failure. Once the HP 9000 nodes and the HyperFabr ic hardware have been configured with no sin gle poin t of f ailur e, MC /ServiceGuard and EM S can be configured to mo nit or an d fai l - ov er nodes and services using ServiceGuard packag es.

If any HyperFabric resource in a cluster fails (adapter card, cable or switch port), the HyperFabric driver transparentl y routes traffic over other available HyperFabric resources with no disruption of service.

The ability of the Hyper Fabric driver to transp aren tly f ail-o ve r traf fic red u ces the complexity of con fi gur ing highly available clusters with M C/S erviceGuard, because MC/ServiceGuard only has to take care of node and service failover.

A “heartbeat” is used by MC/ServiceGuard to monitor the clust er. The HyperFabric links cannot be used for the heart beat. Instead an altern ate LAN connection (100BaseT, Ethernet, Token Ring, FDDI) must be made between the nodes for use as a heartbeat link.

End To End HA: HyperFabric provides End to End HA on t he enti re cluster fabric at the link level. If any of the available routes in the fabric fails, HyperF abric will transparently redirect all the tra ffi c to a function al route and, i f conf i gured, noti fy MC/ServiceGuard or other enter prise management tool s .

Active-Active HA: In configur a tions where there are multiple routes between nodes, the HyperFabric software will use a hashing function to determine which particular adapter/route to send me ssages through. This is done on a mess age- by-me ss age basis. All of the available HyperFabric resources in the fabric are used for communication.

In contrast to Active - Passive HA, where one set of resourc es is no t utilized until another set fails, Active-Active HA provides t he best return on investment because all of the resources are utilized simultaneously. MC/ServiceGuard is not required for Active-Active HA operation .

For more information on setting up HA HyperFabric clusters, see figure 2-3 “TCP/IP High Availability Switch ed Co nfiguration”.

• Dynamic Resource Utilization (DRU): Supported

When a new resource (node, adapter, cable or switch) is added to a cluster, a HyperF ab ric subsystem wi ll dynamic ally identify the added res ource and start u sing it. The same process takes place when a resource is removed from a cluster. The difference between DRU and OL AR is that OLAR only app lies to the addi tion or replacement of adapter cards from node s.

Chapter 2

Planning the Fabric

TCP / IP

• Load Balanci ng: Supported

When a HP 9000 HyperFabric cluster is running TCP/IP applic ations, the HyperFabric driver balances the load across all available resources in the cluster including nod es, adapter cards, links, and multip le link s between switches.

• Switch Management: Not Sup p o rt e d

Switch Management is no t suppor ted. Switc h management will no t operate properly if it is enabled on a HyperFabric cluster.

• Diagnostics: Supp o rt ed

Diagnostics can be run to obtain informat ion on many of the HyperFabric components via the clic_diag, clic_probe and clic_stat commands, as well as the Support Tools Manager (STM).

For more detailed information on Hy per Fabric diagnostics se e “Running Diagnostics” on page 103 on page 149.

Configuration Parameters

This section det ails, in general, the maximum limits for TCP/IP HyperFabric configurati ons. There are numerous variables that can impact the performance of any particular HyperFabric configuration. See the “TCP/IP Supported Configuratio n s” section for guidance on specific HyperF abric configurations for TCP/IP applications.

• HyperFabric is only supported on the HP 9000 series unix servers and workstations.

• TCP/IP is supported for all HyperFabric hard ware and software.

• Maximum Supported Nodes and Adapter Cards:

In point to point configurations the com pl ex ity an d pe rform ance limitations of having a large number of nodes in a cluster make it necessary to include switching in the fabric. Typically, p o int to point configurations co nsis t of only 2 o r 3 nod es.

In switche d co nf igurations, HyperFabric sup p o rts a maximum of 6 4 interconne cted adapte r cards.

A maximum of 8 HyperFabric adapter cards are supported per instance of the HP-UX operating syste m . The actu al nu mbe r of adapter cards a particular node is able to accommodate also depends on slot availability and system resources. See node specific documentation for details.

A maximum of 8 configured IP addresses are supported by the HyperFabric subsystem per in stance of the HP-UX operating system.

• Maximum Number of Switches:

Up to 4 switches (16 port copper, 16 port fibre or Mixed 8 fibre ports / 4 copper ports) can be interconne c te d (me sh ed) in a single HyperFabric cluster.

• Trunking Betwe en Swit ches (multipl e connections)

Trunking betwe en switc hes can be used to increase ban dwid th and cluster throughput. Trunking is also a way to eliminat e a po ssible sin gle poin t of failure. The number of trunked cables between nodes is only limited by port availability . To assess the effects of trunking on the perf orman ce of any particular HyperFabric configur atio n, consult with yo ur HP representative.

Chapter 2

Planning the Fabric

TCP / IP

• Maximum Cable Lengths:

HF1 (copper): The maximum d istan ce be twee n two nod es o r betwee n a node and a switch is 60 ft. (2 standard cable lengths are sold and supported: 35 ft. and 60 ft.)

TCP/IP supports up to four HF1 switches connected in series wit h a maximum cable length of 60 ft. between the switches and 60 ft. betw een switch e s and node s.

HF2 (fibre): The maximum distan ce is 200m (4 standar d cable le ngt hs are sold and supported: 2m, 16m, 50m and 200m).

TCP/IP supports up to four HF2 switches connected in series wit h a maximum cable length of 200m between the switc hes and 200m betwe en switches and nodes.

TCP/IP supports up to 4 hybrid HF1/HF2 switches connected in series with a maximum cable length o f 60 ft. between cop per por ts and 200m be twe en fibre port s.

Chapter 2

Speed and Latency:

Table 2-1 HF1 Speed and Latency w/ TCP/IP Applications

Server Class Maximum Speed Latency

Planning the Fabric

TCP / IP

rp7400 1.28 + 1.28 Gbps full duplex per

link

< 50 microsec

Table 2-2 HF2 Speed and Latency w/ TCP/IP Applications

Server Class Maximum Speed Latency

rp7400 2 + 2 Gbps full duplex per link < 42 microsec

Chapter 2

Planning the Fabric

TCP / IP

Table 2-3 Supported Configurations for A6386A HF2 Adapter On PCI (4X)

Supported HP 9000

Systems

rp24xx (A400 and A500) 11.0, 11i v1, 11iv2 No 2 rp34xx Series 11i v1 and 11i v2 No 2 rp44xx Series 11i v1 and 11i v2 Yes 4 rp54xx Series

(L Class Servers)

rp74x0 (N-Class Series)

rp84x0 11i v1 and 11i v2 Yes 8 (maximum 4 per PCI

rx16x0 Servers 11i v2 Yes 2 rx26x0 Servers 11i v2 No 2 rx4640 Servers 11i v2 Yes 4 rx56xx Series 11i v2 No 2 rx76x0 Servers 11i v2 No 8 (maximum 4 per PCI

HP-UX Version

11.0, 11i v1 and 11i v2 Yes

11.0, 11i v1 and 11i v2 Yes 8

OLAR

Support?

(11iv1 and later)

Maximum Adapters

per System

card cage)

rx86x0 Servers 11i v2 Yes 8 (maximum 4 per PCI

card cage) zx6000 Workstations 11i v2 No 1 B1000, B2000, B2600,

C3000, C3600, C3700, J5000, J5600, J6000, J6700 and J7000 workstations

Superdome servers 11i v1 and 11i v2 Yes 8 (maximum 4 per PCI

SD64A Servers 11i v2 Yes 8 (maximum 4 per PCI

11.0, 11i v1 No 2

card cage)

Chapter 2

+ 126 hidden pages