HP HyperFabric User's Guide

Download

Installing and Administering HyperFabric

HP-UX 11i v3

Edition 13

Manufacturing Part Number: B6257-90055

February 2007

Printed in U.S.A.

Legal Notices

copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license.

The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Oracle is a registered US trademark of Oracle Corporation, Redwood City, California. UNIX is a registered trademark of The Open Group.

1. Overview

Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Notice of non support for Oracle 10g RAC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

HyperFabric Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

HyperFabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Switches and Switch Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Other Product Elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

HyperFabric Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2. Planning the Fabric

Preliminary Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

HyperFabric Functionality for TCP/UDP/IP and HMP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

TCP / UDP / IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Application Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Conﬁguration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

TCP/UDP/IP Supported Conﬁgurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Point-to-Point Conﬁgurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Switched. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

High Availability Switched. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Hyper Messaging Protocol (HMP). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Application Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Conﬁguration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

HMP Supported Conﬁgurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Point to Point. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Enterprise (Database). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Technical Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Contents

3. Installing HyperFabric

Checking HyperFabric Installation Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Installing HyperFabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Online Addition and Replacement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Planning and Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Critical Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Card Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Installing the Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

File Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Loading the Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Installing HyperFabric Switches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Before Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Installing the HF2 Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

With the Rail Kit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Without the Rail Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4. Conﬁguring Hyperfabric

Contents

Conﬁguration Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Information You Need . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Conﬁguration Information Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Performing the Conﬁguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Using the clic_init Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Examples of clic_init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Using SMH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Deconﬁguring a HyperFabric Adapter with SMH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Conﬁguring the HyperFabric EMS Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Conﬁguring HyperFabric with ServiceGuard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

How HyperFabric Handles Adapter Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

Conﬁguring HyperFabric with the ServiceGuard Resource Monitor. . . . . . . . . . . . . . . . . . . . . . . . 80

Conﬁguring ServiceGuard with HyperFabric Using the ASCII File . . . . . . . . . . . . . . . . . . . . . . . . 80

Conﬁguring ServiceGuard with HyperFabric Using SMH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Conﬁguring ServiceGuard for HyperFabric Relocatable IP Addresses . . . . . . . . . . . . . . . . . . . . . . 81

Conﬁguring HMP for Transparent Local Failover Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

How Transparent Local Failover Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Conﬁguring HMP for Transparent Local Failover Support - Using SMH. . . . . . . . . . . . . . . . . . . . . . 88

Deconﬁguring HMP for Local Failover support - Using SMH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Conﬁguring HMP for Transparent Local Failover Support - Using the clic_init command. . . . . . . . 90

5. Managing HyperFabric

Starting HyperFabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Using the clic_start Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Using SMH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

Verifying Communications within the Fabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

The clic_probe Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Examples of clic_probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Displaying Status and Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

The clic_stat Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Examples of clic_stat. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Viewing man Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Stopping HyperFabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Using the clic_shutdown Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Using SMH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

6. Troubleshooting HyperFabric

Running Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

The clic_diag Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Example of clic_diag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

Using Support Tools Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

Useful Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

LED Colors and Their Meanings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Adapter LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

HF2 Switch LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Determining Whether an Adapter or a Cable is Faulty. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Contents

Determining Whether a Switch is Faulty. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Replacing a HyperFabric Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Replacing a HyperFabric Switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Safety Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Regulatory Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Adapters and Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

FCC Statement (USA only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

DOC Statement (Canada only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Europe RFI Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Australia and New Zealand EMI Statement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Radio Frequency Interference (Japan Only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Declarations of Conformity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Physical Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Environmental . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Contents

Figures

Figure 2-1. TCP/UDP/IP Point-To-Point Conﬁgurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Figure 2-2. TCP/UDP/IP Basic Switched Conﬁguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Figure 2-3. TCP/UDP/IP High Availability Switched Conﬁguration . . . . . . . . . . . . . . . . . . . . . . . . 29

Figure 2-4. HMP Point-To-Point Conﬁgurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Figure 2-5. HMP Enterprise (Database) Conﬁguration, Single Connection Between Nodes . . . . . 35

Figure 2-6. Local Failover Supported Enterprise (Database) Conﬁguration, Multiple Connections

Between Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Figure 3-1. HyperFabric File Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Figure 3-2. Front of HF2 Switch (A6388A Switch Module Installed) . . . . . . . . . . . . . . . . . . . . . . . . 53

Figure 3-3. Front of HF2 Switch (A6389A Switch Module Installed) . . . . . . . . . . . . . . . . . . . . . . . . 54

Figure 3-4. Parts of the Rail Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Figure 3-5. The Ends of the Rail Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Figure 4-1. Map for Conﬁguration Information Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Figure 4-2. An ServiceGuard Conﬁguration (with Two HyperFabric Switches) . . . . . . . . . . . . . . . 77

Figure 4-3. Node with Two Active HyperFabric Adapters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Figure 4-4. Node with One Failed HyperFabric Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Figure 4-5. When All HyperFabric Adapters Fail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

Figure 4-6. A Conﬁguration supporting Local Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Figure 4-7. Adapter, Link or Switch Port Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Figure 4-8. Switch Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Figure 4-9. Cable Failover Between Two Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

Figure 4-10. Conﬁguring the Transparent Local Failover feature . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Figures

Tables

Table 2-1. HF2 Speed and Latency w/ TCP/UDP/IP Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Table 2-2. HF2 Speed and Latency w/ HMP Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Table 3-1. Important OLAR Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Table 6-1. LED Names (by Adapter) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Table 6-2. HyperFabric Adapter LED Colors and Meanings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Table 6-3. HF2 Switch LED Colors and Meanings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

Tables

Printing History

The manual printing date and part number indicate its current edition. The printing date will change when a new edition is printed. Minor changes may be made at reprint without changing the printing date. The manual part number will change when extensive changes are made.

Manual updates may be issued between editions to correct errors or document product changes. To ensure that you receive the updated or new editions, you should subscribe to the appropriate product support service. See your HP sales representative for details.

First Edition: March 1998 Second Edition: June 1998 Third Edition: August 1998 Fourth Edition: October 1998 Fifth Edition: December 1998 Sixth Edition: February 1999 Seventh Edition: April 1999 Eighth Edition: March 2000 Ninth Edition: June 2000 Tenth Edition: December 2000 Eleventh Edition: June 2001 Twelfth Edition: September 2002 Thirteenth Edition: March 2006 Fourteenth Edition: November 2006 Fifteenth Edition: February 2007

1 Overview

This chapter contains the following sections that give general information about HyperFabric:

• “Overview” on page 15

• “HyperFabric Products” on page 16

Chapter 1

Overview

• “HyperFabric Concepts” on page 18

Chapter 1

Overview

HyperFabric is a HP high-speed, packet-based interconnect for node-to-node communications. HyperFabric provides higher speed, lower network latency and less CPU usage than other industry standard protocols (e.g. Fibre Channel and Gigabit Ethernet). Instead of using a traditional bus based technology, HyperFabric is built around switched fabric architecture, providing the bandwidth necessary for high speed data transfer. This clustering solution delivers the performance, scalability and high availability required by:

• Parallel Database Clusters:

Oracle 9i Real Application Clusters (RAC) Oracle 8i Parallel Servers (OPS)

• Parallel Computing Clusters

• Client/Server Architecture Interconnects (for example, SAP)

• Multi-Server Batch Applications (for example, SAS Systems)

• Enterprise Resource Planning (ERP)

• Technical Computing Clusters

• Open View Data Protector (earlier known as Omniback)

• Network Backup

• Data Center Network Consolidation

• E-services

Notice of non support for Oracle 10g RAC

HyperFabric product suite was designed to optimize performance of Oracle 9i RAC database running on HP-UX clusters. With the industry moving to standards-based networking technologies for database clustering solutions, HP and Oracle have worked together to optimize features and performance of Oracle 10g RAC database with standards-based interconnect technologies including Gigabit Ethernet, 10Gigabit Ethernet and Inﬁniband.

To align with the market trend for standards-based interconnects, Oracle 10g RAC database is not currently supported on conﬁgurations consisting of HyperFabric product suite and it will not be supported in the future either. As a result, customers must switch to Gigabit Ethernet, 10Gigabit Ethernet or Inﬁniband technology if they plan to use Oracle 10g RAC.

Please note that conﬁgurations comprising HyperFabric and Oracle 9i continue to be supported.

Chapter 1

Overview

HyperFabric Products

HyperFabric hardware consists of host-based interface adapter cards, interconnect cables and optional switches. HyperFabric software resides in Application Speciﬁc Integrated Circuits (ASICs) and ﬁrmware on the adapter cards and includes user space components and HP-UX drivers.

Currently, ﬁbre based HyperFabric hardware is available. In addition, a hybrid switch that has 8 ﬁbre ports is available to support HF2 clusters.

The various HyperFabric products are described below. See the HP HyperFabric Release Note for information about the HP 9000 systems these products are supported on.

NOTE This document uses the term HyperFabric (HF) is used in general to refer to the

hardware and software that form the HyperFabric cluster interconnect product. The term HyperFabric2 (HF2) refers to the ﬁbre based hardware components:

• The A6386A adapter.

• The A6384A switch chassis.

• The A6388A and A6389A switch modules. (Although the A6389A switch module has 4 copper ports it is still considered a HF2 component because it can only be used with the A6384A HF2 switch chassis).

• The C7524A, C7525A, C7526A, and C7527A cables.

HyperFabric Adapters

The HyperFabric adapters include the following:

• A6386A HF2 PCI (4X) adapter with a ﬁbre interface.

Switches and Switch Modules

The HyperFabric2 switches are as follows:

• A6384A HF2 ﬁbre switch chassis with one integrated Ethernet management LAN adapter card, one integrated 8-port ﬁbre card, and one expansion slot. For the chassis to be a functional switch, one of these two switch modules must be installed in the expansion slot:

— The A6388A HF2 8-port ﬁbre switch module. This gives the switch 16 ﬁbre ports

(8 from the integrated ﬁbre card and 8 from the A6388A).

— The A6389A HF2 4-port copper switch module. This gives the switch 12 ports—a

mixture of 8 ﬁbre ports (from the integrated ﬁbre card) and 4 copper ports (from the A6389A module).

The A6384A HF2 switch chassis with either module installed is supported beginning with the following HyperFabric software versions:

• HP-UX 11i v3: HyperFabric software version B.11.31.01

Chapter 1

Overview

HyperFabric Products

NOTE In this manual, the terms HyperFabric2 switch or HF2 switch refer to the functional

switch (the A6384A switch chassis with one of the switch modules installed).

IMPORTANT HF2 adapters and switches are not supported by software versions earlier than those

listed in “HyperFabric Adapters” on page 16 and “Switches and Switch Modules” on page 16.

To determine the version of HyperFabric you have, issue this command:

swlist | grep -i hyperfabric

Other Product Elements

The other elements of the HyperFabric product family are the following:

• HF2 ﬁbre cables:

— C7524A (2m length) — C7525A (16m length) — C7526A (50m length) — C7527A (200m length)

• The HyperFabric software: The software resides in ASICs and ﬁrmware on the adapter cards and includes user space components and HP-UX drivers.

HyperFabric supports the IP network protocol stack, speciﬁcally TCP/IP and UDP/IP.

HyperFabric software includes HyperMessaging Protocol (HMP). HMP provides higher bandwidth, lower CPU overhead, and lower latency (the time it takes a message to get from one point to another). However, these HMP beneﬁts are only available when applications that were developed on top of HMP are running.

Chapter 1

Overview

HyperFabric Concepts

Some basic HyperFabric concepts and terms are brieﬂy described below. The fabric is the physical conﬁguration that consists of all of the HyperFabric adapters,

the HyperFabric switches (if any are used) and the HyperFabric cables connecting them. The network software controls data transfer over the fabric.

A HyperFabric conﬁguration contains two or more HP 9000 systems and optional HyperFabric switches. Each HP 9000 acts as a node in the conﬁguration. Each node has a minimum of one and a maximum of eight HyperFabric adapters installed in it. (See Chapter 2, “Planning the Fabric,” on page 19for information about the maximum number of adapters that can be installed in each system). HyperFabric supports a maximum of 4 HyperFabric switches. HyperFabric switches can be meshed, and conﬁgurations with up to four levels of meshed switches are supported.

A HyperFabric cluster can be planned as a High Availability (HA) conﬁguration, when it is necessary to ensure that each node can always participate in the fabric. This is done by using MC/ServiceGuard, MC/LockManager, and the Event Monitoring Service (EMS). Conﬁgurations of up to eight nodes are supported under MC/ServiceGuard.

Relocatable IP addresses can be used as part of an HA conﬁguration. Relocatable IP addresses permit a client application to reroute through an adapter on a remote node, allowing that application to continue processing without interruption. The rerouting is transparent. This function is associated with MC/ServiceGuard (see “Conﬁguring ServiceGuard for HyperFabric Relocatable IP Addresses” on page 81). When the monitor for HyperFabric detects a failure and the backup adapter takes over, the relocatable IP address is transparently migrated to the backup adapter. Throughout this migration process, the client application continues to execute normally.

When you start HyperFabric (with the clic_start command, through SMH or by booting the HP 9000 system), you start the management process. This process must be active for HyperFabric to run. If the HyperFabric management process on a node stops running for some reason (for example, if it is killed), all HyperFabric-related communications on that node are stopped immediately. This makes the node unreachable by other components in the fabric.

When you start HyperFabric, the fabric is, in effect, veriﬁed automatically. This is because each node performs a self diagnosis and veriﬁcation over each adapter installed in the node. Also, the management process performs automatic routing and conﬁguring for each switch (if switches are part of the fabric). You can, if needed, run the clic_stat command to get a textual map of the fabric, which can be used as another quick veriﬁcation.

Notice that the commands you use to administer HyperFabric all have a preﬁx of clic_ , and some of the other components have CLIC as part of their name (for example, the CLIC ﬁrmware and the CLIC software). CLIC stands for CLuster InterConnect, and it is used to differentiate those HyperFabric commands/components from other commands/components. For example, the HyperFabric command clic_init is different from the HP-UX init command.

Chapter 1

2 Planning the Fabric

This chapter contains the following sections offering general guidelines and protocol speciﬁc considerations for planning HyperFabric clusters that will run TCP/UDP/IP or HMP applications.

• “Preliminary Considerations” on page 21

Chapter 2

Planning the Fabric

• “HyperFabric Functionality for TCP/UDP/IP and HMP Applications” on page 22

• “TCP / UDP / IP” on page 23

• “Hyper Messaging Protocol (HMP)” on page 30

Chapter 2

Planning the Fabric

Preliminary Considerations

Before beginning to physically assemble a fabric, follow the steps below to be sure all appropriate issues have been considered:

Step 1. Read Chapter 1, “Overview,” on page 13 to get a basic understanding of HyperFabric and

its components.

Step 2. Read this chapter, Planning the Fabric, to gain an understanding of protocol speciﬁc

conﬁguration guidelines for TCP/UDP/IP and HMP applications.

Step 3. Read “Conﬁguration Overview” on page 63, “Information You Need” on page 64, and

“Conﬁguration Information Example” on page 66, to gain an understanding of the information that must be speciﬁed when the fabric is conﬁgured.

Step 4. Decide the number of nodes that will be interconnected in the fabric.

Step 5. Decide the type of HP 9000 system that each node will be (for a list of supported HP 9000

systems, see the HP HyperFabric Release Note).

Step 6. Determine the network bandwidth requirements for each node.

Step 7. Determine the number of adapters needed for each node.

Step 8. Determine if a High Availability (ServiceGuard) conﬁguration will be needed.

Remember, If MC/ServiceGuard is used there must be at least two adapters in each node.

Step 9. Decide what the topology of the fabric will be.

Step 10. Determine how many switches will be used based on the number of nodes in the fabric.

Remember, the only conﬁguration that can be supported without a switch is the node-to-node conﬁguration (HA or non-HA). HyperFabric supports meshed switches up to a depth of four switches.

Step 11. Draw the cable connections from each node to the switches (if the fabric will contain

switches). If you use an HA conﬁguration with switches, note that for full redundancy and to avoid a single point of failure, your conﬁguration will require more than one switch. For example, each adapter can be connected to its own switch, or two switches can be connected to four adapters.

Chapter 2

Planning the Fabric

HyperFabric Functionality for TCP/UDP/IP and HMP Applications

The following sections in this chapter deﬁne HyperFabric features, parameters, and supported conﬁgurations for TCP/UDP/IP applications and Hyper Messaging Protocol (HMP) applications. There are distinct differences in supported hardware, available features and performance, depending on which protocol is used by applications running on the HyperFabric.

Chapter 2

Planning the Fabric

TCP / UDP / IP

TCP/UDP/IP applications are supported on all HF2 (ﬁbre) hardware. Although all HyperFabric adapter cards support HMP applications as well, our focus in this section will be on TCP/UDP/IP HyperFabric applications.

Application Availability

All applications that use the TCP/UDP/IP stack are supported such as Oracle 9i.

Features

• OnLine Addition and Replacement (OLAR): Supported The OLAR feature allows the replacement or addition of HyperFabric adapter cards

while the system (node) is running. For a list of systems that support OLAR, see the HyperFabric Release Notes (B6257-90056).

For more detailed information on OLAR, including instructions for implementing this feature, see “Online Addition and Replacement” on page 42 in this manual, as well as Interface Card OL* Support Guide (B2355-90698).

• Event Monitoring Service (EMS): Supported In the HyperFabric version B.11.31.01, the HyperFabric EMS monitor allows the

system administrator to separately monitor each HyperFabric adapter on every node in the fabric, in addition to monitoring the entire HyperFabric subsystem. The monitor can inform the user if the resource being monitored is UP or DOWN. The administrator deﬁnes the condition to trigger a notiﬁcation (usually a change in interface status). Notiﬁcation can be accomplished with a SNMP trap or by logging into the syslog ﬁle with a choice of severity, or by email to a user deﬁned email address.

For more detailed information on EMS, including instructions for implementing this feature, see “Conﬁguring the HyperFabric EMS Monitor” on page 75 in this manual, as well as the EMS Hardware Monitors User’s Guide (B6191-90028).

• ServiceGuard: Supported Within a cluster, ServiceGuard groups application services (individual HP-UX

ServiceGuard via EMS, directly monitors cluster nodes, LAN interfaces, and services (the individual processes within an application). ServiceGuard uses a heartbeat LAN to monitor the nodes in a cluster. It is not possible to use HyperFabric as a heartbeat LAN. Instead a separate LAN must be used for the heartbeat.

Chapter 2

For more detailed information on conﬁguring ServiceGuard, see “Conﬁguring HyperFabric with ServiceGuard” on page 76 in this manual, as well as Managing MC/ServiceGuard Part Number B3936-90065 March 2002 Edition.

• High Availability (HA): Supported

Planning the Fabric

TCP / UDP / IP

To create a highly available HyperFabric cluster, there cannot be any single point of failure. Once the HP 9000 nodes and the HyperFabric hardware have been conﬁgured with no single point of failure, ServiceGuard and EMS can be conﬁgured to monitor and fail-over nodes and services using ServiceGuard packages.

If any HyperFabric resource in a cluster fails (adapter card, cable or switch port), the HyperFabric driver transparently routes trafﬁc over other available HyperFabric resources with no disruption of service.

The ability of the HyperFabric driver to transparently fail-over trafﬁc reduces the complexity of conﬁguring highly available clusters with ServiceGuard, because ServiceGuard only has to take care of node and service failover.

A “heartbeat” is used by MC/ServiceGuard to monitor the cluster. The HyperFabric links cannot be used for the heartbeat. Instead an alternate LAN connection (Ethernet, Gigabit Ethernet, 10Gigabit Ethernet) must be made between the nodes for use as a heartbeat link.

End To End HA: HyperFabric provides End to End HA on the entire cluster fabric at the link level. If any of the available routes in the fabric fails, HyperFabric will transparently redirect all the trafﬁc to a functional route and, if conﬁgured, notify ServiceGuard or other enterprise management tools.

Active-Active HA: In conﬁgurations where there are multiple routes between nodes, the HyperFabric software will use a hashing function to determine which particular adapter/route to send messages through. This is done on a message-by-message basis. All of the available HyperFabric resources in the fabric are used for communication.

In contrast to Active-Passive HA, where one set of resources is not utilized until another set fails, Active-Active HA provides the best return on investment because all of the resources are utilized simultaneously. MC/ServiceGuard is not required for Active-Active HA operation.

For more information on setting up HA HyperFabric clusters, see ﬁgure 2-3 “TCP/UDP/IP High Availability Switched Conﬁguration”.

• Dynamic Resource Utilization (DRU): Supported When a new resource (node, adapter, cable or switch) is added to a cluster, a

HyperFabric subsystem will dynamically identify the added resource and start using it. The same process takes place when a resource is removed from a cluster. The difference between DRU and OLAR is that OLAR only applies to the addition or replacement of adapter cards from nodes.

• Load Balancing: Supported When a HP 9000 HyperFabric cluster is running TCP/UDP/IP applications, the

HyperFabric driver balances the load across all available resources in the cluster including nodes, adapter cards, links, and multiple links between switches.

• Switch Management: Not Supported Switch Management is not supported. Switch management will not operate properly

if it is enabled on a HyperFabric cluster.

• Diagnostics: Supported Diagnostics can be run to obtain information on many of the HyperFabric

components via the clic_diag, clic_probe and clic_stat commands, as well as the Support Tools Manager (STM).

Chapter 2

Planning the Fabric

TCP / UDP / IP

For more detailed information on HyperFabric diagnostics see “Running Diagnostics” on page 115 on page 149.

Conﬁguration Parameters

This section details, in general, the maximum limits for TCP/UDP/IP HyperFabric conﬁgurations. There are numerous variables that can impact the performance of any particular HyperFabric conﬁguration. See the “TCP/UDP/IP Supported Conﬁgurations” section for guidance on speciﬁc HyperFabric conﬁgurations for TCP/UDP/IP applications.

• HyperFabric is only supported on the HP 9000 series unix servers.

• TCP/UDP/IP is supported for all HyperFabric hardware and software.

• Maximum Supported Nodes and Adapter Cards: In point to point conﬁgurations the complexity and performance limitations of

having a large number of nodes in a cluster make it necessary to include switching in the fabric. Typically, point to point conﬁgurations consist of only 2 or 3 nodes.

In switched conﬁgurations, HyperFabric supports a maximum of 64 interconnected adapter cards.

A maximum of 8 HyperFabric adapter cards are supported per instance of the HP-UX operating system. The actual number of adapter cards a particular node is able to accommodate also depends on slot availability and system resources. See node speciﬁc documentation for details.

A maximum of 8 conﬁgured IP addresses are supported by the HyperFabric subsystem per instance of the HP-UX operating system.

• Maximum Number of Switches: You can interconnect (mesh) up to 4 switches (16 port ﬁbre or Mixed 8 ﬁbre ports) in

a single HyperFabric cluster.

• Trunking Between Switches (multiple connections) Trunking between switches can be used to increase bandwidth and cluster

throughput. Trunking is also a way to eliminate a possible single point of failure. The number of trunked cables between nodes is only limited by port availability. To assess the effects of trunking on the performance of any particular HyperFabric conﬁguration, consult with your HP representative.

• Maximum Cable Lengths: HF2 (ﬁbre): The maximum distance is 200m (4 standard cable lengths are sold and

supported: 2m, 16m, 50m and 200m). TCP/UDP/IP supports up to four HF2 switches connected in series with a maximum

cable length of 200m between the switches and 200m between switches and nodes.

Chapter 2

TCP/UDP/IP supports up to 4 hybrid HF2 switches connected in series with a maximum cable length of 200m between ﬁbre ports.

Planning the Fabric

TCP / UDP / IP

• Speed and Latency:

Table 2-1 HF2 Speed and Latency w/ TCP/UDP/IP Applications

Server Class Maximum Speed Latency

rp7420 2 + 2 Gbps full duplex per link < 42 microsec

For a list of HF2 hardware that supports TCP/UDP/IP applications (HP-UXN 11i v3), see HyperFabric Release Notes (B6257-90056).

TCP/UDP/IP Supported Conﬁgurations

Multiple TCP/UDP/IP HyperFabric conﬁgurations are supported to match the cost, scaling and performance requirements of each installation.

In the previous “Conﬁguration Guidelines” section the maximum limits for TCP/UDP/IP enabled HyperFabric hardware conﬁgurations were outlined. In this section the TCP/UDP/IP enabled HyperFabric conﬁgurations that HP supports will be detailed. These recommended conﬁgurations offer an optimal mix of performance, availability and practicality for a variety of operating environments.

There are many variables that can impact HyperFabric performance. If you are considering a conﬁguration that is beyond the scope of the following HP supported conﬁgurations, contact your HP representative.

Point-to-Point Conﬁgurations

Large servers like HP’s Superdome can be interconnected to run Oracle RAC 9i and enterprise resource planning applications. These applications are typically consolidated on large servers.

Point to point connections between servers support the performance beneﬁts of HMP without investing in HyperFabric switches. This is a good solution in small conﬁgurations where the beneﬁts of a switched HyperFabric cluster might not be required (see conﬁguration A and conﬁguration C in Figure 2-1).

If there are multiple point to point connections between two nodes, the trafﬁc load will be balanced over those links. If one link fails, the load will fail-over to the remaining links (see conﬁguration B in Figure 2-1).

Running applications using TCP/UDP/IP on a HyperFabric cluster provides major performance beneﬁts compared to other technologies (such as ethernet). If a HyperFabric cluster is originally set up to run enterprise applications using TCP/UDP/IP and the computing environment stabilizes with a requirement for higher performance, migration to HMP is always an option.

Chapter 2

Figure 2-1 TCP/UDP/IP Point-To-Point Conﬁgurations

Planning the Fabric

TCP / UDP / IP

Chapter 2

Planning the Fabric

TCP / UDP / IP

Switched

This conﬁguration offers the same beneﬁts as the point to point conﬁgurations illustrated in ﬁgure 1, but it has the added advantage of greater connectivity (see Figure 2-2).

Figure 2-2 TCP/UDP/IP Basic Switched Conﬁguration

Chapter 2

High Availability Switched

This conﬁguration has no single point of failure. The HyperFabric driver provides end to end HA. If any HyperFabric resource in the cluster fails, trafﬁc will be transparently rerouted through other available resources. This conﬁguration provides high performance and high availability (see Figure 2-3).

Figure 2-3 TCP/UDP/IP High Availability Switched Conﬁguration

Planning the Fabric

TCP / UDP / IP

Chapter 2

Planning the Fabric

Hyper Messaging Protocol (HMP)

Hyper Messaging protocol (HMP) is HP patented, high performance cluster interconnect protocol. HMP provides reliable, high speed, low latency, low CPU overhead, datagram service to applications running on HP-UX platforms.

HMP was jointly developed with Oracle Corp. The resulting feature set was tuned to enhance the scalability of the Oracle Cache Fusion clustering technology. It is implemented using Remote DMA (RDMA) paradigms.

HMP is integral to the HP-UX HyperFabric driver. It is a functionality that can be enabled or disabled at HyperFabric initialization using clic_init or SMH. The HMP functionality is used by the applications listed in the Application Availability section below.

HMP signiﬁcantly enhances the performance of parallel and technical computing applications.

HMP ﬁrmware on HyperFabric adapter cards provides a “shortcut” that bypasses several layers in the protocol stack, boosting link performance and lowering latency. By avoiding interruptions and buffer copying in the protocol stack, communication task processing is optimized.

Application Availability

Currently there are two families of applications that can use HMP over the HyperFabric interface:

• Oracle 9i Database, Release 1 (9.0.1) and Release 2 (9.2.0.1.0). HMP has been certiﬁed on Oracle 9i Database Release 1 with 11i v3. HMP has been certiﬁed on Oracle 9i Database Release 2 with 11i v3.

• Technical Computing Applications

Features

• OnLine Addition and Replacement (OLAR) The OLAR feature, which allows the replacement or addition of HyperFabric adapter

cards while the system (node) is running, is supported when applications use HMP to communicate.

Chapter 2

Planning the Fabric

Hyper Messaging Protocol (HMP)

• Event Monitoring Service (EMS): Supported The HyperFabric EMS monitor allows the system administrator to separately

monitor each HyperFabric adapter on every node in the fabric, in addition to monitoring the entire HyperFabric subsystem. The monitor can inform the user if the resource being monitored is UP or DOWN. The administrator deﬁnes the condition to trigger a notiﬁcation (usually a change in interface status). Notiﬁcation can be accomplished with a SNMP trap or by logging into the syslog ﬁle with a choice of severity, or by email to a user deﬁned email address.

• ServiceGuard: Supported Within a cluster, ServiceGuard groups application services (individual HP-UX

processes) into packages. In the event of a single service failure (node, network, or other resource), EMS provides notiﬁcation and ServiceGuard transfers control of the package to another node in the cluster, allowing services to remain available with minimal interruption. ServiceGuard via EMS, directly monitors cluster nodes, LAN interfaces, and services (the individual processes within an application). ServiceGuard uses a heartbeat LAN to monitor the nodes in a cluster. ServiceGuard cannot use the HyperFabric interconnect as a heartbeat link. Instead, a separate LAN must be used for the heartbeat.

• High Availability (HA): Supported When applications use HMP to communicate between HP 9000 nodes in a

HyperFabric cluster, MC/ServiceGuard and the EMS monitor can be conﬁgured to identify node failure and automatically fail-over to a functioning HP 9000 node.

For more detailed information on HA when running HMP applications, consult with your HP representative.

• Transparent Local Failover: Supported When a HyperFabric resource (adapter, cable, switch or switch port) fails in a cluster,

HMP transparently fails over trafﬁc using other available resources. This is accomplished using card pairs, each of which is a logical entity that comprises a pair of HF2 adapters on a HP 9000 node. Only Oracle applications can make use of the Local Failover feature. HMP trafﬁc can only fail over between adapters that belong to the same card pair. Trafﬁc does not fail over if both the adapters in a card pair fail. However, administrators do not need to conﬁgure HF2 adapters as card pairs if TCP/UDP/IP is run over HF2.

When HMP is conﬁgured in the local failover mode, all the resources in the cluster are utilized. If a resource fails in the cluster and is restored, HMP does not utilize that resource until another resource fails.

Chapter 2

For more information on Transparent Local Failover while running HMP applications, see “Conﬁguring HMP for Transparent Local Failover Support” on page

96.

Planning the Fabric

Hyper Messaging Protocol (HMP)

• Dynamic Resource Utilization (DRU): Partially Supported When a new HyperFabric resource (node, cable or switch) is added to a cluster

running an HMP application, the HyperFabric subsystem will dynamically identify the added resource and start using it. The same process takes place when a resource is removed from a cluster. The distinction for HMP is that DRU is supported when a node with adapters installed in it is added or removed from a cluster running an HMP application, but DRU is not supported when an adapter is added or removed from a node that is running an HMP application. This is consistent with the fact that OLAR is not supported when an HMP application is running on HyperFabric.

• Load Balancing: Supported When an HP 9000 node that has multiple HyperFabric adapter cards is running

HMP applications, the HyperFabric driver only balances the load across the nodes, available adapter cards on that node, links and multiple links between switches.

• Switch Management: Not Supported Switch Management is not supported. Switch management will not operate properly

if it is enabled on a HyperFabric cluster.

• Diagnostics: Supported Diagnostics can be run to obtain information on many of the HyperFabric

components via the clic_diag, clic_probe and clic_stat commands, as well as the Support Tools Manager (STM).

For more detailed information on HyperFabric diagnostics, see “Running Diagnostics” on page 115 on page 149.

Conﬁguration Parameters

This section details, in general, the maximum limits for HMP HyperFabric conﬁgurations. There are numerous variables that can impact the performance of any particular HyperFabric conﬁguration. See the “HMP Supported Conﬁgurations” section for guidance on speciﬁc HyperFabric conﬁgurations for HMP applications.

• HyperFabric is only supported on the HP 9000 series unix servers.

• The performance advantages HMP offers will not be fully realized unless it is used with A6386A HF2 (ﬁbre) adapters and related ﬁbre hardware. The local failover conﬁguration of HMP is supported only on the A6386AA HF2 adapters.

• Maximum Supported Nodes and Adapter Cards: HyperFabric clusters running HMP applications are limited to supporting a

maximum of 64 adapter cards. However, in local failover conﬁgurations, a maximum of only 52 adapters are supported.

In point to point conﬁgurations running HMP applications, the complexity and performance limitations of having a large number of nodes in a cluster make it necessary to include switching in the fabric. Typically, point to point conﬁgurations consist of only 2 or 3 nodes.

In switched conﬁgurations running HMP applications, HyperFabric supports a maximum of 64 interconnected adapter cards.

Chapter 2

Planning the Fabric

Hyper Messaging Protocol (HMP)

A maximum of 8 conﬁgured IP addresses are supported by the HyperFabric subsystem per instance of the HP-UX operating system.

• Maximum Number of Switches: You can interconnect (mesh) up to 4 switches (16 port ﬁbre or Mixed 8 ﬁbre ports) in

a single HyperFabric cluster.

• Trunking Between Switches (multiple connections). HMP is supported in conﬁgurations where switches are interconnected through

multiple cables. However, with the current release of HMP software, this conﬁguration will not eliminate a single point of failure or increase performance. Instead, all of the trafﬁc will be sent over a single connection with no failover capability and without the performance increase that would come from balancing the load over multiple connections.

• Maximum Cable Lengths: HF2 (ﬁbre): The maximum distance is 200m (4 standard cable lengths are sold and

supported: 2m, 16m, 50m and 200m). HMP supports up to four HF2 switches connected in series with a maximum cable

length of 200m between the switches and 200m between switches and nodes. HMP supports up to 4 hybrid HF2 switches connected in series with a maximum

cable length of 200m between ﬁbre ports.

• HMP is supported on the PCI 4X adapters, A6386A.

• For a list of HF2 hardware that supports HMP on HP-UX 11i v3, see HyperFabric Release Notes (B6257-90056).

• Speed and Latency

Table 2-2 HF2 Speed and Latency w/ HMP Applications

Server Class Maximum Speed Latency

rp7420 2 + 2 Gbps full duplex per link < 22 microsec

For a list of HF2 hardware that supports HMP on HP-UX 11i v3, see HyperFabric Release Notes (B6257-90056).

HMP Supported Conﬁgurations

Chapter 2

Multiple HMP HyperFabric conﬁgurations are supported to match the performance, cost and scaling requirements of each installation.

In the previous “Conﬁguration Guidelines” section, the maximum limits for HMP enabled HyperFabric hardware conﬁgurations were outlined. In this section, the HMP enabled HyperFabric conﬁgurations that HP supports will be detailed. These recommended conﬁgurations offer an optimal mix of performance, availability and practicality for a variety of operating environments.

Planning the Fabric

Hyper Messaging Protocol (HMP)

Point to Point

Large servers like HP’s Superdome can be interconnected to run Oracle RAC 9i and enterprise resource planning applications. These applications are typically consolidated on large servers.

If an HMP application is running over the HyperFabric and another node is added to either of the point to point conﬁgurations illustrated in Figure 2-4, it will be necessary to also add a HyperFabric switch to the cluster.

Figure 2-4 HMP Point-To-Point Conﬁgurations

Chapter 2

Planning the Fabric

Hyper Messaging Protocol (HMP)

Enterprise (Database)

The HMP enterprise conﬁguration illustrated in Figure 2-5 is very popular for running Oracle RAC 9i.

Superdomes or other large servers make up the Database Tier. Database Tier nodes communicate with each other using HMP. Application Tier nodes communicate with each other and to the Database Tier using

TCP/UDP/IP. Although each of the servers in the Application Tier could also have multiple adapter

cards and multiple connections to switches, link and adapter card failover capabilities are not currently available for HMP.

Figure 2-5 HMP Enterprise (Database) Conﬁguration, Single Connection Between

Nodes

Chapter 2

Planning the Fabric

Hyper Messaging Protocol (HMP)

Enterprise (Database) - Local Failover Supported Conﬁguration

The HMP enterprise conﬁguration is a scalable solution. If higher performance is required, or if eliminating single points of failure is necessary, scaling up to the HMP enterprise conﬁguration with multiple connections between nodes is easily accomplished (see Figure 2-6).

Figure 2-6 Local Failover Supported Enterprise (Database) Conﬁguration,

Multiple Connections Between Nodes

Chapter 2

Planning the Fabric

Hyper Messaging Protocol (HMP)

Technical Computing

This conﬁguration is typically used to run technical computing applications. A large number of small nodes are interconnected to achieve high throughput. High availability is not usually a requirement in technical computing environments.

HMP provides the high performance, low latency path necessary for these technical computing applications. As many as 56 nodes can be interconnected using HP’s 16 port switches. Not more than four 16 port switches can be linked in a single cluster.

Chapter 2

Planning the Fabric

Hyper Messaging Protocol (HMP)

Chapter 2

3 Installing HyperFabric

This chapter contains the following sections that describe installing HyperFabric:

• “Checking HyperFabric Installation Prerequisites” on page 41

• “Installing HyperFabric Adapters” on page 42

• “Installing the Software” on page 47

Chapter 3

Installing HyperFabric

• “Installing HyperFabric Switches” on page 52

Chapter 3

Installing HyperFabric

Checking HyperFabric Installation Prerequisites

Before installing HyperFabric, check to make sure the following hardware and software prerequisites have been met:

✓ Check the HP HyperFabric Release Note for any known problems, required patches,

or other information needed for installation.

✓ Conﬁrm the /usr/bin, /usr/sbin, and /sbin directories are in your PATH by

logging in as root and using the echo $PATH command.

✓ Conﬁrm the HP-UX operating system is the correct version. Use the uname -a

command to determine the HP-UX version. See the HP HyperFabric Release Note for information about the required operating

system versions.

✓ If you are installing an HF2 switch, conﬁrm that you have four screws with

over-sized heads.

✓ Conﬁrm there are cables of the proper length and type (ﬁbre) to make each of the

connections in the fabric (adapter to adapter, adapter to switch, or switch to switch).

✓ Conﬁrm there is at least one loopback plug for testing the adapters and switches (a

ﬁbre loopback plug [HP part number A6384-67004] is shipped with each HF2 switch).

✓ Conﬁrm the necessary tools are available to install the HyperFabric switch

mounting hardware. Also check the HP 9000 system documentation to determine if any additional tools may be required for component installation.

✓ Conﬁrm software media is correct. ✓ Create a map of the fabric (optional). ✓ Conﬁrm HP-UX super-user privileges are available, they will be necessary to

complete the HyperFabric installation.

The ﬁrst HyperFabric installation step is installing HyperFabric adapter cards in the nodes. Proceed to the next section “Installing HyperFabric Adapters”.

Chapter 3

Installing HyperFabric

Installing HyperFabric Adapters

This section contains information about installing HyperFabric adapters in HP 9000 systems. Online Addition and Replacement (OLAR) information is provided in the “Online Addition and Replacement” section on page 62.

CAUTION HyperFabric adapters contain electronic components that can easily be damaged by

small amounts of electricity. To avoid damage, follow these guidelines:

• Store adapters in their antistatic plastic bags until installation.

• Work in a static-free area, if possible.

• Handle adapters by the edges only. Do not touch electronic components or electrical traces.

• Use the disposable grounding wrist strap provided with each adapter. Follow the instructions included with the grounding strap.

•Use a suitable ground—any exposed metal surface on the computer chassis.

For speciﬁc instructions see system speciﬁc documentation on “installing networking adapters” for each type of HP 9000 system that HyperFabric adapters will be installed into.

When the HyperFabric adapters have been installed, go to “Installing the Software” on page 47.

Online Addition and Replacement

Online Addition and Replacement (OLAR) allows PCI I/O cards, adapters or

controllers to be replaced or added to HP 9000 systems, without the need for completely shutting down and rebooting the system, or adversely affecting other system components. This feature is only available on HP 9000 systems that are designed to support OLAR. The system hardware uses the per-slot power control combined with OS support to enable this feature.

NOTE OLAR is supported only on TCP/UDP/IP over HF2 adapters.

Not all add-in cards have this capability, but over time many cards will be gaining this capability.

The latest HyperFabric Release Notes contains information about which HP 9000 systems and HyperFabric adapters OLAR is supported for.

Chapter 3

Installing HyperFabric

Installing HyperFabric Adapters

IMPORTANT At this time, Superdome systems are not intended for access by users. HP recommends

that these systems only be opened by a qualiﬁed HP engineer. Failure to observe this requirement can invalidate any support agreement or warranty to which the owner might otherwise be entitled.

There are two methods to add or replace OLAR-compatible cards:

• Using the SAM or SMH1 utility.

NOTE System Administration Manager (SAM) is deprecated in the 11i v3 release of

HP-UX. HP recommends that you use HP System Management Homepage (HP SMH) in an HP-UX 11i v3 system. For OLAR operations, use the new pdweb utility, integrated in SMH.

• Issuing command-line commands, through rad, that refer to the HyperFabric OLAR script (/usr/sbin/olard.d/clicd).

HP recommends that pdweb be used for OLAR procedures, instead of the rad command. This is primarily because pdweb prevents the user from doing things that might have adverse effects. This is not true when the rad command is used.

For detailed information about using either of these two procedures, see Interface Card OL* Support Guide.

Table 3-1 below explains some important OLAR-related terms.

Table 3-1 Important OLAR Terms

Term Meaning

OLAR All aspects of the OLAR feature

Power Domain A grouping of 1 or more interface

target card / target card slot The interface card which will be

including Online Addition (OLA) and Online Replacement (OLR).

card slots that are powered on or off as a unit. (Note: Multi-slot power domains are not currently supported.)

added or replaced using OLAR, and the card slot in which it resides.

Chapter 3

1. The HP System Management Homepage is an enhanced version of System

Administration Manager (SAM) and is introduced for managing HP-UX.

Installing HyperFabric

Installing HyperFabric Adapters

Table 3-1 Important OLAR Terms (Continued)

Term Meaning

affected card / affected card slot Interface cards and the card slots

they reside in, which are in the same power domain as the target slot.

IMPORTANT In many cases, other interface cards and slots within the system are dependent on the

target card. For example, if the target card is a multiple-port card, suspending or deleting drivers for the target card slot also suspends individual drivers for the multiple hardware paths on that card.

During a card replacement operation, pdweb performs a Critical Resource Analysis (CRA), which checks all ports on the target card for critical resources that would be temporarily unavailable while the card is shut down.

Planning and Preparation

As mentioned previously, for the most part, pdweb prevents the user from performing OLAR procedures that would adversely affect other areas of the HP 9000 system. See Interface Card OL* Support Guide for detailed information.

Critical Resources

The effects of shutting down a card’s functions must be considered. Replacing a card that is still operating can have extensive consequences. Power to a slot must be turned off when a card is removed and a new card is inserted.

This is particularly important if there is no online failover or backup card to pick up those functions. For example:

• Which mass storage devices will be temporarily disconnected when a card is shut down?

• Will a critical networking connection be lost?

A critical resource is one that would cause a system crash or prevent an operation from successfully completing if the resource were temporarily suspended or disconnected. For example, if the SCSI controller is connected to the unmirrored root disk or swap space, the system will crash when the SCSI controller is shut down.

During an OLAR procedure, it is essential to check the targeted card for critical resources, as well as the effects of existing disk mirrors and other situations where a card’s functions can be taken over by another card that will not be affected.

Fortunately, as mentioned earlier, OL* performs a thorough CRA automatically, and presents options based on its ﬁndings. If it is determined that critical resources will be affected by the OLAR procedure, the card could be replaced when the system is ofﬂine. If action must be taken immediately, an online addition of a backup card and deletion of the target card could be attempted using rad.

Chapter 3

Installing HyperFabric

Installing HyperFabric Adapters

Card Compatibility

This section explains card compatibility considerations for doing OLAR.

Online Addition (OLA) Multiple cards can be added at the same time. When adding a card online, the ﬁrst issue to resolve is whether the new card is compatible with the system. Each OLAR-capable PCI slot provides a set amount of power. The replacement card cannot require more power than there is available.

The card must also operate at the slot’s bus frequency. A PCI card must run at any frequency lower than its maximum capability, but a card that could operate at only 33 MHz would not work on a bus running at 66 MHz. rad provides information about the bus frequency and power available at a slot, as well as other slot-related data.

If an HP 9000 system has one or more slots that support OLAR and OLA will be used to install a HyperFabric adapter in one of those slots. Refer to the Interface Card OL* Support Guide for information on how to install the adapter in the HP 9000 or Integrity server.

After adding a new HyperFabric adapter, SMH tries to locate the HyperFabric software. If SMH cannot locate the HyperFabric software, the new adapter cannot be used until the software is installed (remember that software installation requires a system reboot). If SMH locates the HyperFabric software, it determines whether the new adapter is functional. If it is not functional, an error message is displayed.

If the new adapter is functional, SMH displays a message telling the user to conﬁgure the adapter and start HyperFabric. If only one adapter is being added, issue the

clic_init -c command or use SMH to conﬁgure the adapter, and then issue the clic_start command or use SMH to start HyperFabric. If multiple adapters are being

added, add all of the adapters ﬁrst, and then run clic_init -c and clic_start or use SMH. See “Performing the Conﬁguration” on page 69 and “Starting HyperFabric” on page 95 for more information about conﬁguring and starting HyperFabric.

CAUTION Do not change any conﬁguration information for an existing HyperFabric adapter or

switch while you are using clic_init -c to conﬁgure a new adapter.

When you have completed the adapter installation, go to “Installing the Software” on page 47.

Online Replacement (OLR) When replacing an interface card online, the replacement card must be identical to the card being replaced (or at least be able to operate using the same driver as the replaced card). This is referred to as like-for-like replacement and should be adhered to, because using a similar but not identical card can cause unpredictable results. For example, a newer version of the target card that is identical to the older card in terms of hardware might contain an updated ﬁrmware version that could potentially conﬂict with the current driver. For example, an A6386A adapter must be replaced with another A6386A adapter. In addition, the old adapter and new adapter must have the same revision levels.

Chapter 3

When a replacement card is added to an HP 9000 system, the appropriate driver for that card must be conﬁgured in the kernel before beginning the replacement operation. SMH ensures the correct driver is present. (In most cases, the replacement card will be the same type as a card already in the system, and this requirement will be automatically met.) Keep the following things in mind:

Installing HyperFabric

Installing HyperFabric Adapters

• If the necessary driver is not present and the driver is a dynamically loadable kernel module (DLKM), it can be loaded manually. See the “Dynamically Loadable Kernel Modules” section in “Interface Card OL* Support Guide” for more information.

• If the driver is static and not conﬁgured in the kernel, then the card cannot be added online. The card could be physically inserted online, but no driver would claim it.

If there is any question about the driver’s presence, or if it is uncertain that the replacement card is identical to the existing card, ioscan can be used together with rad to investigate.

If more than one operational HyperFabric adapter is present when SMH requests the suspend operation for all ports on the target adapter, HyperFabric will redirect the target adapter’s trafﬁc to a local backup adapter using local failover. Client applications using the replaced adapter will not be interrupted in any way.

If the adapter being replacing is active and it is the only operational HyperFabric adapter on the HP 9000 system, SMH displays the following warning message:

WARNING: You have 1 operational HyperFabric card. If you go ahead with this operation you will lose network access via HyperFabric until the on-line replaced HyperFabric card becomes operational.

You are asked if you want to continue. If you reply Yes, client applications are suspended. Replace the adapter according to the procedure described in the Interface Card OL* Support Guide.

When an adapter has been replaced, client application activity resumes unless the TCP timers or the application timers have popped.

CAUTION Do not use the clic_start command or the clic_shutdown command, while an

installed adapter is suspended. Do not use SMH to start or stop HyperFabric while an installed adapter is suspended. The operation will fail and an error message will be displayed.

After a HyperFabric adapter has been replaced, SMH checks the replacement adapter to make sure it is permitted according to the like-for-like rules. If the adapter is permitted, SMH automatically activates it. If it is not permitted, SMH displays an error message.

Chapter 3

Installing the Software

This section describes the HyperFabric ﬁle structure and the steps necessary to load the software. The software must be installed on each instance of the HP-UX operating system in the fabric.

File Structure

The HyperFabric ﬁle structure is shown in Figure 3-1 below. Note that the structure is shown for informational purposes only. The user cannot modify any of the ﬁles or move them to a different directory.

Figure 3-1 HyperFabric File Structure

Installing HyperFabric

Installing the Software

/resmon

/dictionary

/clic_01

/usr

/conf

/lib

/libclic_dlpi_drv.a /libha_drv.a

/opt

/master.d

/etc

/clic

/libclic_mgmt.a

/rc.config.d

/clic_global_conf

/lib

/clic_diag /clic_dump /clic_init /clic_mgmtd /clic_mond /clic_ping

/clic_probe /clic_shutdown /clic_start /clic_stat

/opt

/clic

/bin

/sbin

/init.d

/clic

/firmware

/clic_fw /clic_fw_1x32c

/clic_fw_4x8c /clic_fw_4x32c /clic_fw_hf28c

/clic_fw_hf232c /clic_fw_db

/var/adm

/clic_ip_drv.trc

/clic_ip_drv.trc0

/clic_ip_drv.trc1

/clic_log /clic_log.old

/OLDclic_log

/share

/man

/man1m.Z

Chapter 3

The commands and ﬁles used to administer HyperFabric typically have a preﬁx of clic_. CLIC stands for CLuster InterConnect, and it is used to differentiate those HyperFabric commands/ﬁles from other commands/ﬁles. For example, the HyperFabric command clic_init is different from the HP-UX init command.

Each of the ﬁles shown in Figure 3-1 above is brieﬂy described below:

• /etc/opt/resmon/dictionary/clic_01 The HyperFabric dictionary ﬁle for the Event Monitoring Service (EMS).

• /etc/rc.config.d/clic_global_conf

Installing HyperFabric

Installing the Software

• /sbin/init.d/clic

• /var/adm/clic_ip_drv.trc

• /var/adm/clic_ip_drv.trc0

• /var/adm/clic_ip_drv.trc1

• /var/adm/clic_log

The global conﬁguration ﬁle, which contains the IP addresses for each adapter and each HyperFabric switch (if any) in the fabric.

The system boot startup script for the HyperFabric management process.

One of the software’s trace ﬁles. This ﬁle is created when the clic_diag -D TCP_IP command is run.

One of the HyperFabric software’s trace ﬁles. This is the primary ﬁle that is created when the clic_diag -C TCP_IP command is run.

One of the HyperFabric software’s trace ﬁles. This ﬁle is created when the clic_diag -C TCP_IP command is run, and the primary trace ﬁle (clic_ip_drv.trc0) becomes full.

The global log ﬁle that is updated by the HyperFabric management process.

• /var/adm/clic_log.old The backup copy of the log ﬁle that is created when the log ﬁle grows larger than 100

Kbytes.

• /var/adm/OLDclic_log The log ﬁle from the previous time the clic_start command was executed.

• /usr/conf/lib/libclic_dlpi_drv.a The kernel library that contains the HyperFabric software.

• /usr/conf/lib/libha_drv.a The kernel library that contains the High Availability (HA) software.

• /usr/conf/master.d/clic This ﬁle is described along with the other master ﬁles in the master man page (type

man master at the HP-UX prompt).

• /opt/clic/lib/libclic_mgmt.a The HyperFabric management API library.

• /opt/clic/bin The directory containing the HyperFabric management commands: clic_diag,

clic_init, clic_probe, clic_shutdown, clic_start, clic_stat, and clic_dump. (Note that clic_dump is for HP internal use only.) Also, clic_ping is replaced by clic_probe. This directory also contains the HyperFabric management process (clic_mgmtd) and the HyperFabric EMS monitor process (clic_mond).

• /opt/clic/firmware/clic_fw_4x8c The 4X PCI HyperFabric 8-bit CRC ﬁrmware. This ﬁle must not be modiﬁed for any

reason.

• /opt/clic/firmware/clic_fw_4x32c

Chapter 3

Installing HyperFabric

Installing the Software

The 4X HyperFabric PCI 32-bit CRC ﬁrmware. This ﬁle must not be modiﬁed for any reason.

• /opt/clic/firmware/clic_fw_hf28c The HyperFabric2 8-bit ﬁrmware. This ﬁle must not be modiﬁed for any reason.

• /opt/clic/firmware/clic_fw_hf232c The HyperFabric2 32-bit ﬁrmware. This ﬁle must not be modiﬁed for any reason.

• /opt/clic/firmware/clic_fw_db A binary ﬁle where adapter-speciﬁc conﬁguration information is stored. The

management process creates this ﬁle using default values.

• /opt/clic/share/man/man1m.Z The man pages for the HyperFabric commands.

Chapter 3

Installing HyperFabric

Installing the Software

Loading the Software

Listed below are the steps you must follow to load the HyperFabric software, using the HP-UX swinstall program.

Step 1. Log in as root.

Step 2. Insert the software media into the appropriate drive. If the software is being loaded from

a CD-ROM, go to step 3. Otherwise, go to step 4.

Step 3. Mount the CD-ROM drive by using this command:

mount

where

Step 4. Run the swinstall program using this command:

/usr/sbin/swinstall

This opens the “Software Selection” window.

Step 5. Change the Source Host Name, if necessary, and then enter the mount point of the drive

in the Source Depot Path ﬁeld. Select the OK button to return to the “Software Selection” window.

The “Software Selection” window now contains a list of available software to install.

Step 6. Highlight the HyperFabric software:

• HP-UX 11i v3: HyperFabric-00

Step 7. Choose Mark for Install from the “Actions” menu; this chooses the highlighted

software.

Step 8. From the “Actions” menu, pull down the “Install...” menu, and then choose Install.

This begins product installation and opens the “Install Analysis” window.

Step 9. Select the OK button in the “Install Analysis” window when the Status ﬁeld displays a

“Ready” message.

device_name

is the name assigned to the CD-ROM drive.

Step 10. Select the YES button in the “Conﬁrmation” window to start software installation.

swinstall loads the ﬁleset, runs the control script for the ﬁlesets, and builds the kernel.

When the processing is ﬁnished, the “Status” ﬁeld displays a “Ready” message. Select “Done” and then the “Note” window opens.

Step 11. Select the OK button in the “Note” window to reboot. The user interface disappears and

the system reboots.

Step 12. When the system comes back up, log in as root and view the

/var/adm/sw/swagent.log and /var/adm/sw/swinstall.log ﬁles to view any error or

warning messages that might have occurred during the installation.

Step 13. While still logged in as root, view the /etc/services ﬁle to ensure that these two

HyperFabric-related lines are present:

• hp-clic 3384/tcp #clic management daemon

• hp-clic 3384/udp #clic switch management

Chapter 3

Installing HyperFabric

Installing the Software

Note that these lines are used by the HyperFabric software—and are not comments—so do not remove them from the ﬁle.

Step 14. Verify that all installed HyperFabric adapters have a software state of “CLAIMED,” by

running the ioscan -nf -C clic command.

Note: A check is also done to make sure all of the HyperFabric adapters have been claimed when clic_init is activated or when SMH is used to conﬁgure HyperFabric.

Step 15. If one or more HyperFabric switches are included in the conﬁguration, go to the next

section of this chapter, “Installing HyperFabric Switches”, otherwise, go to Chapter 4, “Conﬁguring HyperFabric,” on page 61.

Chapter 3

Installing HyperFabric

Installing HyperFabric Switches

This section contains the information you need to install HyperFabric switches. As stated earlier, in this manual the term HyperFabric2 (HF2) switch refers to the functional switch (the A6384A switch chassis with one of the switch modules installed).

Before Installation

Before you install the HyperFabric switch, you should be aware of these things:

❏ The A6384A HF2 switch is supported on the following HyperFabric software version:

— HP-UX 11i v3: version B.11.31.01 HyperFabric switches are not supported by software versions earlier than those

mentioned above, respectively. To determine the version of HyperFabric you have, issue this command:

swlist | grep -i hyperfabric

❏ For the HF2 switch, we recommend that you use the rails shipped with the switch

when you mount it in a standard 19-inch rack, even though the switch can be mounted in the rack by itself (without the rails).

WARNING To prevent overheating, you must leave one rack unit (1 EIA) of empty

space above the HyperFabric switch.

❏ After the HyperFabric switch is mounted in the rack, you attach the various cables

to the switch. To avoid damage to any of the cables, follow these guidelines:

— If your cables have dust caps over the connectors, keep them in place until you

are ready to connect them. This prevents dirt and oils from soiling any important surfaces.

— Be careful not to stretch, puncture, or crush the cable.

To install an HF2 switch, see “Installing the HF2 Switch” on page 53.

Chapter 3

Installing HyperFabric Switches

Installing the HF2 Switch

This section contains information for installing an HF2 switch. The front of the HF2 switch has a ﬂange—or “wing”—on each side, with two holes for

attaching the switch to the rack. Note that the two ﬁgures below do not show the ﬂanges. Figure 3-2 below shows the front of the HF2 switch with an A6388A HF2 8-port ﬁbre

switch module installed in the switch’s expansion slot.

Figure 3-2 Front of HF2 Switch (A6388A Switch Module Installed)

Installing HyperFabric

Integrated Ethernet management

LAN card

Status

Powe r

Por t

Ethernet

Por t Main

Por t

Ethernet

Por t

Aux

Label showing Ethernet MAC address

Por t

Port LED colors and meanings

legend

Por t

Integrated 8-port ﬁbre card

Por t

A6388A HF2 8-port ﬁbre switch module in

expansion slot

Figure 3-3 below shows the front of the HF2 switch with an A6389A HF2 4-port copper switch module installed in the switch’s expansion slot.

Por t

Chapter 3

Installing HyperFabric

Installing HyperFabric Switches

Figure 3-3 Front of HF2 Switch (A6389A Switch Module Installed)

Por t

ﬁbre card

Integrated 8-port

Por t

Integrated Ethernet management

LAN card

Status

Powe r

Por t

Ethernet

Por t

Main

Por t

Ethernet

Por t

Aux

Label showing Ethernet MAC address

Por t

Port LED

colors and meanings

legend

A6389A HF2 4-port copper switch module in expansion slot

You can install the HF2 switch in one of these two ways:

• Using the rail kit that is shipped with the switch (see the next section, “With the Rail Kit”). Note that HP strongly recommends installing the HF2 switch this way.

Por t

• Attaching the switch directly to the rack (see “Without the Rail Kit” on page 58).

Chapter 3

With the Rail Kit

HP recommends that you install the HF2 switch using the rail kit that is shipped with the switch. The rail kit includes two adjustable rails, screws, nuts, and washers.

To install the HF2 switch, you need eight screws and four nuts. Use the square cage nuts if you are installing the HF2 switch in a square-hole rack. Use the u-type clip nuts if you are installing the HF2 switch in a round-hole rack.

Figure 3-4 shows various parts that are shipped with the rail kit.

Figure 3-4 Parts of the Rail Kit

Installing HyperFabric

Installing HyperFabric Switches

Chapter 3

The rail kit does not include hold-down brackets for the rear of the switch. HP does not recommend transporting the rack with the switch installed. HP recommends that two people install the HF2 switch.

Installing the HF2 Switch With the Rail kit

When you install the HF2 switch, you must put the front of the switch—the end with the ﬂanges (“wings”)—at the back of the rack. To install the HF2 switch using the rail kit, complete the following steps:

Step 1. Prepare the rack for rail and switch installation.

Step 2. Remove all the screws if you receive the rail kit with all ten screws secured in to the

rails.

Installing HyperFabric

Installing HyperFabric Switches

One end of each rail has six screw-holes (End-A), the other end has two screw holes (End-B). Figure 3-5 shows both the ends of the rail.

Figure 3-5 The Ends of the Rail Kit

Step 3. Orient the rails so that End-A faces the back of the rack and aligns with the front end of

the switch with ﬂanges.

Step 4. Loosen the wing nuts on each rail and adjust the length of each rail to ﬁt the length of

the rack. End A mounts inside the rack column, and End-B mounts outside the rack column.

Step 5. Tighten the wing nuts on each rail after you have adjusted the length properly.

Step 6. Install and secure the rails in the rack, using two screws per rail. Do not secure End-A.

To secure End-B, complete the following steps:

1. If you have square-hole racks, afﬁx two cage nuts inside each rack column. Align these cage nuts with the two holes in End-B of each rail. Secure the assembly with two screws in End-B of each rail.

2. If you have round-hole racks, afﬁx two clip nuts to each rack column. Align these clip nuts with the two holes in End-B of each rail. Secure the assembly with two screws in End-B of each rail.

Step 7. To install the switch, complete the following steps:

1. Orient the front end of the switch (the end with the ﬂanges) toward the back of the rack.

2. Place the switch on the rails and slide it in to the rack until the ﬂanges are snug against the outside of the rack columns.

Chapter 3

Installing HyperFabric

Installing HyperFabric Switches

NOTE HP recommends employing two people to support the weight of the switch because

End-A of the rail is not yet secured.

Step 8. Secure the switch and End-A of each rail by aligning the two holes in each ﬂange with

the two holes in each rack column, and two of the holes in each rail. Secure the entire assembly with two screws in each ﬂange.

Step 9. Attach the cable from the corresponding adapter for each port that is connected to a

HyperFabric adapter in an HP 9000 system.

NOTE Your connections must be copper-to-copper and ﬁbre-to-ﬁbre (including cables).

Step 10. Connect the switch to the Ethernet network.

Step 11. Plug the switch’s power cord in to the rack’s Power Distribution Unit (PDU), if it has

one.

NOTE Ensure that you plug a power card that is compatible with your country’s speciﬁcations

in to a power strip or outlet that you want to use for the switch. In such a scenario, you are responsible for obtaining a compatible power cord.

Step 12. Power on the HF2 switch by plugging the power cord into the AC inlet on the back of the

switch. (There is no power switch.)

Step 13. Once the power is on, check these LEDs on the integrated Ethernet management LAN

adapter card in the top slot of the switch:

✓ The “Operating/Fault” LED shows solid green.

✓ The “Power A” and “Power B” LEDs show solid green.

✓ The “Ethernet Port Main” and “Ethernet Port Aux” LEDs show solid green

(connected) or ﬂashing green. This indicates that ethernet trafﬁc is ﬂowing to the switch. For information about locating the LEDs, see Figure 3-2 on page 53 or Figure 3-3 on page 54.

Step 14. On the integrated 8-port ﬁbre card in the middle slot of the switch, check whether the

LED for each switch port that is connected to an HF2 adapter shows solid green. If the LED shows solid green, it means the connection is operational.

Step 15. On the switch module in the expansion slot in the bottom slot of the switch, check

whether the LED for each switch port that is connected to an HF2 adapter shows solid green. If the LED shows solid green, it means the connection is operational.

Chapter 3

For more information about the switch’s LEDs, see “HF2 Switch LEDs” on page 126.

Repeat steps 1 to 16 to install another HF2 switch using the rail kit. For information about installing an HF2 switch without using a rail kit, see “Without the Rail Kit” on page 58.

Installing HyperFabric

Installing HyperFabric Switches

Without the Rail Kit

As mentioned earlier, HP strongly recommends installing the HF2 switch using the rail kit (described in the previous section, “With the Rail Kit” on page 55).

When you install the HF2 switch, you will be putting the front of the switch—the end with the ﬂanges (“wings”)—at the back of the rack. The steps for installing the HF2 switch without using the rail kit are as follows:

Step 1. Prepare the rack for switch installation.

Step 2. Insert the HF2 switch into the rack, with the front of the switch snug against the back of

the rack.

Step 3. Align the two holes in each ﬂange on the switch’s front with the holes in the rack frame.

Step 4. Fasten each ﬂange of the switch to the rack by putting a screw in each of the four holes

in the ﬂanges. Be sure to use screws with over-sized heads.

Step 5. Tighten all of the screws so that the HF2 switch is ﬁrmly mounted in the rack.

Step 6. For each port that will be connected to an HyperFabric adapter in an HP 9000 system,

attach the cable from the corresponding adapter. Remember, your connections must be copper-to-copper and ﬁbre-to-ﬁbre, including cables.

Step 7. Connect the switch to the Ethernet network.

Step 8. Plug the switch’s power cord into the rack’s power distribution unit (PDU), if it has one.

Alternatively, you can plug a power cord that is compatible with your country’s requirements into a power strip or outlet that you want to use for the switch. (In this case, you are responsible for obtaining a compatible power cord.)

Step 9. Power on the HF2 switch by plugging the power cord into the AC inlet on the back of the

switch. (There is no power switch.)

Step 10. Once the power is on, check these LEDs on the integrated Ethernet management LAN

adapter card (in the top slot of the switch):

✓ The “Operating/Fault” LED shows solid green.

✓ The “Power A” and “Power B” LEDs show solid green.

✓ The “Ethernet Port Main” and “Ethernet Port Aux” LEDs are showing solid green

(connected) or ﬂashing green (Ethernet trafﬁc is ﬂowing to the switch). See Figure 3-2 or Figure 3-3 below for the locations of the LEDs.

Step 11. On the integrated 8-port ﬁbre card (in the middle slot of the switch), check that for each

switch port that is connected to an HF2 adapter, the LED on the port shows as solid green (see Figure 3-2 on page 53 or Figure 3-3 on page 54). This means the connection is operational.

Step 12. On the switch module in the expansion slot (the bottom slot of the switch), check that for

each switch port that is connected to a HyperFabric adapter, the LED on the port shows as solid green (see Figure 3-2 on page 53 or Figure 3-3 on page 54). This means the connection is operational.

For more detailed information about the switch’s LEDs, see “HF2 Switch LEDs” on page 126.

Step 13. If you want to install another HF2 switch without using the rail kit, go to step 1.

Chapter 3

Installing HyperFabric

Installing HyperFabric Switches

If you want to install another HF2 switch using the rail kit, go to “With the Rail Kit” on page 55.

Otherwise, go to Chapter 4, “Conﬁguring HyperFabric,” on page 61.

Chapter 3

Installing HyperFabric

Installing HyperFabric Switches

Chapter 3

4 Conﬁguring HyperFabric

This chapter contains the following sections that describe conﬁguring HyperFabric:

• “Conﬁguration Overview” on page 63

• “Information You Need” on page 64

• “Performing the Conﬁguration” on page 69

Chapter 4

Conﬁguring HyperFabric

• “Deconﬁguring a HyperFabric Adapter with SMH” on page 74

• “Conﬁguring the HyperFabric EMS Monitor” on page 75

• “Conﬁguring HyperFabric with ServiceGuard” on page 76

Chapter 4

Conﬁguring HyperFabric

Conﬁguration Overview

You do not need to conﬁgure the HyperFabric switch because the HyperFabric management process performs automatic routing and conﬁguring for the switch. So, conﬁguring HyperFabric consists only of creating the HyperFabric /etc/rc.config.d/clic_global_conf global conﬁguration ﬁle on each node in the fabric. The conﬁguration ﬁle contains the following information:

• The IP addresses and subnet mask of the HyperFabric adapters installed in the node.

• For each HyperFabric switch in the fabric—the switch’s IP address, and the MAC address of the switch’s Ethernet port. Note that this applies only if you enable switch management. Also note that you cannot enable switch management through SMH—you must use the clic_init command.

• The IP multicast address that all the switches and nodes in the fabric will register to (if you are going to enable switch management).

• The IP address of the local node’s Ethernet LAN interface. This LAN interface must be on the same subnet as the Ethernet port(s) of the HyperFabric switch(es) (if you are going to enable switch management). (Note that a node might have multiple LAN interfaces.)

NOTE HP recommends that you do not enable switch management.

You can create the global conﬁguration ﬁle by either (1) running the clic_init command or (2) using SMH to conﬁgure each HyperFabric adapter.

clic_init and SMH also put the necessary entries into the following three ﬁles:

• The system /etc/rc.config.d/netconf ﬁle.

NOTE In this ﬁle, clic_init and SMH add some HyperFabric-related lines that end with the

characters #clic. These lines are used by the HyperFabric software—and are not comments—so do not remove them from the ﬁle.

• The system /etc/rc.config.d/clic_global_conf ﬁle.

• The /etc/rarpd.conf (Reverse Address Resolution Protocol [RARP]) support ﬁle. This ﬁle is used in the management of the HyperFabric switches (if you are going to enable switch management).

The clic_init command is described in “Using the clic_init Command” on page 70. Using SMH to conﬁgure an adapter is described in “Using SMH” on page 72.

After you have used the clic_init command or SMH, you can conﬁgure HyperFabric with ServiceGuard, if necessary. See “Conﬁguring HyperFabric with ServiceGuard” on page 76 for more information.

Chapter 4

Conﬁguring HyperFabric

Information You Need

When you run the clic_init command or use SMH for conﬁguration, you have to provide certain conﬁguration information. So, before you run clic_init or use SMH, you should get the following information:

❏ For each node in the fabric, determine if that node will need to interoperate with

other nodes that are using; any HP-UX 11i v3 system and HyperFabric versions earlier than B.11.31.00.

❏ For each HyperFabric adapter installed in the local node:

✓ The adapter’s IP address.

NOTE The last 10 bits of each adapter’s IP address must be unique throughout the

entire fabric. And, remember that the last part of the address cannot be 0 (that is, the IP address cannot be n.n.n.0). Also, note that HyperFabric converts these 10 bits to a decimal value called the Virtual route IDentiﬁer (VRID), which is used in some HyperFabric command input and output.

✓ The subnet mask. When you run clic_init or use SMH, if you do not specify a

value for this, a default subnet mask is chosen based on the adapter’s IP address.

When clic_init begins to prompt you for the information for each adapter, it assigns an ID (for example, clic0) to that adapter and displays it as part of the ﬁrst prompt. If you use SMH, it assigns the adapter an ID and displays it in the “Adapter Name” column of the “Conﬁgure HyperFabric Adapter” screen. Note that you can also determine an adapter’s ID by running the clic_stat command (see “The clic_stat Command” on page 101). You should note each adapter’s ID, because it is used as input to other HyperFabric commands.

❏ For using the Transparent Local Failover feature of HMP available with this version

of HyperFabric, you need to deﬁne the card pairs.

❏ For each HyperFabric switch in the fabric (if you are going to enable switch

management):

✓ The IP address of the switch. ✓ The MAC address of the switch’s Ethernet port. If you do not already know the

switch’s MAC address, it is printed on a label on the back of the HF switch and on the front of the HF2 switch.

NOTE Remember, you cannot enable switch management through SMH—you must use the

clic_init command.

When clic_init begins to prompt you for the information for each switch, it assigns an ID (for example, sw_clic0) to that switch and displays it as part of the ﬁrst prompt. Note that you can also determine a switch’s ID by running the clic_stat command (see “The clic_stat Command” on page 101). You should note each switch’s ID, because it is used as input to other HyperFabric commands.

Chapter 4

Conﬁguring HyperFabric

Information You Need

❏ For the entire fabric, you need the IP multicast address that all the switches and

nodes in the fabric will register to. The address must be a class D address. Note that if you do not have switch management enabled, you do not need this information (clic_init will not prompt you for it).

❏ For each node in the fabric, you need the IP address of the node’s Ethernet LAN

interface that is on the same subnet as the switches. (As mentioned earlier, a node might have multiple LAN interfaces.) Note that if you do not have switch management enabled, you do not need this information (clic_init will not prompt you for it).

As stated earlier, HP recommends that you do not enable switch management.

IMPORTANT You should also check your /etc/hosts ﬁle—when you are using ﬁles for host name look

up—to ensure that the entries for all of the systems are in the correct format: the ofﬁcial host name, which is the full domain extended host name, and any alias names. For example:

IP_address bently6.corp3.com bently6 IP_address bently4.corp7.com test1 IP_address bently2.corp4.com test3

Chapter 4

Conﬁguring HyperFabric

Information You Need

Conﬁguration Information Example

For this example, we have added some “dummy” (that is, not valid) addresses to the components in Figure 4-1, Map for Conﬁguration Information Example, below. The “dummy” addresses are used only to show the ﬂow of the information provided as input to the clic_init command and SMH. Do not try to use these addresses in your conﬁguration.

Figure 4-1 Map for Conﬁguration Information Example

Ethernet LAN

Switch ID: sw_clic0 IP address: 193.0.0.20

Ethernet MAC address:

00:60:b0:d0:02:57

IP multicast address:

226.10.1.1

adapter 0

Adapter ID:

clic0

IP address:

192.0.0.1

subnet mask:

255.255.255.0

node A

HF switch 0

adapter 1

Adapter ID:

clic1

IP address:

192.0.8.3

subnet mask:

255.255.255.0

lan0 lan0

Adapter ID:

IP address:

192.0.0.2

subnet mask:

255.255.255.0

switch 1

adapter 0

clic0

Switch ID: sw_clic1

IP address: 193.0.0.21 Ethernet MAC address:

00:60:b0:d0:02:56

IP multicast address:

226.10.1.1

adapter 1

Adapter ID:

clic1

IP address:

192.0.8.4

subnet mask:

node A

255.255.255.0

node B

IP address: 193.0.0.10 IP address: 193.0.0.11

Using the conﬁguration information in Figure 4-1 above, the information you would specify when you run clic_init or SMH on each of the nodes is listed below. Note that this example is not an exact depiction of the prompts produced by clic_init nor the ﬁelds in SMH, but merely an example of the ﬂow of information input. Also, remember that you should not try to use the “dummy” addresses in your actual conﬁguration.

On node A:

1. How many HyperFabric adapters are installed on the node?

2. Do you want this node to interoperate with nodes running any HyperFabric 10.20 version or HyperFabric versions earlier than B.11.00.11 or B.11.11.01?

3. What is the IP address of the ﬁrst adapter (clic0)? (192.0.0.1)

Chapter 4

Conﬁguring HyperFabric

Information You Need

4. What is the subnet mask of the ﬁrst adapter? (255.255.255.0) If you do not specify a value for this, a default mask is chosen. You will most likely

just accept the default. However, in this example, we are showing a value for the subnet mask just to illustrate the correlation between the “dummy” information in Figure 4-1 and where that information is speciﬁed or generated during clic_init and SMH.

5. What is the IP address of the second adapter (clic1)? (192.0.8.3)

6. What is the subnet mask of the second adapter? (255.255.225.0)

7. Do you want to enable switch management? Remember, you cannot enable switch management through SMH (you must use the clic_init command).

As stated earlier, we recommend that you do not enable switch management. However, if you do enable it, you must provide the information in items 8 through 14:

8. If switch management has been enabled, how many switches will be conﬁgured? As stated earlier, we recommend that you do not enable switch management.

9. What is the IP address of the ﬁrst switch (sw_clic0)? (193.0.0.20)

10. What is the Ethernet hardware address of the ﬁrst switch? (0060b0d00257)

11. What is the IP address of the second switch (sw_clic1)? (193.0.0.21)

12. What is the Ethernet hardware address of the second switch? (0060b0d00256)

13. What is the Multicast address for the switches to use? (226.10.1.1)

14. What is the IP address for the LAN card on the same subnet as the switches? (193.0.0.10)

(Looking at Figure 4-1, this is the IP address for lan0 on node A.)

On node B:

1. How many HyperFabric adapters are installed on the node?

2. Do you want this node to interoperate with nodes running any HyperFabric 10.20 version or HyperFabric versions earlier than B.11.00.11 or B.11.11.01?

3. What is the IP address of the ﬁrst adapter (clic0)? (192.0.0.2)

4. What is the subnet mask of the ﬁrst adapter? (255.255.255.0) If you do not specify a value for this, a default mask is chosen. You will most likely

5. What is the IP address of the second adapter (clic1)? (192.0.8.4)

6. What is the subnet mask of the second adapter? (255.255.225.0)

Chapter 4

7. Do you want to enable switch management? Remember, you cannot enable switch management through SMH (you must use the clic_init command).

As stated earlier, we recommend that you do not enable switch management. However, if you do enable it, you must provide the information in items 8 through 14 below.

Conﬁguring HyperFabric

Information You Need

8. If switch management has been enabled, how many switches will be conﬁgured? As stated earlier, we recommend that you do not enable switch management.

9. What is the IP address of the ﬁrst switch (sw_clic0)? (193.0.0.20)

10. What is the Ethernet hardware address of the ﬁrst switch? (0060b0d00257)

11. What is the IP address of the second switch (sw_clic1)? (193.0.0.21)

12. What is the Ethernet hardware address of the second switch? (0060b0d00256)

13. What is the Multicast address for the switches to use? (226.10.1.1)

14. What is the IP address for the LAN card on the same subnet as the switches? (193.0.0.11)

(Looking at Figure 4-1, this is the IP address for lan0 on node B.)

Chapter 4

Conﬁguring HyperFabric

Performing the Conﬁguration

As explained in “Conﬁguration Overview” on page 63, you must create the global conﬁguration ﬁle (/etc/rc.config.d/clic_global_conf) on each node in the fabric. This consists mostly of specifying HyperFabric adapter-related information. (Note that if you are also going to enable switch management—which we do not recommend doing—you need to specify additional conﬁguration information.)

NOTE Specifying conﬁguration information adds or changes only the addresses and other

information in the global conﬁguration ﬁle, based on the information you supply. It does not perform any operations to check the relationships between that information and any physical connections within the fabric.

You need to create the global conﬁguration ﬁle in the following situations:

• You have just installed the HyperFabric hardware and software on the system.

• You want to change the information in the HyperFabric global conﬁguration ﬁle (see the Note above).

IMPORTANT Creating the global conﬁguration ﬁle also modiﬁes the /etc/rc.config.d/netconf ﬁle, adding

some HyperFabric-related lines that end with the characters #clic. These lines are used by the HyperFabric software—and are not comments—so do not remove them from the ﬁle.

You can create the global conﬁguration ﬁle by using (1) the clic_init command (described in the next section, “Using the clic_init Command”) or (2) SMH (described in “Using SMH” on page 72). You cannot enable card pair or switch management through SMH (you must use the clic_init command).

Chapter 4

Conﬁguring HyperFabric

Performing the Conﬁguration

Using the clic_init Command

Run the clic_init command to create the global conﬁguration ﬁle. To use clic_init command to conﬁgure the Transparent Local Failover feature on HMP,

see “Using SMH” on page 72.

IMPORTANT If the global conﬁguration ﬁle already exists and you are running clic_init again (to

change the ﬁle), you have the option of retaining or modifying the existing conﬁguration information, in addition to adding new information pertaining to new hardware.

Also, once you’ve completed your changes and clic_init ends its processing, you must stop HyperFabric (by running the clic_shutdown command or using SMH) and then start HyperFabric (by running the clic_start command or using SMH). Otherwise, your conﬁguration information changes will not take effect. See “Stopping HyperFabric” on page 110 and “Starting HyperFabric” on page 95 for more information.

If you include /opt/clic/bin in your PATH statement, you can run the command as it is shown below. Otherwise, you must include /opt/clic/bin as part of the command name (that is, /opt/clic/bin/clic_init).

You must be logged in as root to run this command. The syntax is as follows: clic_init [-c] [-?] where

• -c speciﬁes that you want to create the global conﬁguration ﬁle. You are prompted for the information described in “Information You Need” on page 64. Note that if the global conﬁguration ﬁle already exists (for example, when you are adding an adapter to an existing fabric), clic_init prompts you with the existing conﬁguration information. As you are prompted with each piece of information, you can then conﬁrm that you want to keep it or you can change it.

• -? displays the online help for clic_init.

If you do not specify any of the above parameters, the online help for clic_init is displayed. After you have entered the information for all the adapters in the node and all of the

switches (if any) in the fabric, a summary of the conﬁguration information is displayed. Once clic_init has ﬁnished, you do one of the following things:

• If you want to conﬁgure HyperFabric with ServiceGuard, complete the conﬁguration described in “Conﬁguring HyperFabric with ServiceGuard” on page 76, then run clic_start or use SMH to start HyperFabric.

• If you have just created the global conﬁguration ﬁle on the local node for the ﬁrst time (and you are not conﬁguring ServiceGuard), run clic_start or use SMH to start HyperFabric.

• If you have just changed an existing conﬁguration ﬁle on the node, run clic_shutdown or use SMH to stop HyperFabric, and then run clic_start or use SMH to start HyperFabric. Until you do those two things, your conﬁguration changes will not take effect.

Chapter 4

Conﬁguring HyperFabric

Performing the Conﬁguration

See “Stopping HyperFabric” on page 110 and “Starting HyperFabric” on page 95 for more information.

Examples of clic_init

Some examples of using the clic_init command are shown below.

• Example 1 To create the global conﬁguration ﬁle on the local node, issue this command:

clic_init -c

• Example 2 To display the online help for the clic_init command, issue this command:

clic_init -?

or this command:

clic_init

Chapter 4

Conﬁguring HyperFabric

Performing the Conﬁguration

Using SMH

This section describes how to use SMH1 to conﬁgure HyperFabric. For information on how to use SMH to conﬁgure and deconﬁgure local failover feature on HMP, see “Conﬁguring HMP for Transparent Local Failover Support - Using SMH” on page 88.

IMPORTANT If the global conﬁguration ﬁle already exists, and you are running SMH again (to change

the ﬁle), you can keep or modify the existing conﬁguration information, in addition to adding new information pertaining to new hardware.

Also, once you’ve completed your changes and SMH ends its processing, you must stop HyperFabric (by running the clic_shutdown command or using SMH) and then start HyperFabric (by running the clic_start command or using SMH). Otherwise, your conﬁguration information changes will not take effect. See “Stopping HyperFabric” on page 110 and “Starting HyperFabric” on page 95 for more information.

To use SMH to create the global conﬁguration ﬁle on an HP 9000 system running HP-UX 11i v3, follow these steps:

Step 1. Start SMH.

Step 2. Select the “Networking and Communications” area.

Step 3. Select the “Network Interfaces” option.

Step 4. Select “HyperFabric.”

All HyperFabric adapters installed in the system are listed; installed adapters that are not yet conﬁgured show Not Configured in the “Status” ﬁeld.

Step 5. Highlight the adapter you want to conﬁgure.

Step 6. Pull down the “Actions” menu and select Configure Adapter.

Step 7. In the “Conﬁgure HyperFabric Adapter” screen, specify information for the following

ﬁelds:

• Internet Address—Required. The IP address of the adapter.

• Subnet Mask—Optional. The adapter’s subnet mask. If you do not specify this, a default mask is chosen based on the adapter’s IP address.

• Interoperability Enabled—Required. Whether you want the adapter to be able to interoperate with adapters that are using; any HP-UX 10.20 version of HyperFabric, any HP-UX 11.0 HyperFabric versions earlier than B.11.00.11 or any HP-UX 11i HyperFabric versions earlier than B.11.11.01. Note that if you select No, the HyperFabric software on the system will not be backwards compatible with previous releases. This means you must update all of the other systems in the fabric to the version that is running on the system.

Default: No.

Step 8. Select OK (remember, you cannot enable switch management within SMH).

1. The HP System Management Homepage (HP SMH) is an enhanced version of

System Administration Manager (SAM) and is introduced for managing HP-UX.

Chapter 4

Step 9. Exit SMH.

Once SMH has ﬁnished, you do one of the following things:

• If you want to conﬁgure HyperFabric with MC/ServiceGuard, complete the conﬁguration described in “Conﬁguring HyperFabric with ServiceGuard” on page 76, then run clic_start or use SMH to start HyperFabric.

• If you have just created the global conﬁguration ﬁle on the local node for the ﬁrst time (and you are not conﬁguring ServiceGuard), run clic_start or use SMH to start HyperFabric.

See “Stopping HyperFabric” on page 110 and “Starting HyperFabric” on page 95 for more information.

Conﬁguring HyperFabric

Performing the Conﬁguration

Chapter 4

Conﬁguring HyperFabric

Deconﬁguring a HyperFabric Adapter with SMH

To use SMH to deconﬁgure a HyperFabric adapter on an HP 9000 system running HP-UX 11i v3, follow these steps:

Step 1. Start SMH.

Step 2. Select the “Networking and Communications” area.

Step 3. Select the “Network Interfaces” option.

Step 4. Select “HyperFabric.”

All HyperFabric adapters installed in the system are listed. Installed adapters that are conﬁgured show Configured in the “Status” ﬁeld, and installed adapters that are not yet conﬁgured show Not Configured in the “Status” ﬁeld. You can deconﬁgure only an adapter with a status of Configured.

Step 5. Highlight the adapter you want to deconﬁgure.

Step 6. Pull down the “Actions” menu and select Deconfigure Adapter.

Step 7. In the pop-up window, if you want to deconﬁgure the adapter, select OK to conﬁrm it.

If you do not want to deconﬁgure the adapter, select Cancel.

Step 8. If you selected OK, the entry for the adapter is deleted from the HyperFabric

conﬁguration ﬁles (/etc/rc.config.d/clic_global_conf and /etc/rc.config.d/netconf).

If you selected Cancel, you remain in the main “HyperFabric Conﬁguration” screen.

Step 9. Exit SMH.

NOTE If you have conﬁgured HMP for Transparent Local Failover support and if you select

Deconﬁgure Adapter, HyperFabric will verify if the selected adapter is conﬁgured to be part of any card pair. If yes, the user is informed and the card pair entry is removed from the /etc/rc.config.d/netconf and /etc/rc.config.d/clic_global_conf ﬁles.

Chapter 4

Conﬁguring HyperFabric

Conﬁguring the HyperFabric EMS Monitor

The HyperFabric Event Monitoring Service (EMS) monitor allows system administrators to separately monitor each HyperFabric adapter on every node in the fabric, in addition to monitoring the entire HyperFabric subsystem.

The monitor can inform the user if the resource being monitored is UP or DOWN. The administrator deﬁnes the condition to trigger a notiﬁcation (usually a change in interface status). Notiﬁcation can be accomplished with a SNMP trap or by logging into the syslog ﬁle with a choice of severity, or by email to a user deﬁned email address.

To conﬁgure the HyperFabric EMS monitor, it is necessary to have the EMS HA monitor product installed (Product Number B7609BA). This product is available on the applications CD media.

Use SMH to initiate monitoring of any particular HyperFabric resource. following the procedure outlined below:

1. Start SMH (Use the online help at any time for details)

2. Select “Resource Management”

3. Select “Event Monitoring Service”

4. Select “Action” and “Add Monitoring Request”

5. Select the location /net/interfaces/clic (class for HyperFabric resources)

6. Select a resource instance (either all instances or a speciﬁc instance from the list)

7. Validate your choice by clicking on OK at the bottom of the screen

8. A Monitoring Request Parameters window opens, showing the resource and its status (if All instances have been selected, then no value is displayed)

9. Deﬁne a condition that will trigger a notiﬁcation (for instance, “When Value is”, “equal to”, “UP”)

10. Deﬁne a polling interval (default is 300 seconds)

11. Deﬁne a way of notiﬁcation: SNMP trap, log in syslog with a choice of severity, or email to a user deﬁned email address

12. Validate by pressing OK

NOTE Although EMS is able to monitor each HyperFabric adapter on every node in the

fabric, as well as the entire HyperFabric subsystem, EMS is not able to monitor HyperFabric switches.

Chapter 4

For more detailed information on EMS, including instructions for implementing this feature, see the EMS Hardware Monitors Users Guide Part Number B6191-90028 September 2001 Edition.

Conﬁguring HyperFabric

Conﬁguring HyperFabric with ServiceGuard

HyperFabric supports the ServiceGuard HA product.

NOTE If you plan to conﬁgure HyperFabric with ServiceGuard, please read this section.

Otherwise, skip this section and go on to the next chapter, Chapter 5, “Managing HyperFabric,” on page 93.

ServiceGuard lets you create HA clusters of HP 9000 server systems. Within the cluster, ServiceGuard allows you to group your application services (individual HP-UX processes) into packages. In the event of a single service, node, network, or other resource failure, ServiceGuard can transfer control of the package to another node in the cluster, allowing services to remain available with minimal interruption.

CAUTION When applications use HMP to communicate between HP 9000 nodes in a HyperFabric

cluster, the EMS monitor in conjunction with ServiceGuard can be conﬁgured to identify node failure and automatically fail-over to a functioning HP 9000 node. Although failure of an adapter card or a link will be detected, there will not be automatic fail-over if an adapter card or a link fails. See “Features” on page 23 for details on features available when HMP applications are run over HyperFabric.

ServiceGuard directly monitors cluster nodes, LAN interfaces, and services, which are the individual processes within an application. In addition, specialized monitors might be supplied by the developers of other components. The HyperFabric monitor is supplied with the HyperFabric product and is installed with it. To use the HyperFabric monitor with ServiceGuard, you conﬁgure the monitor as an ServiceGuard package dependency.

Although HyperFabric can be used by an application within a package to communicate with other nodes, it is not possible to use HyperFabric as a heartbeat LAN. So, in a package control script, do not specify HyperFabric IPs/subnets in the lines that contain the keywords IP[n] and SUBNET[n]. Also, cmquerycl will not “discover” and report HyperFabric IPs and subnets.

After you have conﬁgured HyperFabric as a package dependency, ServiceGuard’s package manager calls the Event Monitoring Service (EMS) to launch an external monitor for HyperFabric. The package will not start unless the monitor reports that HyperFabric is available, and the package will fail when HyperFabric’s status is DOWN (that is, when all HyperFabric adapters on a node become non-functional).

Complete instructions for conﬁguring ServiceGuard clusters and packages are provided in Managing MC/ServiceGuard.

Figure 4-2 below shows a HyperFabric switch conﬁguration with ServiceGuard. This example shows a four-node conﬁguration with two HyperFabric switches, and redundant heartbeat Ethernet LANs.

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HyperFabric with ServiceGuard

NOTE Because the HyperFabric network does not currently support ServiceGuard heartbeat

connections, you must use an alternative type of connection for the heartbeat, such as FDDI, Token Ring, 100BaseT, or Ethernet (as shown in Figure 4-2 below).

Figure 4-2 An ServiceGuard Conﬁguration (with Two HyperFabric Switches)

Ethernet Heartbeat LAN 1

Ethernet Heartbeat LAN 0

node A

adapter 1

adapter 0

Ethernet Port

node C

adapter 0

adapter 1

switch 0

HF adapter 0

adapter 1

switch 1

HF adapter 0

node B

Ethernet Port

node D

adapter 1

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HyperFabric with ServiceGuard

How HyperFabric Handles Adapter Failures

HyperFabric adapters are handled differently than other types of networking adapters (such as Ethernet, FDDI, and Fibre Channel) in the ServiceGuard environment. In the non-HyperFabric cases, two network links are in a node, and one will be active and one will be idle or in standby. In the case of an active link failure, ServiceGuard is notiﬁed and the network trafﬁc is switched to the standby adapter (which then becomes active).

However, in the case of HyperFabric, if two adapters are in a node, both will be active. If one active HyperFabric adapter fails, its network trafﬁc is switched to the other active HyperFabric adapter in the node. (Throughput might be slower because only one active adapter is now handling the network trafﬁc.) This rearrangement is handled by the HyperFabric software, and ServiceGuard is not notiﬁed. However, note that if all of the HyperFabric adapters fail, HyperFabric does notify ServiceGuard. In both cases, though, the events are logged to /var/adm/clic_log and /var/adm/syslog.log.

Example 1:

This example, illustrated by Figure 4-3 below, presents an HA conﬁguration using ServiceGuard with HyperFabric. Both of the HyperFabric adapters are active on node A. The HyperFabric Resource Monitor reports the active status of the HyperFabric resource to the Event Monitoring Service (EMS), which lets ServiceGuard know that the HyperFabric resource is available to Packages A and B.

Figure 4-3 Node with Two Active HyperFabric Adapters

node A

HyperFabric Resource Active

Package

Active Adapter

adapter 1

adapter 0

Adapter IP address:

172.16.10.11

Adapter IP address:

172.16.20.21

Example 2:

This example, illustrated by Figure 4-4 below, shows the same node after the failure of one of the HyperFabric adapters. The remaining adapter in node A is now handling all HyperFabric network trafﬁc for the node. Because the HyperFabric resource is still

Chapter 4

Conﬁguring HyperFabric with ServiceGuard

available, ServiceGuard has not been notiﬁed; HyperFabric handles the local HyperFabric adapter failover. However, the failure of adapter 1 has been logged to /var/adm/clic_log.

Figure 4-4 Node with One Failed HyperFabric Adapter

node A

HyperFabric Resource Active

Package

Failed Adapter

adapter 1

Conﬁguring HyperFabric

Package

Active Adapter

adapter 0

Adapter IP addresses:

172.16.10.11

172.16.20.21

After the failover, if you issue a netstat -in command, you will see that an IP address is still assigned to each adapter. For example:

Name MTU network Address Ipkts Opkts clic1 31744 172.16.10.0 172.16.10.11 711 12 clic0 31744 172.16.20.0 172.16.20.21 1222 333

Example 3:

This ﬁnal example, illustrated by Figure 4-5 below, shows a situation in which all of the HyperFabric adapters on node A fail. The HyperFabric Resource Monitor reports to the Event Monitoring Service (EMS). The EMS then notiﬁes the ServiceGuard cmcld daemon that the HyperFabric resource on node A is unavailable. Because HyperFabric is conﬁgured as a package dependency for Packages A and B, ServiceGuard causes the packages to failover to node B. In a four-node conﬁguration (note that only two nodes are shown in Figure 4-5 below), Packages A and B can continue to communicate through the HyperFabric network with the other active nodes in the ServiceGuard cluster.

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HyperFabric with ServiceGuard

Figure 4-5 When All HyperFabric Adapters Fail

Packages failover to Node B

node A

HyperFabric

Resource Failed

adapter 1

adapter 0

switch 0

Ethernet Port

switch 1

Ethernet Port

Ethernet Heartbeat LAN 0

HyperFabric Resource Active

adapter 0

HF adapter 1

node B

Package

Ethernet Heartbeat LAN 1

Conﬁguring HyperFabric with the ServiceGuard Resource Monitor

You can conﬁgure the HyperFabric Resource Monitor with ServiceGuard in either of these ways:

• Editing an ASCII ﬁle.

• Using the SMH GUI. For more details, please see the manual Using EMS HA Monitors.

NOTE You should conﬁgure HyperFabric with ServiceGuard before running the clic_start

command or using SMH to start HyperFabric.

Conﬁguring ServiceGuard with HyperFabric Using the ASCII File

When using the ServiceGuard commands (for example, cmapplyconf) to specify the use of the HyperFabric Resource Monitor, the section of the package ASCII conﬁguration ﬁle that has the keyword RESOURCE_NAME must be uncommented and set to the following values:

RESOURCE_NAME /net/interfaces/clic/status

RESOURCE_POLLING_INTERVAL 10

Chapter 4

RESOURCE_UP_VALUE =UP

Conﬁguring ServiceGuard with HyperFabric Using SMH

You must perform the following steps when using SMH to conﬁgure the HyperFabric Resource Monitor with ServiceGuard:

smh

Clusters

High Availability Clusters

Cluster Configuration (go through all the steps to create a

cluster)

Package Configuration

Create/Add Package (if creating new packages)

Specify Package Name and Nodes Specify Package SUBNET Address Specify Package Services Specify Package Failover Options Specify Package Control Script Location Specify Package Control Script Information

Conﬁguring HyperFabric

Conﬁguring HyperFabric with ServiceGuard

Specify Package Resources Dependencies

Add

Resource Name (Navigate the Resource Subclass by double-clicking on /net until /net/interfaces/clic/ status shows up in the selection box Resource Name,then select it and click OK.)

Resource Parameters

- Input the Resource Polling Interval (for example, 10 seconds).

- Select “UP” from the “Available Resource Values” and click “Add”.

- Click OK to accept the values.

Conﬁguring ServiceGuard for HyperFabric Relocatable IP Addresses

If you want to use relocatable IP addresses, conﬁgure the relocatable IP addresses with the IP[n] command in the package control script.

For example, to conﬁgure the relocatable address 192.0.0.3 for adapter 0 and 192.0.8.5 for adapter 1, specify this:

IP[0]= 192.0.0.3

IP[1]= 192.0.8.5

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

HMP supports Local Failover on HP-UX 11i v3. If a HyperFabric resource (adapter, cable, switch or switch port) fails in a cluster, HMP

transparently fails over trafﬁc (Local Failover) using another available resource from the card pair. A card pair can be deﬁned as a logical entity comprising of a pair of HF2 adapters on a HP 9000 node. For example, if there are four HF2 adapters installed and conﬁgured in a node, then there would be two card pairs.

IMPORTANT Remember the following points while conﬁguring HMP for Local Failover support:

• Only Oracle applications can make use of the local failover feature. Other middleware can continue using HMP without local failover support.

• HMP supports the local failover conﬁguration from HyperFabric version B.11.11.03.

• All nodes in the cluster need to be conﬁgured either in the local failover mode or the non-local failover mode. While using clic_init, if you answer ‘y’ to the question, “Do you want to conﬁgure Local Failover on this node?”, then you have conﬁgured HMP for the local failover mode. Otherwise, your HMP conﬁguration is for the non-local failover mode. Do not mix these two modes. For any incorrect conﬁgurations, HP recommends that you clic_shutdown and clic_start the cluster.

• The Transparent Local Failover feature over HMP is supported only in a switch-based environment. If two nodes are connected through multiple point-to-point links, local failover cannot be achieved.

• There must be even number of HF2 adapters on any given node, and all adapters installed must be conﬁgured. In addition, all the adapters must belong to a card pair.

• HMP can fail over trafﬁc only between adapters that belong to the same card pair.

• HMP does not support backward compatibility in the local failover and non-local failover mode. However, TCP/UDP/IP supports backward compatibility and interoperability.

• When HMP is conﬁgured in the local failover mode, all the resources in the cluster are utilized. If a resource fails in the cluster and is restored, HMP does not utilize that resource until another resource fails.

• Before running clic_start on all the nodes in the cluster, ensure that all the conﬁgured cards are connected in the cluster. In other words, before running the clic_start command, verify that all the cables are connected to the adapters and switches.

• If a resource in the cluster fails and is restored, perform the following steps to ensure full utilization of that resource:

— Shutdown Oracle RAC — Execute the clic_shutdown command — Execute the clic_start command

— Restart Oracle RAC

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

• After executing the clic_start command on all node in the cluster, ensure that you run Oracle RAC only after one minute (approximately).

• If all the trunks between the switches are down, then execute clic_shutdown followed by clic_start on all the nodes in the cluster, after replacing at least one trunk between the switches.

• Maintain at least one trunk between the switches when Oracle RAC is running in the cluster.

• If any of the nodes is shut down for administrative reasons, shut down Oracle RAC before executing clic_shutdown on that node. In such a case, Oracle RAC continues to be operational on all the other nodes in the cluster. When you bring up the node that was shut down, it joins the cluster.

How Transparent Local Failover Works

Consider a hypothetical HyperFabric conﬁguration in a 4-node cluster, with each node having two adapters (see Figure 4-6). In this conﬁguration, there is no single point of failure, and all adapters that are installed on any given node are conﬁgured as part of a card pair.

Figure 4-6 A Conﬁguration supporting Local Failover

node A

adapter 1

{

Card Pair 0

adapter 0

HF Switch 0

node C

adapter 0

adapter 1

HF Switch 1

adapter 1

node B

node D

Card Pair 0 Card Pair 0

{

Chapter 4

{

Card Pair 0

adapter 1

adapter 0

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

The details of how HyperFabric handles failures of the adapters, links, switch ports, switches, and cables between switches in a cluster are discussed in the following sections.

Chapter 4

Conﬁguring HMP for Transparent Local Failover Support

Case 1: Adapter, Link or Switch Port Failure (see Figure 4-7) If an adapter or a link or a switch port fails, HMP transparently fails over trafﬁc through

the other available link.

Figure 4-7 Adapter, Link or Switch Port Failover

Conﬁguring HyperFabric

node A

adapter 1

{

Card Pair 0

adapter 0

HF Switch 0

node C

adapter 0

{

Card Pair 0

adapter 1

adapter 0

adapter 1

HF Switch 1

adapter 1

adapter 0

node B

node D

Card Pair 0 Card Pair 0

{

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

Case 2: Switch Failure (see Figure 4-8) Consider the following illustration where node A is connected to node D with trafﬁc

being routed through the HF adapter 1 on both the nodes (A and D), and the HF switch 1 fails. HMP transparently fails over trafﬁc through the other available switch (HF switch

0). This is possible only if at least one adapter of the card pairs on both the nodes (HF

adapter 0 of the Card Pair 0 on nodes A and D) is physically reachable (through the HF switch 0).

Figure 4-8 Switch Failover

node A

adapter 1

{

Card Pair 0

adapter 0

HF Switch 0

node C

adapter 0

{

Card Pair 0

adapter 1

adapter 0

adapter 1

HF Switch 1

adapter 1

adapter 0

node B

node D

Card Pair 0 Card Pair 0

{

Thus, if a switch fails, HMP transparently fails over trafﬁc only if at least one member of the card pair is physically reachable through the other switch.

Case 3: Cable Failure Between Two Switches (see Figure 4-9)

Chapter 4

Conﬁguring HMP for Transparent Local Failover Support

If a cable between two switches fails, HMP trafﬁc fails over to the other available cable between those two switches.

Figure 4-9 Cable Failover Between Two Switches

Conﬁguring HyperFabric

node A

adapter 1

{

Card Pair 0

adapter 0

HF Switch 0

node C

adapter 0

{

Card Pair 0

adapter 1

adapter 0

adapter 1

HF Switch 1

adapter 1

adapter 0

node B

node D

Card Pair 0 Card Pair 0

{

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

Conﬁguring HMP for Transparent Local Failover Support - Using SMH

To use SMH to conﬁgure HMP for Local Failover Support, complete the following steps:

Step 1. Start SMH.

Step 2. Select the “Networking and Communications” area.

Step 3. Select the “Network Interfaces” area.

Step 4. Select “HyperFabric.”

All HyperFabric adapters installed in the system are listed; installed adapters that are not yet conﬁgured show Not Configured in the “Status” ﬁeld. Before proceeding to the next step, conﬁgure all the available adapters.

Step 5. Pull down the "Actions" menu and select "Conﬁgure Local Failover for HMP". SMH now

veriﬁes the following:

• If not all the adapters are conﬁgured

• If the number of adapters conﬁgured are odd

• If “Interoperability Enabled” is set to YES

If any of the above conditions is true, then SMH displays an appropriate error message. Otherwise, the “Conﬁgure Local Failover for HMP” window pops up.

Step 6. In this window, specify the conﬁgured adapters for each card pair and press OK.

On pressing OK, SMH veriﬁes the following:

• If HyperFabric subsystem is not running on the machine

• If all card pairs are not conﬁgured

• If you have chosen the same adapter for different card pairs

If any of the above conditions is true, SMH displays an appropriate error message. Otherwise, the /etc/rc.config.d/clic_global_conf ﬁle is updated with information about the conﬁgured card pairs. If Card-Pair 0 comprises of adapters clic0 and clic1 and if the Card-Pair 1, comprises of adapters clic2 and clic3, then the following entries are added to the clic_global_conf ﬁle.

CARD_PAIR[0] = clic0-clic1 CARD_PAIR[1] = clic2-clic3

If you press Cancel, you remain in the main “HyperFabric Conﬁguration” screen.

If you press Help, help text for this task appears.

Step 7. Exit SMH.

NOTE To view the card pair information from the /etc/rc.config.d/clic_global_conf file, select “View

Local Failover for HMP” option in the “HyperFabric Conﬁguration” screen.

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

Deconﬁguring HMP for Local Failover support - Using SMH

To use SMH to deconﬁgure HMP for Local Failover support, complete the following steps:

Step 1. Start SMH.

Step 2. Select the “Networking and Communications” area.

Step 3. Select the “Network Interfaces” area.

Step 4. Select “HyperFabric”.

All HyperFabric adapter card pairs installed in the system are listed.

Step 5. Pull down the “Actions” menu and select Deconfigure Local Failover for HMP to remove all

the card pair entries from the /etc/rc.config.d/clic_global_conf ﬁle. In the next conﬁrmation window that pops-up, select YES to conﬁrm it.

If you select NO. you remain in the main “HyperFabric Conﬁguration” screen.

Step 6. Exit SMH.

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

Conﬁguring HMP for Transparent Local Failover Support - Using the clic_init command

You can conﬁgure the Transparent Local Failover feature of HMP using clic_init also. Let us consider the following example where we have discussed the conﬁguration in

detail. This example uses some “dummy” (that is, not valid) addresses to the components in Figure 4-10. The dummy addresses are used only to show the ﬂow of the information provided as input to the clic_init command and SMH. Do not try to use these addresses in your conﬁguration.

Figure 4-10 Conﬁguring the Transparent Local Failover feature

Ethernet LAN

Switch ID: sw_clic0 IP address: 193.0.0.20

Ethernet MAC address:

00:60:b0:d0:02:57

IP multicast address:

226.10.1.1

HF2

adapter 0

Adapter ID:

clic0

IP address:

192.0.0.1

subnet mask:

255.255.255.0

IP address: 193.0.0.10 IP address: 193.0.0.11

node A

switch 0

HF2

adapter 1

Adapter ID:

clic1

IP address:

192.0.8.3

subnet mask:

255.255.255.0

lan0

switch 1

HF2

adapter 0

Adapter ID:

clic0

IP address:

192.0.0.2

subnet mask:

255.255.255.0

lan0

Switch ID: sw_clic1

IP address: 193.0.0.21 Ethernet MAC address:

00:60:b0:d0:02:56

IP multicast address:

226.10.1.1

HF2

adapter 1

Adapter ID:

clic1

IP address:

192.0.8.4

subnet mask:

255.255.255.0

node A

node B

Using the conﬁguration information in Figure 4-10, the information you would specify when you run clic_init on each of the nodes is listed below. This example is not an exact depiction of the prompts produced by clic_init, but merely an example of the ﬂow of information input. In addition, you should not try to use the dummy addresses in your actual conﬁguration.

On node A:

1. How many HyperFabric adapters are installed on the node?

2. Do you want this node to interoperate with nodes running any HyperFabric versions earlier than B.11.00.11 or B.11.11.01? (n)

Chapter 4

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

You must answer ‘no’ if you want to run applications using HMP (Local Failover or Non-Local Failover) for communication over HyperFabric. In that case, all nodes in the cluster must be running version B.11.00.11 (or) B.11.11.01 (or) later version of HyperFabric software.

3. What is the IP address of the ﬁrst adapter (clic0)? (192.0.0.1)

4. What is the subnet mask of the ﬁrst adapter? (255.255.255.0) If you do not specify a value for this, a default mask is chosen. You will most likely

just accept the default. However, in this example, we are showing a value for the subnet mask just to illustrate the correlation between the “dummy” information in Figure 4-10 and where that information is speciﬁed or generated during clic_init and SMH.

5. What is the IP address of the second adapter (clic1)? (192.0.8.3)

6. What is the subnet mask of the second adapter? (255.255.255.0)

7. Do you want to conﬁgure Local Failover on this node? (y)

Enter ‘y’ if you are using Oracle RAC with HMP.

8. Select any two of the following clic adapters for CARD_PAIR[0]: clic0 clic1

Enter the ﬁrst clic adapter from above listed adapters: (clic0) Enter the second clic adapter from above listed adapters: (clic1)

9. Do you want to enable switch management? (n)

On node B:

1. How many HyperFabric adapters are installed on the node?

2. Do you want this node to interoperate with nodes running any HyperFabric versions earlier than B.11.00.11 or B.11.11.01? (n)

3. What is the IP address of the ﬁrst adapter (clic0)? (192.0.0.2)

4. What is the subnet mask of the ﬁrst adapter? (255.255.255.0) If you do not specify a value for this, a default mask is chosen. You will most likely

just accept the default. However, in this example, we are showing a value for the subnet mask just to illustrate the correlation between the dummy information in Figure 4-10 and where that information is speciﬁed or generated during clic_init and SMH.

5. What is the IP address of the second adapter (clic1)? (192.0.8.4)

Chapter 4

6. What is the subnet mask of the second adapter? (255.255.255.0)

7. Do you want to conﬁgure Local Failover on this node? (y)

Enter ‘y’ if you are using Oracle RAC with HMP.

Conﬁguring HyperFabric

Conﬁguring HMP for Transparent Local Failover Support

8. Select any two of the following clic adapters for CARD_PAIR[0]: clic0 clic1

Enter the ﬁrst clic adapter from above listed adapters: (clic0) Enter the second clic adapter from above listed adapters: (clic1)

9. Do you want to enable switch management? (n)

Chapter 4

5 Managing HyperFabric

This chapter contains the following sections that give information about managing HyperFabric:

• “Starting HyperFabric” on page 95

• “Verifying Communications within the Fabric” on page 97

Chapter 5

Managing HyperFabric

• “Displaying Status and Statistics” on page 101

• “Viewing man Pages” on page 109

• “Stopping HyperFabric” on page 110

Chapter 5

Managing HyperFabric

Starting HyperFabric

HyperFabric is started in one of these three ways:

• As part of the normal local node boot process (HP 9000 system).

• By running the HyperFabric clic_start command (described below).

• By starting HyperFabric through SMH (described in “Using SMH” on page 96). HyperFabric needs to be started in the following situations:

• If HyperFabric hardware and software have just been installed on the system and the clic_init command has been used to conﬁgure the HyperFabric adapters on this node.

• If the HyperFabric conﬁguration has been changed by using the clic_init command or using SMH (HP-UX 11i v3 only). In this situation, you must have run clic_shutdown or used SMH to stop HyperFabric (HP-UX 11i v3 only), before restarting HyperFabric.

• If a new HyperFabric adapter has been added to a system online and conﬁgured using clic_init. In this situation, it is not necessary to run clic_shutdown before running clic_start (see “Online Addition and Replacement” on page 42in Ch. 3 of this user guide).

NOTE Starting HyperFabric launches the HyperFabric CLuster InterConnect (CLIC) daemon

(clic_mgmtd). This daemon process must be running for the HyperFabric product to operate correctly. It is possible that other daemons will be running, but it is essential that at least one CLIC daemon is running. To check if a CLIC daemon is running, use the following command:

ps -ef | grep clic

If the CLIC daemon is not running, start the HyperFabric subsystem by executing the following command:

/opt/clic/bin/clic_start

Using the clic_start Command

Run the clic_start command on each node to start the HyperFabric management process on that node.

You must be logged in as root to run this command. The syntax is as follows:

Chapter 5

clic_start

Managing HyperFabric

Starting HyperFabric

Step 1. Start SMH.

Step 2. Select the “Networking and Communications” area.

The clic_start -? command can be issued to display the online help for clic_start, or look at the clic_start (1m) man page by issuing the man clic_start command.

If HyperFabric is already running, you will receive an informational (FYI) message telling you so. Your reaction to this message depends on the situation:

• If you have simply forgotten (or did not know) that HyperFabric was already running, you do not have to do anything.

• If you have changed the HyperFabric conﬁguration with the clic_init command or SMH, you must stop HyperFabric (by running the clic_shutdown command or using SMH) and then start HyperFabric (by running the clic_start command or using SMH). See either “Using the clic_shutdown Command” on page 110 or “Using SMH” on page 111, whichever is appropriate.

Using SMH

To use SMH to start HyperFabric on an HP 9000 system running HP-UX 11i v3, follow these steps:

Step 3. Select “HyperFabric.”

Step 4. Pull down the “Actions” menu and select Start HyperFabric.

When HyperFabric starts, a conﬁrmation message displays. Also, the status “HyperFabric: Running” is displayed above the adapter conﬁguration area of the screen.

Step 5. Exit SMH.

Chapter 5

Managing HyperFabric

Verifying Communications within the Fabric

You can verify the communications within the fabric by running the clic_probe command, which is described below. You can also use clic_probe to verify the status of speciﬁc adapters.

IMPORTANT You should also check your /etc/hosts ﬁle—when you are using ﬁles for host name look

up—to ensure that the entries for all of the systems are in the correct format: the ofﬁcial host name, which is the full domain extended host name, and any alias names. For example:

IP_address IP_address IP_address

bently6.corp3.com bently6 bently4.corp7.com test1 bently2.corp4.com test3

The clic_probe Command

Run the clic_probe command to send 256-byte packets to verify the link out to and back from a speciﬁc destination, optionally using a speciﬁc adapter for the veriﬁcation. The destination can be either a node or a switch (if a switch is part of the fabric).

You do not have to be logged in as root to run this command. The syntax is as follows:

clic_probe [-c [-l -c [-p

Note that some of the lines in the above syntax are indented for readability purposes only. When you actually type the command, you do not indent anything.

node_name

adapter_ID-rVRID switch_hopcount

packet_count

[-c

adapter_ID

] [-s -c

] [-?]

]

adapter_ID

]

Chapter 5

The command parameters are as follows:

•

node_name

speciﬁes the node you want to verify. This value is conditionally required—you must specify it when you are verifying trafﬁc to a remote node, unless you use the -r parameter (described below).

• -c speciﬁes that you want to use the adapter identiﬁed by

adapter_ID

for the

veriﬁcation.

• -r speciﬁes that To determine the

VRID switch_hopcount

VRID

and

switch_hopcount

is the routing information for the adapter.

to specify, ﬁrst run the clic_stat -d VRID command (see “The clic_stat Command” on page 101). Note that if you specify this parameter (-r the -c

adapter_ID

parameter (described above).

VRID switch_hopcount

), you must also specify

Managing HyperFabric

Verifying Communications within the Fabric

• -l speciﬁes that you want to do local loopback testing on a particular adapter. Note that if you specify this parameter (-l), you must also specify the -c parameter (described above).

• -s speciﬁes that you want to loopback at the switch port attached to a particular adapter. Note that if you specify this parameter (-s), you must also specify the

-c

adapter_ID

parameter (described above).

• -p speciﬁes that you want to send

packet_count

can be any positive integer. This parameter is useful for building scripts for debugging or for hardware veriﬁcation. If you do not specify this parameter, one packet is sent each second, until you stop the command with a

CTRL-C.

• -? displays the online help for clic_probe. If you do not specify any of the above parameters, the online help for clic_probe is

displayed.

NOTE Also see the clic_diag command to:

Probe a speciﬁc remote node. Dump and format trace data. Set the tracing level for the HyperFabric software and ﬁrmware.

The clic_diag command is detailed in the Running Diagnostics section of Chapter 6,

Troubleshooting.

Examples of clic_probe

Some examples of using clic_probe are shown below.

packet_count

number of 256-byte packets.

• Example 1 If the local node is bently6 and you want to send ﬁve packets to verify that the

adapter clic0 (which is on bently6) is able to handle trafﬁc, issue this command:

clic_probe -l -c clic0 -p 5

The generated output could look like this:

CLIC_PROBE: 256 byte packets Local Loopback: Source and Target Adapter ID: bently6.corp3.com:clic0

--------- bently6.corp3.com CLIC_PROBE Statistics -------5 packets transmitted, 5 packets received, 0% packet loss.

• Example 2 If the local node is bently6, and you want to verify communications with the remote

node bently4, issue this command:

Chapter 5

Managing HyperFabric

Verifying Communications within the Fabric

clic_probe bently4

CLIC_PROBE: 256 byte packets Source adapter id: bently6.corp3.com:clic0 Target adapter id: bently4.corp7.com:clic3

256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. 256 bytes: seq_num = 5. Packet Acknowledged. 256 bytes: seq_num = 6. Packet Acknowledged. 256 bytes: seq_num = 7. Packet Acknowledged. 256 bytes: seq_num = 8. Packet Acknowledged.

--------- bently6.corp3.com CLIC_PROBE Statistics -------8 packets transmitted, 8 packets received, 0% packet loss.

• Example 3 If the local node is bently6, and you want to send ﬁve packets to verify

communications with the remote node bently7, using the adapter clic0 (which is on bently6), issue this command:

clic_probe bently7 -c clic0 -p 5

CLIC_PROBE: 256 byte packets Source adapter id: bently6.corp3.com:clic0 Target adapter id: bently7.corp4.com:clic1

--------- bently7.corp4.com CLIC_PROBE Statistics -------5 packets transmitted, 5 packets received, 0% packet loss.

• Example 4 If the local node is bently6, and you want to send ﬁve packets to verify

communications with the remote node bently7, using the adapter clic0 (which is on bently6) and the route identiﬁed by VRID 194 and switch hopcount 1, issue this command:

clic_probe -c clic0 -r 194 1 -p 5

(Remember, because you speciﬁed the -r not need to also specify the

node_name

VRID switch_hopcount

parameter, you do

The generated output could look like this:

Chapter 5

CLIC_PROBE: 256 byte packets sent Source adapter id: bently6.corp3.com:clic0 Target adapter id: bently7.corp4.com:clic1

Managing HyperFabric

Verifying Communications within the Fabric

--------- bently7.corp4.com CLIC_PROBE Statistics -------5 packets transmitted, 5 packets received, 0% packet loss.

Note that the VRID you speciﬁed (194) actually went to the adapter clic1 on bently7. And, as explained earlier, you run the clic_stat -d VRID command to

determine the VRID and switch hopcount to specify.

100

Chapter 5

HP HyperFabric User's Guide

Specifications and Main Features

Frequently Asked Questions

User Manual

1 Overview

Overview

Notice of non support for Oracle 10g RAC

HyperFabric Products

HyperFabric Adapters

Switches and Switch Modules

Other Product Elements

HyperFabric Concepts

2 Planning the Fabric

Preliminary Considerations

HyperFabric Functionality for TCP/UDP/IP and HMP Applications

TCP / UDP / IP

Application Availability

Features

Switched

High Availability Switched

Hyper Messaging Protocol (HMP)

Application Availability

Features

Point to Point

Enterprise (Database)

Technical Computing

3 Installing HyperFabric

Checking HyperFabric Installation Prerequisites

Installing HyperFabric Adapters

Online Addition and Replacement

Planning and Preparation

Critical Resources

Card Compatibility

Installing the Software

File Structure

Loading the Software

Installing HyperFabric Switches

Before Installation

Installing the HF2 Switch

With the Rail Kit

Without the Rail Kit

Information You Need

Using the clic_init Command

Examples of clic_init

Using SMH

How HyperFabric Handles Adapter Failures

How Transparent Local Failover Works

5 Managing HyperFabric

Starting HyperFabric

Using the clic_start Command

Using SMH

Verifying Communications within the Fabric

The clic_probe Command

Examples of clic_probe