The information in this document is subject to change without notice.
Hewlett-Packard makes no warranty of any kind with regard to this
manual, including, but not limited to, the implied warranties of
merchantability and fitness for a particular purpose. Hewlett-Packard
shall not be held liable for errors contained herein or direct, indirect,
special, incidental or consequential damages in connection with the
furnishing, performance, or use of this material.
Warranty
A copy of the specific warranty terms applicable to your Hewlett-Packard
product and replacement parts can be obtained from your local Sales and
Service Office.
U.S. Government License
Proprietary computer software. Valid license from HP required for
possession, use or copying. Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation,
and Technical Data for Commercial Items are licensed to the U.S.
Government under vendor's standard commercial license.
Copyright Notice
Copyright 2004 Hewlett-Packard Development Company L.P. All
rights reserved. Reproduction, adaptation, or translation of this
document without prior written permission is prohibited, except as
allowed under the copyright laws.
Trademark Notices
Oracle is a registered trademark of Oracle Corporation.
UNIX is a registered trademark in the United States and other
countries, licensed exclusively through The Open Group.
This document describes how to install, configure, and troubleshoot the
HyperFabric product on the HP-UX 11i v2 (HP-UX 11.23) operating
system.
Before you install the HyperFabric software, ensure that the operating
system software and the appropriate files, scripts, subsets are installed.
The document printing date and part number indicate the document’s
current edition. The printing date will change when a new edition is
printed. Minor changes may be made at reprint without changing the
printing date. The document part number will change when extensive
changes are made.
Document updates may be issued between editions to correct errors or
document product changes. To ensure that you receive the updated or
new editions, you should subscribe to the appropriate product support
service. See your HP sales representative for details.
The latest version of this manual can be found online at
This manual is intended for system and network administrators
responsible for installing, configuring, and managing the HyperFabric
software and hardware. Administrators are expected to have knowledge
of operating system concepts, commands, and configuration.
It is helpful to have knowledge of Transmission Control Protocol/Internet
Protocol (TCP/IP) networking concepts and network configuration.
This document is not a tutorial.
xi
HP-UX Release Name and Release Identifier
Each HP-UX 11i release has an associated release name and release
identifier. The uname (1) command with the -r option returns the
release identifier. Table 1 shows the releases available for HP-UX 11i.
Table 1HP-UX 11i Releases
Supported
Release IdentifierRelease Name
B.11.11HP-UX 11i v1PA-RISC
B.11.20HP-UX 11i v1.5Intel Itanium
B.11.22HP-UX 11i v1.6Intel Itanium
B.11.23HP-UX 11i v2Intel Itanium
Processor
Architecture
Publishing History
Table 2 provides the publication date for the pertinent edition number.
Table 2Publishing History Details
Edition NumberPublication Date
xii
FirstMarch 1998
SecondJune 1998
ThirdAugust 1998
FourthOctober 1998
FifthDecember 1998
SixthFebruary 1999
SeventhApril 1999
EighthMarch 2000
NinthJune 2000
TenthDecember 2000
Table 2Publishing History Details (Continued)
Edition NumberPublication Date
EleventhJune 2001
TwelfthSeptember 2002
ThirteenthJuly 2003
FourteenthMarch 2004
What Is in This Document
HyperFabric Administrator’s Guide is divided into several chapters, each
of which contains information about installing, configuring, or
troubleshooting HyperFabric. The appendixes contain supplemental
information.
The following list describes the content in more detail.
Table 3Organization
ChapterDescription
Overview of the
HyperFabric Product
Planning the FabricDescribes the steps that need to be
Installing HyperFabricDescribes the tasks to install the
Configuring HyperFabricDescribes the tasks to configure
Managing HyperFabricDescribes the tasks to start, stop and
Troubleshooting
HyperFabric
Presents an overview of HyperFabric
and lists the components that the
HyperFabric product contains
followed while planning the fabric
HyperFabric products on the HP-UX
11i v2 operating system
HyperFabric
manage HyperFabric
Describes how to troubleshoot
HyperFabric
xiii
New and Changed Information in This Edition
This edition includes information about the transparent local failover
feature of Hyper Messaging Protocol (HMP). This feature is available
with the HyperFabric version B.11.23.01.
Typographic Conventions
This document uses the following typographic conventions:
Book TitleItalic (slanted) type indicates document and book
names.
daemonCourier font type indicates daemons, files, commands,
•Using a feedback form located at the following URL:
http://docs.hp.com/assistance/feedback.html
Please include the following information along with your comments:
•The complete title of the manual and the part number. (The part
number appears on the title page of printed and PDF versions of a
manual.)
•The section numbers and page numbers of the information on which
you are commenting.
•The version of HP-UX that you are using.
Please note that the HP-UX networking communications publications
group does not provide technical support for HP products.
1Overview of the HyperFabric
Product
This chapter contains the following sections that give general
information about HyperFabric:
Chapter 11
Overview of the HyperFabric Product
•“About HyperFabric” on page 3
•“HyperFabric Products” on page 4
•“HyperFabric Concepts” on page 7
Chapter 12
Overview of the HyperFabric Product
About HyperFabric
About HyperFabric
HyperFabric is an HP high-speed, packet-based interconnect for
node-to-node communications. HyperFabric provides higher speed, lower
network latency and uses less CPU than other industry standard
protocols (for example, Fibre Channel and Gigabit Ethernet). Instead of
using a traditional bus-based technology, HyperFabric is built around
switched fabric architecture, providing the bandwidth necessary for high
speed data transfer. This clustering solution delivers the performance,
scalability, and high availability required by the following:
•Client/Server Architecture Interconnects (for example, SAP)
•Multi-Server Batch Applications (for example, SAS Systems)
•Enterprise Resource Planning (ERP)
•Technical Computing Clusters
•HP Message Passing Interface (MPI) based applications
•OpenView Data Protector (earlier known as Omniback)
•Network Backup
•Data Center Network Consolidation
•E-services
Chapter 13
Overview of the HyperFabric Product
HyperFabric Products
HyperFabric Products
HyperFabric hardware consists of host-based interface adapter cards,
interconnect cables, and optional switches. HyperFabric software resides
in Application Specific Integrated Circuits (ASICs) and firmware on the
adapter cards and includes user-space components and HP-UX drivers.
Currently, fiber-based HyperFabric hardware are available. In addition,
a hybrid switch that has 8-fiber ports is available to support HF2
clusters.
This section describes the various HyperFabric products. For more
information on HP 9000 systems that support HyperFabric products, see
the HyperFabric Release Notes, available at
NOTEThis document uses the term HyperFabric (HF) to refer to the
hardware and software that form the HyperFabric cluster interconnect
product.
The term HyperFabric2 (HF2) refers to the following fiber-based
hardware components:
•The A6386A adapter
•The A6384A switch chassis
•The A6388A and A6389A switch modules. (Although the A6389A
switch module has 4-copper ports, it is still considered an HF2
component because it can only be used with the A6384A HF2 switch
chassis).
•The C7524A, C7525A, C7526A, and C7527A cables
HyperFabric Adapters
The HyperFabric adapters are as follows:
•A6386A HF2 PCI (4X) adapter with a fiber interface.
The A6092A HyperFabric adapter is supported beginning with the
following HyperFabric software versions:
Chapter 14
Overview of the HyperFabric Product
HyperFabric Products
•HP-UX 11.0: HyperFabric software version B.11.00.09
•HP-UX 11i v1: HyperFabric software version B.11.11.00
•HP-UX 11i v2: HyperFabric software version B.11.23.00
The A6386A HyperFabric2 adapter is supported beginning with the
following HyperFabric software versions:
•HP-UX 11.0: HyperFabric software version B.11.00.11
•HP-UX 11i v1: HyperFabric software version B.11.11.01
•HP-UX 11i v2: HyperFabric software version B.11.23.00
Switches and Switch Modules
The HyperFabric2 switches are as follows:
•A6384A HF2 fiber switch chassis with one integrated Ethernet
management LAN adapter card, one integrated 8-port fiber card, and
one expansion slot. For the chassis to be a functional switch, install
one of the following switch modules in the expansion slot:
— The A6388A HF2 8-port fiber switch module. This gives 16-fiber
ports to the switch (8 from the integrated fiber card and 8 from
the A6388A switch module).
— The A6389A HF2 4-port copper switch module. This gives 12
ports to the switch - a mixture of 8-fiber ports (from the
integrated fiber card) and 4-copper ports (from the A6389A
module).
The A6384A HF2 switch chassis with either module installed is
supported beginning with the following HyperFabric software versions:
•HP-UX 11.0: HyperFabric software version B.11.00.11
•HP-UX 11i v1: HyperFabric software version B.11.11.01
•HP-UX 11i v2: HyperFabric software version B.11.23.00
NOTEIn this manual, the terms HyperFabric2 switch or HF2 switch refer
to the functional switch (the A6384A switch chassis with one of the
switch modules installed).
Chapter 15
Overview of the HyperFabric Product
HyperFabric Products
IMPORTANTHF2 adapters and switches are not supported by software versions
earlier than those listed in “HyperFabric Adapters” on page 4 and
“Switches and Switch Modules” on page 5.
To determine the version of HyperFabric, issue the following command:
$ swlist | grep -i hyperfabric
Other Product Elements
The following are the other elements of the HyperFabric product family:
•The HyperFabric software: The software resides in ASICs and
firmware on the adapter cards and includes user-space components
and HP-UX drivers.
HyperFabric supports the IP network protocol stack, specifically
TCP/IP and UDP/IP.
HyperFabric software includes HyperMessaging Protocol (HMP).
HMP provides higher bandwidth, lower CPU overhead, and lower
latency (the time a message takes to get from one point to another).
However, these HMP benefits are available only when applications
that are developed on top of HMP are running. HMP can only be
used on HP 9000 systems running HP-UX 11.0 or 11i v1, provided
HyperFabric A6092A or A6386A (PCI 4X) adapter cards are installed
on those systems.
Chapter 16
Overview of the HyperFabric Product
HyperFabric Concepts
HyperFabric Concepts
This section briefly describes some of the basic HyperFabric concepts and
terms.
The fabric is the physical configuration that consists of all the
HyperFabric adapters, the HyperFabric switches (if any), and the
HyperFabric cables connecting them. The network software controls data
transfer over the fabric.
The HyperFabric configuration contains two or more HP 9000 systems
and optional HyperFabric switches. Each HP 9000 acts as a node in the
configuration. Each node has a minimum of one and a maximum of eight
HyperFabric adapters installed in it. (For information on the maximum
number of adapters that can be installed in each system, see Chapter 2,
“Planning the Fabric,” on page 9.) Each HF2 switch can be configured
with 12 or 16 ports. HyperFabric supports a maximum of 4 HyperFabric
switches. You can mesh HyperFabric switches and configurations with
up to four levels of meshed switches are supported.
You can plan a HyperFabric cluster as a High Availability (HA)
configuration, when it is necessary to ensure that each node can always
participate in the fabric. This is done by using ServiceGuard (earlier
known as MC/ServiceGuard), ServiceGuard OPS Edition (earlier known
as MC/LockManager), and the Event Monitoring Service (EMS).
Configurations of up to 8 nodes are supported under ServiceGuard.
Beginning with HyperFabric software versions B.11.00.05 and
B.11.11.00,you can use relocatable IP addresses as part of an HA
configuration. Relocatable IP addresses permit a client application to
reroute through an adapter on a remote node, allowing that application
to continue processing without interruption. The rerouting is
transparent. This function is associated with ServiceGuard (see
“Configuring ServiceGuard for HyperFabric Relocatable IP Addresses”
on page 95). When the monitor for HyperFabric detects a failure and the
backup adapter takes over, the relocatable IP address is transparently
migrated to the backup adapter. Throughout this migration process, the
client application continues to execute normally.
When you start HyperFabric (with the clic_start command, through
SAM, or by booting the HP 9000 system), you start the managementprocess. This process must be active for HyperFabric to run. If the
HyperFabric management process on a node stops running for some
Chapter 17
Overview of the HyperFabric Product
HyperFabric Concepts
reason (for example, if it is killed), all HyperFabric-related
communications on that node are stopped immediately. This makes the
node unreachable by other components in the fabric.
When you start HyperFabric, the fabric is verified automatically. This is
because each node performs a self diagnosis and verification over each
adapter installed in the node. In addition, the management process
performs automatic routing and configuring for each switch (if switches
are part of the fabric). You can, if needed, run the clic_stat command
to get a textual map of the fabric, which can be used as another method
of quick verification.
Notice that the commands to administer HyperFabric have a prefix of
clic_, and some of the other components have CLIC as part of their
name (for example, the CLIC firmware and the CLIC software). CLIC
stands for CLuster InterConnect, and it is used to differentiate those
HyperFabric commands or components from other commands or
components. For example, the HyperFabric command clic_init is
different from the HP-UX init command.
Chapter 18
2Planning the Fabric
This chapter contains the following sections that include general
guidelines and protocol-specific considerations for planning HyperFabric
clusters that run TCP/UDP/IP or HMP applications.
Chapter 29
Planning the Fabric
•“Preliminary Considerations” on page 11
•“HyperFabric Features, Parameters and Supported Configurations
for TCP/UDP/IP and HMP Applications” on page 13
•“TCP/UDP/IP” on page 14
•“Hyper Messaging Protocol (HMP)” on page 26
Chapter 210
Planning the Fabric
Preliminary Considerations
Preliminary Considerations
Before assembling a fabric physically, do the following to address all of
the appropriate issues:
Step 1. Read Chapter 1, “Overview of the HyperFabric Product,” on page 1, to
get a basic understanding of HyperFabric and its components.
Step 2. Read this chapter, Planning the Fabric, to gain an understanding of
protocol specific configuration guidelines for TCP/UDP/IP and HMP
applications.
Step 3. Read “Configuration Overview” on page 69, “Information You Need” on
page 71, and “Configuration Information Example” on page 74, to
understand how to configure the fabric.
Step 4. Decide the number of nodes that will be interconnected in the fabric.
Step 5. Decide the type of HP 9000 system for each node (for a list of supported
HP 9000 systems, see the HyperFabric Release Notes available at
Step 6. Determine the network bandwidth requirements for each node.
Step 7. Determine the number of adapters needed for each node.
Step 8. Determine if a High Availability (ServiceGuard) configuration will be
needed. If ServiceGuard is used, each node should have at least two
adapters.
Step 9. Decide the topology of the fabric.
Step 10. Determine how many switches will be used based on the number of
nodes in the fabric. The only configuration that can be supported without
a switch is the node-to-node configuration (HA or non-HA). HyperFabric
supports meshed switches up to a depth of four switches, starting with
the following versions of the HyperFabric software:
•For HF2 switches: software versions B.11.00.11, B.11.11.01, and
B.11.23.00.
Chapter 211
Planning the Fabric
Preliminary Considerations
Step 11. Draw the cable connections from each node to switches (if the fabric will
contain switches). If you use an HA configuration with switches, it
requires more than one switch for complete redundancy and to avoid a
single point of failure. For example, each adapter can be connected to its
own switch, or two switches can be connected to four adapters.
Chapter 212
Planning the Fabric
HyperFabric Features, Parameters and Supported Configurations for TCP/UDP/IP and HMP Applications
HyperFabric Features, Parameters and
Supported Configurations for TCP/UDP/IP
and HMP Applications
The following sections in this chapter define HyperFabric functionality
for TCP/UDP/IP applications and Hyper Messaging Protocol (HMP)
applications. There are distinct differences in supported hardware,
available features and performance, depending on which protocol is used
by applications running on the HyperFabric.
Chapter 213
Planning the Fabric
TCP/UDP/IP
TCP/UDP/IP
TCP/UDP/IP is supported on all HF2 hardware. Although some of the
HyperFabric adapter cards support both HMP and TCP/UDP/IP
applications, in this section, the focus is on TCP/UDP/IP HyperFabric
applications.
Application Availability
All applications, including Oracle 9i and HP-MPI, that use the
TCP/UDP/IP stack are supported.
Features
This section discusses the following HyperFabric features on
TCP/UDP/IP:
•OnLine Addition and Replacement (OLAR): Supported
The OLAR feature allows the replacement or addition of
HyperFabric adapter cards while the system (node) is running.
HyperFabric supports this functionality on the SD64A, rx8620,
rx4640, rp54xx (L-class), rp74xx (N-class), rp8400 and Superdome
systems, running on the HP-UX 11i v2 platform.
For more information on OLAR, including instructions for
implementing this feature, see “Online Addition and Replacement”
on page 44 and Configuring HP-UX for Peripherals Part Number
B2355-90698 November 2000 Edition.
•Event Monitoring Service (EMS): Supported
In the HyperFabric version B.11.23.01, the HyperFabric EMS
monitor enables the system administrator to separately monitor
each HyperFabric adapter on every node in the fabric, in addition to
monitoring the entire HyperFabric subsystem. The monitor can
inform the user if the resource being monitored is UP or DOWN. The
administrator defines the condition to trigger a notification (usually
a change in interface status). Notification can be accomplished with
one of the following:
— A Simple Network Management Protocol (SNMP) trap
Chapter 214
Planning the Fabric
— Logging into a user specified log file with a choice of severity
— Email to a user defined email address.
For more information on EMS, including instructions for
implementing this feature, see “Configuring the HyperFabric EMS
Monitor” on page 85 and the EMS Hardware Monitors User’s Guide
Part Number B6191-90028 September 2001 Edition.
•ServiceGuard: Supported
Within a cluster, ServiceGuard groups application services
(individual HP-UX processes) into packages. In the event of a single
service failure (node, network, or other resource), EMS provides
notification and ServiceGuard transfers control of the package to
another node in the cluster, allowing services to remain available
with minimal interruption.
ServiceGuard via EMS, directly monitors cluster nodes, LAN
interfaces, and services (the individual processes within an
application). ServiceGuard uses a heartbeat LAN to monitor the
nodes in a cluster. ServiceGuard cannot use the HyperFabric
interconnect as a heartbeat LAN. Instead, use a separate LAN for
the heartbeat.
TCP/UDP/IP
For more information on configuring ServiceGuard, see “Configuring
HyperFabric with ServiceGuard” on page 87, and ManagingMC/ServiceGuard Part Number B3936-90065 March 2002 Edition.
•High Availability (HA): Supported
To create a highly available HyperFabric cluster, there should not be
any single point of failure. Once the HP 9000 nodes and the
HyperFabric hardware have been configured with no single point of
failure, ServiceGuard and EMS can be configured to monitor and fail
over nodes and services using ServiceGuard packages.
If any HyperFabric resource in a cluster fails (adapter card, cable or
switch port), the HyperFabric driver transparently routes traffic over
other available HyperFabric resources with no disruption of service.
The ability of the HyperFabric driver to transparently fail over traffic
reduces the complexity of configuring highly available clusters with
ServiceGuard, because ServiceGuard has to take care of node and
service failover only.
Chapter 215
Planning the Fabric
TCP/UDP/IP
ServiceGuard uses a “heartbeat” to monitor the cluster. The
HyperFabric links cannot be used for the heartbeat. Instead, an
alternate LAN connection such as 100BaseT, Ethernet, Token Ring,
or FDDI must be made between the nodes for use as a heartbeat link.
End-To-End HA: HyperFabric provides end-to-end HA on the entire
cluster fabric at the link level. If any of the available routes in the
fabric fails, HyperFabric transparently redirects all the traffic to a
functional route and, if configured, notifies ServiceGuard or other
enterprise management tools.
Active-Active HA: In configurations where there are multiple
routes between nodes, the HyperFabric software uses a hashing
function to determine an adapter or a route through which it sends
messages. This is done on a message-by-message basis. All of the
available HyperFabric resources in the fabric are used for
communication.
In contrast to Active-Passive HA, where one set of resources is not
utilized until another set fails, Active-Active HA provides the best
return on investment because all of the resources are utilized
simultaneously. ServiceGuard is not required for Active-Active HA
operation.
For more information on setting up HA HyperFabric clusters, see
Figure 2-3 “TCP/UDP/IP High Availability Switched Configuration”
on page 24.
•Dynamic Resource Utilization (DRU): Supported
If you add a new resource (node, adapter, cable or switch) to a cluster,
the HyperFabric subsystem dynamically identifies the added
resource and starts using it. The same process takes place when a
resource is removed from a cluster. The difference between DRU and
OLAR is that OLAR applies only to the addition or replacement of
adapter cards from nodes.
•Load Balancing: Supported
When an HP 9000 HyperFabric cluster is running TCP/UDP/IP
applications, the HyperFabric driver balances the load across all
available resources in the cluster, including nodes, adapter cards,
links, and multiple links between switches.
•Switch Management: Not Supported
Chapter 216
Planning the Fabric
TCP/UDP/IP
Switch Management is not supported. Switch management will not
operate properly if you enable it on a HyperFabric cluster.
•Diagnostics: Supported
Diagnostics can be run to obtain information on many of the
HyperFabric components using the clic_diag, clic_probe and
clic_stat commands, as well as the Support Tools Manager (STM).
For more information on HyperFabric diagnostics, see “Running
Diagnostics” on page 133.
Configuration Parameters
This section describes the maximum limits for TCP/UDP/IP HyperFabric
configurations. There are numerous variables that can impact the
performance of any HyperFabric configuration. For guidance on specific
HyperFabric configurations for TCP/UDP/IP applications, see the
section, “TCP/UDP/IP Supported Configurations” on page 21.
•HyperFabric is supported only on the HP 9000 series servers and
workstations.
•TCP/UDP/IP is supported for all HyperFabric hardware and
software.
•Maximum Supported Nodes and Adapter Cards
In point-to-point configurations, the complexity and performance
limitations of having a large number of nodes in a cluster make it
necessary to include switching in the fabric. Typically, point-to-point
configurations consist of only 2 or 3 nodes.
In switched configurations, HyperFabric supports a maximum of 64
interconnected adapter cards.
A maximum of 8 HyperFabric adapter cards are supported per
instance of the HP-UX operating system. The actual number of
adapter cards a particular node is able to accommodate also depends
on slot availability and system resources. See node specific
documentation for details.
HyperFabric subsystem supports a maximum of 8 configured IP
addresses per instance of the HP-UX operating system.
•Maximum Number of Switches
Chapter 217
Planning the Fabric
TCP/UDP/IP
You can interconnect (mesh) up to 4 switches (16-port fiber, or Mixed
8 fiber ports) in a single HyperFabric cluster.
•Trunking Between Switches (multiple connections)
You can use trunking between switches to increase bandwidth and
cluster throughput. Trunking is also a way to eliminate a possible
single point of failure. The number of trunked cables between nodes
is limited only by port availability. To assess the effects of trunking
on the performance of any particular HyperFabric configuration,
contact your HP representative.
•Maximum Cable Lengths
HF2 (fiber): The maximum distance is 200m (Four standard cable
lengths are sold and supported: 2m, 16m, 50m and 200m).
TCP/UDP/IP supports up to four HF2 switches connected in series
with a maximum cable length of 200m between the switches and
200m between switches and nodes.
TCP/UDP/IP supports up to four hybrid HF2 switches connected in
series with a maximum cable length of 200m between fiber ports.
Chapter 218
Planning the Fabric
TCP/UDP/IP
•Throughput and Latency
Table 2-1HF2 Throughput and Latency with TCP/UDP/IP Applications
Server ClassMaximum ThroughputLatency
rp74002 + 2 Gbps full duplex per link< 42 microsec
Multiple TCP/UDP/IP HyperFabric configurations are supported to
match the cost, scaling, and performance requirements of each
installation.
In the previous section, “Configuration Parameters” on page 17, the
maximum limits for TCP/UDP/IP enabled HyperFabric hardware
configurations were outlined. In this section the TCP/UDP/IP enabled
HyperFabric configurations that HP supports are explained. These
recommended configurations offer an optimal mix of performance and
availability for a variety of operating environments.
There are many variables that can impact HyperFabric performance. If
you are considering a configuration that is beyond the scope of the
following HP supported configurations, contact your HP representative.
Point-to-Point Configurations
You can interconnect large servers like the HP Superdome to run Oracle
RAC 9i and enterprise resource planning applications. These
applications are typically consolidated on large servers.
Point-to-point connections between servers support the performance
benefits of HMP without investing in HyperFabric switches. This is a
good solution in small configurations where the benefits of a switched
HyperFabric cluster might not be required (see configuration A and
configuration C in Figure 2-1).
If there are multiple point-to-point connections between two nodes,
traffic load is balanced over those links. If one link fails, the load fails
over to the remaining links (see configuration B in Figure 2-1).
Running applications using TCP/UDP/IP on a HyperFabric cluster
provides major performance benefits compared to other technologies
such as Ethernet. If a HyperFabric cluster is originally set up to run
enterprise applications using TCP/UDP/IP and the computing
environment stabilizes with a requirement for higher performance,
migration to HMP is always an option.
Chapter 221
Planning the Fabric
TCP/UDP/IP
Figure 2-1TCP/UDP/IP Point-To-Point Configurations
Chapter 222
Switched Configuration
This configuration offers the same benefits as the point-to-point
configurations illustrated in Figure 2-1, but it has the added advantage
of greater connectivity (see Figure 2-2).
Figure 2-2TCP/UDP/IP Basic Switched Configuration
Planning the Fabric
TCP/UDP/IP
Chapter 223
Planning the Fabric
TCP/UDP/IP
High Availability Switched Configuration
This configuration has no single point of failure. The HyperFabric driver
provides end-to-end HA. If any HyperFabric resource in the cluster fails,
traffic is transparently rerouted through other available resources. This
configuration provides high performance and high availability (see
Figure 2-3).
Figure 2-3TCP/UDP/IP High Availability Switched Configuration
Chapter 224
Hybrid Configuration
You can interconnect servers and workstations in a single heterogeneous
HyperFabric cluster.
In this configuration, the servers are highly available. In addition, the
workstations and the servers can run the same application or different
applications (see Figure 2-4).
Figure 2-4TCP/UDP/IP Hybrid Configuration
Planning the Fabric
TCP/UDP/IP
Chapter 225
Planning the Fabric
Hyper Messaging Protocol (HMP)
Hyper Messaging Protocol (HMP)
Hyper Messaging Protocol (HMP) is an HP patented, high performance
cluster interconnect protocol. HMP provides reliable, high speed, low
latency, low CPU overhead, datagram service to applications running on
the HP-UX operating system.
HMP was jointly developed with Oracle Corp. The resulting feature set
was tuned to enhance the scalability of the Oracle Cache Fusion
clustering technology. It is implemented using Remote DMA (RDMA)
paradigms.
HMP is integral to the HP-UX HyperFabric driver. It can be enabled or
disabled at HyperFabric initialization using the clic_init command or
SAM. The HMP functionality is used by the applications listed in the
following “Application Availability” section.
HMP significantly enhances the performance of parallel and technical
computing applications.
HMP firmware on HyperFabric adapter cards provides a shortcut that
bypasses several layers in the protocol stack, boosting link performance
and lowering latency. By avoiding interruptions and buffer copying in the
protocol stack, communication task processing is optimized.
Application Availability
The following are the two families of applications that can use HMP over
the HyperFabric interface:
•Oracle 9i Database, Release 1 (9.0.1) and Release 2 (9.2.0.1.0).
HMP has been certified on Oracle 9i Database Release 1 with HP-UX
11.0, 11i v1, and 11i v2.
HMP has been certified on Oracle 9i Database Release 2 with HP-UX
11.0, 11i v1, and 11i v2.
•Technical Computing Applications that use the HP Message Passing
Interface (HP-MPI).
Chapter 226
Planning the Fabric
Hyper Messaging Protocol (HMP)
HP MPI is a native implementation of version 1.2 of the
Message-Passing Interface Standard. It has become the industry
standard for distributed technical applications and is supported on
most technical computing platforms.
Features
The following are the HyperFabric features on HMP:
•OnLine Addition and Replacement (OLAR)
The OLAR feature, which allows the replacement or addition of
HyperFabric adapter cards while the system (node) is running, is
supported when applications use HMP to communicate.
•Event Monitoring Service (EMS): Supported
In the HyperFabric version B.11.23.01, the HyperFabric EMS
monitor enables the system administrator to separately monitor
each HyperFabric adapter on every node in the fabric, in addition to
monitoring the entire HyperFabric subsystem. The monitor can
inform the user if the resource being monitored is UP or DOWN. The
administrator defines the condition to trigger a notification (usually
a change in interface status). Notification can be accomplished with a
SNMP trap, or by logging into a user specified log file with a choice of
severity, or by email to a user defined email address.
For more information on EMS, including instructions for
implementing this feature, see “Configuring the HyperFabric EMS
Monitor” on page 85 in this manual, and the EMS HardwareMonitors User’s Guide Part Number B6191-90028 September 2001
Edition.
•ServiceGuard: Supported
Within a cluster, ServiceGuard groups application services
(individual HP-UX processes) into packages. In the event of a single
service failure (node or network), EMS provides notification and
ServiceGuard transfers control of the package to another node in the
cluster, allowing services to remain available with minimal
interruption. ServiceGuard using EMS, directly monitors cluster
nodes, LAN interfaces, and services (the individual processes within
an application). ServiceGuard uses a heartbeat LAN to monitor the
Chapter 227
Planning the Fabric
Hyper Messaging Protocol (HMP)
nodes in a cluster. ServiceGuard cannot use the HyperFabric
interconnect as a heartbeat link. Instead, a separate LAN must be
used for the heartbeat.
For more information on configuring ServiceGuard, see “Configuring
HyperFabric with ServiceGuard” on page 87, as well as ManagingMC/ServiceGuard Part Number B3936-90065 March 2002 Edition.
•High Availability (HA): Supported
When applications use HMP to communicate between HP 9000
nodes in a HyperFabric cluster, you can configure ServiceGuard and
the EMS monitor to identify node failure and automatically failover
to a functioning HP 9000 node.
For more information on HA when running HMP applications,
contact your HP representative.
•Transparent Local Failover: Supported
HMP supports Transparent Local Failover in the HyperFabric
version B.11.23.01.
When a HyperFabric resource (adapter, cable, switch or switch port)
fails in a cluster, HMP transparently fails over traffic using other
available resources. This is accomplished using card pairs, each of
which is a logical entity that comprises a pair of HF2 adapters on a
HP 9000 node. Only Oracle applications can make use of the Local
Failover feature. HMP traffic can only fail over between adapters
that belong to the same card pair. Traffic does not fail over if both the
adapters in a card pair fail. However, administrators do not need to
configure HF2 adapters as card pairs if TCP/UDP/IP is run over HF2
or MPI uses HMP.
When HMP is configured in the local failover mode, all the resources
in the cluster are utilized. If a resource fails in the cluster and is
restored, HMP does not utilize that resource until another resource
fails.
For more information on Transparent Local Failover while running
HMP applications, see “Configuring HMP for Transparent Local
Failover Support” on page 96.
•Dynamic Resource Utilization (DRU): Partially Supported
If you add a new HyperFabric resource (node, cable or switch) to a
cluster running an HMP application, the HyperFabric subsystem
will dynamically identify the added resource and start using it. The
Chapter 228
Planning the Fabric
Hyper Messaging Protocol (HMP)
same process takes place when a resource is removed from a cluster.
However, DRU is not supported if you add or remove an adapter from
a node that is running an HMP application. This is consistent with
the fact that OLAR is not supported when an HMP application is
running on HyperFabric.
•Load Balancing: Supported
When an HP 9000 HyperFabric cluster is running HMP applications,
the HyperFabric driver balances the load across all available
resources in the cluster, including nodes, adapter cards, links, and
multiple links between switches.
•Switch Management: Not Supported
Switch Management is not supported. Switch management will not
operate properly if it is enabled on a HyperFabric cluster.
•Diagnostics: Supported
You can run diagnostics to obtain information on many of the
HyperFabric components using the clic_diag, clic_probe and
clic_stat commands, as well as the Support Tools Manager (STM).
For more information on HyperFabric diagnostics, see “Running
Diagnostics” on page 149.
Configuration Parameters
This section discusses the maximum limits for HMP HyperFabric
configurations. There are numerous variables that can impact the
performance of any particular HyperFabric configuration. For more
information on specific HyperFabric configurations for HMP
applications, see “HMP Supported Configurations” on page 33.
•HyperFabric is supported on the HP 9000 series servers and
workstations only.
•HMP is supported on the HF2 adapter, A6386A, only.
•The performance advantages that HMP offers are not completely
realized unless HMP is used with A6386A HF2 (fiber) adapters and
related fiber hardware. See Table 2-2 on page 20 for details. The local
failover configuration of HMP is supported only on the A6386A HF2
adapters.
•Maximum Supported Nodes and Adapter Cards
Chapter 229
Planning the Fabric
Hyper Messaging Protocol (HMP)
HyperFabric clusters running HMP applications are limited to
supporting a maximum of 64 adapter cards. However, in local
failover configurations, a maximum of only 52 adapters are
supported.
In point-to-point configurations running HMP applications, the
complexity and performance limitations of having a large number of
nodes in a cluster make it necessary to include switches in the fabric.
Typically, point-to-point configurations consist of only 2 or 3 nodes.
In switched configurations running HMP applications, HyperFabric
supports a maximum of 64 interconnected adapter cards.
A maximum of 8 HyperFabric adapter cards are supported per
instance of the HP-UX operating system. The actual number of
adapter cards a particular node is able to accommodate also depends
on slot availability and system resources. See node specific
documentation for details.
A maximum of 8 configured IP addresses are supported by the
HyperFabric subsystem per instance of the HP-UX operating system.
•Maximum Number of Switches
You can interconnect (mesh) up to 4 switches (16-port fiber or Mixed
8 fiber ports) in a single HyperFabric cluster.
•Trunking Between Switches (multiple connections)
Trunking between switches can be used to increase bandwidth and
cluster throughput. Trunking is also a way to eliminate a possible
single point of failure. The number of trunked cables between nodes
is only limited by port availability. To assess the effects of trunking
on the performance of any particular HyperFabric configuration,
contact your HP representative.
•Maximum Cable Lengths
HF2 (fiber): The maximum distance is 200m (4 standard cable
lengths are sold and supported: 2m, 16m, 50m and 200m).
HMP supports up to 4 HF2 switches connected in series with a
maximum cable length of 200m between the switches and 200m
between switches and nodes.
HMP supports up to 4 hybrid HF2 switches connected in series with
a maximum cable length of 200m between fiber ports.
Chapter 230
Planning the Fabric
Hyper Messaging Protocol (HMP)
•HMP is supported on A400, A500, rp2400, rp2450, rp54xx (N-class),
rp74xx (L-class), rp8400, and Superdome servers running 64-bit
HP-UX.
•HMP is supported on HyperFabric starting HyperFabric versions
B.11.00.11, B.11.11.01, and B.11.23.00.
•HMP is not supported on the A180 or A180C server.
•HMP is not supported on 32-bit versions of HP-UX.
•Throughput and Latency
Table 2-3HF2 Throughput and Latency with HMP Applications
Server ClassMaximum ThroughputLatency
rp 74002 + 2 Gbps full duplex per link< 22 microsec
NOTEThe local failover configuration on HMP is supported only on the A6386A
HF2 adapters.
Chapter 232
Planning the Fabric
Hyper Messaging Protocol (HMP)
HMP Supported Configurations
Multiple HMP HyperFabric configurations are supported to match the
performance, cost and scaling requirements of each installation.
In the section, “Configuration Parameters” on page 29, the maximum
limits for HMP enabled HyperFabric hardware configurations were
outlined. This section discusses the HMP enabled HyperFabric
configurations that HP supports. These recommended configurations
offer an optimal mix of performance and availability for a variety of
operating environments.
There are many variables that can impact HyperFabric performance. If
you are considering a configuration that is beyond the scope of the
following HP supported configurations, contact your HP representative.
Point-to-Point Configuration
You can interconnect large servers like the HP Superdome to run Oracle
RAC 9i and enterprise resource planning applications. These
applications are typically consolidated on large servers.
Point-to-point connections between servers support the performance
benefits of HMP without investing in HyperFabric switches. This is a
good solution in small configurations where the benefits of a switched
HyperFabric cluster might not be required (see configuration A in
Figure 2-5).
If an HMP application is running over HyperFabric and another node or
adapter is added to either of the nodes, then it is necessary to also add a
HyperFabric switch to the cluster (see configuration B in Figure 2-5).
Chapter 233
Planning the Fabric
Hyper Messaging Protocol (HMP)
Figure 2-5HMP Point-To-Point Configurations
Chapter 234
Planning the Fabric
Hyper Messaging Protocol (HMP)
Enterprise (Database) Configuration
The HMP enterprise configuration illustrated in Figure 2-6 is very
popular for running Oracle RAC 9i.
Superdomes or other large servers make up the Database Tier. Database
Tier nodes communicate with each other using HMP.
Application Tier nodes communicate with each other and to the
Database Tier using TCP/UDP/IP.
Figure 2-6HMP Enterprise (Database) Configuration, Single Connection
Between Nodes
Chapter 235
Planning the Fabric
Hyper Messaging Protocol (HMP)
Enterprise (Database) - Local Failover Supported Configuration
The HMP enterprise configuration is a scalable solution. For high
availability and performance, you can easily scale the HMP enterprise
configuration with multiple connections between the HyperFabric
resources. Any single point of failure in the database tier of the fabric is
eliminated in Figure 2-7.
In this configuration, if a HyperFabric resource (adapter, cable, switch or
switch port) fails in a cluster, HMP transparently fails over traffic using
another available resource. For more information, see “Configuring HMP
for Transparent Local Failover Support” on page 96.
Chapter 236
Planning the Fabric
Hyper Messaging Protocol (HMP)
Technical Computing (Work Stations) Configuration
This configuration is typically used to run technical computing
applications with HP-MPI. A large number of small nodes are
interconnected to achieve high throughput (see Figure 2-8). High
availability is not usually a requirement in technical computing
environments.
HMP provides the high performance, low latency path necessary for
these technical computing applications. You can interconnect up to 56
nodes using HP 16-port switches. You cannot link more than four 16-port
switches in a single cluster (see Figure 2-9).
The HP “J”, “B”, and “C” class workstations provide excellent
performance and return on investment in technical computing
configurations.
Chapter 237
Planning the Fabric
Hyper Messaging Protocol (HMP)
Figure 2-8Technical Computing Configuration
Chapter 238
Hyper Messaging Protocol (HMP)
Figure 2-9Large Technical Computing Configuration
Planning the Fabric
Chapter 239
Planning the Fabric
Hyper Messaging Protocol (HMP)
Chapter 240
3Installing HyperFabric
This chapter contains the following sections that describe the
HyperFabric installation:
•“Checking HyperFabric Installation Prerequisites” on page 43.
Chapter 341
Installing HyperFabric
•“Installing HyperFabric Adapters” on page 44.
•“Installing the Software” on page 51.
•“Installing HyperFabric Switches” on page 57.
Chapter 342
Installing HyperFabric
Checking HyperFabric Installation Prerequisites
Checking HyperFabric Installation
Prerequisites
Before installing HyperFabric, ensure that the following hardware and
software prerequisites are met:
✓Check the HyperFabric Release Notes for known problems, required
patches, or other information needed for installation.
✓Confirm that the /usr/bin, /usr/sbin, and /sbin directories are in
your PATH by logging in as root and using the echo $PATH
command.
✓Confirm that the HP-UX operating system is the correct version. Use
the uname -a command to determine the HP-UX version.
For more information about the required operating system versions,
see HyperFabric Release Notes.
✓ If you are installing an HF2 switch, confirm that you have four
screws with over-sized heads.
✓Confirm that there are cables of proper length and type (fiber) to
make connections in the fabric (adapter to adapter, adapter to
switch, or switch to switch).
✓Confirm that there is at least one loopback plug for testing the
adapters and switches (a fiber loopback plug [HP part number
A6384-67004] is shipped with each HF2 switch).
✓Confirm that necessary tools are available to install the HyperFabric
switch mounting hardware. In addition, check the HP 9000 system’s
documentation to determine if additional tools may be required for
component installation.
✓Confirm that the software media is correct.
✓Create a map of the fabric (optional).
✓Confirm that HP-UX super-user privileges are available; they are
necessary to complete the HyperFabric installation.
The first HyperFabric installation step is installing HyperFabric adapter
cards in the nodes.
Chapter 343
Installing HyperFabric
Installing HyperFabric Adapters
Installing HyperFabric Adapters
This section contains information about installing HyperFabric adapters
in HP 9000 systems. Online Addition and Replacement (OLAR)
information is provided in the section, “Online Addition and
Replacement” on page 44.
CAUTIONHyperFabric adapters contain electronic components that can easily be
damaged by small amount of electricity. To avoid damage, follow these
guidelines:
•Store adapters in their antistatic plastic bags until installation.
•Work in a static-free area, if possible.
•Handle adapters by the edges only. Do not touch electronic
components or electrical traces.
•Use the disposable grounding wrist strap provided with each
adapter. Follow the instructions included with the grounding strap.
•Use a suitable ground—any exposed metal surface on the computer
chassis.
For specific instructions, see system-specific documentation on
“installing networking adapters” for each type of HP 9000 system that
HyperFabric adapters will be installed into.
When the HyperFabric adapters have been installed, go to “Installing the
Software” on page 51.
Online Addition and Replacement
Online Addition and Replacement (OLAR) allows PCI I/O cards,
adapters or controllers to be replaced or added to HP 9000 systems,
without the need for completely shutting down and rebooting the system,
or adversely affecting other system components. This feature is only
Chapter 344
Installing HyperFabric
Installing HyperFabric Adapters
available on HP 9000 systems that are designed to support OLAR. The
system hardware uses the per-slot power control combined with OS
support to enable this feature.
NOTEOLAR is supported only on TCP/UDP/IP over HF2 adapters.
Not all add-in cards have this capability, but over time many cards will
be gaining this capability.
The HyperFabric Release Notes contains information about which
HP 9000 systems and HyperFabric adapters OLAR is supported for.
IMPORTANTAt this time, Superdome systems are not intended for access by users.
HP recommends that these systems only be opened by a qualified HP
engineer. Failure to observe this requirement can invalidate any support
agreement or warranty to which the owner might otherwise be entitled.
There are two methods to add or replace OLAR-compatible cards:
•Using the SAM utility.
•Issuing command-line commands, through olrad, that refer to the
HyperFabric OLAR script (/usr/sbin/olard.d/clicd).
HP recommends that SAM be used for OLAR procedures, instead of the
rad command. This is primarily because SAM prevents the user from
doing things that might have adverse effects. This is not true when the
rad command is used.
For detailed information about using either of these two procedures, see
Configuring HP-UX For Peripherals. You can order that document from
HP, or you can view, download, and print it from the following URL:
http://www.docs.hp.com.
Chapter 345
Installing HyperFabric
Installing HyperFabric Adapters
Table 3-1 below explains some important OLAR-related terms.
Table 3-1Important OLAR Terms
TermMeaning
OLARAll aspects of the OLAR feature
Power DomainA grouping of 1 or more interface
target card / target card slotThe interface card which will be
affected card / affected card slotInterface cards and the card slots
including Online Addition (OLA)
and Online Replacement (OLR).
card slots that are powered on or
off as a unit. (Note: Multi-slot
power domains are not currently
supported.)
added or replaced using OLAR,
and the card slot in which it
resides.
they reside in, which are in the
same power domain as the target
slot.
IMPORTANTIn many cases, other interface cards and slots within the system are
dependent on the target card. For example, if the target card is a
multiple-port card, suspending or deleting drivers for the target card slot
also suspends individual drivers for the multiple hardware paths on that
card.
During a card replacement operation, SAM performs a CriticalResource Analysis (CRA), which checks all ports on the target card for
critical resources that would be temporarily unavailable while the card is
shut down.
Chapter 346
Installing HyperFabric
Installing HyperFabric Adapters
Planning and Preparation
As mentioned previously, for the most part, SAM prevents the user from
performing OLAR procedures that would adversely affect other areas of
the HP 9000 system. See Configuring HP-UX For Peripherals for detailed
information.
Critical Resources
The effects of shutting down a card’s functions must be considered.
Replacing a card that is still operating can have extensive consequences.
Power to a slot must be turned off when a card is removed and a new
card is inserted.
This is particularly important if there is no online failover or backup
card to pick up those functions. For example:
•Which mass storage devices will be temporarily disconnected when a
card is shut down?
•Will a critical networking connection be lost?
A critical resource is one that would cause a system crash or prevent an
operation from successfully completing if the resource were temporarily
suspended or disconnected. For example, if the SCSI controller is
connected to the unmirrored root disk or swap space, the system will
crash when the SCSI controller is shut down.
During an OLAR procedure, it is essential to check the targeted card for
critical resources, as well as the effects of existing disk mirrors and other
situations where a card’s functions can be taken over by another card
that will not be affected.
As mentioned earlier, SAM performs a thorough CRA automatically, and
presents options based on its findings. If it is determined that critical
resources will be affected by the OLAR procedure, the card could be
replaced when the system is offline. If action must be taken immediately,
an online addition of a backup card and deletion of the target card could
be attempted using the rad command.
Card Compatibility
This section explains card compatibility considerations for doing OLAR.
Chapter 347
Installing HyperFabric
Installing HyperFabric Adapters
Online Addition (OLA) Multiple cards can be added at the same time.
When adding a card online, the first issue to resolve is whether the new
card is compatible with the system. Each OLAR-capable PCI slot
provides a set amount of power. The replacement card cannot require
more power than there is available.
The card must also operate at the slot’s bus frequency. A PCI card must
run at any frequency lower than its maximum capability, but a card that
could operate at only 33 MHz would not work on a bus running at 66
MHz. rad provides information about the bus frequency and power
available at a slot, as well as other slot-related data.
If an HP 9000 system has one or more slots that support OLAR and OLA
will be used to install a HyperFabric adapter in one of those
slots—install the adapter in the HP 9000 system according to the
procedure described in the “Managing PCI Cards with OLAR” chapter of
the “Configuring HP-UX Peripherals” manual.
After adding a new HyperFabric adapter, SAM tries to locate the
HyperFabric software. If SAM cannot locate the HyperFabric software,
the new adapter cannot be used until the software is installed (software
installation requires a system reboot). If SAM locates the HyperFabric
software, SAM determines whether the new adapter is functional. If it is
not functional, SAM displays an error message.
If the new adapter is functional, SAM displays a message telling the user
to configure the adapter and start HyperFabric. If only one adapter is
being added, issue the clic_init -c command or use SAM to configure
the adapter, and then issue the clic_start command or use SAM to
start HyperFabric. If multiple adapters are being added, add all of the
adapters first, and then run clic_init -c and clic_start or use SAM.
See “Performing the Configuration” on page 78 and “Starting
HyperFabric” on page 109 for more information about configuring and
starting HyperFabric.
CAUTIONDo not change any configuration information for an existing HyperFabric
adapter or switch while you are using clic_init -c to configure a new
adapter.
When you have completed the adapter installation, go to “Installing the
Software” on page 51.
Chapter 348
Installing HyperFabric
Installing HyperFabric Adapters
Online Replacement (OLR) When replacing an interface card online,
the replacement card must be identical to the card being replaced (or at
least be able to operate using the same driver as the replaced card). This
is referred to as like-for-like replacement and should be adhered to,
because using a similar but not identical card can cause unpredictable
results. For example, a newer version of the target card that is identical
to the older card in terms of hardware might contain an updated
firmware version that could potentially conflict with the current driver.
An A6386A adapter must be replaced with another A6386A adapter, etc.
Also, the old adapter and new adapter must have the same revision
levels.
When a replacement card is added to an HP 9000 system, the
appropriate driver for that card must be configured in the kernel before
beginning the replacement operation. SAM ensures the correct driver is
present. (In most cases, the replacement card will be the same type as a
card already in the system, and this requirement will be automatically
met.) Keep the following things in mind:
•If the necessary driver is not present and the driver is a dynamically
loadable kernel module (DLKM), it can be loaded manually. See the
“Dynamically Loadable Kernel Modules” section in “ConfiguringHP-UX For Peripherals” for more information.
•If the driver is static and not configured in the kernel, then the card
cannot be added online. The card could be physically inserted online,
but no driver would claim it.
If there is any question about the driver’s presence, or if it is uncertain
that the replacement card is identical to the existing card, ioscan can be
used together with rad to investigate.
If more than one operational HyperFabric adapter is present when SAM
requests the suspend operation for all ports on the target adapter,
HyperFabric will redirect the target adapter’s traffic to a local backup
adapter using local failover. Client applications using the replaced
adapter will not be interrupted in any way.
If the adapter being replacing is active and it is the only operational
HyperFabric adapter on the HP 9000 system, SAM displays the following
warning message:
WARNING: You have 1 operational HyperFabric card. If you go
ahead with this operation you will lose network access via
HyperFabric until the on-line replaced HyperFabric card
becomes operational.
Chapter 349
Installing HyperFabric
Installing HyperFabric Adapters
You are asked if you want to continue. If you reply Yes, client applications
are suspended. Replace the adapter according to the procedure described
in the “Managing PCI Cards with OLAR” chapter of the ConfiguringHP-UX Peripherals manual.
When an adapter has been replaced, client application activity resumes
unless the TCP timers or the application timers have popped.
CAUTIONDo not use the clic_start command or the clic_shutdown command,
while an installed adapter is suspended. Do not use SAM to start or stop
HyperFabric while an installed adapter is suspended. The operation will
fail and an error message will be displayed.
After a HyperFabric adapter has been replaced, SAM checks the
replacement adapter to ensure that it is permitted according to the
like-for-like rules. If the adapter is permitted, SAM automatically
activates it. If it is not permitted, SAM displays an error message.
Chapter 350
Installing the Software
This section describes the HyperFabric file structure and the steps
necessary to load the software. The software must be installed on each
instance of the HP-UX operating system in the fabric.
File Structure
The HyperFabric file structure is shown in Figure 3-1 below. The
structure is shown for informational purposes only. The user cannot
modify any of the files or move them to a different directory.
The commands and files used to administer HyperFabric typically have a
prefix of clic_. CLIC stands for CLuster InterConnect, and it is used to
differentiate those HyperFabric commands/files from other
commands/files. For example, the HyperFabric command clic_init is
different from the HP-UX init command.
Each of the files shown in Figure 3-1 above is briefly described below:
•/etc/opt/resmon/dictionary/clic_01
•/etc/rc.config.d/clic_global_conf
•/sbin/init.d/clic
•/var/adm/clic_ip_drv.trc
The HyperFabric dictionary file for the Event Monitoring Service
(EMS).
The global configuration file, which contains the IP addresses for
each adapter and each HyperFabric switch (if any) in the fabric.
The system boot startup script for the HyperFabric management
process.
One of the software’s trace files. This file is created when the
clic_diag -D TCP_IP command is run.
•/var/adm/clic_ip_drv.trc0
One of the HyperFabric software’s trace files. This is the primary file
that is created when the clic_diag -C TCP_IP command is run.
•/var/adm/clic_ip_drv.trc1
One of the HyperFabric software’s trace files. This file is created
when the clic_diag -C TCP_IP command is run, and the primary
trace file (clic_ip_drv.trc0) becomes full.
•/var/adm/clic_log
The global log file that is updated by the HyperFabric management
process.
•/var/adm/clic_log.old
The backup copy of the log file that is created when the log file grows
larger than 100 Kbytes.
Chapter 352
Installing HyperFabric
Installing the Software
•/var/adm/OLDclic_log
The log file from the previous time the clic_start command was
executed.
•/usr/conf/lib/libclic_dlpi_drv.a
The kernel library that contains the HyperFabric software.
•/usr/conf/lib/libha_drv.a
The kernel library that contains the High Availability (HA) software.
•/usr/conf/master.d/clic
This file is described along with the other master files in the master
man page (type man master at the HP-UX prompt).
•/opt/clic/lib/libclic_mgmt.a
The HyperFabric management API library.
•/opt/clic/bin
The directory containing the HyperFabric management commands:
clic_diag, clic_init, clic_probe, clic_shutdown, clic_start,
clic_stat, and clic_dump. (Note that clic_dump is for HP internal
use only.) Also, clic_ping is replaced by clic_probe.This directory
also contains the HyperFabric management process (clic_mgmtd)
and the HyperFabric EMS monitor process (clic_mond).
•/opt/clic/firmware/clic_fw
The 1X HSC HyperFabric 8-bit CRC firmware. This file must not be
modified.
•/opt/clic/firmware/clic_fw_1x32c
The 1X HSC HyperFabric 32-bit CRC firmware. This file must not be
modified.
Chapter 353
Installing HyperFabric
Installing the Software
•/opt/clic/firmware/clic_fw_4x8c
•/opt/clic/firmware/clic_fw_4x32c
•/opt/clic/firmware/clic_fw_hf28c
•/opt/clic/firmware/clic_fw_hf232c
•/opt/clic/firmware/clic_fw_db
•/opt/clic/share/man/man1m.Z
The 4X PCI HyperFabric 8-bit CRC firmware. This file must not be
modified.
The 4X HyperFabric PCI 32-bit CRC firmware. This file must not be
modified.
The HyperFabric2 8-bit firmware. This file must not be modified.
The HyperFabric2 32-bit firmware. This file must not be modified.
A binary file where adapter-specific configuration information is
stored. The management process creates this file using default
values.
The man pages for the HyperFabric commands.
Chapter 354
Installing HyperFabric
Installing the Software
Loading the Software
Listed below are the steps you must follow to load the HyperFabric
software, using the HP-UX swinstall program.
Step 1. Log on to the system as root.
Step 2. Insert the software media into the appropriate drive. If the software is
being loaded from a CD-ROM, go to step 3; otherwise, go to step 4.
Step 3. Mount the CD-ROM drive by using the following command at the
command prompt:
$ mount
where
Step 4. Run the swinstall program using the following command:
$ /usr/sbin/swinstall
This opens the “Software Selection” window.
Step 5. Change the Source Host Name, if necessary, and then enter the mount
point of the drive in the Source Depot Path field. Select the OK button to
return to the “Software Selection” window.
The “Software Selection” window now contains a list of available
software to install.
Step 6. Highlight the HyperFabric software:
•HP-UX 11i v2: HyprFabrc-00
Step 7. Choose Mark for Install from the “Actions” menu; this chooses the
highlighted software.
Step 8. From the “Actions” menu, select the “Install...” menu, and then choose
Install. This begins product installation and opens the “Install
Analysis” window.
device_name
device_name
is the name assigned to the CD-ROM drive.
Step 9. Select the OK button in the “Install Analysis” window when the Status
field displays a “Ready” message.
Step 10. Select the YES button in the “Confirmation” window to start software
installation.
Chapter 355
Installing HyperFabric
Installing the Software
swinstall loads the fileset, runs the control script for the filesets, and
builds the kernel. When the processing is finished, the “Status” field
displays a “Ready” message. Select “Done” and then the “Note” window
opens.
Step 11. Select the OK button in the “Note” window to reboot. The user interface
disappears and the system reboots.
Step 12. When the system comes back up, log on to the system as root and view
the /var/adm/sw/swagent.log and /var/adm/sw/swinstall.log files
to view error or warning messages that may have occurred during the
installation.
Step 13. While still logged in as root, view the /etc/services file to ensure that
these two HyperFabric-related lines are present:
•hp-clic 3384/tcp #clic management daemon
•hp-clic 3384/udp #clic switch management
These lines are used by the HyperFabric software, and are not
comments; therefore, do not remove them from the file.
Step 14. Verify that all installed HyperFabric adapters have a software state of
“CLAIMED,” by running the ioscan -nf -C clic command.
A check is also done to make sure all of the HyperFabric adapters have
been claimed when clic_init is activated or when SAM is used to
configure HyperFabric.
Step 15. If one or more HyperFabric switches are included in the configuration, go
to the next section of this chapter, “Installing HyperFabric Switches”;
otherwise, go to Chapter 4, “Configuring HyperFabric,” on page 67.
Chapter 356
Installing HyperFabric
Installing HyperFabric Switches
Installing HyperFabric Switches
This section contains the information you need to install HyperFabric
switches. As stated earlier, the term HyperFabric2 (HF2) switch
refers to the functional switch (the A6384A switch chassis with one of the
switch modules installed).
Before Installation
Before you install the HyperFabric switch, you should be aware of the
following:
❏The A6384A HF2 switch is supported beginning with the following
HyperFabric software versions:
— HP-UX 11.0: version B.11.00.11
— HP-UX 11i v1: version B.11.11.01
— HP-UX 11i v2: version B.11.23.00.
HyperFabric switches are not supported by software versions earlier
than those mentioned above, respectively.
To determine the version of HyperFabric you have, issue the
following command at the command prompt:
$ swlist | grep -i hyperfabric
❏For the HF2 switch, HP recommends that you use the rails shipped
with the switch when you mount it in a standard 19-inch rack, even
though the switch can be mounted in the rack by itself (without the
rails).
WARNINGTo prevent overheating, you must leave one rack unit (1 EIA)
of empty space above the HyperFabric switch.
❏After the HyperFabric switch is mounted in the rack, attach the
various cables to the switch.
To avoid damage to any of the cables, follow these guidelines:
Chapter 357
Installing HyperFabric
Installing HyperFabric Switches
— If your cables have dust caps over the connectors, keep them in
— Be careful not to stretch, puncture, or crush the cable.
To install an HF2 switch, see “Installing the HF2 Switch” on page 59.
place until you are ready to connect them. This prevents dirt and
oils from soiling any important surfaces.
Chapter 358
Installing HyperFabric
Installing HyperFabric Switches
Installing the HF2 Switch
This section contains information on installing an HF2 switch.
The front of the HF2 switch has a flange—or “wing”—on each side, with
two holes for attaching the switch to the rack. The following figures do
not show the flanges.
Figure 3-2 shows the front of the HF2 switch with an A6388A HF2 8-port
fiber switch module installed in the switch’s expansion slot.
Figure 3-2Front of HF2 Switch (A6388A Switch Module Installed)
Integrated Ethernet management
LAN card
Status
Status
Status
Powe r
A
B
Por t
7
Por t
15
Ethernet
Por t
Main
Por t
6
Por t
14
A6388A HF2 8-port fiber
switch module in
expansion slot
Ethernet
Por t
Por t
5
Por t
13
Aux
Label showing
Ethernet MAC
address
Por t
4
Por t
12
Por t
3
Por t
11
Port LED
colors and
meanings
legend
Por t
2
Por t
10
Integrated 8-port
fiber card
Por t
Por t
1
9
Por t
0
Por t
8
Chapter 359
Installing HyperFabric
Installing HyperFabric Switches
Figure 3-3 shows the front of the HF2 switch with an A6389A HF2 4-port
copper switch module installed in the switch’s expansion slot.
Figure 3-3Front of HF2 Switch (A6389A Switch Module Installed)
Integrated Ethernet management
LAN card
Status
Status
Status
Powe r
A
B
Por t
7
Ethernet
Por t
Main
Por t
6
Por t
11
A6389A HF2 4-port copper
switch module in
expansion slot
You can install the HF2 switch in one of the following ways:
•Using the rail kit that is shipped with the switch (see the next
section, “With the Rail Kit”). HP strongly recommends installing the
HF2 switch this way.
Ethernet
Por t
5
Por t
Aux
Label showing
Ethernet MAC
address
Por t
4
Por t
10
Por t
3
Port LED
colors and
meanings
legend
Por t
2
Por t
9
Integrated 8-port
fiber card
Por t
1
Por t
0
Por t
8
•Attaching the switch directly to the rack (see “Without the Rail Kit”
on page 65).
Chapter 360
Installing HyperFabric
Installing HyperFabric Switches
With the Rail Kit
As mentioned earlier, HP strongly recommends installing the HF2
switch using the rail kit.
When you install the HF2 switch, you will be putting the front of the
switch—the end with the flanges (“wings”)—at the back of the rack. The
steps for installing the HF2 switch using the rail kit are as follows:
Step 1. Prepare the rack for rail and switch installation.
Step 2. Install and secure the rails in the rack, using two screws per rail. The
following figure shows the rack with the rails installed.
rack
back
screws
screws
rack
front
Step 3. From the front of the rack, slide the switch—with the front of the switch
facing the back of the rack—into the rack, on the rails. Move it until it is
snug against the back of the rack. You might not have enough clearance
between the switch and the rail screws; this may prevent you from easily
Chapter 361
Installing HyperFabric
Installing HyperFabric Switches
sliding the switch. If so, try lifting the switch over the rail screws. If you
cannot do this, remove the rail screws, slide the switch into position, and
put the rail screws back in.
Step 4. Align the two holes in each flange (“wing”) on the switch’s front with the
holes in the rack frame. Fasten each flange of the switch to the rack by
putting a screw in each of the four holes in the flanges. Be sure to use
screws with over-sized heads. The following figure shows the rack with
the switch in this position.
rack
back
front of switch
(with flanges)
rack
front
Step 5. From the front of the rack, install a bracket on the outside of each rail,
using two screws per bracket. Be sure to use the upper screw holes on
each bracket. Put the screws in the two square holes closest to the back
of the switch, so that the brackets—referred to as “bracket 1” and
“bracket 2” in these steps—are snug against the switch. Be sure that the
brackets make contact with the beveled (sloped) part of the switch (not
Chapter 362
Installing HyperFabric
Installing HyperFabric Switches
the flat part). Otherwise, the brackets will not secure the back of the
switch properly. The following figure shows the rack with these two
brackets installed.
rack
back
bracket 2
front of switch
(with flanges)
rack
front
bracket 1
Step 6. When you are sure the brackets are snug against the beveled part of the
switch, tighten all four screws in brackets 1 and 2.
Step 7. For each port that will be connected to an HyperFabric adapter in an
HP 9000 system, attach the cable from the corresponding adapter. Your
connections must be copper-to-copper and fiber-to-fiber (including
cables).
Step 8. Connect the switch to the Ethernet network.
Chapter 363
Installing HyperFabric
Installing HyperFabric Switches
Step 9. Plug the switch’s power cord into the rack’s PDU, if it has one.
Alternatively, you can plug a power cord that is compatible with your
country’s requirements into a power strip or outlet that you want to use
for the switch. (In this case, you are responsible for obtaining a
compatible power cord.)
Step 10. Power on the HF2 switch by plugging the power cord into the AC inlet on
the back of the switch. (There is no power switch.)
Step 11. Once the power is on, check these LEDs on the integrated Ethernet
management LAN adapter card (in the top slot of the switch):
✓The “Operating/Fault” LED shows solid green.
✓The “Power A” and “Power B” LEDs show solid green.
✓The “Ethernet Port Main” and “Ethernet Port Aux” LEDs are
showing solid green (connected) or flashing green (Ethernet traffic is
flowing to the switch). See Figure 3-2 on page 59 or Figure 3-3 on
page 60 for the locations of the LEDs.
Step 12. On the integrated 8-port fiber card (in the middle slot of the switch),
check that for each switch port that is connected to an HF2 adapter, the
LED on the port shows as solid green (see Figure 3-2 on page 59 or
Figure 3-3 on page 60). This means the connection is operational.
Step 13. On the switch module in the expansion slot (the bottom slot of the
switch), check that for each switch port that is connected to an
HyperFabric adapter, the LED on the port shows as solid green (see
Figure 3-2 on page 59 or Figure 3-3 on page 60). This means the
connection is operational.
For more information about the switch’s LEDs, see “HF2 Switch LEDs”
on page 148.
Step 14. If you want to install another HF2 switch using the rail kit, go back to
step 1.
If you want to install another HF2 switch without using the rail kit, go to
the next section, “Without the Rail Kit”.
Otherwise, go to Chapter 4, “Configuring HyperFabric,” on page 67.
Chapter 364
Installing HyperFabric
Installing HyperFabric Switches
Without the Rail Kit
As mentioned earlier, HP strongly recommends installing the HF2
switch using the rail kit (described in the previous section, “With the
Rail Kit” on page 61).
When you install the HF2 switch, you will be putting the front of the
switch—the end with the flanges (“wings”) — at the back of the rack. The
steps for installing the HF2 switch without using the rail kit are as
follows:
Step 1. Prepare the rack for switch installation.
Step 2. Insert the HF2 switch into the rack, with the front of the switch snug
against the back of the rack.
Step 3. Align the two holes in each flange on the switch’s front with the holes in
the rack frame.
Step 4. Fasten each flange of the switch to the rack by putting a screw in each of
the four holes in the flanges. Be sure to use screws with over-sized heads.
Step 5. Tighten all of the screws so that the HF2 switch is firmly mounted in the
rack.
Step 6. For each port that will be connected to an HyperFabric adapter in an
HP 9000 system, attach the cable from the corresponding adapter. Your
connections must be copper-to-copper and fiber-to-fiber (including
cables).
Step 7. Connect the switch to the Ethernet network.
Step 8. Plug the switch’s power cord into the rack’s PDU, if it has one.
Alternatively, you can plug a power cord that is compatible with your
country’s requirements into a power strip or outlet that you want to use
for the switch. (In this case, you are responsible for obtaining a
compatible power cord.)
Step 9. Power on the HF2 switch by plugging the power cord into the AC inlet on
the back of the switch. (There is no power switch.)
Chapter 365
Installing HyperFabric
Installing HyperFabric Switches
Step 10. Once the power is on, check these LEDs on the integrated Ethernet
management LAN adapter card (in the top slot of the switch):
✓The “Operating/Fault” LED shows solid green.
✓The “Power A” and “Power B” LEDs show solid green.
✓The “Ethernet Port Main” and “Ethernet Port Aux” LEDs are
showing solid green (connected) or flashing green (Ethernet traffic is
flowing to the switch). See Figure 3-2 or Figure 3-3 below for the
locations of the LEDs.
Step 11. On the integrated 8-port fiber card (in the middle slot of the switch),
check that for each switch port that is connected to an HF2 adapter, the
LED on the port shows as solid green (see Figure 3-2 on page 59 or
Figure 3-3 on page 60). This means the connection is operational.
Step 12. On the switch module in the expansion slot (the bottom slot of the
switch), check that for each switch port that is connected to a
HyperFabric adapter, the LED on the port shows as solid green (see
Figure 3-2 on page 59 or Figure 3-3 on page 60). This means the
connection is operational.
For more information about the switch’s LEDs, see “HF2 Switch LEDs”
on page 148.
Step 13. If you want to install another HF2 switch without using the rail kit, go to
step 1.
If you want to install another HF2 switch using the rail kit, go to “With
the Rail Kit” on page 61.
Otherwise, go to Chapter 4, “Configuring HyperFabric,” on page 67.
Chapter 366
4Configuring HyperFabric
This chapter contains the following sections that describe configuring
HyperFabric:
•“Configuration Overview” on page 69.
Chapter 467
Configuring HyperFabric
•“Information You Need” on page 71.
•“Performing the Configuration” on page 78.
•“Deconfiguring a HyperFabric Adapter with SAM” on page 83.
•“Configuring the HyperFabric EMS Monitor” on page 85.
•“Configuring HyperFabric with ServiceGuard” on page 87.
•“Configuring HMP for Transparent Local Failover Support” on
page 96.
Chapter 468
Configuring HyperFabric
Configuration Overview
Configuration Overview
You do not need to configure the HyperFabric switch because the
HyperFabric management process performs automatic routing and
configuring for the switch. So, configuring HyperFabric consists only of
creating the HyperFabric /etc/rc.config.d/clic_global_conf global
configuration file on each node in the fabric. The configuration file
contains the following information:
•The IP addresses and subnet mask of the HyperFabric adapters
installed in the node.
•For each HyperFabric switch in the fabric — the switch’s IP address,
and the MAC address of the switch’s Ethernet port. This applies only
if you enable switch management. In addition, you cannot enable
switch management through SAM — you must use the clic_init
command.
•The IP multicast address that all the switches and nodes in the
fabric will register to (if you are going to enable switch
management).
•The IP address of the local node’s Ethernet LAN interface. This LAN
interface must be on the same subnet as Ethernet ports of the
HyperFabric switches (if you are going to enable switch
management). (A node might have multiple LAN interfaces.)
NOTEHP recommends that you do not enable switch management.
You can create the global configuration file by either running the
clic_init command or using SAM to configure each HyperFabric
adapter.
The clic_init command and SAM also place the necessary entries into
the following three files:
•The system /etc/rc.config.d/netconf file.
Chapter 469
Configuring HyperFabric
Configuration Overview
IMPORTANTIn this file, clic_init and SAM add some HyperFabric-related lines
that end with the characters #clic. These lines are used by the
HyperFabric software — and are not comments — so do not remove
them from the file.
•The system /etc/rc.config.d/clic_global_conf file.
•The /etc/rarpd.conf (Reverse Address Resolution Protocol
[RARP]) support file. This file is used in the management of the
HyperFabric switches (if you are going to enable switch
management).
The clic_init command is described in “Using the clic_init Command”
on page 79. Using SAM to configure an adapter is described in “Using
SAM” on page 81.
After you have used the clic_init command or SAM, you can configure
HyperFabric with ServiceGuard, if necessary. For more information, see
“Configuring HyperFabric with ServiceGuard” on page 87.
You can configure card pairs for the Transparent Local Failover Feature
of HMP available with the B.11.23.01 release of HyperFabric. For more
information, see “Configuring HMP for Transparent Local Failover
Support” on page 96.
Chapter 470
Configuring HyperFabric
Information You Need
Information You Need
When you run the clic_init command or use SAM for configuration,
you have to provide certain configuration information. So, before you run
clic_init or use SAM, you should have the following information:
❏For each node in the fabric, determine if that node will need to
interoperate with other nodes that are using; any HP-UX 11.0
HyperFabric versions earlier than B.11.00.11 or any HP-UX 11i v1
HyperFabric versions earlier than B.11.11.01.
❏For each HyperFabric adapter installed in the local node:
✓The adapter’s IP address.
IMPORTANTThe last 10 bits of each adapter’s IP address must be unique
throughout the entire fabric. And, remember that the last part of
the address cannot be 0 (that is, the IP address cannot be
n.n.n
.0). Also, note that HyperFabric converts these 10 bits to a
decimal value called the Virtual Route IDentifier (VRID),
which is used in some HyperFabric command input and output.
✓The subnet mask. When you run clic_init or use SAM, if you
do not specify a value for this, a default subnet mask is chosen
based on the adapter’s IP address.
When clic_init begins to prompt you for the information for each
adapter, it assigns an ID (for example, clic0) to that adapter and
displays it as part of the first prompt. If you use SAM, it assigns the
adapter an ID and displays it in the “Adapter Name” column of the
“Configure HyperFabric Adapter” screen. You can also determine an
adapter’s ID by running the clic_stat command (see “The clic_stat
Command” on page 117). You should note each adapter’s ID, because
it is used as input to other HyperFabric commands.
❏For using the Transparent Local Failover feature of HMP available
in the version B.11.23.01 of HyperFabric, you need to define the card
pairs.
❏For each HyperFabric switch in the fabric (if you are going to enable
switch management):
Chapter 471
Configuring HyperFabric
Information You Need
✓The IP address of the switch.
✓The MAC address of the switch’s Ethernet port. If you do not
already know the switch’s MAC address, it is printed on a label
on the back of the HF switch and on the front of the HF2 switch.
See Figure 3-2 on page 66 for the location of the label on the HF
switch, and Figure 3-2 on page 59 and Figure 3-3 on page 60 for
the location of the label on the HF2 switch.
IMPORTANTYou cannot enable switch management through SAM — you must
use the clic_init command.
When clic_init begins to prompt you for the information for each
switch, it assigns an ID (for example, sw_clic0) to that switch and
displays it as part of the first prompt. Note that you can also
determine a switch’s ID by running the clic_stat command (see
“The clic_stat Command” on page 117). You should note each switch’s
ID, because it is used as input to other HyperFabric commands.
❏For the entire fabric, you need the IP multicast address that all the
switches and nodes in the fabric will register to. The address must be
a class D address. Note that if you do not have switch management
enabled, you do not need this information (clic_init will not
prompt you for it).
❏For each node in the fabric, you need the IP address of the node’s
Ethernet LAN interface that is on the same subnet as the switches.
(As mentioned earlier, a node might have multiple LAN interfaces.)
Note that if you do not have switch management enabled, you do not
need this information (clic_init will not prompt you for it).
As stated earlier, HP recommends that you do not enable switch
management.
IMPORTANTYou should also check your /etc/hosts file — when you are using files
for host name look up — to ensure that the entries for all of the systems
are in the correct format: the official host name, which is the full domain
extended host name, and any alias names. For example:
This example uses some “dummy” (that is, not valid) addresses to the
components in Figure 4-1. The dummy addresses are used only to show
the flow of the information provided as input to the clic_init command
and SAM. Do not try to use these addresses in your configuration.
To use clic_init to configure the Transparent Local Failover feature on
HMP, see the section, “Configuring HMP for Transparent Local Failover
Support - Using the clic_init command” on page 104.
Figure 4-1Map for Configuration Information Example
Ethernet LAN
S
Switch ID: sw_clic0
IP address: 193.0.0.20
Ethernet MAC address:
00:60:b0:d0:02:57
IP multicast address:
226.10.1.1
HF
switch 0
HF
switch 1
Switch ID: sw_clic1
IP address: 193.0.0.21
Ethernet MAC address:
00:60:b0:d0:02:56
IP multicast address:
226.10.1.1
HF
adapter 0
Adapter ID:
clic0
IP address:
192.0.0.1
subnet mask:
255.255.255.0
node A
node A
IP address: 193.0.0.10IP address: 193.0.0.11
HF
adapter 1
Adapter ID:
clic1
IP address:
192.0.8.3
subnet mask:
255.255.255.0
lan0lan0
Adapter ID:
clic0
IP address:
192.0.0.2
subnet mask:
255.255.255.0
HF
adapter 0
node A
node B
HF
adapter 1
Adapter ID:
clic1
IP address:
192.0.8.4
subnet mask:
255.255.255.0
S
Chapter 474
Configuring HyperFabric
Information You Need
Using the configuration information in Figure 4-1, the information you
would specify when you run clic_init or SAM on each of the nodes is
listed below. This example is not an exact depiction of the prompts
produced by clic_init nor the fields in SAM, but merely an example of
the flow of information input. In addition, you should not try to use the
dummy addresses in your actual configuration.
On node A:
1. How many HyperFabric adapters are installed on the node?
2. Do you want this node to interoperate with nodes running any
HyperFabric versions earlier than B.11.00.11 or B.11.11.01?
3. What is the IP address of the first adapter (clic0)? (192.0.0.1)
4. What is the subnet mask of the first adapter? (255.255.255.0)
If you do not specify a value for this, a default mask is chosen. You
will most likely just accept the default. However, in this example, we
are showing a value for the subnet mask just to illustrate the
correlation between the “dummy” information in Figure 4-1 and
where that information is specified or generated during clic_init
and SAM.
5. What is the IP address of the second adapter (clic1)? (192.0.8.3)
6. What is the subnet mask of the second adapter? (255.255.225.0)
7. Do you want to enable switch management? You cannot enable
switch management through SAM (you must use the clic_init
command).
As stated earlier, HP recommends that you do not enable switch
management. However, if you do enable it, you must provide the
information in items 8 through 14.
8. If switch management has been enabled, how many switches will be
configured? As stated earlier, HP recommends that you do not enable
switch management.
9. What is the IP address of the first switch (sw_clic0)? (193.0.0.20)
10. What is the Ethernet hardware address of the first switch?
(0060b0d00257)
11. What is the IP address of the second switch (sw_clic1)?
(193.0.0.21)
Chapter 475
Configuring HyperFabric
Information You Need
12. What is the Ethernet hardware address of the second switch?
(0060b0d00256)
13. What is the Multicast address for the switches to use? (226.10.1.1)
14. What is the IP address for the LAN card on the same subnet as the
switches? (193.0.0.10)
(In Figure 4-1, this is the IP address for lan0 on node A.)
On node B:
1. How many HyperFabric adapters are installed on the node?
2. Do you want this node to interoperate with nodes running any
HyperFabric versions earlier than B.11.00.11 or B.11.11.01?
3. What is the IP address of the first adapter (clic0)? (192.0.0.2)
4. What is the subnet mask of the first adapter? (255.255.255.0)
If you do not specify a value for this, a default mask is chosen. You
will most likely just accept the default. However, in this example, we
are showing a value for the subnet mask just to illustrate the
correlation between the dummy information in Figure 4-1 and where
that information is specified or generated during clic_init and
SAM.
5. What is the IP address of the second adapter (clic1)? (192.0.8.4)
6. What is the subnet mask of the second adapter? (255.255.225.0)
7. Do you want to enable switch management? You cannot enable
switch management through SAM (you must use the clic_init
command).
As stated earlier, HP recommends that you do not enable switch
management. However, if you do enable it, you must provide the
information in items 8 through 14.
8. If switch management has been enabled, how many switches will be
configured? As stated earlier, HP recommends that you do not enable
switch management.
9. What is the IP address of the first switch (sw_clic0)? (193.0.0.20)
10. What is the Ethernet hardware address of the first switch?
(0060b0d00257)
11. What is the IP address of the second switch (sw_clic1)?
(193.0.0.21)
Chapter 476
Configuring HyperFabric
Information You Need
12. What is the Ethernet hardware address of the second switch?
(0060b0d00256)
13. What is the Multicast address for the switches to use? (226.10.1.1)
14. What is the IP address for the LAN card on the same subnet as the
switches? (193.0.0.11)
(In Figure 4-1, this is the IP address for lan0 on node B.)
Chapter 477
Configuring HyperFabric
Performing the Configuration
Performing the Configuration
As explained in “Configuration Overview” on page 69, you must create
the global configuration file (/etc/rc.config.d/clic_global_conf)on
each node in the fabric. This consists mostly of specifying HyperFabric
adapter-related information. (If you are also going to enable switch
management — which HP does not recommend doing — you need to
specify additional configuration information.)
NOTESpecifying configuration information adds or changes only the addresses
and other information in the global configuration file, based on the
information you supply. It does not perform any operations to check the
relationships between that information and any physical connections
within the fabric.
You need to create the global configuration file in the following
situations:
•You have just installed the HyperFabric hardware and software on
the system.
•You want to change the information in the HyperFabric global
configuration file (see the preceding note).
IMPORTANTCreating the global configuration file also modifies the
/etc/rc.config.d/netconf file, adding some HyperFabric-related lines
that end with the characters #clic. These lines are used by the
HyperFabric software — and are not comments — so do not remove them
from the file.
You can create the global configuration file by using the clic_init
command (described in “Using the clic_init Command” on page 79) or
SAM (described in “Using SAM” on page 81). You cannot enable switch
management through SAM (you must use the clic_init command).
Chapter 478
Configuring HyperFabric
Performing the Configuration
Using the clic_init Command
Run the clic_init command to create the global configuration file.
To view the man page for clic_init see “Viewing man Pages” on
page 127 of this manual.
To use clic_init to configure the Transparent Local Failover feature on
HMP, see the section, “Configuring HMP for Transparent Local Failover
Support - Using the clic_init command” on page 104.
IMPORTANTIf the global configuration file already exists and you are running
clic_init again (to change the file), you have the option of retaining or
modifying the existing configuration information, in addition to adding
new information pertaining to new hardware.
Also, once you have completed your changes and clic_init ends its
processing, you must stop HyperFabric (by running the clic_shutdown
command or using SAM) and then start HyperFabric (by running the
clic_start command or using SAM). Otherwise, your configuration
information changes will not take effect. See “Stopping HyperFabric” on
page 128 and “Starting HyperFabric” on page 109 for more information.
If you include /opt/clic/bin in your PATH statement, you can run the
command as it is shown below. Otherwise, you must include
/opt/clic/bin as part of the command name (that is,
/opt/clic/bin/clic_init).
You must be logged in as root to run this command.
The syntax is as follows:
clic_init [-c] [-?]
where
•-c specifies that you want to create the global configuration file. You
are prompted for the information described in “Information You
Need” on page 71. Note that if the global configuration file already
exists (for example, when you are adding an adapter to an existing
fabric), clic_init prompts you with the existing configuration
information. As you are prompted with each piece of information, you
can then confirm that you want to keep it or you can change it.
•-? displays the online help for clic_init.
Chapter 479
Configuring HyperFabric
Performing the Configuration
If you do not specify any of the above parameters, the online help for
clic_init is displayed.
After you have entered the information for all the adapters in the node
and all of the switches (if any) in the fabric, a summary of the
configuration information is displayed.
Once clic_init has finished, you do one of the following things:
•If you want to configure HyperFabric with ServiceGuard, complete
the configuration described in “Configuring HyperFabric with
ServiceGuard” on page 87, then run clic_start or use SAM to start
HyperFabric.
•If you have just created the global configuration file on the local node
for the first time (and you are not configuring ServiceGuard), run
clic_start or use SAM to start HyperFabric.
•If you have just changed an existing configuration file on the node,
run clic_shutdown or use SAM to stop HyperFabric, and then run
clic_start or use SAM to start HyperFabric. Until you do those two
things, your configuration changes will not take effect.
See “Stopping HyperFabric” on page 128 and “Starting HyperFabric” on
page 109 for more information.
Examples of clic_init
Some examples of using the clic_init command are shown below.
•Example 1
To create the global configuration file on the local node, issue this
command:
$ clic_init -c
•Example 2
To display the online help for the clic_init command, issue this
command:
$ clic_init -?
or this command:
$ clic_init
Chapter 480
Configuring HyperFabric
Performing the Configuration
Using SAM
This section describes how to use SAM to configure HyperFabric. For
information on how to use SAM to configure and deconfigure local
failover feature on HMP, see “Configuring HMP for Transparent Local
Failover Support - Using SAM” on page 102 and “Deconfiguring HMP for
Local Failover support - Using SAM” on page 103.
IMPORTANTIf the global configuration file already exists, and you are running SAM
again (to change the file), you can keep or modify the existing
configuration information, in addition to adding new information
pertaining to new hardware.
Also, once you’ve completed your changes and SAM ends its processing,
you must stop HyperFabric (by running the clic_shutdown command or
using SAM) and then start HyperFabric (by running the clic_start
command or using SAM). Otherwise, your configuration information
changes will not take effect. See “Stopping HyperFabric” on page 128 and
“Starting HyperFabric” on page 109 for more information.
To use SAM to create the global configuration file on an HP 9000 system
running HP-UX 11i v2, follow these steps:
Step 1. Start SAM.
Step 2. Select the “Networking and Communications” area.
Step 3. Select “HyperFabric.”
All HyperFabric adapters installed in the system are listed; installed
adapters that are not yet configured show Not Configured in the
“Status” field.
Step 4. Highlight the adapter you want to configure.
Step 5. Pull down the “Actions” menu and select Configure Adapter.
Step 6. In the “Configure HyperFabric Adapter” screen, specify information for
the following fields:
•Internet Address—Required. The IP address of the adapter.
Chapter 481
Configuring HyperFabric
Performing the Configuration
•Subnet Mask—Optional. The adapter’s subnet mask. If you do not
specify this, a default mask is chosen based on the adapter’s IP
address.
•Interoperability Enabled—Required. Whether you want the
adapter to be able to interoperate with adapters that are using; any
HP-UX 11.0 HyperFabric versions earlier than B.11.00.11 or any
HP-UX 11i v1 HyperFabric versions earlier than B.11.11.01. Note
that if you select No, the HyperFabric software on the system will not
be backwards compatible with previous releases. This means you
must update all of the other systems in the fabric to the version that
is running on the system.
Default: No.
Step 7. Select OK (remember, you cannot enable switch management within
SAM).
Step 8. Exit SAM.
Once SAM has finished, you do one of the following things:
•If you want to configure HyperFabric with ServiceGuard, complete
the configuration described in “Configuring HyperFabric with
ServiceGuard” on page 87, then run clic_start or use SAM to start
HyperFabric.
•If you have just created the global configuration file on the local node
for the first time (and you are not configuring ServiceGuard), run
clic_start or use SAM to start HyperFabric.
•If you have just changed an existing configuration file on the node,
run clic_shutdown or use SAM to stop HyperFabric, and then run
clic_start or use SAM to start HyperFabric. Until you do those two
things, your configuration changes will not take effect.
See “Stopping HyperFabric” on page 128 and “Starting HyperFabric” on
page 109 for more information.
Chapter 482
Deconfiguring a HyperFabric Adapter with SAM
Deconfiguring a HyperFabric Adapter with
SAM
To use SAM to deconfigure a HyperFabric adapter on an HP 9000 system
running HP-UX 11i v2, complete the following steps:
Step 1. Start SAM.
Step 2. Select the “Networking and Communications” area.
Step 3. Select “HyperFabric.”
All HyperFabric adapters installed in the system are listed. Installed
adapters that are configured show Configured in the “Status” field, and
installed adapters that are not yet configured show Not Configured in
the “Status” field. You can deconfigure only an adapter with a status of
Configured.
Step 4. Highlight the adapter you want to deconfigure.
Configuring HyperFabric
Step 5. Pull down the “Actions” menu and select Deconfigure Adapter.
Step 6. In the pop-up window, if you want to deconfigure the adapter, select OK to
confirm it.
If you do not want to deconfigure the adapter, select Cancel.
Step 7. If you selected OK, the entry for the adapter is deleted from the
HyperFabric configuration files (/etc/rc.config.d/clic_global_conf
and /etc/rc.config.d/netconf).
If you selected Cancel, you remain in the main “HyperFabric
Configuration” screen.
Step 8. Exit SAM.
NOTEIf you have configured HMP for Transparent Local Failover support and
if you select Deconfigure Adapter, HyperFabric will verify if the
selected adapter is configured to be part of any card pair. If yes, the user
Chapter 483
Configuring HyperFabric
Deconfiguring a HyperFabric Adapter with SAM
is informed and the card pair entry is removed from the
/etc/rc.config.d/netconf and
/etc/rc.config.d/clic_global_conf files.
Chapter 484
Configuring HyperFabric
Configuring the HyperFabric EMS Monitor
Configuring the HyperFabric EMS Monitor
In the HyperFabric version B.11.23.01, the HyperFabric Event
Monitoring Service (EMS) monitor allows system administrators to
separately monitor each HyperFabric adapter on every node in the
fabric, in addition to monitoring the entire HyperFabric subsystem.
The monitor can inform the user if the resource being monitored is UP or
DOWN. The administrator defines the condition to trigger a notification
(usually a change in interface status). Notification can be accomplished
with a SNMP trap or by logging into the syslog file with a choice of
severity, or by email to a user defined email address.
To configure the HyperFabric EMS monitor, it is necessary to have the
EMS HA monitor product installed (Product Number B7609BA). This
product is available on the applications CD media.
Use SAM to initiate monitoring of any particular HyperFabric resource.
following the procedure outlined below:
1. Start SAM (Use the online help at any time for details)
2. Select “Resource Management”
3. Select “Event Monitoring Service”
4. Select “Action” and “Add Monitoring Request”
5. Select the location /net/interfaces/clic (class for HyperFabric
resources)
6. Select a resource instance (either all instances or a specific instance
from the list)
7. Validate your choice by clicking on OK at the bottom of the screen
8. A Monitoring Request Parameters window opens, showing the
resource and its status (if All instances have been selected, then no
value is displayed)
9. Define a condition that will trigger a notification (for instance, “When
Value is”, “equal to”, “UP”)
10. Define a polling interval (default is 300 seconds)
11. Define a way of notification: SNMP trap, log in syslog with a choice of
severity, or email to a user defined email address
Chapter 485
Configuring HyperFabric
Configuring the HyperFabric EMS Monitor
12. Validate by pressing OK
NOTEAlthough EMS is able to monitor each HyperFabric adapter on every
node in the fabric, as well as the entire HyperFabric subsystem,
EMS is not able to monitor HyperFabric switches.
For more detailed information on EMS, including instructions for
implementing this feature, see the EMS Hardware Monitors Users GuidePart Number B6191-90028 September 2001 Edition.
Chapter 486
Loading...
+ hidden pages
You need points to download manuals.
1 point = 1 manual.
You can buy points or you can get point for every manual you upload.